Method and system for managing and displaying virtual content in a mixed reality system

Abstract

A method is disclosed for managing and displaying, by each application, virtual content independently on a one-at-a-time basis in a mixed reality environment, each virtual content being rendered by its respective application into a bounded volume referred to as a "prism". Each prism may have features and characteristics that allow a global application to manage and display the prism in a mixed reality environment such that the global application may manage the placement and display of virtual content in the mixed reality environment by managing the prism itself.

Description

[0001] This application is a divisional application of the patent application with international application number PCT / US2018 / 066331, international application date December 18, 2018, Chinese national application number 201880088453.0, and title “Methods and systems for managing and displaying virtual content in a mixed reality system”. Technical Field

[0002] The present disclosure generally relates to systems and methods configured to facilitate an interactive virtual or augmented reality environment for one or more users. Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" (VR), "augmented reality" (AR) experiences, and / or "mixed reality" experiences (hereinafter collectively referred to as "mixed reality" and / or "MR"), in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. VR scenes typically involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input, while AR or MR scenes typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the real world around the user, such that the digital or virtual image (e.g., virtual content) may appear to be part of the real world. However, MR may integrate virtual content in a contextually meaningful manner, while AR may not.

[0004] Therefore, a method for managing and displaying virtual content in a mixed reality environment is needed. Summary of the invention

[0005] According to some embodiments, instead of each application independently managing and displaying virtual content in a mixed reality environment on a one-at-a-time basis, each virtual content is rendered by its corresponding application into a bounded volume, which may be referred to hereinafter as a "prism." Each prism may have features and characteristics that allow an application (sometimes referred to as a universe application) to manage and display the prism in the mixed reality environment, such that the universe application can manage the placement and display of virtual content in the mixed reality environment by managing the prism itself.

[0006] One embodiment relates to a method for displaying virtual content in a 3D spatial environment, the method comprising: 1) receiving a request from an application to display virtual content in the 3D spatial environment; 2) creating a prism for managing the display of the virtual content, wherein the prism is a cubic and / or rectangular spatial volume configured to confine the virtual content to the boundaries of the prism; 3) receiving the virtual content from the application; 4) rendering the virtual content within the boundaries of the prism, thereby associating the prism with an object in the 3D spatial environment based at least in part on user input, and 6) anchoring the prism to the 3D spatial environment.

[0007] In one or more embodiments, a border of the boundary of the prism is not displayed. The 3D space environment can be a physical real-world environment of the user. The prism can be automatically created with a set of functions. The set of functions can include a minimum / maximum size allowed for the prism and / or an aspect ratio for changing the size of the prism. The set of functions can include an association between the prism and an object in the 3D space environment. The application can render additional virtual content into the additional prism, wherein each virtual content can be rendered into a separate prism.

[0008] In one or more embodiments, the prisms do not overlap with other prisms in the 3D spatial environment. The prisms may include one or more general features to ensure that different applications interact appropriately with each other, and / or one or more application-specific features, wherein the one or more general features and the one or more application-specific features are selected from a pre-approved list of options.

[0009] Another embodiment relates to a display system for displaying virtual content in a 3D spatial environment, which display system may include an augmented reality head-mounted display system and one or more modules for processing data, wherein the one or more modules are stored in one or more memories and are configured to perform the following operations: 1) receive a request from an application to display virtual content in a 3D spatial environment, 2) create a prism, wherein the prism is a spatial volume configured to confine the virtual content to within the boundaries of the prism, 3) receive the virtual content from the application, 4) render the virtual content within the boundaries of the prism, and 5) associate the prism with an object in the 3D space.

[0010] In one or more embodiments, a border of the boundary of the prism is not displayed. The 3D space environment can be a physical real-world environment of the user. The prism can be automatically created with a set of functions. The set of functions can include a minimum / maximum size allowed for the prism and / or an aspect ratio for changing the size of the prism. The set of functions can include an association between the prism and an object in the 3D space environment. The application can render additional virtual content into the additional prism, wherein each virtual content can be rendered into a separate prism.

[0011] In one or more embodiments, the prisms do not overlap with other prisms in the 3D spatial environment. The prisms may include one or more general features to ensure that different applications interact appropriately with each other, and / or one or more application-specific features, wherein the one or more general features and the one or more application-specific features are selected from a pre-approved list of options.

[0012] Another embodiment relates to a method for initiating a mixed reality system, which may include: determining a user's current location; retrieving one or more prisms previously deployed at the current location; restoring the one or more prisms at the user's current location; and displaying the one or more prisms restored at the user's current location.

[0013] In one or more embodiments, a prism is a cubic and / or rectangular volume of space into which virtual content from an application is displayed, wherein more than one prism may be rendered. In other words, in some embodiments, a single application may correspond to more than one prism. In some embodiments, a single application corresponds to a single prism. A prism represents a subtree of a multi-application scene graph for the user's current location. Retrieving one or more prisms previously deployed at the user's current location may include, for example, retrieving instance data for one or more prisms from an external database, and rebuilding a local prism database using instance data about one or more prisms, wherein the instance data for each prism includes a data structure defining prism characteristics of the prism, the prism characteristics including at least one of position, orientation, range width, range height, range depth, anchor type, and / or anchor position, wherein the instance data about each prism includes a key-value pair of application-specific characteristics, the application-specific characteristics including state information of virtual content previously rendered into the prism by the application. In some embodiments, the data is stored locally and no external database is required.

[0014] In one or more embodiments, restoring one or more prisms includes launching corresponding applications corresponding to one or more prisms previously deployed at the current location, creating one or more new prisms corresponding to the one or more prisms previously deployed, and rendering corresponding virtual content into the one or more new prisms.

[0015] In one or more embodiments, the method further includes: updating the user's local prism database with updated prism instance data when the user interacts with one or more prisms, and synchronizing the local prism database with an external database.

[0016] Some embodiments relate to a method for managing application state of virtual content in a mixed reality system, which may include segmenting a 3D volume into a volume grid. The method also includes determining a first position of a user within the volume grid. The method also includes determining a second position of the application within the 3D volume. The method also includes calculating a distance between the user and the application within the 3D volume. The method may additionally include modifying the state of the application based at least in part on the distance calculated between the user and the application.

[0017] Another embodiment relates to a method for managing application state of virtual content in a mixed reality system, the method comprising recording the spatial location of one or more applications in a volumetric grid, the one or more applications providing content displayed within one or more corresponding prisms, the volumetric grid corresponding to a coarse representation of a physical environment on x, y, and z axes. Identifying a cell of the volumetric grid, the cell including an application located within the cell, wherein a width of the cell is equal to or greater than a radius of an active area. Determining a distance of each application to a known location within the cell and adjacent cells as a user using a mixed reality device moves within the cell. And modifying the state of the application based at least in part on the distance between the mixed reality device and each application within the cell and adjacent cells.

[0018] In one or more embodiments, the radius of the active zone defines a circular / spherical area around a user using a mixed reality device, where the user may be located at the center of the circle / sphere. In some embodiments, modifying the state of an application may be based at least in part on whether the application is obscured by another application. In some embodiments, modifying the state of an application is based at least in part on the user's head posture. The user's head posture is a measurement of the position and / or orientation of the user's head. The head posture can be used to render the scene to match the user's dynamically changing head position and orientation to provide an increased sense of immersion in a virtual / augmented / hybrid space. In some embodiments, the head posture may be determined at least in part by an inertial measurement unit mounted on a head-mounted display system or a user's head, although other suitable methods may also be used. The distance to a known position of each application within a cell may be determined only for the cell in which the user using the mixed reality device is located and for adjacent cells.

[0019] In one or more embodiments, there may be a buffer zone around the outside of the active area, where the buffer zone prevents intermittent or rapid changes in the state of the application.

[0020] Another embodiment relates to a method for launching an application from a launcher menu to display virtual content in a mixed reality system, the method comprising receiving a request to launch an application. A prism is created by a global application for displaying and managing virtual content, wherein the prism is a volumetric display space with boundaries for the application to render virtual content into it. The application is launched through the prism with the help of a lifecycle service. The unique identifier (UID) of the application is determined through the prism with the help of a package manager service. A listener is registered through the application with the global application. The UID of the application is determined through the global application. The prism is associated with the listener through the global application. The prism is assigned to an application using the listener of the application through the global application. And the prism is placed in a subgroup of a 3D displayable space of the mixed reality system.

[0021] Some embodiments relate to a method for opening and placing an application in an augmented reality environment, the method may include receiving a first user input from a user, the first user input indicating a request for new content. The method may also include launching the application to generate content. The method may further include creating a small display volume of a 3D display volume management unit, wherein a page preview is displayed in the small display volume, wherein the 3D display volume management unit is created simultaneously with the launching of the application. The method may further include receiving a second user input, the second user input indicating movement of the small display volume. The method may also include receiving a third user input indicating placement of the small display volume at a location in the augmented reality environment; and expanding the 3D display volume management unit at the location to replace the small display volume, the 3D display volume management unit displaying content fully loaded within the 3D display volume management unit.

[0022] In one or more embodiments, the first user input may be a cursor movement over a link on a web page, wherein the second user input is a selection of the link and movement of the small display volume. The small display volume may be an initial default size of the 3D display volume management unit. Content may be loaded into the small display volume while the user is moving and placing the small display volume. The position may be fixed to an object in the augmented reality environment, wherein the object is the user.

[0023] Some embodiments may involve a method for managing virtual content, which may include receiving content from a content generation application. The method may also include continuously managing the display of the content in a 3D space environment by a global application.

[0024] Some embodiments may be directed to a method comprising accessing a scene graph for a scene, wherein the scene graph comprises one or more transformation trees, each tree comprising a plurality of nodes. The method may further comprise adding a label to one or more nodes from a plurality of nodes within the one or more transformation trees, wherein the labeled one or more nodes form a transformation group, wherein the labeled one or more nodes comprise a first labeled node and a second labeled node. The method may further comprise moving the first labeled node, wherein moving the first labeled node causes the second labeled node to move.

[0025] Another embodiment relates to a method of displaying virtual content in a 3D shared space, which may include: generating virtual content in the 3D shared space by a first application, and displaying the virtual content generated by the first application by a second application, the first application and the second application being different applications.

[0026] Some embodiments may be directed to a method for assigning common features and application-selected features from a pre-approved list of options to a prism, the pre-approved list of options for display customization by the application. Another embodiment is directed to a method for displaying virtual content into one or more prisms, wherein the one or more prisms do not overlap each other. Another embodiment is directed to a method for changing the state of a prism based at least in part on the relative positioning and location of the prism to a user. Another embodiment is directed to a method for managing content creation in an application and managing content display in a separate application. Another embodiment is directed to a method for opening an application that provides content into a prism while placing the prism in a mixed reality environment.

[0027] Some embodiments may relate to a method for assigning position, orientation, and range data to a prism for displaying virtual content within the prism, the virtual content being 3D virtual content.

[0028] In one or more embodiments, the position is the coordinates of an anchor position of the prism in the mixed reality environment. The extent data defines the dimensions of the prism.

[0029] Some embodiments may relate to a method for pinning a launcher application to a real-world object within a mixed reality environment.

[0030] In one or more embodiments, the pinned launcher application launches the content of the application within the prism in the same location as the pinned launcher application.

[0031] Some embodiments may be directed to a method for assigning a behavior type to each prism, the behavior type comprising at least one of world lock, billboard, edge billboard, follow head lock, external sensor based follow, or fade (described in detail below). Some embodiments may be directed to a method for identifying the most frequently used content specific to a placement location of a launcher app. Some embodiments may be directed to a method for displaying favorite apps via a placed launcher app, the favorite apps being based at least in part on a range relative to a location of a placed launcher.

[0032] Additional and other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings illustrate the design and practicality of preferred embodiments of the present disclosure, wherein similar elements are referred to by common reference numerals. In order to better understand how to obtain the above and other advantages and purposes of the present disclosure, a more specific description of the present disclosure briefly described above will be presented by reference to the specific embodiments of the present disclosure shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered as limiting the scope thereof, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0034] The drawings use similar reference numerals to identify similar elements. A letter following a reference numeral, such as "120a," indicates that the text refers specifically to the element with that particular reference numeral. A reference numeral in the text, such as "120," without a following letter, refers to any or all elements in the drawings bearing that reference numeral (e.g., "120" in the text refers to reference numerals "120a" and / or "120b" in the drawings).

[0035] Figure 1 An example user physical environment and system architecture for managing and displaying virtual content in a mixed reality system is shown in accordance with some embodiments.

[0036] Figure 2 A system architecture for managing and displaying virtual content in a mixed reality system is shown in accordance with some embodiments.

[0037] Figure 3 A prism / bounded volume is shown according to some embodiments.

[0038] Figure 4 A launcher menu for launching an application to display virtual content is shown in accordance with some embodiments.

[0039] Figure 5A-5B A panel carousel replacement is shown in accordance with some embodiments.

[0040] Figure 6 A flow chart of a method for launching a mixed reality system according to some embodiments is shown.

[0041] Figure 7 A flow chart of a method for displaying virtual content in a mixed reality environment according to some embodiments is shown.

[0042] Figure 8 is a graph for managing application state according to some embodiments.

[0043] Fig. 9 A flow diagram for managing application state relative to a user's location is shown in accordance with some embodiments.

[0044] Fig. 10A Shown are tree nodes of a scene graph according to some embodiments.

[0045] Figure 10B-10AA Various transform trees and group trees are shown in accordance with some embodiments.

[0046] Fig.11 A diagram showing placed launchers in accordance with some embodiments.

[0047] Fig.12 Shown are the types of content that may be displayed while the launcher is in an idle / sleeping state according to some embodiments.

[0048] Fig.13 A secondary UI volume is shown according to some embodiments.

[0049] Fig.14 Examples of body dynamics in accordance with some embodiments are shown.

[0050] Fig.15 Different types of body dynamics are shown according to some embodiments.

[0051] Fig.16 A flow chart for simultaneously launching and placing applications in a mixed reality environment is shown in accordance with some embodiments.

[0052] Fig.17 is a block diagram of an exemplary computing system 1400 suitable for implementing one or more of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0053] The present disclosure is directed to managing and displaying virtual content in a mixed reality system. Instead of allowing multiple applications to manage and display virtual content in a mixed reality system independently of each other, embodiments of the present disclosure disclose a global application that manages (e.g., how and where) virtual content is displayed and managed in a mixed reality system using a 3D window called a prism.

[0054] This disclosure provides a description of an exemplary mixed reality system that can be used to practice some embodiments of the disclosure, followed by a description of one or more embodiments of processes and mechanisms for managing and displaying virtual content in the mixed reality system.

[0055] The following description relates to an exemplary mixed reality system in which the present disclosure may be practiced. However, it should be understood that the present disclosure is also applicable to applications in other types of AR and virtual reality (VR) systems, and therefore, the present disclosure is not limited to the exemplary system disclosed herein.

[0056] picture( Figure 1 ) shows an example user physical environment and system architecture for managing and displaying virtual content in a mixed reality system. The representative environment 100 includes a user's landscape 110 viewed by a user 103 through a head-mounted system 160. The user's landscape 110 is a 3D view of the world, where user-placed content can be composited on top of the real world. The representative environment 100 also includes access to global applications 130 via a processor 170 operably coupled to a network (not shown). Although the processor 170 is shown as an isolated component separate from the head-mounted system 160, in alternative embodiments, the processor 170 may be integrated with one or more components of the head-mounted system 160 and / or may be integrated into other system components within the representative environment 100, such as a network accessing a computing network (not shown) and an external storage device 150. In some embodiments, the processor 170 may not be connected to a network. The processor 170 may be configured with software (e.g., the global application 130) for receiving and processing information such as video, audio, and / or other data (e.g., depth camera data) received from the head-mounted system 160, the local storage device 137, the application 140, the computing network, and / or the external storage device 150.

[0057] The global application 130 may be a 3D Windows manager similar to a 2D Windows manager running on, for example, a desktop computer, which is used to manage 2D windows displayed on a display screen of the desktop computer. However, the global application 130 (hereinafter referred to as "global") manages the creation, placement and display of virtual content 115 in a 3D spatial environment and the interaction between multiple virtual content 115 displayed in the user's landscape 110. The virtual content 115 from the application 140 is presented to the user 103 within one or more 3D window display management units, such as bounding volumes and / or 3D windows (hereinafter referred to as prisms 113).

[0058] Bounded volume / 3D window / prism 113 can be a volume of a rectangle, cube, cylinder or any other shape that can be positioned and oriented in space. Prism 113 can be a volume display space with a boundary for presenting / displaying content (e.g., virtual content) therein, wherein the boundary is not displayed. In some embodiments, the boundary can be displayed. Prism 113 can provide a standard basic level of interaction and control over the content and position of the application. Prism 113 can represent a subtree of a multi-application scene graph, which can be embedded inside the universe or can be outside the universe but can be accessed by the universe. The scene graph is a common data structure commonly used by vector-based graphics, editing applications and modern game software, which can arrange the logical representation of the graphic scene and the usual (but not necessarily) spatial representation. The scene graph can be considered as a data structure that defines how the content is positioned and transformed relative to each other within its structure. Application 140 gives an instance of prism 113 to place content therein. Applications can render 2D / 3D content in prism 113 using relative placement algorithms and arbitrary transformations, but the universe can ultimately still be responsible for the overall interaction mode such as content extraction. Multiple applications may render to the universe via prisms 113, with process boundaries separating the prisms 113. There may be n bounding volumes / prisms 113 per application process, but this is clearly an n:1 relationship, so that only one process per application may be running per bounding volume / prism 113, but there may be a large number of m processes running, each with its own bounding volume / prism 113.

[0059] The universe operates using a prismatic / distributed scene graph approach for 2D and / or 3D content. For each application, a portion of the universe scene graph is reserved for rendering. Each interaction with an application can be accomplished through a multi-application scene graph, such as a launcher menu, landscape, or body-centric application area (all described in detail below). Each application is assigned 1 to n rectangular prisms representing subtrees of the scene graph. Prisms are not allocated by client applications, but are created through user interactions within the universe (for example, when a user opens a new application in the landscape by clicking a button on a controller). In some embodiments, an application can request a prism from the universe, but the request can be denied. In some embodiments, if an application requests and is allowed a new prism, the application can transform the new prism only relative to one of its other prisms.

[0060] The universe encloses virtual content 115 from applications 140 in objects called prisms 113. Each application process or instance can render its virtual content into its own single prism 113 or group of prisms. The universe manages the world space, sometimes referred to as the landscape, in which the prisms 113 are displayed. In some embodiments, the universe provides the ability to attach applications to walls and surfaces, place prisms anywhere in space, register them with the world database of the mixed reality system, and / or control content sharing between multiple users of the mixed reality system.

[0061] In some embodiments, the purpose of the prism 113 is to provide behavior and control the rendering and display of content. Just as a window can be used to define a 2D display of the position, menu structure, and display of 2D content within a 2D window, with the help of a 3D virtual display, the prism allows the mixed reality system (e.g., the universe) to surround the display of 3D content with controls related to, for example, content position, 3D window behavior, and / or menu structure. For example, the control may include at least placing virtual content at a specific location in the user's landscape 110, removing virtual content from the landscape 110, copying virtual content, and / or placing the copy in a different location, etc. In some embodiments, the prism can be created and destroyed by the user and only by the user. This can be done explicitly to help control abuse of the provided interface and help the user maintain control of the user's content. In addition, in some embodiments, the application 140 does not know where their volume is placed in the landscape—just that they exist. In some embodiments, an application can request one or more prisms, and the request may or may not be granted. After creating a new prism, the user can change the location and / or the application can automatically position the new prism relative to the currently existing prism associated with the application. In some embodiments, it may be necessary for each application 140 to utilize global services to render 3D content (e.g., synthesized 3D content) into a global process so that a listener is first registered with the global. The listener can be used to notify the application 140 of the creation and destruction of rendering prisms based on the user's movement and user interaction with these prisms. A listener is an interface object that receives messages from an inter-process communication system. For example, in the Android operating system, a listener is an object that receives messages through a binder interface. However, any IPC system can be used so that a binder is not always used.

[0062] In some embodiments, a prism may be created from the following interactions: (1) a user has extracted content from an extractable node (disclosed further below); (2) a user has launched an application from a launcher; (3) a user has downloaded a nearby passable world map tile that includes placed instances of applications that the user has permission to view; (4) a user has downloaded a nearby passable world map tile that includes objects that the passable world object recognizer infrastructure has detected for which a given application must render content; and / or (5) a user has triggered a dispatch from another application that must be processed in a different application. In some embodiments, the passable world model allows a user to effectively pass a piece of a user's world (e.g., surroundings, interactions, etc.) to another user.

[0063] Extractable content is content inside a prism (including but not limited to icons, 3D icons, words in a text display, and / or images) that can be pulled out of the prism and placed in the landscape using an input device. For example, a prism may display a web page showing running shoes for sale. To extract the running shoes, the shoes can be selected and "pulled" using an input device. A new prism will be created using a 3D model representing the shoes, and the prism will be moved out of the original prism and toward the user. Like any other prism, the user can use an input device to move, grow, shrink, or rotate the new prism containing the shoes in the 3D space of the landscape. Extractable nodes are nodes in the prism scene graph that have been marked as extractable. In the global domain, extracting content means selecting an extractable node and pulling the content out of the prism using an input device. The input to initiate this pull can be to aim a 6-DOF pointing device at the extractable content and pull a trigger on the input device.

[0064] Each user's respective individual mixed reality system (e.g., mixed reality device) captures information as the user passes through or inhabits the environment, which the mixed reality system processes to generate a transferable world model. More details about the transferable world are described in U.S. Patent Application No. 14 / 205,126, filed on March 11, 2014, entitled "SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY," which has been previously incorporated by reference. The individual mixed reality system can communicate or transfer the transferable world model to a public or shared data collection called the cloud. The individual mixed reality system can communicate or transfer the transferable world model to other users directly or via the cloud. The transferable world model provides the ability to effectively communicate or transfer information that substantially contains at least the user's field of view. In one embodiment, the system uses posture and orientation information and the above-mentioned collected 3D points to create a transferable world. In some embodiments, the transferable world model allows users the ability to integrate content (e.g., virtual and / or physical content) with the real world. The transferable world system may include one or more mixed reality systems or mixed reality user devices that can be connected to a cloud network, a transferable world model, a group of object identifiers, and a database (e.g., an external database 150). The transferable world model may be configured to receive information from the mixed reality user devices and also send data to them over the network. For example, based on input from the user, a piece of the transferable world can be transferred from one user to another. The transferable world model can be considered as a collection of images, points, and other information (e.g., real-world information), based on which the mixed reality system can build, update, and establish a virtual world on the cloud, and effectively transfer pieces of the virtual world to various users. For example, a set of real-world points collected from mixed reality user devices can be collected in a transferable world model. Various object identifiers can crawl through the transferable world model to identify objects, mark images, etc., and attach semantic information to objects. The transferable world model can use a database to build its knowledge about the world, attach semantic information, and store data associated with the transferable world.

[0065] In the case of a Prism that is visible to the user but whose controlling app is not currently installed, the universe may render a temporary placeholder for that app which, when interacted with, redirects the user to the app store page for that app.

[0066] In some embodiments, the prism can be destroyed in a similar interactive manner: (1) the user has walked far enough away from the transferable world map tile that the placed application instance has been unloaded (i.e., removed) from volatile memory; (2) the user has destroyed the placed instance of the application; and / or (3) the application requests that the prism be closed.

[0067] In some embodiments, if no prisms for an application are visible and / or loaded, the process associated with those prisms may be paused or ended. Once the placed prisms for that application are visible again, the process may be restarted. Prisms may also be hidden, but in some embodiments, this may only happen under the command of the universe and the user. In some embodiments, multiple prisms may be placed in the same exact location. In such an embodiment, the universe may display only one instance of a placed prism in one place at a time, and manage the rendering by hiding the visibility of the prism (and its associated content) until a user interaction is detected (e.g., a user "slides" to the next visible element (e.g., a prism) in that location).

[0068] In some embodiments, each prism 113 may be exposed to the application 140 via a volume listener interface with methods for accessing properties of the prism 113 and registering the content in a scene graph subtree for shared resources such as meshes, textures, animations, and the like.

[0069] In some embodiments, since the application 140 does not know where a given prism 113 is placed in 3D space, the volume listener interface can provide accessor methods to a set of hints that help define where a given prism exists in the universe, such as centered on the hand, stuck in the landscape, centered on the body, etc. These characteristics additionally specify the expected behavior of the prism and can be controlled in limited ways by the user, the application 140, or the universe. A given prism can be positioned relative to another prism owned by the application. The application can specify that when a prism from the application is placed, the prisms should be stuck together (the two sides of their bounding volumes are touching). In addition, the prism can provide an API for key-value data storage. Some of these key-value pairs are writable only by privileged applications.

[0070] In some embodiments, application 140 is a client software application that provides content to be displayed to user 103 in the user's landscape 110. For example, application 140 may be a video streaming application, where video data may be streamed to the user for display on a 2D plane. As another example, application 140 may be a Halcyon application that provides 3D imaging of physical objects that may represent a time period in the past that was idyllic, happy and peaceful to the user. Application 140 provides content that the user may wish to include in the user's landscape 110. The placement and management of content generated by application 140 is managed via the universe of prism 113.

[0071] When a non-immersive application is executed / launched in the user's landscape 110, its content (e.g., virtual content) is presented inside the prism 113. A non-immersive application may be an application that can run and / or display content simultaneously with one or more other applications in a shared 3D environment. Although the virtual content may be contained in the prism, the user may still interact with the virtual content, e.g., hover over an object, click on an object, etc. The prism 113 may also delimit the displayed content of an application 140 so that different applications 140 do not interfere with each other or other objects in the user's landscape 110. The prism 113 may also provide a useful abstraction for suspending, pausing, and / or minimizing virtual content from an application 140 that is out of view or too far from the user.

[0072] The prism 113 may be anchored / attached / pinned to various objects within the user's landscape 110, including snapping or anchoring to another prism. For example, a prism 113a displaying virtual content 115 (e.g., video 115a from a video streaming application) may be anchored to a vertical wall 117a. As another example, in Figure 1 , a prism 113b is shown that displays a 3D tree 115b from the Halcyon application, which is anchored to a table 117b. In addition, the prism 113 can be anchored relative to the user 103 (e.g., body-centered), where the prism 113 displaying the virtual content 115 can be anchored to the user's body so that as the user's body moves, the prism 113 moves relative to the movement of the user's body. Body-centric content can be application content that follows the user and remains positioned consistently with the user, such as a plane, a grid, etc. For example, a small dialog box that follows the user around exists relative to the user's spine rather than the landscape 110. In addition, the prism 113 can also be anchored to a virtual object, such as a virtual display monitor displayed within the user's landscape 110. The prism 113 can be anchored in different ways, which are disclosed below.

[0073] The universe may include a local database 137 to store characteristics and features of prisms 113 for the user. The stored prism information may include prisms activated by the user within the user's landscape 110. The local database 137 may be operably coupled to an external database 150, which may reside in the cloud or an external storage facility. The external database 150 may be a persistent database that maintains information about the mixed reality environment of the user and other users.

[0074] For example, when a user launches a new application to display virtual content in the user's physical environment, the local database 137 may store information corresponding to a prism created by the universe and placed at a specific location, where the application 140 may render the content into the prism 113 for display in the user's landscape 110. The information corresponding to the prism 113, virtual content 115, and application 140 stored in the local database 137 may be synchronized to the external database 150 for persistent storage.

[0075] In some embodiments, persistent storage may be important because data stored in the local database 137 may be erased, deleted, or not persistent when the mixed reality system is turned off. Therefore, when the user turns on the mixed reality system, before turning off the mixed reality system, the universe may synchronize with the external database 150 to retrieve an instance of the local database 137 corresponding to the user 103 and the user's landscape 110. The local database 137 may be an instance of the external database 150, wherein the instance of the local database 137 includes information related to the user 103 and the user's current environment. The external database 150 may additionally store instances of local databases for other users, multiple users, the same user over time, and / or other environments. The external database 150 may contain information for managing and sharing virtual content between multiple users of the mixed reality system, while the local database 137 stores and maintains information corresponding to the user 103.

[0076] Each time an application 140 needs to render virtual content 115 onto the user's landscape 110, the universe can create a prism 113 for the application 140. In some embodiments, the prism 113 created by the universe allows the application 140 to focus on rendering virtual content for display, while the universe focuses on creating and managing the placement and display of the prism 113 with the virtual content 115 displayed by the application 140 within the boundaries of the prism.

[0077] Each virtual content 115 rendered by the application 140 displayed in the user's landscape 110 may be displayed in a single prism 113. For example, if the application 140 needs to render two virtual contents (e.g., 115a and 115b) to be displayed within the user's landscape 110, the application 140 may render the two virtual contents 115a and 115b. Since the virtual content 115 includes only rendered virtual content, the universe may create prisms 113a and 113b to correspond to each of the virtual contents 115a and 115b, respectively. The prism 113 may include 3D window management properties and features of the virtual content 115 so that the universe manages the placement and display of the virtual content 115 inside the prism 113 and the prism 113 in the user's landscape 110.

[0078] The universe may be the first application that the user 103 sees when the user 103 turns on the mixed reality device. The universe may be responsible for at least (1) rendering the user's world view; (2) 2D window management for flat applications and 3D window (e.g., prism) management; (3) displaying and executing an application launcher menu; (4) allowing the user to place virtual content into the user's world 110; and / or (5) managing different states of the display of prisms 113 within the user's world 110.

[0079] The head mounted system 160 may be a mixed reality head mounted system that includes a display system (e.g., a user interface) located in front of the eyes of the user 103, a speaker coupled to the head mounted system and positioned adjacent to the ear canal of the user, a user sensing system, an environment sensing system, and a processor (all not shown). The head mounted system 160 presents the display system (e.g., a user interface) to the user 103 for interacting with and experiencing the digital world. Such interactions may involve the user and the digital world, one or more other users connected to the representative environment 100, and objects located within the digital and physical worlds.

[0080] The user interface may include viewing, selecting, locating, and managing virtual content via user input through the user interface. The user interface may be at least one or a combination of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, an audio input device, a smartphone, a tablet, or a head mounted system 160. A tactile interface device is a device that allows a person to interact with a computer through physical sensations and movements. Haptics refers to a type of human-computer interaction technology that includes tactile feedback or other physical sensations to perform actions or processes on a computing device.

[0081] An example of a haptic controller may be a totem (not shown). In some embodiments, a totem is a handheld controller that tracks its position and orientation relative to the headset 160. In this example, the totem may be a six-degree-of-freedom (six DOF) controller in which a user may move a prism around in altitude and azimuth (on a spherical shell) by moving the totem up or down. In some embodiments, to move an object closer or further away, the user may use a joystick on the totem to "push" or "pull" the prism, or may simply move the totem forward or backward. This may have the effect of changing the radius of the shell. In some embodiments, two buttons on the totem may cause the prism to grow or shrink. In some embodiments, rotating the totem itself may rotate the prism. Other totem manipulations and configurations may be used, and should not be limited to the above-described embodiments.

[0082] The user sensing system may include one or more sensors 162 operable to detect certain features, characteristics, or information related to the user 103 wearing the head mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical features / measurements of the user 103, such as one or more of the following: pupil constriction / dilation, angle measurement / positioning of each pupil, sphericity, eye shape (change in eye shape over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that may be used by the head mounted system 160 to enhance the user's viewing experience.

[0083] The environment sensing system may include one or more sensors 164 for acquiring data from the user's landscape 110. Objects or information detected by the sensors 164 may be provided as input to the head mounted system 160. In some embodiments, the input may represent the user's interaction with the virtual world. For example, a user (e.g., user 103) viewing a virtual keyboard on a table (e.g., table 188) may gesture with their fingers as if the user were typing on the virtual keyboard. The motion of the finger movement may be captured by the sensors 164 and provided as input to the head mounted system 160, where the input may be used to change the virtual world or create new virtual objects.

[0084] The sensor 164 may include, for example, a generally outward-facing camera or scanner for capturing and interpreting scene information, for example, by continuously and / or intermittently projecting infrared structured light. The environment sensing system may be used to map one or more elements of the user's landscape 110 around the user 103 by detecting and registering one or more elements from the local environment (including static objects, dynamic objects, people, gestures, and various lighting, atmospheric and acoustic conditions, etc.). Thus, in some embodiments, the environment sensing system may include image-based 3D reconstruction software embedded in a local computing system (e.g., processor 170) and operable to digitally reconstruct one or more objects or information detected by the sensor 164.

[0085] In some embodiments, the environment sensing system provides one or more of the following: motion capture data (including gesture recognition), depth sensing, facial recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors. It should be understood that the environment sensing system may include other components in addition to those discussed above.

[0086] As described above, in some embodiments, processor 170 may be integrated with other components of head mounted system 160, integrated with other components of the system of representative environment 100, or may be a processor such as a Figure 1 The processor 170 may be connected to the various components of the head mounted system 160 via a physical, wired connection or via a wireless connection, such as a mobile network connection (including cellular telephone and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. The processor 170 may include a storage module, an integrated and / or additional graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of transforming data from a source (e.g., a computing network and from a user sensing system and an environmental sensing system of the head mounted system 160) into image and audio data, where the image / video and audio can be presented to the user 103 via a user interface (not shown).

[0087] Processor 170 handles data processing for various components of head mounted system 160 and data exchange between head mounted system 160 and software applications such as a universe, external database 150, etc. Processor 170 may be used to buffer and process data flow between user 103 and a computing network including software applications, thereby enabling a smooth, continuous, and high-fidelity user experience. Processor 170 may be configured to execute a set of program code instructions. Processor 170 may include a memory to store the set of program code instructions, wherein the set of program code instructions includes program code to display virtual content in a subset of an available 3D displayable space by displaying the virtual content within the volumetric display space, wherein the boundaries of the volumetric display space are not displayed. In some embodiments, the processor may be two or more processors operably coupled.

[0088] In some embodiments, the mixed reality system may be configured to be assigned to prism general features and application-selected / application-specific features from a list of pre-approved options for display customization of the application. For example, general features ensure that different applications interact well. Some examples of general features may include maximum / minimum size, non-overlapping prisms (excluding temporary overlap caused by collision behavior), no display content outside the prism boundary, and the application requires permission from the user if the application wants to access sensors or sensitive information. Application-selected / application-specific features can achieve an optimized application experience. Application-selected / application-specific features may include maximum / minimum size (within system limitations), default size (within system limitations), body dynamic type (e.g., no / world lock, billboard, edge billboard, follow / lazy head lock, follow based on external sensors, fade-discussed below), sub-prism generation location, sub-head pose highlighting, sub-prism related behavior, on-surface behavior, independent transformation control, change size vs change scale, idle state timeout, collision behavior, permission / password to access application, etc. In another embodiment, the mixed reality system may be configured to display virtual content into one or more prisms, where in some embodiments, the one or more prisms do not overlap each other. In some embodiments, one or more prisms may overlap to provide specific interactions. In some embodiments, one or more prisms may overlap, but only with other prisms from the same application. In another embodiment, the mixed reality system may be configured to change the state of the prism based at least in part on the relative positioning and position of the prism relative to the user. In another embodiment, the mixed reality system may be configured to manage content creation in applications and manage content display in separate applications. In another embodiment, the mixed reality system may be configured to open an application that will provide content to the prism and place the prism in a mixed reality environment.

[0089] In some embodiments, the mixed reality system may be configured to assign position, orientation, and range data to a prism to display virtual content within the prism, wherein the virtual content is 3D virtual content. In some embodiments, the mixed reality system may be configured to pin a launcher application to a real-world object within a mixed reality environment. In some embodiments, the mixed reality system may be configured to assign a behavior type to each prism, the behavior type including at least one of world lock, billboard, edge billboard, follow head lock, follow based on external sensors, or fade (described in detail below). In some embodiments, the mixed reality system may be configured to identify the most frequently used content or applications specific to the placement location of the launcher application, and thus reorder the applications, for example, from the most frequently used to the least frequently used order. In another embodiment, the mixed reality system may be configured to display favorite applications at the placed launcher application, the favorite applications being based at least in part on the context of the location relative to the placed launcher.

[0090] Figure 2 A system architecture for managing and displaying virtual content in a mixed reality system according to some embodiments is shown. System 200 includes universe 130, applications 140, icon grid application 260, status bar application 270, social panel application 280, and store panel application 290. These applications may represent a basic level of applications on system 200, however, in some embodiments, more or fewer applications may be part of system 200.

[0091] As above Figure 1 As discussed in , the universe can be viewed as a 3D window (e.g., Prism) manager, similar to the 2D window manager that manages 2D windows in conventional computer desktop systems and the like. Figure 2 Available from Figure 1 Here, the global application 130 may also include a global server 205, a loader volume 210, a secondary UI volume 220, a global client 225, a launcher application 230, and a global server 205. The global server 205 may be a processing thread of the global in a multithreaded processing environment for multi-parallel processing.

[0092] The loader volume 210 is a placeholder volume that is displayed to the user while the universe creates a prism for displaying virtual content in the user's landscape 110. For example, when a user selects an application to be displayed in the user's landscape 110 at a particular location (e.g., on a vertical wall of the user's landscape 110), while the universe is setting up the prism and launching the application to render virtual content to the prism, the universe may display the loader volume 210 with a default icon as a placeholder volume to indicate to the user that the universe is setting up the prism for display. Once the application finishes rendering the virtual content into the prism for display in the user's landscape, the loader volume 210 is replaced by the actual prism containing the rendered virtual content.

[0093] In some embodiments, while the universe is launching an application for displaying virtual content, the user 103 can move the loader volume 210 to a different desired location. In some embodiments, the user can move the loader volume 210 to a different location than the location of the loader volume / prism that was initially selected. Once the universe has finished creating the prism and the application has rendered virtual content into the prism, the universe can replace the loader volume 210 with the display virtual content prism, regardless of where the user has placed the loader volume 210.

[0094] The secondary UI volume 220 is another prism that may be created when the prism 113 (e.g., its "parent prism") is created. The secondary UI volume 220 provides a common interface for the prism to the user. For example, since the secondary UI volume 220 provides a mechanism for managing the prism (e.g., close / remove, share, follow, take a screenshot of the prism content, etc.), the secondary UI volume 220 may be considered a window decoration. When the prism is created, if the prism is not part of the launcher (the launcher application may not have a secondary UI volume), a secondary UI volume 220 may be created for the prism. The secondary UI volume 220 provides space / volume to display graphical user interface icons such as close / remove, share, follow, screenshot, etc. for the user to interact with and manage the prism. The secondary UI volume 220 is associated with the parent prism and can be grouped with the parent prism. The secondary UI volume 220 lifecycle ends when its lifecycle is associated with the parent prism and ends.

[0095] In some embodiments, the secondary UI volume 220 may have at least three states: (1) displaying nothing when the parent prism is not in focus; (2) displaying the "visible name" of the component when the parent prism is in focus; (3) displaying a "carousel" of application menu options icons when a specific user interaction is detected, such as when the home button of a handheld controller (e.g., a totem or other suitable user interaction controller) is held down for a certain number of seconds, where the carousel displays a collection of icons, one of which may be a large "X" icon for closing the prism. In some embodiments, the secondary UI volume 220 receives input via its parent prism. In other words, the parent prism may determine whether the secondary UI volume 220 is displaying its carousel, and if so, the parent prism redirects the user input to the secondary UI. The carousel of the secondary UI volume 220 is disclosed below.

[0096] In some embodiments, the launcher can be the default "home" menu for the application mixed reality system. The launcher can place multiple content panels next to the system status bar. Each panel can represent a different content type. Apps can be pulled out of the launcher and pinned in the landscape for quick access. The launcher itself can be placed in the landscape for customization and / or quick access to each location.

[0097] The launcher 230 provides the user with the ability to launch new applications into the user's landscape 110. The launcher 230 can be an application consisting of a series of body-centered prisms called panels. The panels can be scrolled vertically and horizontally, and, for example, the user can switch between panels by a sliding action. In some embodiments, one panel (e.g., the central panel) can be visible at a time, and its two adjacent panels can be seen as placeholder panels at its sides. When the user slides to the next panel, the placeholder panel may expand to show the entire panel. The panel may include an icon grid application 260, a social panel 280, and a store panel 290. In some embodiments, when the user slides to the next panel, the panel itself is not moved or changed, but the content (e.g., icons) in the different panels can be animated in and out of the central panel (e.g., the active panel). In addition, applications can be pulled from the launcher 230 and pinned to the user's landscape 110 for customization of each location, which will be discussed further below.

[0098] In some embodiments, the applications 140 can communicate with the universe via a centralized rendering service client 250 on each application 140. The centralized rendering service client 250 can communicate with a universe server 205 within the universe 130. The centralized rendering service client 250 can be a client service of a centralized rendering system that allows the applications 140 and other applications that generate content for display in the user's landscape to communicate with the universe via the universe server 205.

[0099] The global server 205 may include services of a centralized rendering system that allows the global to communicate with applications that provide global content to be displayed in the user's landscape. In some embodiments, the communications may include more than the following: rendering data (e.g., input data), requesting security privileges, requesting to show or hide a virtual keyboard, etc.

[0100] In some embodiments, a centralized rendering system may be a system of hardware and software resources dedicated to receiving graphics data from multiple applications to be displayed on a single display (e.g., in a user's view of a mixed reality system). The centralized rendering system combines the graphics data from multiple applications 140 into a "centralized" data structure (e.g., a scene graph) that can be used to render a scene reflecting the graphics data from multiple applications to a display in a realistic and efficient manner. To implement the centralized rendering system, in some embodiments, an application may communicate with a client prism (e.g., Figure 2 The centralized rendering system may be referred to as a "Cali" or "Kali" system hereinafter. For example, because the universe can manage prisms in the real world, the universe may be considered an enhanced version of a Cali server.

[0101] In some embodiments, each application 140 that creates virtual content 115 for the universe communicates with the centralized rendering system and the universe via a centralized rendering service client 250 (hereinafter referred to as a "Cali client") installed on each of the corresponding applications 140. More information may be disclosed in related application serial No. 62 / 479,134, entitled "CENTRALIZED RENDERING," filed on March 30, 2017, the entire contents of which are incorporated herein by reference. The centralized rendering system improves the user's experience by ensuring that virtual content from multiple different applications is properly analyzed and processed, if necessary, to ensure that the virtual content is displayed to the user in a realistic manner. In some embodiments, the universe is an instance of a Cali server with other functions (such as managing prisms). In some embodiments, a client prism is an instance of a Cali client volume, and a server prism is an instance of a Cali server volume, which has other functions, such as the ability to call up an application options display, display a loader volume while a prism is loading its content, collide with other prisms, and be part of a "transform tree."

[0102] The client prisms 215a and client prisms 215b include virtual content generated by the application 140 and sent by the Cali client 250a to the global server 205 to be displayed in the user's environment. In some embodiments, when the application 140 makes changes to the virtual content 115a and 115b, the changes to the virtual content are transmitted from the client prism 215 to the global server 205, and the information is stored inside the global in the corresponding server prism data structure 113. In some embodiments, the application 140 is unaware of where the virtual content 115a is displayed in the user's landscape. The global can manage the display location of the virtual content 115a (e.g., the virtual content 115a after the virtual content 115a has been processed by the centralized rendering system) via the corresponding server prism 113a associated with the client prism 215a. The application 140 can request a new prism by accessing the global server 205. In some embodiments, the global server 205 may be a software module in the global that communicates with a centralized rendering service client 250 from an application that provides virtual content for display in the user's landscape 110. For example, when a user wants to launch an application and display virtual content from the application in the user's landscape, the application may provide the virtual content to the global to the global centralized rendering service on the global via the centralized rendering service client from the application to be displayed in a prism that may be anchored in the user's landscape.

[0103] In some embodiments, the icon grid application 260 may include a recent applications section (not shown) and / or a general applications section (not shown). The general applications section includes icons that represent each application installed on the mixed reality system. The general applications section may initially populate a call to a package manager (not shown) to determine a list of installed packages. An icon will be added for each application in each package. When the package manager notifies the world of package installations and uninstalls, the icon grid application 260 adjusts its icons accordingly. The package manager service manages the installation of applications and maintains information about these applications, such as their names, icon graphics, security permissions, executable files, and data files.

[0104] The recent icon portion may be initially rebuilt from the log on disk and then updated by calls from other services. The package name may be logged to disk when the lifecycle service notifies the launcher of an application launch event and when the package manager notifies the launcher of a package uninstall event. The user may interact with the icon grid application 260 by selecting an icon to launch or extracting an icon to place in the landscape.

[0105] The lifecycle service can be a centralized service that manages the process of starting, stopping, sleeping, and waking up applications. The lifecycle service also knows when an application terminates unexpectedly (crashes). When any of these events occurs, the service's listeners are notified, and the entire domain is one of the listeners. The entire domain accesses this service to start, stop, sleep, and wake up applications. In some embodiments, the lifecycle service provides an application programming interface (API) for controlling the lifecycle of application processes running in a mixed reality system. The lifecycle service may generate a new process to run application binaries with a set of permissions and call APIs on predefined interfaces implemented by the application to control its lifecycle. The lifecycle service also provides a listener interface through which other modules can keep track of applications being started / stopped / paused / resumed. The lifecycle service can be a program separate from the launcher or the entire domain. In some embodiments, the lifecycle service can be middleware.

[0106] In some embodiments, Figure 2 As shown, the icon grid application 260 includes a centralized rendering service client 250b and a client prism 215c. As described above, in some embodiments, an application that displays content within a user's landscape may send its content to the universe via a centralized rendering service client 250 that communicates with the universe server 205. Here, the icon grid application 260 that provides icons of applications installed on the mixed reality system for a launcher menu is like any other application that provides content to be displayed in the user's landscape. However, in some embodiments, an icon within the icon grid application, when selected by a user, may indicate that the universe is starting and launching a new application, at which point the new application may request the universe to create a new prism (e.g., via the universe server 205) so that the application can provide content to be displayed in the new prism. If the application is already executing, the universe may request the application to open a new prism.

[0107] The status bar application 270 includes a status indicator for the mixed reality system. The status indicator and the status bar application 270 are not adjustable by the user. The status indicator can be initially populated by querying a first service for operating and maintaining WiFi services, a second service for maintaining Bluetooth services, and a third service for status. When these services notify the status bar application 270 of the updated status, the status bar can be adjusted accordingly. The status bar provides users with quickly browseable information, and they can respond quickly and effectively from anywhere in the system. In some embodiments, the status bar can be displayed above the launcher. The four main sections in the status bar can be (1) global search, (2) notifications, (3) quick settings, and (4) power. When needed, other temporary sections can be added to the status bar, such as music, calls, sharing, etc.

[0108] When the user is in the launcher menu, the status bar will be reduced to browsable icons. When the user swipes up to the top, it may trigger an animation and the status bar may expand. The status bar may stay above the launcher while the user swipes left and right in the launcher panel. When the status bar is highlighted, it may expand and animate forward. By default, a child selection highlight may appear on the left, such as in global search. If there is another section with more pressing content (e.g., recent notifications, low battery, etc.), the child selection highlight may appear on that section instead.

[0109] The social panel application 280 may consist of a series of contacts that the user may interact with. The social panel may initially be populated by a call to the contact service for available contacts. Each contact may be added to the social panel and displayed to the user as an icon. When the social panel application 280 receives new contacts, updated contacts, and removed contact events, the social panel application 280 may adjust its contact information accordingly. The user may interact with the contact icon by clicking on the contact icon to pop up an options menu with various available contact providers. When the user selects a provider, the launcher application may launch an associated application with the contact's information.

[0110] The store panel application 290 can allow users to search, download, and install applications 140 for the mixed reality system. When a user requests to download and install an application, the launcher application 230 can use an authentication service (not shown) to verify the identity of the user, and then use the package manager to install the application. If the user launches an application from the panel, the life cycle service can be called. In some embodiments, each panel in the launcher can be used as a separate application instead of a launcher application.

[0111] In some embodiments, the global client 225 renders content specific to the global domain. The global server 205 does not render third-party applications. This is because the content within the prism can only be rendered by the global client 225, not by the global server 205. Therefore, to render the infinite (infinity) prism, the loader volume / prism and / or the secondary UI prism, it may be necessary to delegate work to the global client 225 to render those specific types of content for the server. For example, when two prisms collide, the global domain can use the infinite prism to render other graphics around the prism. The infinite prism is further discussed below. Using the loader prism and the secondary UI prism, there may be specific communications between the global server 205 and the global client 225 to coordinate certain functions. For example, the global server 205 can be informed that the application has finished loading. Then, the global server 205 can notify the client loader prism that is currently loading the application. The loader prism will have to react to the event that the application has finished loading by displaying an animation. Once the client loader prism has completed the animation, the loader prism can notify the global server 205 that the animation is complete. The global server 205 can then notify the loader prism of the completion of the animation by forcing the loader prism to be dropped, destroying the loader prism, and displaying the application prism with the rendered animation in place of the loader prism. What has been disclosed is only one example of how the global client 225 works. It will be appreciated by those skilled in the art that there may be other examples when the global client 225 can assist the universe 130.

[0112] Figure 3 Example bounded volumes / prisms in accordance with some embodiments are shown. Application content is presented to the user inside one or more bounded volumes called "prisms". As described above, when a non-immersive application is executed in a mixed reality system, its content will be presented inside a prism. The properties and characteristics of the prisms enable the universe to consistently manage prisms within the user's landscape.

[0113] The volume of the prism 113 may have a Figure 31. The borders provide a bounded volume for the virtual content 115 to be displayed only within the borders of the prism 113. The borders of the prism prevent content from the application displayed within the prism from flowing out or overflowing outside the prism and into the user's landscape. When the user sees the virtual content 115 displayed within the prism 113, the borders of the prism 113 may not be displayed to the user. This is an important feature because in order to maintain a realistic display of 3D content in the user's landscape, it is important that the borders of the prism surrounding the virtual content 115 are not displayed. Those of ordinary skill in the art recognize the importance of not displaying the borders of the prism surrounding the virtual content 115 so that the virtual content 115 can be displayed in a more realistic manner in the user's landscape. In contrast to 2D windows, the borders and borders of 2D windows are usually displayed so that a computer user displaying the 2D windows can clearly distinguish between the content in one 2D window and the content in another 2D window. However, in some embodiments, it may be beneficial to at least temporarily display the borders of the prism, for example, to help solve problems with one or more applications.

[0114] The universe gives the Prism 113 instance to place content within. Applications can render 2D and / or 3D content within the Prism 113 using relative placement algorithms and / or arbitrary transformations, but the universe is ultimately still responsible for the overall interaction model, such as content extraction. Multiple applications can present to the universe via the Prism 113, with process boundaries separating the prisms.

[0115] Each prism allocated in the universe has an associated set of key-value properties that can be adjusted and corresponding bits that can determine behavior or convey information about why a given prism exists. Some properties are read-only for regular applications, but are writable for applications with private APIs. Prisms 113 may include prism properties 310, application-specific properties 320, and virtual content 115. In addition, some prisms 113 include a secondary UI volume 330 for providing additional prism management options to the user. However, in some embodiments, for example, a prism may not have a secondary UI volume 330 because these other types of prisms (e.g., launcher menu prisms) may not require the features provided by the secondary UI volume 330. The secondary UI volume 330 may also not be displayed to the user because of the boundaries of the prism. When a user wants to make changes to the prism, the user can initiate a request to display an application options menu that displays the UI controls of the prism within the volume space of the secondary UI volume.

[0116] Depending on the application being used, Prism may require different characteristics in order to provide appropriate feedback and behavior for its content. Application developers can choose from many pre-programmed options for Prism when creating an application, so their content can be represented correctly according to their preferences. The following are examples of some of these options.

[0117] The prism properties 310 at least partially define the prisms and allow the world to manage and maintain the prisms within the user's landscape. For example, the prism properties 310 may include a default size, a maximum size, a minimum size, an anchoring / placement type (e.g., an option for a billboard, etc.), the behavior of a given prism with respect to anchoring type, anchoring location, sub-prism generation location, sub-head pose highlighting, surface behavior, independent transform control, change size vs. change scale indicators, idle state timeout variables, etc. The prism properties 310 allow the world to track and manage the ability of each prism within the user's landscape. Having a single application manage virtual content displayed in a user's landscape can ensure that content displayed in the user's landscape is displayed in a consistent and reliable manner. Some of the prism properties 310 are further disclosed below.

[0118] Maximum, minimum, and default sizes: An application may have upper and lower bounds specified by the application developer (optionally with other constraints from the global domain). In addition, the application developer may have default sizes on first launch.

[0119] Options for billboarding during movement sequences: Certain objects (e.g., the contents of a plane) can be made meaningful to the user for billboarding during movement sequences to encourage legibility and less management. For example, certain content displayed on a plane can be positioned at a specific location and / or relative to an object, but their orientation is automatically calculated so that the content displayed on the plane is always oriented in the direction that the user is viewing the content displayed on the plane. Other optional body dynamic behaviors can also be added.

[0120] Sub-prism spawning location: Prisms can spawn sub-prisms to create flexible layouts. Application developers should be able to determine the response range of the sub-prism relative to the location where the parent prism may spawn.

[0121] Child head pose highlighting: Apps can choose whether to treat head pose highlights on child prisms as separate highlights, or continue to highlight all child / parent prisms as a single unit.

[0122] Child prism related behavior: A prism can determine whether its child prisms can be anchored to its parent prism in translation, rotation, and scaling, and can also choose whether the child prisms are closed to the main prism.

[0123] On the surface: Prisms can be snapped to a surface and queried to determine if they want a size / scale change. If the surface has space, the prisms can be resized to fit all or part of the surface and factors in the user's field of view (FOV).

[0124] Independent transform controls:An application can request independent control of its translation, rotation, and scale. This allows the application to move and transform itself.

[0125] Resizing vs Scaling: Some applications may choose to change the size of their bounds, rather than just scaling their content. This can accommodate more content to be displayed within its bounds. This may work more like existing computer 2D windows.

[0126] Idle state timeout: Apps may be able to choose how long they spend in an idle state. This handles situations where an app may want to continue playing content even when they are out of sight. For example, an app showing live video may want to continue showing content and playing audio even if the user temporarily looks away.

[0127] Application-specific features 320 may be a list of key-value pairs that store application-specific state information about each prism. The list of key-value pairs is application-specific, and the key-value pairs provide state information of the content of the application being displayed or rendered within the prism. Depending on the application being rendered into the prism, the list of key-value pairs for each prism may be different. For example, if the application is a video streaming application, some key-value pairs may include the name of the video, the viewing time of the video, the aspect ratio used to display the video, etc.

[0128] Both prism characteristics 310 and application-specific characteristics 320 for each prism may be stored in a data structure in the local database 137. operate When the mixed reality system interacts with the prism, the prism data is continuously updated. As described above, the prism instance data of the local database 137 can be maintained by periodically synchronizing with the external database 150. In some embodiments, the local database 137 and the external database 150 can be synchronized in near real time.

[0129] When the user launches the application in the universe, the user can pull the prism out of the launcher menu and then put the resulting volume into the space. Other methods of launching the application can be used, such as clicking on the application icon. In some embodiments, the user can move the prism in altitude and azimuth (on the spherical shell) by moving the controller / input device (e.g., totem) up and down. In order to move the object closer or away, the user can use the joystick on the totem to "push" or "pull" the prism, or can slide the user's finger on the touch-sensitive part of the totem. This has the effect of changing the radius of the shell. In some embodiments, the two buttons on the totem can cause the prism to increase or decrease. Finally, rotating the totem itself can rotate the prism. It is assumed that the totem may have six degrees of freedom (DOF). For example, this is consistent with the type of control used in VR painting applications, but the totem can be any suitable user input device.

[0130] In some embodiments, a prism may not allow itself to be placed in a way that it fully or partially intersects with other prisms. The prisms may not intersect at all, or may not occupy / actively display in the exact same location (anchor point), but for physical purposes, the prisms may overlap a small amount, as described below. If more than one prism is placed in the exact same location, the active application may be displayed, while other applications located in the exact same location may be hidden. The user may be able to tell that there are multiple applications in a certain location by, for example, the dots displayed in the volume. For example, if there are three prisms / applications at a particular location, there may be three dots. If the user is viewing application #2 of the three, the second dot may be brightened, while the other dots may be dimmed. The user can then slide or scroll through the different applications. The graphics can be switched and the dots can be updated (e.g., by brightening the active dot) to show which application is currently active.

[0131] In some embodiments, several prisms may be co-located at the same anchor location. At first glance, this may seem strange. With all the 3D space available to place apps in the user's environment, why would you want to place them in the same place? In fact, it makes perfect sense. For example, a user's favorite place to play virtual tabletop games may be on the kitchen table. In the morning, the user may enjoy playing "Ticket To Ride" while eating breakfast. However, when the user gets home from get off work, the user may enjoy playing "Risk" on the computer. The user can have multiple tabletop games in the same place and switch between them when necessary.

[0132] In some embodiments, a prism can be placed anywhere in space. In this case, the prism can be anchored by the center point of the cubic / rectangular volume. However, if (e.g., during placement) the prism is moved near a horizontal surface in the landscape, the prism may try to snap to that surface. The anchor point may then become the center of the bottom plane of the prism. Similarly, if the prism is moved toward a vertical surface (e.g., a wall), it may try to snap to it, and the anchor point may become the side of the prism that is close to the vertical surface.

[0133] The purpose of an anchor point can be to place a prism so that it does not interpenetrate the surface to which the prism is anchored. An anchor point can also move with the object to which it is anchored. When multiple prisms share the same location, that location may be an anchor point rather than the center point of their respective volumes. Applications do not know or need to know their locations, but applications may ask their respective prisms to review the way to anchor their respective prisms. Applications can also specify which anchor types are valid. For example, it does not make sense to anchor a Halcyon to a vertical surface.

[0134] All of the content (graphics) about the application can be contained within the volume of the prism. The universe may automatically block graphics that extend outside the prism. Since applications do not know about other applications in the world, the universe can manage the interactions that occur between different prisms of different applications.

[0135] A user interface design for placing a prism may require that the prism physically sways (like an object on a string) when the prism is moved in a placed state. Rather than trying to predict what physical behavior different applications want, the prism can feed movement information back to the application as it is placed (through the binder interface). The application can then behave appropriately.

[0136] There may also be physics behavior between prisms when they are placed. This may override your app's physics implementation and your app may stop receiving motion data.

[0137] The prisms may initially resist intersecting. If the user continues to push the two prisms to the same location, the prism may snap to the anchor position of the prism it intersects with. This can be done in a way that feels springy (e.g., similar to how soap bubbles interact with each other) and is roughly based on physics.

[0138] Audio emitters can be placed as child nodes in the application's scene graph. These nodes can be local to the root node transform. Therefore, the prism can be moved, where the movement of the prism does not require the application to update the transform of the audio node. The universe can be responsible for the final transformation of the audio emitter to world space. The prism can also be responsible for constraining the audio nodes to its bounds. Applications may not be able to emit audio from points outside of their respective prisms.

[0139] In some embodiments, spatialized audio may not be desired. For example, if a user places a virtual TV on a wall and focuses on the TV image, the TV's audio may be provided directly to the user without modification. This is likely to provide a better audio experience for the user. In the case of surround sound, the audio signal already has spatial information. Sound may emanate from virtual speakers placed in optimal locations relative to the TV.

[0140] In some embodiments, when pressing a button to control the user's audio intensity, the universe can check the head gesture to determine which prism the user is looking at and send a volume up or volume down event to the corresponding prism. The prism can forward this information to the application running in the prism, and the application can decide how to interpret it. If no application is focused in the landscape, the volume button setting can adjust the global volume.

[0141] In some embodiments, one difference between a conventional 2D window and prisms 113 is that in the case of a 2D window, the borders that set the boundaries of the 2D window tend to be seen by the user to provide a specific border for containing the content within the 2D window, separate from the content outside the 2D window border. However, in some embodiments, the borders of a 3D window (e.g., prism 113) are not visible. If the user could see the outline of each prism (e.g., border), it would break the illusion of "reality", and virtual content displayed inside the prism with its border displayed would look like computational / digital / virtual content, rather than real content. In some embodiments, the borders may be displayed, for example, to enable the user to manipulate as needed.

[0142] Another difference is that 2D windows are typically meant to be controlled and / or interacted with by the user. For example, a close button may always appear in the upper right corner of a traditional 2D window, or a menu bar may be displayed at the top border of a 2D window. However, in the case of prisms, the user typically does not interact with the prism and its borders. Instead, a secondary menu (e.g., an application options menu) may be temporarily pulled down for the user to control and manage / manipulate the prism from a list of options.

[0143] Furthermore, a 2D window is independent of its surroundings. For example, if a user moves the screen, the content displayed on the computer screen does not automatically change. However, the prisms need to be placed in the background with the real world. For example, each prism can be placed in the real world relative to: (1) objects in the real environment (e.g., walls, tables, etc.); (2) virtual objects created to provide a backdrop or canvas for the prism to anchor; and / or (3) the user. In some embodiments, the prisms can be placed in the background with the transferable world and the real world.

[0144] Further, in some embodiments, prisms may not be allowed to overlap / interpenetrate each other, except that for physical purposes, prisms may overlap a small amount. For example, in some embodiments, when virtual content within two or more prisms collides, the virtual content may appear to bounce between the two virtual content when the two virtual content appear to collide with each other. Here, the prisms may overlap a small amount to create the effect of bouncing between the two virtual content. In some embodiments, when the bounding boxes for two or more prisms collide, the prisms and the contents of the prisms may bounce. However, 2D windows on a computer do overlap, and in many cases, 2D windows may be layered on top of each other, hiding each other from the user's view. In some embodiments, if two prisms are anchored at the same location in the user's landscape 110, one of the prisms may be displayed, while the other prism is minimized from the display, with an icon or text or image (or any other visual indicator) displayed to indicate to the user that the other prism is anchored at the exact same location. In some embodiments, the infinite prism may be implemented, for example, to render additional graphics around the prisms when the prisms collide. In some embodiments, an infinite prism may be a prism whose bounds are set to infinity. For example, if two prisms are close to colliding, the universe may render a halo in the area of ​​the space between the two prisms. To handle these exceptions, the universe may create an infinite prism that may contain all of the space around / around the two prisms, the user's entire field of view (what the user can currently see), the user's entire field of view (what the user can see when moving around), etc. This may allow the universe to draw graphics anywhere between the two prisms. In some embodiments, the infinite prisms may not collide or interact in any way. In some embodiments, the infinite prisms do not have secondary UIs, etc. In some embodiments, only the universe may access the infinite prisms. The infinite prisms may be created when the universe is initialized and may exist until the universe is closed. In a second example, the infinite prisms may be used to move characters (e.g., avatars, personal assistants, butterflies, animals, etc.) between other landscape applications, for example, explaining to the user what each application is and / or how to use the application.

[0145] Figure 4An example launcher menu for launching applications to display virtual content as seen by a user wearing, for example, a head mounted system 160, according to some embodiments is shown. The launcher menu may exist as its own application, since launching other applications requires elevated privileges. The launcher menu 400 may include a status bar 410 and may have a body-centered bounded volume (e.g., a prism) referred to as a panel. In some embodiments, the launcher panel includes several different applications that give the appearance of a single application. For example, each panel may be a separate application, but may be given the appearance of a single launcher application. During a launcher menu summon (e.g., a request to open a launcher menu), the universe may make an explicit create call for each panel.

[0146] In some embodiments, a separate application process may be rendered into each panel. These processes may use the same content rendering techniques as the rest of the universe, but in some embodiments, with one or more of the following exceptions: (1) the launcher menu may trigger the launch of other applications; (2) the launcher menu icon and widget icons may be extracted and placed in the landscape - the placed icons may be called shortcuts; (3) the launcher menu itself may be placed in the landscape and moved around; (4) when content extraction begins on the launcher icon, the launcher may load placeholder content that is anchored to the user's hand, 6DOF controller, or other controller, so that once the extraction is complete, the universe launches the application into a prism that replaces the placeholder content; (5) the user may swipe between panels in the launcher menu and expand the corresponding panels. In some embodiments, the panels may not be implemented as a turntable. Alternatively, the launcher panels may be co-located so that when the user performs a swipe left, right, up, or down, the main panel is always fixed as the center panel and content from the minimized panels may be animated in and out of the center panel. In some embodiments, the launcher application itself may be placed by the user within the user's transferable world, thereby creating a launcher shortcut or placing the launcher within the user's transferable world and / or landscape.

[0147] In some embodiments, the launcher menu 400 can organize a series of body-centered panels into a panel carousel. The panel carousel can display one panel at a time so that each panel is running its own application and / or each panel is a separate application. In some embodiments, only one current panel 405 can be expanded and visible at a time. Two body-centered placeholder panels (e.g., 440a and 440b) can be a fixed distance from the currently expanded panel 405 on either side of the currently expanded panel 405. The placeholder panels 440a and 440b can be separate prisms created by the universe, and the placeholder panels 440a and 440b represent the abbreviated form of two panels adjacent to the currently expanded panel 405. In some embodiments, when the user slides to the next panel, the panel represented by the placeholder can become the current panel. Regardless of which panel is represented, the physical model of the placeholder panel can remain the same. In some embodiments, the only visual difference between the placeholders representing different panels is the label (e.g., Figure 4 Recent and Social shown). Other embodiments may have different panels in the same location, but with different icons specific to the panel, see Fig.11 .

[0148] In some embodiments, the launcher menu 400 manages the flow of panels, ideally caching the left and right panels. The panel carousel may include a recently visited application panel 420, a universal application panel 425, a social application panel 430, and a store panel (not shown). The recently visited application panel 420 may include icons of applications recently visited by the user. When the current panel 405 about the universal application panel 425 is currently displayed, the recently visited application panel 420 may be shown in a minimized state. When the user slides in the corresponding direction (e.g., to the right) to make the recently visited application panel 420 located in the front and center, the recently visited application panel 420 may be expanded to display its icon, or according to other embodiments, the content related to the corresponding panel may be animated inside and outside the central panel. In some embodiments, the recently visited application panel 420 may be displayed first and may include an application ring. Any number of panels may be used, and these panels may be displayed in any order. The order and number of panels may be based on user testing, user convenience, and / or user preferences.

[0149] In some embodiments, the social application panel 430 may be shown in a minimized state while the current panel 405 (or any other panel) regarding the general application 425 is currently displayed. If the user swipes in the corresponding direction (e.g., left) to bring the social application panel 430 front and center, the social application panel 430 may be expanded to display its icons, or in some embodiments, content associated with the social application may be animated / displayed in the center panel while the label on the placeholder panel is changed accordingly.

[0150] In some embodiments, the icon 450 of the general application panel 425 may represent Figure 2 450 may represent an application installed in the mixed reality system. A user may launch an application by selecting a particular icon 450. In some embodiments, a user may extract an icon 450 from the panel and place the icon 450 in the user's landscape, at which point the universe may initialize a launch process for the application corresponding to the extracted icon 450. When the corresponding application completes rendering of its content, the extracted icon 450 may be replaced with a prism, wherein the rendered content of the application is displayed within the prism.

[0151] As described above, in some embodiments, the status bar 410 may include a status indicator for the mixed reality system. The status indicator may be initially populated by querying, for example, a first service for operating and maintaining a WiFi service, a second service for maintaining a Bluetooth service, and a third service for status. When these services notify the status bar of updated status, the status bar may adjust accordingly. The status bar may include status indicators for services, such as audio volume, current time, WiFi connection / strength, Bluetooth, and power. Other services may be combined with the above services or alternatively included in the status indicator.

[0152] In some embodiments, based on a request to launch an application from the launcher menu 400, the icon grid application may communicate with the universe, for example, through the prism service and / or the universe service, to launch the application. The universe may create a prism using the specified package name and / or component name. The universe may grant the prism to the application using the application's listener. The prism may now be in "placement mode". Once the user has placed the prism, the user may modify the prism in a variety of ways (e.g., using the "secondary UI" application options menu).

[0153] Figure 5A-5B An example panel carousel replacement is shown in accordance with some embodiments. Figure 5A An example view of a launcher menu is shown with the launcher menu having a portion of the menu represented as “left 520” (eg, Figure 4 440a) and "right 530" (e.g., Figure 4440b) panel in the active panel 505. The currently active panel 505 shows an example icon grid, which depicts the icons within the icon grid as "BB" within the active panel 505. In some embodiments, the icon grid can be a grid, a list, one or more columns with one or more rows, an icon circle, and / or one or more icons forming any other shape. The micro panel 535 shows the available panels in the panel carousel (in some embodiments, the micro panel 535 is not shown to the user). The panel position indicator 540 shows the center panel, which is represented as "C", which points to the "B" panel; the left panel, which is represented as "L", which points to the "A" panel; and the right panel, which is represented as "R", which points to the "C" panel, where there is no "D" panel currently in view, so it is not specified with the panel position indicator 540 (in some embodiments, it is also not displayed to the user).

[0154] Figure 5B Shows how the screen will look like after sliding from left to right. Figure 5A An example view of the launcher menu is shown. The new active panel 555 is now the "A" panel, with "AA" as the icon within the icon grid of the new active panel 555. The two placeholder panels, the left panel 570, shows the "D" panel, and the right panel 580 shows the Figure 5A 505. In addition, the micropanel 585 now displays the available panels in the panel carousel as panels A, B, C, D. However, the panel position indicators 590 have changed accordingly, now showing that the center panel, represented by "C", is pointing to the "A" panel, the left panel, represented by "L", is now pointing to the "D" panel, and the right panel, represented by "R", is now pointing to the "B" panel.

[0155] In some embodiments, the placeholder panel itself can remain constant, with only the label reflecting the base panel changing. Figure 4 In , if the user swipes from left to right, similar to Figure 5B , the new active panel will be the Recent panel, with the right placeholder panel showing as a new panel titled General and the left placeholder panel showing as a new panel titled Store (assuming the panel carousel only has Figure 4 The image of the placeholder panel may remain unchanged, but the titles of the left and right placeholder panels may be changed accordingly. The panel shift may be based on a switching action of a user action, such as sliding from left to right. It will be appreciated by those skilled in the art that any other method of switching between panels may also be used.

[0156] Figure 6A flow chart of a method for starting a mixed reality system after a previous use of the mixed reality system is shown in accordance with some embodiments. Startup process 600 shows a process for powering on / starting a mixed reality system after a user has previously used the mixed reality system and powered off the mixed reality system. At 610, the mixed reality system identifies the location of the user and establishes a head pose to determine the location and / or orientation of the user (e.g., the user's head). The head mounted system 160 may include a GPS for determining, at least in part, the location of the user.

[0157] At 620, the universe may access a transferable world system (e.g., external database 150) and retrieve an instance of a prism that was previously displayed in the current user's landscape. The transferable world may return the instance data, so the universe may reconstruct the previous state of the prisms in the user's landscape by rebuilding the user's local database 137 using the retrieved prism instance data. The transferable world system persistently stores information about the user's environment, such as the prisms that were previously active in the user's landscape 110 and the state of each prism when the user previously shut down the system. Because the user's prism instance data stored in the local database 137 may be erased when the user shuts down the mixed reality system, upon restarting the mixed reality system, the prism instance data may be restored based on data retrieved from the transferable world system (e.g., external database 150). Figure 1 The user prism instance data in the local database 137 is reconstructed using instance data retrieved from the external database 150).

[0158] In some embodiments, for each prism, the instance data may include one or more of the following: (1) prism characteristics, where the prism characteristics may include position, orientation, range width, range height, range depth, anchor type, and / or anchor position; and (2) application-specific characteristics including state information of virtual content previously rendered by the corresponding application. For example, if the application rendered into the prism is a video streaming application, the application-specific information may include the title of the video, the time point at which the video is played, and the aspect ratio of the video being rendered. The application-specific information may be a list of key-value pairs that store state information about each application. The list of key-value pairs may be specific to the corresponding application, and the key-value pairs may provide state information for the content of the application being displayed or rendered within the prism.

[0159] In some embodiments, each user 103 may have their own prism instance data stored in their local database 137, while the master prism database instance data is stored in a transferable world system (e.g., external database 150) that includes other prism instance data for other users applying the mixed reality system and / or prism instance data for locations other than the user's current location. Prism data for users of the mixed reality system may be stored in the transferable world system. In some embodiments, a first user may want to share virtual content displayed within a prism from the first user's landscape with a second user using the mixed reality system who is located at the same location or, in some embodiments, at a different physical location than the first user. Since the prism information is available to the first user at the first user's local database, the instance data for the prism from the first user may be uploaded to the transferable world system when the prism is shared to the second user. The instance data about the prism may then be sent to or retrieved by the second user to be stored in the second user's local database so that the second user's global application can reconstruct the instance data of the prism into the second user's local data and thus display the prism into the second user's landscape.

[0160] At 630, the prisms may be restored at the user's current location. Once instance data regarding the prisms previously deployed at the user's location is rebuilt in the user's local database 137, the corresponding applications corresponding to the prisms previously deployed at the user's current location may be launched. Since the mixed reality system has just been launched, it may be necessary to launch applications that previously provided content to the mixed reality system, but at this time, only applications that previously provided content to the user's landscape need to be launched. As part of the launch of the application, the application may request the universe to create a prism for the application to render content into, where the prism to be created corresponds to the prism that was previously deployed before the user closed the mixed reality system for display in the user's landscape. Once the application is launched and the prisms are created, the application may provide content to the universe to render the content into the corresponding prism.

[0161] In some embodiments, an application may provide content to more than one prism deployed at the user's location. In such embodiments, an application may deploy multiple instances or processes of an application to render content about each instance or process of an application into one or more prisms.

[0162] At 640, the prisms may be displayed into the user's landscape at the location where the prisms were displayed before the user closed the mixed reality system. Because the prisms and applications rendered into the prisms are persistent, the mixed reality system may reconstruct the user's landscape by deploying / displaying the prisms and corresponding virtual content at the location of the user's landscape based at least in part on each of the prismatic properties of each of the prisms. In some embodiments, such as Figure 2 A placeholder prism for the loader volume 210 of the present invention may be displayed at startup in the location of the currently loaded prism. In such an embodiment, once the new prism is created and restored, displaying the new prism in place of the placeholder prism may be as simple as replacing the corresponding placeholder prism with the newly created prism in the location of the placeholder prism. If the prism has a property that marks it as being anchored to the user rather than the landscape, the prism may be restored in a location relative to the user's current location.

[0163] In some embodiments, applications deployed in prisms may be rendered in a state consistent with the state when the user previously closed the mixed reality system. For example, a user's field of view of a user landscape may have four prisms deployed, where one of the four prisms has an application in a dormant state, and each of the other three prisms has an application that is actively rendering. When the mixed reality system is restarted and the four prisms are displayed, three prisms may have an active rendering state, and the fourth prism may have an application in a dormant state until the user reactivates the dormant state of the fourth prism or the prism is configured such that the state can be automatically changed to active rendering if the user is viewing it in the general vicinity. Changing the state of a prism is further disclosed below. In some embodiments, other methods may be used to change the state of a prism.

[0164] At 650, the user's local database may be constant (or at regular intervals, or triggered by events) and automatically updated with prismatic information as the user interacts with prisms in the user's landscape. For example, when an application on certain prisms changes from an active rendering state to an active but not rendering state or to a dormant state, the user's local database will be updated with updated state information about the application within the prism. The updated information from the user's local database may update the deliverable world system (e.g., external database 150) immediately or periodically to ensure that the deliverable world system is kept up to date with the changing states of the applications and prisms within the user's landscape. One of ordinary skill in the art will appreciate that this is just one example of how to change prism characteristics or application-specific characteristics and how to update the local database and then the external database, and that many other similar types of changes may be made to prisms or applications that provide content to prisms.

[0165] By making a program (for example, Figure 1Application 140) manages content creation and enables separate programs (e.g., Figure 1 The global application 130) manages content display, which can solve the following problem: virtual content from one application displayed in a 3D space environment is kept privately related to virtual content from a different application displayed in the 3D space environment while still being able to interact with it.

[0166] Figure 7 A flow chart of a method for displaying virtual content provided by an application into a mixed reality 3D spatial environment by a global application in accordance with some embodiments is shown. At step 710, a global application of a user may receive a request to display virtual content in the user's landscape (e.g., a 3D spatial environment). In some embodiments, the request may have been initiated by a user's interaction with a launcher menu, where the user selected an icon from a grid of icons to launch an application to provide virtual content into the user's landscape. As described above, an application (e.g., Figure 1 Application 140 in the can manage content creation, while separate applications (e.g. Figure 1 The global application 130 in the user's landscape can manage the display of content. This separation of processing can allow multiple applications to create content to be displayed within the user's landscape, while the display management of the content is managed by a separate application to ensure that the multiple virtual content displayed within the user's landscape is accurately and realistically displayed to the user, especially when the virtual content begins to interact with each other.

[0167] At 720, the universe may create a prism so that content created by the application may be displayed within the boundaries of the prism in the user's landscape. It is important to note that although the prism may include a border that defines the boundaries of the volumetric space of the prism, the border of the border of the prism may not be displayed to the user. This may be important because virtual content rendered within the prism to be displayed in the user's landscape should be as realistic as possible to the user viewing the virtual content in the user's landscape. If the border that limits the bounding volumetric space of the virtual content is visible to the user, the virtual content may inevitably appear to be computer generated within the confined space. Therefore, although the prism may include a border and a border to confine the virtual content within the boundaries of the prism, the border may not be displayed to the user. The border of the prism may be displayed in certain circumstances, such as during system development and testing. In some embodiments, the border / border of the prism may be displayed to the user, for example, during placement, so that the user can more easily understand where the prism will be located.

[0168] In some embodiments, a user may request a single application to display multiple virtual contents within the user's landscape. Each virtual content displayed by a single application may be associated with and displayed by its own independent prism, or may also share a single prism. Thus, in some embodiments, if an application requests additional virtual content to be displayed, the universe may create one or more additional prisms for the application, where each of the additional content may require its own prism to be deployed / anchored in the user's landscape.

[0169] The created prism may be provided with a unique identifier of the application requesting display of content, so that when the application provides content to be displayed in the user's landscape, the universe may be able to uniquely identify the prism to render the content received from the application. In addition, the prism may be created to automatically have a predefined set of functions, which in some embodiments may be predefined by the application and / or the universe. For example, the set of functions may include a default minimum / maximum size allowed for the prism, an aspect ratio for changing the size of the prism based on how far or near the prism can be anchored in the user's landscape relative to the user, an anchor type for anchoring the prism in the user's landscape, etc. The set of functions may be stored in a local database as prism properties. In addition, depending on the application that called for the creation of the prism, certain application-specific properties may also be created for the prism.

[0170] At 730, the universe may receive virtual content created by the application. After receiving the content, the universe may render the virtual content to be displayed within the boundaries of the prism. Figure 2 As discussed in, in some embodiments, an application may create content and may send the content to a global centralized rendering service running in the global domain via a centralized rendering service client. From there, the global domain may render the content into a prism created for the application requesting display of the content. The global domain may also be responsible for ensuring that the prism is correctly displayed in the user's landscape. For example, the global domain may ensure that the prism does not overlap with other prisms in the user's landscape. The global domain ensures that new prisms will not be placed in an overlapping position relative to existing prisms or real-world content unless an exception (e.g., a common anchor point) as described above is applied. The global domain may not allow users to place prisms in positions where prisms overlap. In some embodiments, a client / server communicating with an application may not be required. If the operating system (e.g., a portion of the system that manages rendering) is configured as multi-threaded, the application may directly render itself into the prism without a client / server structure. A central agency, such as a global server, may then be responsible for managing collisions, movement, input, etc.

[0171] At 740, based on the user's input or default anchoring rules that may be applicable to a particular application or prism, the prism may be associated with: (1) a fixed position in space of the user's landscape, (2) a physical object within the user's landscape, (3) a virtual object displayed within the user's landscape, or (4) a fixed distance relative to a particular object (e.g., the user's body or body part, also referred to below as a body-centric anchor). In some embodiments, the body-centric anchor may be relative to a body part of the user (e.g., body-fixed) or relative to a head / field of view position (e.g., head-fixed). For example, a prism may be associated with a body-fixed position such that the prism is fixed relative to the user's body. If the user moves his head, the content may not move, but if the user walks, the prism may move with the user to maintain a fixed position relative to the user's body. As another example, a prism may be associated with a head-fixed position such that the prism is fixed relative to the user's head or posture. If the user rotates his head, the prism may move relative to the user's head movement. Similarly, if the user walks, the content may also move relative to the user's head.

[0172] At 750, the prism is placed / anchored in the user's landscape. Depending on how the user launched the application, the prism may already be associated with a specific anchor / location. For example, the user may launch the application by dragging the application icon from a grid of icons in a launcher menu and placing the icon at a specific location in the user's landscape, at which point the prism may be created at that location and the content rendered into the prism at that specific location.

[0173] Figure 8 An example of a diagram for managing application state is shown in accordance with some embodiments. The problem of high computational load for managing, rendering, and displaying augmented reality content may be addressed by automatically changing the state of the prisms based on the relative positioning and position of the prisms relative to the user and, in some embodiments, relative to the user's field of view. In accordance with some embodiments, diagram 800 shows a bird's-eye view of the user's location. User 805 is shown as being located within a volumetric grid, which is a rough representation of a planet (e.g., a physical environment) along the x, y, and z axes. Grid lines are shown for exemplary purposes to indicate the distance of the user 805 to the prisms having different prismatic states 870. Here, a 2D representation of the user's current location is shown. One of ordinary skill in the art will appreciate that the view could be a 3D representation of the user's current location. However, for simplicity purposes, Figure 8 Shown as showing a 2D space.

[0174] The user 805 is shown as being at the center of a three by three cell outer region 810. In addition, the outer region 810 is segmented into two circular regions, including an active region 820 and a buffer region 830. The outer region 810 includes a three by three cell grid, with the user 805 at the center of the grid. As the user 805 moves through the user's landscape, other prisms associated with corresponding applications may enter and leave the outer region 810 as shown in FIG. Figure 8 The activity zone 820 shows an area (e.g., a circular area) around the user 805 to indicate certain prism states 870 of the prisms located within the activity zone 820. For example, dormant prisms within the activity zone 820 of the user 805 need to be activated, while other running prisms must remain running even if their states are active but not rendered 870b located within the activity zone 820.

[0175] Buffer 830 is an area farther from the user than active area 820. Prisms in this area may not need a state change. For example, running prisms remain running, for example, one prism located in the buffer has a prism state of active but not rendering 870b. As another example, dormant prisms may remain dormant as in two prisms with dormant prism state 870a in buffer 830. Prisms located outside buffer 830 but within external area 810 may remain in a dormant state because they are far enough away from user 805, so they may not need to change state. Finally, prisms located in external related units 815 may have unloaded, dormant or background prism states because they are too far away from user 805, so that the mixed reality system may be beneficial to save processing resources by not running those specific prisms until user 805 is located at a closer distance from the corresponding prisms.

[0176] The prism state legend 860 discloses various different prism states of the prism based on a color scheme, where a red prism indicates a dormant prism state 870a, an orange prism indicates an active, non-rendering prism state 870b, a green prism indicates an active and rendered prism state 870c, a purple prism indicates an unloaded prism state 870d, and a blue prism indicates a background, active, non-rendering prism state 870e.

[0177] The user 805 may have a particular field of view 840, wherein the prisms within the field of view 840 may be located in a frustum 850. The frustum is the 3D area visible on the screen of the user 805 (e.g., via Figure 1870c). The running prism located within the frustum 850 should be rendered, as indicated by the prism with the prism state of active and rendering 870c. Since there is a prism with the dormant prism state 870a within the frustum 850 and the buffer 830, the particular prism may be started and rendered when it is located within the buffer 830 and the frustum 850. In some embodiments, the head pose frustum can be used to remove rendering of prisms, where frustum removal is a method of determining hidden surfaces (such as hidden surfaces of prisms).

[0178] Table 1 below shows a prism state change diagram, where the rows in the first column describe the current state of the prism. The rows in subsequent columns describe the change of the prism state from the current state to the new state as the prism moves from the current area to the new area, as shown in subsequent columns. The values ​​in the rows in subsequent columns describe the new state that the prism may become (if any) based on the relative distance of the prism to the user 805.

[0179] Table 1 - Prism state change chart

[0180]

[0181]

[0182] For example, if the prism is in a current "dormant" state and based on the movement of the user 805 the prism may be placed within the active area 820 relative to the user 805, the prism state of the prism may be changed to "active no rendering" because the dormant prism should be activated when it is within the active area 820 of the user 805. Thus, following the same example, if the prism is in a current "dormant" state and once the prism is within both the buffer 830 and the frustum 850, the state remains unchanged because a dormant application may remain dormant even if within the buffer 830 and the frustum 850. However, once the dormant prism transitions to the active area 820 and the frustum, the prism may begin to activate and the prism state may be changed to active and rendering 870c.

[0183] Fig. 9A flow chart for managing prism states relative to a user's location is shown in accordance with some embodiments. At 910, the spatial location of a prism may be recorded in a volumetric grid. The spatial location may be an anchor location of a particular prism placed / anchored within a user's landscape. For example, a prism may be anchored to an object within the user's landscape, such as a wall or table. As another example, a prism may be anchored at a fixed location in space that is not associated with any particular object within the user's landscape (e.g., a floating 3D globe of planet Earth). When a prism is activated, its spatial location is recorded in a volumetric grid that may have x, y, and z axes relative to the world to identify the location of the prism into which the prism content may be rendered. Each prism placed in the user's landscape may have a specific location stored in a data structure of the prism stored in a navigable world system (e.g., as a volumetric grid). As described above, the volumetric grid may represent a rough representation of the planet in an x, y, and z axis format. Any other reasonable coordinate or relative placement system may also be used.

[0184] At 920, a cell of a volumetric grid representing a 3D spatial region is identified. A cell may include one or more prisms if the prisms are spatially adjacent to each other. In some embodiments, the width of the cell may be equal to or greater than the radius of the active zone. In some embodiments, the width of the cell may be less than the radius of the active zone. The radius of the active zone represents a measurement of the distance from a user using the mixed reality device to a preconfigured distance (e.g., 25 meters) that would define a circular and / or spherical (or other suitable shape) spatial region as such. Figure 8 The active area 820 is shown. Optionally, the volume grid may be divided into units of predetermined lengths to further establish boundaries and borders corresponding to the distance from the user using the mixed reality device to establish different areas to implement the embodiments of the present disclosure.

[0185] At 930, as the user using the mixed reality device moves into a location in the cell, the distance to the known location of each prism within the cell and adjacent cells is determined. As the user using the mixed reality device moves through the user's landscape, a distance check to the last known location of each prism is checked to manage the state of prisms near the user. To prevent distance checks from being performed on prisms throughout the world, the system can minimize distance checks / calculations / determinations by using a volumetric grid and limiting the search / checks / calculations / determinations to only cells surrounding the mixed reality device (e.g., the cell and adjacent cells). In some embodiments, distance checks are performed only on prisms in the cell occupied by the user. In some embodiments, distance checks are performed only on prisms in the cell occupied by the user or adjacent to the cell occupied by the user. In some embodiments, distance checks are performed on prisms in all cells within the user's FOV / viewing frustum. In some embodiments, distance checks may be performed on prisms within a distance of the user, such as within 10 feet or 25 feet of the user.

[0186] At 940, depending on the distance between the prism and the user, the state of the prism may be modified. In addition to the distance, other factors may also determine whether the state of the prism is modifiable, such as the current state of the prism relative to different zones and the user's field of view. For example, if the current state of the prism is dormant and the user moves to a position where the prism is located within the active zone 820 and the frustum 850, the state of the prism may be changed from dormant to active and rendered according to Table 1. However, if the dormant prism is located within the buffer zone 830 and the frustum 850, the prism state may not change because according to Table 1, when the prism moves into the buffer zone and the frustum zone, the current dormant state remains dormant. As described above, the buffer zone primarily prevents intermittent or rapid changes in the state of the prism. The user's head posture may modify the state of the prism because the user's head posture helps define the user's field of view, which ultimately helps define the frustum zone 850. As described above, the user's head posture is a measurement of the position and / or orientation of the user's head. Head pose information can be used to render the scene to match the user's dynamically changing head position and orientation to provide increased immersion in the virtual / augmented / hybrid space.

[0187] In some embodiments, a wall may separate the user from the prism so that the prism may remain in a dormant state even if the user's distance places the prism within the active zone 820 and / or the user's field of view orientation places the prism within the frustum zone 850. The state of the prism may not change from dormant due to occlusion by the wall, which prevents the user from seeing the prism. Further computational processing may be avoided by the system recognizing that even though distance and head pose may indicate a change in the state of the prism, other factors such as a wall or object blocking the prism may cause the state of the prism to remain unchanged.

[0188] Managing Prism Relationships and Grouping

[0189] In some embodiments, the centralized rendering system may not support transforming / managing client prisms 215 as part of a larger (e.g., global space) scene graph. A global prism may be an instance of a client prism with other logic and functionality. In the global domain, the ability to transform a group of client prisms 215 may be handled via a transformation tree. For example, each panel of a launcher may be a separate application, each of which corresponds to its own prism. It may be beneficial to be able to group multiple prisms associated with a launcher into a group to manage multiple prisms together so that when one prism in the group is modified, other prisms in the group may be affected and may also need to be modified, which may be beneficial to the global domain. This can make multiple applications appear and behave as if they were a single application. For example, if the launcher is anchored to the user's body (e.g., body dynamics), the position of the launcher and the position of its panels corresponding to multiple prisms may also need to be modified / transformed into a group as the user moves in his environment.

[0190] As another example, the application requesting the prism may have a secondary UI (with a close button, pin / unpin, settings, etc.) that is positioned relative to the prism near the application. The universe may group the two prisms into a group and manage the two prisms as a group with two nodes. The association of managing multiple prisms in a hierarchical / relational manner and / or in a logical grouping structure may be handled with a transform tree and / or a group tree (discussed below).

[0191] In some embodiments, a transform tree is a tree data structure configured to transform / organize prisms and / or prism groups in a global domain, and the transform tree can be rendered by a central rendering system. The transform tree structure can have a scene graph of nodes, each of which provides a movable local space corresponding to a prism. The nodes of the scene graph can correspond to prisms. Prisms can be tree nodes. There may be other tree nodes that are not prisms. These "other tree nodes" can be used to group content or serve as intermediate transformations. In addition, nodes can be grouped with other nodes so that when one node is transformed, other nodes in the same group are also transformed. In some embodiments, a group tree can manage the grouping of nodes.

[0192] A group tree may be a tree data structure that stores logical groupings of nodes.

[0193] In some embodiments, since the prisms may not overlap and may be viewed as independent content displayed in the user's landscape, this may introduce the problem of virtual content from one application not being able to interact closely with virtual content from a different application. However, creating multiple smaller prisms that tightly surround virtual content from a single application may solve this problem. For example, there may be multiple prisms opened for a single application, but the user may view the multiple prisms as a unified system (e.g., parent / child prisms of a transform tree). In addition, prisms may also be grouped together so that some prisms may essentially be equivalent to each other. If all content from one application is located within a large box, there may be more "air" or dead space within the prism. This may prevent content from different applications from interacting with content in those areas, thereby preventing certain types of "realistic" interactions from occurring.

[0194] For example, an interaction may be a simple collision of two 3D cars. If each of the prisms into which the two 3D cars are rendered has such a large "dead space" between the prisms relative to the invisible boundaries of the 3D cars that the user may not be able to see that the two cars are actually touching each other, then the collision may not be "real" to the user. However, if the prisms are small and tightly surround each of the 3D cars, then from certain viewing angles, it may appear to the user that the two 3D cars are slightly touching each other and are bouncing off each other due to the collision.

[0195] Fig. 10A An example tree node of a scene graph according to some embodiments is shown. Tree node 1000 depicts an example tree node for a launcher menu. Global node 1002 is the root node in this tree node of the scene graph of the user's mixed reality system. When the mixed reality system is started, a launcher application can be launched to be displayed to the user as a launcher menu. When the launcher application is launched, a launcher tree node 1004 representing the location of the launcher menu in the scene graph can be created. Launcher tree node 1004 can be used as an anchor for other nodes that depend on or are children of the launcher menu.

[0196] For example, an icon grid application may also be launched. An icon grid tree node 1009a may be added to the tree node using the scene graph identifier (SGID) of the launcher application. Since the icon grid application may be displayed as a panel in a turntable of panels for the launcher menu, the icon grid tree node 1009 may be a child of the launcher tree node 1004, as shown in the child node 1008 indicated by a dashed border marked as a launcher pane. More subordinate applications of the launcher may be launched and added to the tree node as child nodes of the launcher tree node 1004 (e.g., the recently evaluated application tree node 1027 and the social application tree node 1029). The node groups associated with the launcher menu may be grouped together as a launcher group 1006.

[0197] When the launcher menu is moved (e.g., an ad), the nodes within the launcher group 1006 can be moved as a group. In order to provide an exact method to accurately position / ad the launcher menu, rather than applying logic to one of the launcher panels and hoping to best maintain the grouping of other related panels, it may be desirable to set the launcher node as an anchor node. Child nodes (e.g., icon grids, recently accessed applications, and social applications) are automatically repositioned relative to the launcher tree node. The universe can own the entire scene graph of multiple prisms. In some embodiments, each prism can start with an empty parent and zero children, effectively making it a scene graph with one node. In some embodiments, when an application requests a prism, the universe can create (1) a node for the requested prism and (2) a node for the secondary UI. The universe can then position the secondary UI relative to the prism. In some embodiments, the universe can place the secondary UI node as a child of the prism node.

[0198] Transform Tree

[0199] As described above, a transformation tree is a tree data structure configured to transform / organize prisms and / or prism groups in a global domain. The transformation tree structure can be considered as a scene graph including nodes, each of which provides a movable local space that can correspond to a prism. The nodes of the scene graph may correspond to prisms, or may be nodes that are used only to transform a group of prisms or other tree nodes. In this case, the nodes can be grouped together with other nodes under the parent node so that when one node is transformed, other nodes in the same group are also transformed together. In some embodiments, the prism is located in a transformation tree with only one prism (i.e., itself), but when it is necessary to parent a prism with another prism, the ability is there. In some embodiments, the prism scene graph / transformation tree may differ from a conventional scene graph in that when a child prism changes its parent, the child's world transform may not change. For example, if a prism is rendered in front of a user, the prism will not move from its current position when the prism's parent changes. In some embodiments, the matrix for each prism within the transform tree may be column-major, which means that the world matrix for a prism within the tree must be computed by appending the parent transform to the left of the prism's local transform. For example, the prism's world transform may be equal to the parent's world transform multiplied by the prism's local transform.

[0200] In some embodiments, basic tree logic may be defined as one or more of the following: (1) a node may set itself as a child of another node; (2) a node may set another node as a child of itself; (3) a node may get its parent; (4) a node may get its children; (5) a node may remove any of its children (this does not isolate these children - they may be attached only to the node's parent or to any other node in the tree, including the root global node, which may be placed into world space); (6) a node may remove its parent. Inserting and removing nodes may update the local transform of the node and its children. This means that changing a node's parent may not change its world transform and may in fact preserve the world transform. In some embodiments, any operation that changes a node's local transform may update the world transform of the node and its children.

[0201] In some embodiments, each node in the transform tree may contain one or more of the following matrix components: (1) a local transform; (2) a rotation matrix (which may be used to calculate the local transform); (3) a scale vector (which may be used to calculate the local transform); (4) a position vector (which may be used to calculate the local transform); (5) a local to world transform (which may be cached for convenience); and (6) a local to world scale (which may be cached for convenience). In some embodiments, a local transform may be considered to be the transform applied to its parent node. If a node has no parent, the local transform may be equal to the local to world transform. The local transform may be calculated as follows:

[0202] Local transformation = position * rotation matrix * scale

[0203] This formula can be understood as, "Using the parent node as its origin, transform the node by first scaling it, then rotating it, and then changing its position relative to its origin node." Rotation, scaling, and positioning can be kept independent so that getters / setters can be easily provided through the interface to modify any of them separately. When a node changes position, the child node world transform may be updated accordingly. When these matrices are multiplied in sequence, a local-to-world matrix can be constructed. For efficiency, the calculated local-to-world matrix can also be stored. In some embodiments, to calculate the local-to-world transform of a node, the following formula can be used:

[0204] World Transform = Parent World Transform * Local Transform

[0205] Note the order of matrix multiplication. In some embodiments, the matrix is ​​column-major, which means that x, y, z, and position are arranged in each column. This means that the parent node matrix always appears to the left of the child matrix. Fig. 10B A small transform tree 1010 is shown and how each world transform for each node is calculated.

[0206] Other suitable methods or systems utilizing transform trees may be used.

[0207] Group Tree

[0208] In some embodiments, an additional feature of the transform tree is the ability to group any prism within a transform tree with any other prism. Prisms in a group may be in the same or different transform trees. In addition, a prism may be a child or descendant of another prism within the same group and "transform tree". In some embodiments, when a transform is applied to one of the prisms in a group, all other prisms in the group are transformed along with it. The prism group logic is responsible for updating the transforms in any transform tree to which the group members belong. Any prism that is not in a group but is a child of a prism in the group may also be transformed with the group. When a prism in a group is transformed, its children are also transformed. For example, if prisms A and B are in a group, and prism C is a child of prism B but not in the group, and prism B moves 1 meter to the left, then both prisms B and C will move 1 meter to the left. However, if prism C moves 1 meter to the left, only prism C will move 1 meter to the left. The present disclosure shares some features with conventional scene graphs, but may also have additional functionality / behavior, such as 1) the ability to group indirect descendants of a node together with the indirect descendants, so that the node and the indirect descendants undergo the same transformation (if they are grouped), and / or 2) the ability to group two nodes that are on completely independent transformation trees. In some embodiments, an indirect descendant may be a node that is not directly connected to a parent node, for example, may be separated by two or more intermediate nodes. In some embodiments, the grouping logic may be implemented by creating a root group node for all prisms belonging to the current group. If any prisms in the same group are descendants of other prisms in the group, they are also descendants of these prisms in the transformation group tree. Note that in some embodiments, a separate data structure may not be created for a group; instead, the group may have an additional flag attached to the prism. In some embodiments, if any prism in the group is transformed, the transformation may be calculated and applied to the root group node. The transformation may be propagated down the transformation tree like any other transformation, where the transformation is propagated down to each child in the transformation tree and each prism in the group.

[0209] Following are various embodiments of adding nodes to and removing nodes from a group tree. Figures 10C-10E One embodiment for creating a new group is shown. Fig. 10C In the transformation tree 1020 in FIG. 1020 , this embodiment will show the grouping of nodes A and E together, which are currently in separate transformation trees. (1) Determine whether A or E is already in a group. Currently, neither A nor E has a parent group, so neither of them is in a group. Therefore, a group root is created. Fig. 10D , create a group tree 1022 with a group root 1024. (2) Fig.10E As shown, the parent groups of A and E are assigned to group root 1024. (3) Determine whether E is an ancestor (or parent) of A. A has no parent, so E is not an ancestor of A. (4) Determine whether A is an ancestor (or parent) of E. E has no parent, so A is not an ancestor of E.

[0210] Figures 10F-10I Another embodiment for creating a new group is shown. Fig.10F , this embodiment shows the grouping of nodes A and C together, where node C is a child of node A, as shown in the transformation tree 1030. (1) Determine whether A or C is already in the group. Currently, neither A nor C has a parent group, so they are not in the group. Therefore, create a group root. Figure 10G In , a group tree 1032 with a group root 1034 is created. (2) The parent group of A and C is the group root 1034. Fig. 10H As shown, A and C are assigned to group root 1034. (3) Determine whether C is an ancestor (or parent) of A. A has no parent, so C is not an ancestor of A. (4) Determine whether A is an ancestor (or parent) of E. A is an ancestor of C. Therefore, reassign C's parent group to A.

[0211] Figures 10J-10K An embodiment is shown in which an ungrouped node is added to an existing group. for Fig.10J The transform tree 1040 and group tree 1042 in FIG. 1 show that nodes C and E from the transform tree 1040 are grouped into the same group. (1) Determine whether C or E is already in the group. Fig.10J As shown, E is in the same group as A. (2) Determine the parent group of C – Find the parent or ancestor of C that is in the same group as E. Here, A is in the same group as E. A is an ancestor of C. Assign the parent group of C to A (see Figure 10K ). (3) Determine whether C has an ancestor or parent of any other node in the group. C has no children. Processing stops.

[0212] Figure 10L-10N Another embodiment of adding an ungrouped node to an existing group is shown. for Fig.10L 1050 and group tree 1052 in the embodiment, which shows that nodes B and E from transform tree 1050 are grouped into the same group (e.g., group tree 1052). (1) Determine whether B or E is already in a group. Here, E is in a group with A and C. (2) Determine the parent group of B by finding the parent or ancestor of B that is in the same group as E. Here, A is in a group with E and A is the parent of B. Assign the parent group of B to A (see Figure 10M ). (3) Determine that B is the parent or ancestor of any other node in the group. Here, C is the child or descendant of B. Change the parent group of C to B (see Fig.10N ).

[0213] Figure 10O-10R Another embodiment of adding an ungrouped node to an existing group is shown.For transform tree 1060 and group tree 1062, this embodiment shows grouping nodes A and E from transform tree 1060 into the same group (e.g., group tree 1062). (1) Determine if A or E is already in a group. Here, E is in a group with B and F. (2) Determine the parent group of A by finding the parent or ancestor of A that is in the same group as E. Here, A has no parent, so A is assigned to group root 1064 (see Figure 10P ). (3) Determine if A is the parent or ancestor of any other node in the group. Here, A finds that B is a child. Therefore, B sets its parent group to A (see Figure 10Q ). In addition, A finds F as a descendant. A and F belong to the same group (e.g., group tree 1062). Therefore, F sets its parent group to A (see Figure 10R ).

[0214] Figure 10S-10W An embodiment in which a grouped node is added to another group is shown. for Fig. 10S , this embodiment shows that nodes C and G are grouped into the same group. (1) Determine whether C or G is already in a group. Here, C is in a group with D. G is in a group with B and F. (2) Since both C and G are in a group, C and G can be merged into a single group by arbitrarily selecting the last group as the group for C and D (e.g., group root 1074). (3) Only direct children of the old group root 1076 are added. This means that only B and F are added to C's group. (4) B is added to C's group. (4a) Determine the parent group of B by finding the parent or ancestor of B that belongs to the same group as C. Here, there is no parent group. Therefore, B is added as a child of group root 1074 (see Figure 10T ). (4b) Determine if B is the parent or ancestor of any other node in the group. Here, B finds C as a child. B and C belong to the same group. C sets its parent group to B (see Figure 10U ). (5) Add F to C's group (note: this is not arbitrary. G is never chosen to be added next, it is only pulled out with F). (5a) Determine F's parent by finding a parent or ancestor of F that is in the same group as C. Here, F has D as its parent. F sets its parent group to D (see Figure 10V ). Note that G is automatically placed in the same group. (5b) Determine if F is the parent or ancestor of any other node in the group. Here, F finds G, but G's parent is already F. (6) Delete the old group root 1076 (see Figure 10W ).

[0215] Transform a group

[0216] In some embodiments, any transformation applied directly to a node belonging to a group may transform the group along with the node. Indirect transformations applied to a group node may not transform the group. For example, if a parent node is not in a group, its children are in a group, and the parent is transformed, then even if the child's transform changes, the nodes in the child's group will not transform along with it. If the entire group needs to be moved, the parent should be added to the group.

[0217] In order to compute a transform to apply to a group, the transform to be used must be applied to each direct child of the group root.

[0218] Figure 10X-10Y Sample calculations according to some embodiments of the present disclosure are shown. Figure 10X Given the transformation tree 1080 and group tree 1082 shown, this embodiment applies a transformation to node A. Since A belongs to the group tree 1082, it is expected that nodes C, D, and E will be transformed with node A. In addition, node B, which does not belong to the group, must also be transformed with the group because it is a child of A. We denote A's new world matrix as: Maw', where Maw' stands for "matrix a world new". We denote A's old world matrix as: Maw, where Maw stands for "matrix a world old". We denote the matrix to be applied to A to obtain Maw' as: Mgw, where Mgw stands for "matrix group world". The goal is to find a matrix that can be set to the group root (Mgw) that, when applied to the world matrix of node A, will be equal to the new matrix of node A. This same matrix is ​​applied to other direct children of the group root.

[0219] Figure 10Y A sample calculation is shown applying the transformation to node A. Figure 10Y In the example above, we care about node A because it is the node where the matrix changes. So:

[0220] Maw'=Mgw*Maw

[0221] We know Maw' and Maw to solve Mgw

[0222] Mgw=Maw'*Maw^-1

[0223] Therefore, for each child node of the group root, set the node's world transform to: childnodenewworldmatrix = Mgw*(childnodeoldworldmatrix). For this example, Mgw is applied to the two direct children of the group root, A and E:

[0224] Maw'=Mgw*Maw

[0225] =(Maw'*Maw^-1)*Maw

[0226] =Maw

[0227] Mew'=Mgw*Mew

[0228] =(Maw'*Maw-1)*Mew

[0229] Note that the final world transform of A is what we wanted, Maw'. And since the transform propagates up the transform tree, all child nodes (regardless of whether they are part of the same group or not) are updated. This means that node B is also updated.

[0230] Figure 10Z-10AA Sample calculations according to some embodiments of the present disclosure are shown. Figure 10Z Given the transformation tree 1090 and group tree 1092 shown, this embodiment applies a transformation to node C. Since C belongs to the group, nodes A, D, and E are expected to transform with it. In addition, node B, which does not belong to the group, must also be transformed with the group because it is a child of A. We denote the new world matrix of C as: Mcw', where Mcw' stands for "matrix c world new". We denote the old world matrix of C as: Mcw, where Mcw stands for "matrix c world old". We denote the matrix to be applied to c to obtain Maw' as: Mgw, where Mgw stands for "matrix group". The goal is to find a matrix that can be set as the group root (Mgw), where when it is applied to the world matrix of node C, it will be equal to the new matrix of node C. This same matrix is ​​also applied to other direct children in the group root.

[0231] In reference Figure 10AA , we care about node C since it is the node where the matrix changes. Therefore, Mcw'=Mgw*Mcw. We know Mcw' and Mcw, so solve for Mgw. Mgw=Mcw'*Mcw^-1. Therefore, for each child node of the group root, set the node's world transform to: "child node new world matrix=Mgw*(child node old world matrix). For this example, recall that Mcw is equal to (Mcw=Ma1*Mb1*Mc1). Combining this with Mgw=Mcw*Mcw^-1, we get Mgw=Mcw'*(Ma1*Mb1*Mc1)^-1, where Mgw=Mcw'*Mc1^-1*Mb1^-1*Ma1^-1. And don't forget, since A and E are root nodes, Maw=Ma1 and Mew=Me1. In this way, we can find the new transform for each node. Apply Mgw to the two direct children of the group root, E and A:

[0232] (E is the root node of the transformation tree, so Mew == Me1).

[0233] Mew'=Mgw*Mew

[0234] =(Mcw'*Mc1^-1*Mb1^-1*Ma1^-1)*Mew

[0235] =(Mcw'*Mc1^-1*Mb^-1*Ma1^-1)*Me1

[0236] (A is the root node of the transformation tree, so Maw == Ma1)

[0237] Maw'=Mgw*Maw

[0238] =(Mcw'*Mc1^-1*Mb1^-1*Ma1^-1)*Maw

[0239] =(Mcw'*Mc1^-1*Mb1^-1*Ma1^-1)*Ma1

[0240] (Since A is the parent of other nodes, the world transform of child nodes B and C is updated)

[0241] Mbw'=Maw'*Mb1

[0242] =(Mcw'*Mc1^-1*Mb1^-1)*Mb1

[0243] =Mcw'*Mc1^-1

[0244] Mcw'=Mbw'*Mc1

[0245] =(Mcw'*Mc1^-1)*Mc1

[0246] =Mcw'

[0247] Note that the final world transform of node C is what we wanted, Mcw'.

[0248] In some embodiments, the transform tree and transform group functionality may be implemented within the universe itself. In some embodiments, other applications (e.g., application 140, icon grid application 260, status bar application 270, social panel application 280, store panel application 290, etc.) may not need such functionality because the universe may be managing this functionality. Other suitable methods or systems utilizing group trees and / or transform groups may be used.

[0249] Placed launcher

[0250] In some embodiments, the user may be able to place the launcher in the landscape where they want to launch the application or permanently pin the launcher. This makes the launcher a natural extension of the user's landscape and creates a clear connection between the digital world and the physical world. The placed launcher in the open state may have the same layout and structure as the ordinary launcher, except that it is tied to a certain location and can be scaled up and / or reduced. In some embodiments, when the user creates a placed launcher, the user may have the option of pinning the launcher to the location permanently. In this case, the launcher may stay in the location until the user moves it or removes it. When the user presses the home button, even if the launcher has been pinned, the ordinary launcher may still be generated. In some embodiments, the user is able to pin multiple launchers (so, for example, they can place it where they launch the most content). The pinned launcher can display the latest application launched from the location. This allows each launcher to be customized for the space in which it is located. In some embodiments, the behavior of pinning the launcher application to the location is simulated by an extractable content manager (ECM), which gives the appearance of a pinned launcher icon. It does this by literally duplicating the icon into a separate prism and allowing the user to move that prism around.

[0251] In some embodiments, since the launcher can be an application, and the launcher carousel's panels and launcher menu can be presented within a separate prism, the prism itself can be placed within the user's landscape.

[0252] Fig.11 An example diagram of a placed launcher according to some embodiments is shown. Fig.11 As shown, the launcher menu 1100 may include an icon grid panel 1120, a minimized social panel 1110, and a minimized store panel 1130. The icon grid panel 1120 may actively display icons of applications that the user may launch, wherein the minimized social panel 1110 is on the left side of the icon grid panel 1120, and the minimized store panel 1130 is located on the right side of the icon grid panel 1120. The image icon 1140 may indicate to the user that the launcher has been placed in a specific location, in this example, on top of the stool. In contrast to a placed launcher, a different image icon 1140 may indicate a pinned launcher. In some embodiments, different numbers of panels and / or different content for the panels may be used.

[0253] In some embodiments, the pinned launcher menu is persistently stored and may remain pinned in that location until the user moves or removes the launcher from that location. Even if the user shuts down the mixed reality system, upon restarting the mixed reality system, assuming a set of conditions are met (e.g., the user is near the pinned launcher menu), the launcher menu may be displayed in that location after the system is fully booted. In some embodiments, an application launched from a placed or pinned launcher may be generated and displayed in the same location as the pinned or placed launcher. In some embodiments, once the user selects a landscape application, the launcher may disappear and the application (e.g., prism) may appear in the same location.

[0254] If the user chooses to pin the launcher permanently to that location, the launcher may remain there, and the user may interact with the launcher and launch apps from that location until the user moves or removes the launcher from that location. If the user does not interact with the launcher, it may enter its idle minimized state. When the user does not interact with the launcher, the state of the placed or pinned launcher may change to a minimized / idle state. Launchers placed here may display the time, featured store content, featured social contacts, brand moments / experiences, and / or other content.

[0255] Fig.12 Example types of content that can be displayed when a placed launcher is in an idle / dormant state according to some embodiments are shown. A standard view of a placed launcher in its minimized / dormant state can display a time 1210 that represents the current time and date. When a user hovers over a placed launcher with a head gesture and / or clicks / selects a placed launcher, the launcher can change state from dormant to active and render and launch into its full active view, e.g., displaying at least one icon grid and corresponding panel. In some embodiments, featured store content 1220 can be displayed in place of the standard viewing time and date 1210. Featured store content 1220 can be a pre-downloaded preview of an application that gives a quick interactive flash of a product. Other types of content that can be displayed can include magical experiences, such as children's toys 1230 or balloons 1240 with daily messages. These magical experiences can include content and inspiration for users of the device. Artists, musicians, and influencers can create these contents.

[0256] Secondary UI – Application options displayed in the carousel

[0257] Fig.13 An example secondary UI volume is shown in accordance with some embodiments. Application options menu 1310 may illustrate different interactions a user may have with content provided by an application rendered as a prism. Application options menu 1310 may be displayed on Figure 2's volume space of the secondary UI volume 220. In some embodiments, the application options menu can be a carousel with a central object that is always highlighted. When the user slides to the next item, the icon may become more visible and display appropriate text describing the option. Application title 1320 can help the user identify the application that is providing content. Here, as an example, the application is a 2D Internet browser that displays a web page. Settings 1330 can display application settings and / or contextually associated connections to the global settings application. Close application 1340 can close / remove the application from the user's landscape. Share 1350 can allow the user to use the application to start a sharing session with other users. Download 1360 can download content displayed in the prism. For example, the user may want to download an article displayed in the 2D Internet browser. It will be understood by those of ordinary skill in the art that the application options menu 1310 may include more options or fewer options, and may be in an order different from that already disclosed and shown.

[0258] Body Dynamics

[0259] Body dynamics refers to how virtual content anchored relative to the user's body is managed and displayed to the user.

[0260] Fig.14 An example of body dynamics according to some embodiments is shown. Body dynamics may refer to how the user application / user interface / content 1480 moves relative to the user 1490. There may be different types of body dynamics, which may have different types of user interface effects. Body dynamic types may also be referred to as anchoring types, which are used for how the prism and / or applications in the prism are anchored and displayed in the user's landscape.

[0261] Fig.15Different types of body dynamics according to some embodiments are shown. One example of a type of body dynamics is world lock 1520, where there is no body dynamics. In some embodiments, the universal prism is world locked and may not have body dynamics unless the user chooses to activate the follow behavior from the application options. Another example of a type of body dynamics is billboard 1530. In some embodiments, a billboard is when the angle and / or orientation of the application content changes to maintain a fixed orientation and / or angle relative to the user. The relative angle and / or orientation can be determined in any number of ways. For example, the virtual content may have a predefined forward direction or forward direction. The system may require that the forward vector of the virtual content points in the forward direction of the user, as determined by the head posture. In some embodiments, the billboard behavior maintains a fixed position in 3D space, while other embodiments may allow the position to change. In some embodiments, billboard dynamics may also include edge billboards 1540, which only begin to turn towards the user if the user is viewing the content at an angle that is too oblique. For example, the application remains static until the user views the web page from a horizontal angle that makes the text difficult to read. The application follows the edge of the location where the user pushes the frustum cone as if projected from the planar surface of the front face of the prism. Parameters for edge billboarding (e.g., application-specific parameters) may include frustum angle, follow behavior (e.g., centering on the user or moving to the edge of the location where the user pushes the frustum cone), force field (push) Boolean, force field activation distance, speed, and / or flip Boolean.

[0262] The body dynamic type of following 1550 is another type of body dynamic. In some embodiments, following 1550 is a specific mode in which a user can put a single object (e.g., a prism), for example, temporarily attaching the object to the user's head position. This is useful when the user wants the object to move with its head position. The feature of following is that it ignores the pitch (e.g., up and down rotation) of the user's head, which gives the user greater freedom to look around in his physical space. If the user looks left or right, the object may be re-centered in its field of view. In some embodiments, this re-centering has an interpolation (lerp value), in which the movement takes time and feels a little loosely attached to the user. In some embodiments, the following behavior can directly imitate the user's head movement, such as one-to-one movement. In some embodiments, following can be based on a prism and then the prism is accessed and activated by the application options menu. In some embodiments, the following mode can be a default assignment to a prism and / or can be a non-temporary characteristic of a prism (e.g., fixed to follow the prism until the user, for example, appropriately changes the prism).

[0263] In some embodiments, the follow 1550 body dynamic type may have one of two settings, such as lazy headlock 1560 and external sensor 1570. During the lazy headlock 1560 setting, a single object, prism, or application may be attached to the user's head position similar to the follow 1550 setting, however, the pitch parameter may be unlocked by the user, so if the user looks up and down, the object will also follow. During the external sensor 1570 setting, the object, prism, or application may follow the user based on data from the external sensor. In some embodiments, an inertial measurement unit (IMU) may be included in the user's waist pack group unit. In some embodiments, the IMU is a unit in the electronic module that collects angular velocity and linear acceleration data that is sent to the main processor. The IMU housing may contain two separate sensors. The first sensor may be an accelerometer triad. It may generate three analog signals that describe the acceleration generated by the user and acting on the user along each of its axes. The second sensor may be an angular rate sensor triad. It may also output three analog signals. These signals may describe the angular velocity of the user around each sensor axis. This IMU allows the mixed reality system to have prisms relative to the body, rather than necessarily relative to the user's head pose.

[0264] For the fade 1580 body dynamics setting, at a specified distance before the clipping plane, the content may begin to dissolve / disappear. In some embodiments, at a specified distance before the clipping plane, where the mixed reality system can no longer comfortably render the content and / or as the user approaches this cutoff point, the content should fade out smoothly rather than abruptly shut off. In some embodiments, the fade occurs as the user approaches the clipping plane (e.g., the exact measurement may depend on the hardware). Virtual content in the landscape may have this effect when the user passes through or gets too close to the content.

[0265] Launch and place applications simultaneously

[0266] In some embodiments, it may be beneficial to launch an application and place it in a mixed reality environment at the same time. In one embodiment, each time the cursor moves over a web page, the browser engine may perform a hit test on the web page content to see what is underneath the cursor position. The hit test may detect an element in the web page and then analyze what kind of element it is. Each html element may have different parameters, some of which are common to all elements, some of which are common to multiple elements, and some of which are element specific. For links, the most common and recommended element is to use an anchor tag.<ahref=”mylink”> . This anchor tag can be used with any element, image or text. At any time, we can request the type of element that the cursor is currently over. The browser engine can cache the results of the hit test and pass the node or node type to the browser application. In some embodiments, this caching of the results of the hit test can be performed when the user selects a link using an extraction trigger.

[0267] In another embodiment, when the cursor moves over any element in a clickable web page, the web page engine may call an API to the browser application to request that the cursor change from an "arrow" or "i-beam" to a "hand". When this API is called, the browser application may then request information about what kind of element the cursor is over.

[0268] Fig.16 A flow chart for simultaneously launching and placing applications in a mixed reality environment according to some embodiments is shown. At 1610, a user may move a cursor to a link on a web page displayed in a browser application, which is displayed within a prism of the user's landscape. The cursor may be moved by a handheld controller device such as a mouse, a totem, or the like, or by eye gaze, or any other suitable method, wherein with respect to eye gaze, the user's eyes may be focused on the web page and sensors from the user's head-mounted system may detect that the user is viewing a link on the web page. The link may be a uniform resource locator (URL). Movement of the cursor over the link and / or eye gaze (or any other suitable user selection method) focused on the web page with the link may send user input to the application rendering the content indicating that the user may be interested in the content corresponding to the link.

[0269] At 1615, the web engine and / or browser application may detect a link that the user moves over using a cursor or eye gaze. Upon detecting the link, the web engine may notify the browser application that the user may be interested in the content so that the browser application may request the universe to create a small display volume (e.g., a small prism). The small prism may be an initial default size of a standard 3D display volume management unit (e.g., a prism) to serve as a placeholder prism for the browser application as the browser application is loading content from the link into the small prism.

[0270] At 1620, the universe creates a mini-prism and begins loading the mini-prism to display a page preview of the content being loaded. At 1625, the user can select a link using an extraction trigger to indicate that the user is interested in viewing the content to which the link points. The use of an extraction trigger can indicate to the application that the user may be very interested in viewing the content corresponding to the link. The browser can cache the results of the hit test and pass the node or node type to the browser application to load the prism. At 1630, the user can move the mini-prism away from the web page using a handheld controller. This movement of the mini-prism can send user input to the universe indicating that the user is moving the mini-prism.

[0271] At 1635, the mini-prism is in placement mode, and the user may move the mini-prism to place the mini-prism at a location in the user's landscape. At 1640, the user may release the mouse button or extract the trigger to place the mini-prism at a location in the user's landscape. Releasing the mouse button or extracting the trigger may send user input to the universe to indicate a location to place the mini-prism in the augmented reality environment (e.g., the user's landscape).

[0272] At 1645, the small prism may be expanded to a regular size (e.g., not a small prism size) at the drop location, with the linked page fully loaded within the prism for display to the user. In fact, the user may simultaneously open / launch an application in the augmented reality environment by selecting a link and dropping the linked small prism in the mixed reality environment, while in the background the mixed reality system is loading and preparing the prism for display.

[0273] In an alternative embodiment, after 1615, at 1670, the user may select a link using an extraction trigger before the mini-prism is requested by the application and created by the universe at 1675. This may help reduce system processing for creating the mini-prism before the user actually decides to extract content from the link for display.

[0274] In an alternative embodiment, after the user selects a link with an extraction trigger, the user may release the extraction trigger without moving the mini-prism at 1680. In this embodiment, instead of expanding the mini-prism to a full-sized prism fully loaded with content from the link, the world may use the link to simply create a new tab within the web browser application, where the content of the link was already loaded and cached when the user selected the link with the extraction trigger equivalent to the user input, at 1685.

[0275] A system and method for managing and displaying virtual content in a mixed reality environment has been disclosed. In particular, the system and method discloses a global application that can act as a 3D window manager by managing 3D windows such as prisms. A prism is a bounded volume positioned in space (e.g., by a user). An application can render graphics into a prism via the global. The global works with Kali to render a scene graph and has full control over the placement, scaling, etc. of each prism. In some embodiments, the global may provide the ability to attach prisms to walls and surfaces, register prisms within a transferable world system, and / or control content sharing between multiple users of a mixed reality system. In some embodiments, the global also manages the prisms themselves (e.g., creates prisms, manages placement and / or snapping rules, provides basic user interface controls - close button, action bar, etc.), does not care what is inside a prism, and / or keeps track of the prism's record (e.g., what application owns the prism, where the prism is placed, how the prism is anchored - body-centered, world-fixed, etc.). In some embodiments, prism behavior may be based in part on anchoring. In some embodiments, prismatic behavior may be based in part on placement (e.g., user placement via user interaction) and / or body dynamics (e.g., billboarding, body centric, lazy head lock, etc.).

[0276] As a 3D Windows manager, the universe may provide more functionality than a standard 2D Windows manager. In some embodiments, as a 3D Windows manager (e.g., on top of a 2D Windows manager), one or more other features (e.g., functionality) that the universe may provide may include:

[0277] Persistence: An application sitting on the user's kitchen counter can appear on the kitchen counter unless the user changes it. The user does not have to restart the application every time the system is turned on / off or every time the user leaves the room and comes back. Since the universe stores the prismatic information in a transferable world system, the universe can restart the application that is located on the user's kitchen every time the user uses the mixed reality system and is in close proximity to the application in the user's kitchen.

[0278] Application state relative to the user: Applications can be started, stopped / paused, restarted automatically - without explicit user action. In contrast to 2D Windows Manager, where user interaction is required in order to change the operational state of the application (e.g., user clicks a close button).

[0279] Locality - Real World vs 2D Window: A 2D window manager may be limited to the screen, regardless of where the screen may be placed. However, in a mixed reality environment, applications may be placed relative to the real world, e.g., relative to other things such as physical objects in the user's environment, the user's body, fixed locations, etc.

[0280] Physical: The prism is movable. Therefore, the movement of the prism and the tracking of the movement need to be managed (e.g. to user / body centric ads, lazy ads, swaying when moving, collision bounce, etc.).

[0281] Private but interactive: 2D windows are private but they are not interactive, or they are interactive but not private. However, the universe can enable both privacy and interactivity for 3D windows (e.g., prisms).

[0282] Other embodiments

[0283] Other embodiments of the present disclosure are described below. These additional embodiments may combine elements of the embodiments disclosed above.

[0284] 1. A method for starting a mixed reality system, the method comprising:

[0285] Determine the user's current location;

[0286] retrieving one or more prisms previously deployed at the current location;

[0287] restoring one or more prisms at the user's current location; and

[0288] Displays one or more prisms restored at the user's current location.

[0289] 2. The method of embodiment 1, wherein the prism is a volume of space containing virtual content from an application.

[0290] 3. The method of embodiment 2, wherein when an application is rendered into more than one prism, the application is an application instance of a draft application.

[0291] 4. The method of embodiment 1, wherein the prism represents a portion of a scene graph corresponding to a current location of the user.

[0292] 5. A method according to embodiment 4, wherein the scene graph includes data from the first application and the second application.

[0293] 6. The method of embodiment 1, wherein retrieving one or more prisms previously deployed at the current location comprises:

[0294] retrieving instance data for one or more prisms from an external database; and

[0295] The local prism database is rebuilt using instance data for one or more prisms.

[0296] 7. A method according to embodiment 6, wherein the instance data for each prism includes a data structure that defines prism characteristics of the prism, the prism characteristics including at least one of position, orientation, range width, range height, range depth, body dynamics, anchor type, or anchor position.

[0297] 8. The method of embodiment 7, wherein the instance data for each prism includes application-specific data including state information of virtual content previously rendered into the prism by the application.

[0298] 9. The method of embodiment 1, wherein restoring one or more prisms comprises:

[0299] Launching respective applications corresponding to one or more prisms previously deployed at the current location;

[0300] creating one or more new prisms corresponding to one or more previously deployed prisms; and

[0301] Render the corresponding virtual content into one or more new prisms.

[0302] 10. The method according to embodiment 9 further includes displaying a placeholder prism at a position corresponding to the position of one or more prisms previously deployed, and displaying one or more prisms further includes replacing the corresponding placeholder prism with one or more new prisms having corresponding virtual content.

[0303] 11. The method according to embodiment 1, further comprising:

[0304] When a user interacts with one or more Prisms, updating the user's local Prism database with updated Prism instance data; and

[0305] Synchronize your local Prism database with an external database.

[0306] 12. A method for displaying virtual content in a 3D space environment, the method comprising:

[0307] receiving a request from an application to display virtual content in a 3D spatial environment;

[0308] creating a prism, wherein a prism is a volume of space configured to confine virtual content to within the boundaries of the prism;

[0309] receiving virtual content from the application;

[0310] Rendering virtual content within the boundaries of the prism; and

[0311] Associate the prism with an object in a 3D space environment.

[0312] 13. The method of embodiment 12, wherein the boundaries of the prisms are not displayed.

[0313] 14. The method of embodiment 12, wherein the 3D spatial environment is the user's physical real-world environment.

[0314] 15. The method of embodiment 12, wherein the prism is automatically created to have a set of functions.

[0315] 16. The method of embodiment 15, wherein the set of functions includes associations between prisms to objects in a 3D spatial environment.

[0316] 17. The method of embodiment 15, wherein the set of functions includes one or more of: a minimum size allowed for a prism, a maximum size allowed for a prism, and an aspect ratio for changing the size of a prism.

[0317] 18. The method of embodiment 12, wherein the application renders the first virtual content into a first prism and renders the second virtual content into a second prism, the first prism and the second prism being different prisms.

[0318] 19. The method of embodiment 18, wherein the prisms do not overlap with other prisms in the 3D spatial environment.

[0319] 20. The method of embodiment 12, wherein the prisms are placed relative to objects in a 3D spatial environment.

[0320] 21. The method of embodiment 20, wherein the subject is a user of an augmented reality device.

[0321] 22. The method of embodiment 12, wherein the prism comprises:

[0322] One or more common functions;

[0323] One or more application-specific functions; and

[0324] Therein, one or more common features and one or more application-specific features are selected from a list of pre-approved options.

[0325] 23. The method of embodiment 22, wherein one or more common features ensure that different applications interact with each other in a consistent manner.

[0326] 24. A method for managing application state of virtual content in a mixed reality system, the method comprising:

[0327] Split the 3D volume into a volume grid;

[0328] determining a first position of the user within the volumetric grid;

[0329] determining a second position of the application within the 3D volume;

[0330] Calculating the distance between the user and the application within the 3D volume; and

[0331] The state of the application is modified based at least in part on the distance calculated between the user and the application.

[0332] 25. The method of embodiment 24, wherein the radius of the activity zone defines a circular / spherical area around a user using the mixed reality device.

[0333] 26. The method of embodiment 25, further comprising a buffer around the outside of the active area, wherein the buffer prevents intermittent or rapid changes to the state of the application.

[0334] 27. The method of embodiment 24, wherein modifying the state of the application is based at least in part on whether the application is obscured by another application.

[0335] 28. The method of embodiment 24, wherein modifying the state of the application is based at least in part on a head posture of the user.

[0336] 29. A method according to embodiment 24, wherein the distance to the known position within the unit is determined only for the unit where the user using the mixed reality device is located and the adjacent units.

[0337] 30. A method for opening and placing an application in an augmented reality environment, the method comprising:

[0338] receiving a first user input from a user, the first user input indicating a request for new content;

[0339] Launch the app to generate content;

[0340] creating a small display volume of a 3D display volume management unit, wherein a page preview is displayed in the small display volume, wherein the 3D display volume management unit is created simultaneously with the launching of the application;

[0341] receiving a second user input indicating movement of the small display volume;

[0342] receiving a third user input indicating placement of the small display volume at a location in the augmented reality environment; and

[0343] The 3D display volume management unit is expanded at this location to replace the small display volume, and the 3D display volume management unit displays the content fully loaded within the 3D display volume management unit.

[0344] 31. The method of embodiment 30, wherein the first user input is a cursor movement over a link on the web page.

[0345] 32. The method of embodiment 31, wherein the second user input is a selection of a link and a movement of the small display volume.

[0346] 33. The method of embodiment 30, wherein when the 3D display volume management unit is expanded to replace the small display, the 3D display volume management unit replaces the small display.

[0347] 34. The method of embodiment 30, wherein the content is loaded into the 3D display volume management unit when the user moves the small display.

[0348] 35. A method according to embodiment 30, wherein the position is fixed to the object in the augmented reality environment.

[0349] 36. The method of embodiment 35, wherein the object is a user.

[0350] 37. A method for managing virtual content, the method comprising:

[0351] receiving content from a content generating application;

[0352] Displaying content in a 3D spatial environment through global applications; and

[0353] The display of content in a 3D spatial environment is continuously managed by global applications.

[0354] 38. A system, method, and computer program product for managing and displaying virtual content in a mixed reality system according to any inventive concept disclosed herein.

[0355] 39. The system, method, and computer program product of embodiment 38, further comprising displaying virtual content in a subset of the available 3D displayable space by displaying the virtual content within the volumetric display space, wherein a boundary of the volumetric display space is not displayed.

[0356] 40. The system, method, and computer program product of embodiment 38 further comprising assigning common features and application-selected features from a pre-approved list of options for display customization by the application to the prism.

[0357] 41. The system, method, and computer program product of embodiment 38, further comprising displaying virtual content into one or more prisms, wherein the one or more prisms do not overlap each other.

[0358] 42. The system, method, and computer program product of embodiment 38, further comprising changing the state of the prism based at least in part on the relative positioning and location of the prism to the user.

[0359] 43. The system, method, and computer program product of embodiment 38 further comprising managing content creation in applications and managing content display in separate applications.

[0360] 44. The system, method, and computer program product of embodiment 38 further comprising opening an application that will provide content to the prism when the prism is placed in a mixed reality environment.

[0361] 45. The system, method, and computer program product of embodiment 38, further comprising: assigning position, orientation, and range data to a prism for displaying virtual content within the prism, the virtual content being 3D virtual content.

[0362] 46. ​​The system, method, and computer program product of embodiment 45, wherein the location is the coordinates of an anchor position of the prism in the mixed reality environment.

[0363] 47. The system, method, and computer program product of embodiment 45, wherein the orientation defines how the prism is rotated relative to an object, wherein the object is a wall.

[0364] 48. The system, method, and computer program product of embodiment 45, wherein the range data defines the dimensions of the prism.

[0365] 49. The systems, methods, and computer program products of embodiment 38 further include pinning the launcher application to a real-world object within the mixed reality environment.

[0366] 50. The system, method, and computer program product of embodiment 49, wherein the pinned launcher application launches the content of the application within the prism at the same location as the pinned launcher application.

[0367] 51. The system, method, and computer program product of embodiment 38 further comprises assigning a behavior type to each prism, the behavior type comprising at least one of world lock, billboard, edge billboard, follow head lock, external sensor based follow, or fade.

[0368] 52. The system, method, and computer program product of embodiment 38, further comprising identifying the most frequently used content specific to the placement location of the launcher application.

[0369] 53. The system, method, and computer program product of embodiment 38, further comprising displaying favorite applications via the placed launcher application based at least in part on a range relative to the location of the placed launcher.

[0370] 54. A method comprising:

[0371] Accessing a scene graph for the scene, wherein the scene graph includes one or more transform trees, each tree including a plurality of nodes;

[0372] adding a label to one or more nodes from a plurality of nodes within one or more transform trees, wherein the labeled one or more nodes form a transform group, wherein the labeled one or more nodes include a first labeled node and a second labeled node; and

[0373] A first marked node is moved, wherein moving the first marked node causes a second marked node to move.

[0374] 55. The method of embodiment 54, wherein the second marked node is not a direct descendant of the first marked node.

[0375] 56. The method of embodiment 54, wherein the first marked node and the second marked node are not from the same transform tree.

[0376] 57. A method according to embodiment 54, wherein the multiple nodes include a first node and a second node, and the method further includes setting the first node as the parent of the second node, wherein setting the first node as the parent of the second node does not cause the parent node to move.

[0377] 58. A method for displaying virtual content in a 3D shared space, the method comprising:

[0378] Generating virtual content in the 3D shared space by the first application; and

[0379] The virtual content generated by the first application is displayed by the second application, and the first application and the second application are different applications.

[0380] System Architecture Overview

[0381] Fig.17 14 is a block diagram of an exemplary computing system 1400 suitable for implementing one or more embodiments of the present disclosure. The computing system 1400 includes a bus 1406 or other communication mechanism for communicating information, which interconnects subsystems and devices such as a processor 1407, a main memory 1408 (e.g., RAM), static storage devices 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., a modem or Ethernet card), a display 1411 (e.g., a CRT or LCD), an input device 1412 (e.g., a keyboard), and a cursor control.

[0382] According to some embodiments, computing system 1400 performs specific operations by processor 1407 executing one or more sequences of one or more instructions contained in main memory 1408. Such instructions may be read into main memory 1408 from another computer readable / usable medium (e.g., static storage device 1409 or disk drive 1410). In alternative embodiments, hard-wired circuits may be used instead of software instructions or in combination with software instructions to implement the present disclosure. Therefore, the embodiments are not limited to any specific combination of hardware circuits and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or part of the present disclosure.

[0383] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1407 for execution. Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 1410. Volatile media include dynamic memory, such as main memory 1408. In some embodiments, the system may use solid state drive (SSD) memory.

[0384] Common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0385] In one embodiment, the execution of the sequence of instructions for practicing the present disclosure is performed by a single computing system 1400. According to other embodiments, two or more computing systems 1400 coupled by a communication link 1415 (e.g., a LAN, PTSN, or wireless network) may cooperatively execute the sequence of instructions required to practice the present disclosure.

[0386] The computing system 1400 can send and receive messages, data, and instructions, including programs, such as application code, via the communication interface 1414 over the communication link 1415. The received program code can be executed by the processor 1407 as it is received and / or stored in the disk drive 1410 or other non-volatile storage for later execution. The computing system 1400 can communicate with a database 1432 on an external storage device 1431 through a data interface 1433.

[0387] In the foregoing description, the present disclosure has been described with reference to specific embodiments of the present disclosure. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above-described process flow is described with reference to a specific order of processing actions. However, the order of many of the described processing actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings should be considered to be exemplary and not restrictive.

Claims

1. A method for starting a mixed reality system, the method comprising: Determine the user's current location; retrieving first and second prisms previously deployed at the current location, wherein each of the first and second prisms is configured to provide an outer boundary for displaying virtual content in a 3D volume within the outer boundary; restoring the first and second prisms at the current location of the user; and The first and second prisms restored at the current position of the user are displayed and the first and second prisms are associated as corresponding parent nodes and child nodes using a transformation tree so that When the parent node changes position, the child node also changes position, and When the child node changes position, the parent node does not change position.

2. A method for starting a mixed reality system, the method comprising: Determine the user's current location; retrieving one or more prisms previously deployed at the current location; restoring the one or more prisms at the current location of the user; as well as The one or more prisms restored at the current location of the user are displayed.

3. A method for managing application state of virtual content in a mixed reality system, the method comprising: Split the 3D volume into volume grids; determining a first position of a user within the volumetric grid; determining a second position for application within the 3D volume; calculating a distance between the user and the application within the 3D volume; as well as A state of the application is modified based at least in part on the distance calculated between the user and the application.

4. A method for opening and placing an application in an augmented reality environment, the method comprising: receiving a first user input from a user, the first user input indicating a request for new content; launching an application to generate the content; creating a small display volume of a 3D display volume management unit, wherein a page preview is displayed in the small display volume, wherein the 3D display volume management unit is created simultaneously with the launching of the application; receiving a second user input indicating movement of the small display volume; receiving a third user input indicating placement of the small display volume at a location in the augmented reality environment; and The 3D display volume management unit is expanded at the position to replace the small display volume, and the 3D display volume management unit displays the content fully loaded in the 3D display volume management unit.

5. A method for managing virtual content, the method comprising: receiving content from a content generating application; Displaying the content in a 3D spatial environment by a global application; as well as Display of the content in the 3D spatial environment is continuously managed by the global application.

6. A method comprising: Accessing a scene graph for a scene, wherein the scene graph comprises one or more transform trees, each tree comprising a plurality of nodes; adding a label to one or more nodes from a plurality of nodes within the one or more transform trees, wherein the one or more labeled nodes form a transform group, wherein the one or more labeled nodes include a first labeled node and a second labeled node; and The first marked node is moved, wherein moving the first marked node causes the second marked node to move.

7. A method for displaying virtual content in a 3D shared space, the method comprising: Generating virtual content in the 3D shared space by a first application; as well as The virtual content generated by the first application is displayed by a second application, and the first application and the second application are different applications.

8. A method for displaying virtual content in a 3D space environment, the method comprising: receiving a request from an application to display virtual content in a 3D spatial environment; creating a prism, wherein the prism is a volume of space configured to confine the virtual content to within the boundaries of the prism, the prism having pre-programmed key values ​​callable by the application for managing display of the virtual content by the application; receiving the virtual content from the application; and The virtual content is rendered within the boundaries of the prism.

9. A display system for displaying virtual content in a 3D space environment, the display system comprising: Augmented reality head mounted display system; as well as One or more modules for processing data, wherein the one or more modules are stored in a memory, and the one or more modules are configured to perform the following processes: receiving a request from an application to display virtual content in a 3D spatial environment; creating a prism, wherein the prism is a volume of space configured to confine the virtual content to within the boundaries of the prism, the prism having pre-programmed key values ​​callable by the application for managing display of the virtual content by the application; receiving the virtual content from the application; and The virtual content is rendered within the boundaries of the prism.

Images