Management and display of web page in virtual three-dimensional space using mixed reality system

Abstract

To provide a method of using an extended reality system to manage and display web pages and web resources in a virtual three-dimensional (3D) space, a system and a product.SOLUTION: A technique includes: receiving input for 3D conversion for a web page or a web page panel therefor. A browser engine, in response to the input, to be combined with a processor of an extended reality system determines 3D conversion data for the web page or the web page panel on the basis of, at least partially the 3D conversion of the web page or the web page panel, where the 3D conversion includes a change in a 3D position, rotation, or a scale of the web page in the virtual 3D space or the web page panel therefor. A universe browser engine can present content of the web page in virtual 3D space on the basis of at least partially the 3D conversion data.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (Copyright Notice) A portion of the disclosure of this patent document contains material that is subject to copyright protection. There is no objection to anyone copying this patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but any and all copyrights otherwise reserved. [Background technology]

[0002] In the modern computing era, the Internet provides a multitude of resources to devices worldwide through the Internet Protocol Suite. For example, the World Wide Web (or simply, the Web) provides interconnected documents, services, resources, etc., through hyperlinks and Uniform Resource Locators (URLs) so that users can, for example, view Web pages, access Web resources, etc., using a Web browser.

[0003] Web pages are typically displayed within a two-dimensional (2D) stationary window. This flat representation of web pages, web resources, etc. imposes limitations on website design and the like, and therefore provides a limited user experience and usability. Some entities have envisioned and created so-called three-dimensional (3D) browsers, such as Microsoft's SurfCube and Mootools' 3D Browser (or 3DB), which present a limited perspective view of a web page on a computer display screen. However, these so-called "3D browsers" render content in a perspective view on a two-dimensional display screen; thus, the rendered content, although involving a perspective view, is still presented in a flat view and merely appears to have some degree of three-dimensional effect. These 3D browsers therefore do not actually address, mitigate, or alleviate the limitations imposed by traditional 2D web pages. Some older techniques allow the browser to rotate within a plane. For example, these traditional techniques may rotate the browser window, along with all open web pages within it, relative to the normal of the browser's planar view. However, these traditional techniques are limited not only to two-dimensional displays, but also to rotating the entire browser window (and therefore the open web pages), and therefore lack the ability to rotate each web page individually, and in both an in-plane and out-of-plane manner.

[0004] Modern computing and display technology has facilitated the development of systems for so-called "virtual reality" (VR), "augmented reality" (AR), and / or "mixed reality" experiences (hereinafter collectively referred to as "mixed reality" and / or "MR"), in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or may be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to other actual, real-world visual input, while AR or MR scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user, such that the digital or virtual image (e.g., virtual content) may appear to be part of the real world. However, while MR may integrate virtual content in a contextually meaningful way, AR may not.

[0005] The uses of mixed reality technology are expanding from, for example, gaming, military training, simulation-based training, etc., to productivity and content creation and management. Mixed reality systems have the ability to create virtual objects that appear or are perceived as real. When applied to Internet technology, such capabilities may further extend and enhance the capabilities and user experience of the Internet, such that use of web resources is no longer limited by flat, two-dimensional representations of web pages.

[0006] Therefore, there is a need for methods, systems, and computer program products for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system. Summary of the Invention [Means for solving the problem]

[0007] Disclosed, in one or more embodiments, are methods, systems, and articles of manufacture for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system. Some embodiments are directed to methods for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system.

[0008] In some embodiments, these techniques receive input for a 3D transformation for a web page or a web page panel therefor. In response to the input, a browser engine coupled to a processor of the extended reality system determines 3D transformation data for the web page or web page panel based, at least in part, on the 3D transformation of the web page or web page panel, the 3D transformation comprising a change in 3D position, rotation, or scale of the web page or web page panel therefor within the virtual 3D space. The universe browser engine may present content of the web page within the virtual 3D space based, at least in part, on the 3D transformation data.

[0009] In some of these embodiments, the browser engine further identifies a web page to be displayed within the virtual 3D space, and the virtual 3D space is or includes a prism. The browser engine further stores the 3D transformation data in a non-transitory computer-readable medium accessible by the browser engine. The 3D transformation data may be transmitted from the browser engine to the universe browser engine.

[0010] In some embodiments, these techniques further optionally receive input for 3D transformation for the web page from the web page or a website hosting the web page, where the boundaries of the web page are constrained by a configurable web browser to be displayed or constrained within a virtual 3D space. The input for 3D transformation for the web page may alternatively be optionally received from a user, where the boundaries of the web page are constrained by a configurable web browser to be displayed or constrained within a virtual 3D space. Additionally or alternatively, the input for 3D transformation for the web page or web page panel may be identified from the web page's code, where a web page developer constrains the 3D position, rotation, or scale of the web page to be displayed by the browser engine.

[0011] Additionally or alternatively, the 3D transformation data may be transmitted from the browser engine to the universe browser engine, and changes in the 3D position, rotation, or scale of the web page or web page panel therefor may be received at the universe browser engine. In some of the immediately above embodiments, the universe browser engine may further update the 3D transformation data with updated 3D transformation data and synchronize the updated 3D transformation data with the 3D transformation data accessible to the browser engine.

[0012] A virtual three-dimensional volume having a three-dimensional boundary within the virtual 3D space created for a web page by the extended reality system may be identified or generated, in part or in whole, based on one or more behavior types, and these techniques further initialize a browser engine and determine a unique identifier for the browser engine. A listener instance for the browser engine may then be registered with the universe browser engine.

[0013] The universe browser engine may further associate the listener instance with a virtual three-dimensional volume, which may also be assigned to the browser engine, and which may then be positioned or located within the virtual 3D space by using at least the scene graph structure.

[0014] Some embodiments receive a first user input indicating a user's interest in the content of the web page. In response to the first user input, at least a browser engine may be executed to generate the content, and a virtual three-dimensional volume may be determined within the virtual three-dimensional space for rendering the content therein. In some of these embodiments, a second user input indicating a 3D transformation for the web page or a web page panel therefor may be received, and the content may be rendered within the virtual three-dimensional volume based, at least in part, on the second user input.

[0015] Additionally or alternatively, a transformation tree structure and a node within the transformation tree structure may be identified for the web page, a determination may be made whether the node has a parent node within the transformation tree structure, 3D transformation data may be received from a browser engine for the web page or web page panel, and 3D position or rotation input may be received from a universe browser engine for the web page.

[0016] In some of these embodiments, the universe browser engine may determine an updated 3D position, rotation, or scale for the web page or web page panel based, at least in part, on the 3D position, rotation, or scale input, and synchronize the updated 3D position, rotation, or scale data with 3D position, rotation, or scale data accessible by the browser engine.

[0017] To determine updated 3D position, rotation, or scale data, one or more transformations of a parent of the web page may be identified, one or more transformation characteristics for the web page may also be identified based, at least in part, on the 3D position, rotation, or scale input, and the 3D position, rotation, or scale input may be identified for the web page.

[0018] In some of the immediately above embodiments, these techniques determine a set of transformations for a web page or web page panel therefor based, at least in part, on one or more of a 3D position, rotation, or scale input, one or more transformations of a parent of the web page or web page panel therefor, or one or more transformation properties for the web page or web page panel therefor. The updated 3D position, rotation, or scale data may be determined by applying the set of transformations to the web page or web page panel therefor. In some embodiments, the 3D transformations are limited to modifying only the orientation of the web page or web page panel therefor.

[0019] Some embodiments are directed to hardware systems that can be invoked to perform any of the methods, processes, or subprocesses disclosed herein. The hardware system, in some embodiments, may include a mixed reality system having at least one processor or at least one processor core that executes one or more threads of execution to perform any of the methods, processes, or subprocesses disclosed herein. The hardware system may further include one or more forms of non-transitory machine-readable storage media or devices to store various types of data or information, either temporarily or persistently. Some example modules or components of the hardware system may be found in the System Architecture Overview section below.

[0020] Some embodiments are directed to an article of manufacture that includes a non-transitory machine-accessible storage medium having stored thereon sequences of instructions that, when executed by at least one processor or at least one processor core, cause the at least one processor or at least one processor core to perform any of the methods, processes, or sub-processes disclosed herein. Some example forms of non-transitory machine-readable storage media may also be found in the System Architecture Overview section below. The present invention provides, for example, the following. (Item 1) 1. A computer-implemented method comprising: receiving, in a browser engine, input for a three-dimensional (3D) transformation for a web page or a web page panel therefor; determining, by at least the browser engine coupled to a microprocessor of an extended reality system, 3D transformation data for the web page in response to the input based at least in part on the 3D transformation of the web page or a web page panel therefor, the 3D transformation comprising a change in 3D position, rotation, or scale of the web page or a web page panel therefor; presenting, at least by a universe browser engine coupled to a browser engine of the extended reality system, content of the web page within a virtual 3D space based at least in part on the 3D transformation data; 11. A computer-implemented method comprising: (Item 2) identifying, in the browser engine, a web page to be displayed within the virtual 3D space, the virtual 3D space comprising a virtual three-dimensional volume; storing, by the browser engine, the 3D transformation data in a non-transitory computer-readable medium accessible by the browser engine; transmitting the 3D transformation data from the browser engine to the universe browser engine; Item 1. The computer-implemented method of item 1, further comprising: (Item 3) receiving input for a 3D transformation for the web page from the web page or a website hosting the web page, wherein the boundaries of the web page are constrained by a web browser application instance that is configurable to be displayed or constrained within the virtual 3D space; or receiving input from a user for a 3D transformation for the web page, the boundaries of the web page being constrained by the web browser being configurable to be displayed or constrained within the virtual 3D space; or identifying inputs for 3D transformations for the web page from the web page's code, wherein the web page developer constrains the 3D position, rotation, or scale of the web page to be displayed by the browser engine; Item 3. The computer-implemented method of item 2, further comprising: (Item 4) transmitting the 3D transformation data from the browser engine to the universe browser engine; receiving, in the universe browser engine, a change in 3D position, rotation, or scale of the web page or a web page panel therefor; Item 3. The computer-implemented method of item 2, further comprising: (Item 5) receiving, in the browser engine, data from a website regarding a 3D position, rotation, or scale of the web page or the web page panel; transmitting data or 3D transformation data regarding a 3D position, rotation, or scale of the web page or the web page panel from the browser engine to the universe browser engine; Item 3. The computer-implemented method of item 2, further comprising: (Item 6) determining an existing prism or generating a new prism in the virtual 3D space in the universe browser engine; modifying the data or the 3D transformation data into updated 3D transformation data in the universe browser engine; synchronizing the 3D transformation data determined by the browser engine with the updated 3D transformation data; Item 6. The computer-implemented method of item 5, further comprising: (Item 7) updating the 3D transformation data by at least modifying the 3D transformation data into updated 3D transformation data by the universe browser engine; synchronizing the 3D transformation data accessible to the browser engine with the updated 3D transformation data updated by the universe browser engine; Item 5. The computer-implemented method of item 4, further comprising: (Item 8) identifying or generating a virtual three-dimensional volume having a three-dimensional boundary within the virtual 3D space created by the extended reality system for the web page or the web page panel based, in part or in whole, on one or more behavior types; initializing the browser engine; determining a unique identifier for the browser engine; registering a listener instance for said browser engine with said universe browser engine; Item 1. The computer-implemented method of item 1, further comprising: (Item 9) associating, by the universe browser engine, the virtual three-dimensional volume with the listener instance; assigning the virtual three-dimensional volume to the browser engine; locating a virtual three-dimensional volume within said virtual 3D space by at least using a scene graph structure; Item 9. The computer-implemented method of item 8, further comprising: (Item 10) receiving a first user input indicating a user interest in content of the web page; responsive to the first user input, executing at least a browser engine for generating the content; determining a virtual three-dimensional volume within the virtual 3D space into which the content is to be rendered; Item 1. The computer-implemented method of item 1, further comprising: (Item 11) receiving a second user input indicating a 3D transformation for the web page or a web page panel therefor; rendering the content within a virtual three-dimensional volume within the virtual 3D space based at least in part on the second user input; and Item 11. The computer-implemented method of item 10, further comprising: (Item 12) identifying a transformation tree structure for the web page or a web page panel therefor and a node within the transformation tree structure; determining whether the node has a parent node in the transform tree structure; receiving 3D transformation data for the web page or the web page panel from the browser engine; receiving 3D position, rotation, or scale input for the web page or the web page panel from the universe browser engine; Item 1. The computer-implemented method of item 1, further comprising: (Item 13) determining, in the universe browser engine, an updated 3D position, rotation, or scale for the web page or a web page panel therefor based, at least in part, on the 3D position, rotation, or scale input; synchronizing the updated 3D position, rotation, or scale data with the 3D position, rotation, or scale data accessible by the browser engine; and Item 13. The computer-implemented method of item 12, further comprising: (Item 14) Determining the updated 3D position, rotation, or scale data includes: identifying one or more transformations of a parent of the web page; identifying one or more transformation characteristics for the web page based at least in part on the 3D position, rotation, or scale input; and identifying a 3D position, rotation, or scale input for said web page or said web page panel; Item 14. The computer-implemented method of item 13, comprising: (Item 15) Determining the updated 3D position or rotation data includes: determining a set of transformations for the web page or web page panel therefor based at least in part on one or more of the 3D position, rotation, or scale input, one or more transformations of a parent of the web page or web page panel therefor, or one or more transformation properties for the web page or web page panel therefor; determining the updated 3D position, rotation, or scale data by applying a set of transformations to the web page or a web page panel therefor; Item 10. The computer-implemented method of item 9, comprising: (Item 16) Item 10. The computer-implemented method of item 1, wherein the 3D transformation is limited to modifying only the orientation of the web page or a web page panel therefor. (Item 17) 1. A system comprising: at least one processor; a non-transitory computer-accessible storage medium having stored thereon program code that, when executed by the at least one processor, causes the at least one processor to: receiving, in a browser engine, input for a three-dimensional (3D) transformation for a web page; determining, by at least the browser engine coupled to a microprocessor of an extended reality system, 3D transformation data for the web page in response to the input based at least in part on the 3D transformation of the web page or a web page panel therefor, the 3D transformation comprising a change in 3D position, rotation, or scale of the web page or a web page panel therefor; presenting, at least by a universe browser engine coupled to a browser engine of the extended reality system, content of the web page within a virtual 3D space based at least in part on the 3D transformation data; a non-transitory computer-accessible storage medium for causing A system comprising: (Item 18) The at least one processor further comprises at least identifying, in the browser engine, a web page to be displayed within the virtual three-dimensional (3D) space, the virtual 3D space comprising a virtual three-dimensional volume; storing, by the browser engine, the 3D transformation data in a non-transitory computer-readable medium accessible by the browser engine; transmitting the 3D transformation data from the browser engine to the universe browser engine; Item 18. The system of item 17, which executes the program code to perform the above. (Item 19) The at least one processor further comprises at least receiving input for a 3D transformation for the web page from the web page or a website hosting the web page, wherein the boundaries of the web page are constrained by a web browser application instance that is configurable to be displayed or constrained within the virtual 3D space; or receiving input from a user for a 3D transformation for the web page, the boundaries of the web page being constrained by the web browser being configurable to be displayed or constrained within the virtual 3D space; or identifying inputs for 3D transformations for the web page from the web page's code, wherein the web page developer constrains the 3D position, rotation, or scale of the web page to be displayed by the browser engine; Item 19. The system of item 18, which executes the program code to perform the above. (Item 20) The at least one processor further comprises at least transmitting the 3D transformation data from the browser engine to the universe browser engine; receiving, in the universe browser engine, a change in 3D position, rotation, or scale of the web page or a web page panel therefor; Item 19. The system of item 18, which executes the program code to perform the above. (Item 21) The at least one processor further comprises at least receiving, in the browser engine, data from a website regarding a 3D position, rotation, or scale of the web page or the web page panel; transmitting data or 3D transformation data regarding a 3D position, rotation, or scale of the web page or the web page panel from the browser engine to the universe browser engine; Item 19. The system of item 18, which executes the program code to perform the above. (Item 22) The at least one processor further comprises at least determining an existing prism or generating a new prism in the virtual 3D space in the universe browser engine; modifying the data or the 3D transformation data into updated 3D transformation data in the universe browser engine; synchronizing the 3D transformation data determined by the browser engine with the updated 3D transformation data; 22. The system of claim 21, wherein the system executes the program code to perform the above. (Item 23) The at least one processor further comprises at least updating the 3D transformation data with updated 3D transformation data by the universe browser engine; synchronizing the 3D transformation data accessible to the browser engine with the updated 3D transformation data; 21. The system of claim 20, wherein the system executes the program code to perform the above. (Item 24) The at least one processor further comprises: identifying or generating a virtual three-dimensional volume having a three-dimensional boundary within the virtual 3D space created by the extended reality system for the web page or the web page panel based, in part or in whole, on one or more behavior types; initializing the browser engine; determining a unique identifier for the browser engine; registering a listener instance for said browser engine with said universe browser engine; Item 19. The system of item 18, which executes the program code to perform the above. (Item 25) The at least one processor further comprises: associating, by the universe browser engine, the virtual three-dimensional volume with the listener instance; assigning the virtual three-dimensional volume to the browser engine; positioning said virtual three-dimensional volume within said virtual 3D space by at least using a scene graph structure; 25. The system of claim 24, wherein the system executes the program code to perform the steps. (Item 26) The at least one processor further comprises: receiving a first user input indicating a user interest in content of the web page; responsive to the first user input, executing at least a browser engine for generating the content; determining a virtual three-dimensional volume within the virtual 3D space into which the content is to be rendered; Item 19. The system of item 18, which executes the program code to perform the above. (Item 27) The at least one processor further comprises: receiving a second user input indicating a 3D transformation for the web page or a web page panel therefor; rendering the content within a virtual three-dimensional volume within the virtual 3D space based at least in part on the second user input; and 27. The system of claim 26, wherein the system executes the program code to perform the above. (Item 28) The at least one processor further comprises: identifying a transformation tree structure for the web page or a web page panel therefor and a node within the transformation tree structure; determining whether the node has a parent node in the transform tree structure; receiving 3D position, rotation, or scale data for the web page or a web page panel therefor from the browser engine; receiving 3D position, rotation, or scale input for the web page or the web page panel from the universe browser engine; Item 19. The system of item 18, which executes the program code to perform the above. (Item 29) The at least one processor further comprises: determining, in the universe browser engine, an updated 3D position, rotation, or scale for the web page or a web page panel therefor based, at least in part, on the 3D position, rotation, or scale input; synchronizing the updated 3D position, rotation, or scale data with the 3D position, rotation, or scale data accessible by the browser engine; and 29. The system of claim 28, wherein the system executes the program code to perform the steps. (Item 30) The at least one processor further comprises: identifying one or more transformations of a parent of the web page; identifying one or more transformation characteristics for the web page based at least in part on the 3D position, rotation, or scale input; and identifying a 3D position, rotation, or scale input for said web page or a web page panel therefor; 30. The system of claim 29, wherein the system executes the program code to perform the above. (Item 31) The at least one processor further comprises: determining a set of transformations for the web page or web page panel therefor based at least in part on one or more of the 3D position, rotation, or scale input, one or more transformations of a parent of the web page or web page panel therefor, or one or more transformation properties for the web page or web page panel therefor; determining the updated 3D position, rotation, or scale data by applying at least a set of transformations to the web page or a web page panel therefor; 30. The system of claim 29, wherein the system executes the program code to perform the above. (Item 32) Item 18. The system of item 17, wherein the 3D transformation is limited to modifying only the orientation of the web page or a web page panel therefor. (Item 33) 1. An extended reality system, comprising: at least one processor; a browser engine coupled to the at least one processor; a universe browser engine coupled to the at least one processor and the browser engine; a non-transitory computer-accessible storage medium having stored thereon program code that, when executed by the at least one processor, causes the at least one processor to: receiving, at the browser engine, one or more requests from a website for presentation of a web page on the extended reality system; receiving, in the universe browser engine, a request to configure a 3D (three-dimensional) transformation of a web page or a web page panel therefor to be displayed within a virtual 3D space provided by the extended reality system; determining 3D transformation data in the universe browser engine in response to a request to set the 3D transformation of the web page or the web page panel; a non-transitory computer-accessible storage medium for causing An extended reality system comprising: (Item 34) The non-transitory computer-accessible storage medium, when executed by the at least one processor, further causes the at least one processor to: synchronizing the 3D transformation data between the universe browser engine and the browser engine; transmitting at least a portion of the 3D transformation data from the browser engine to the website; Item 35. The extended reality system of item 34, having stored thereon the program code that causes the system to perform the above. (Item 35) Item 35. The extended reality system of item 34, wherein the browser engine is further configured to receive 3D position, rotation, or scale data from the website, the universe browser engine is further configured to receive the 3D position, rotation, or scale data or modified 3D position, rotation, or scale data from the browser engine that modifies the 3D position, rotation, or scale data into modified 3D position, rotation, or scale data, and the universe browser engine is further configured to determine an existing prism or generate a new prism within the virtual 3D space provided by the extended reality system. (Item 36) Item 35. The extended reality system of item 34, wherein the one or more requests comprise a first request for a current 3D position, rotation, or scale for the web page to be rendered and displayed by the extended reality system, or a second request for setting a different 3D position, rotation, or scale for the web page to be rendered and displayed by the extended reality system. [Brief explanation of the drawings]

[0021] The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by like reference numerals throughout the drawings. To better understand how the above-listed and other advantages and objects of the various embodiments of the present invention are obtained, further details of the invention briefly described above will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. With the understanding that these drawings depict only typical embodiments of the invention and therefore should not be considered limiting of its scope, the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0022] [Figure 1A] FIG. 1A illustrates a high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0023] [Figure 1B] FIG. 1B illustrates another high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0024] [Figure 1C] 1C-1E illustrate further detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. [Figure 1D] 1C-1E illustrate further detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. [Figure 1E] 1C-1E illustrate further detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0025] [Figure 1F] FIG. 1F illustrates an example of a universe browser prism in one or more embodiments.

[0026] [Figure 1G] 1G-1I illustrate projected views of some simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space in some embodiments. [Figure 1H] 1G-1I illustrate projected views of some simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space in some embodiments. [Figure 1I] 1G-1I illustrate projected views of some simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space in some embodiments.

[0027] [Figure 1J] 1J-1L illustrate three simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space. [Figure 1K] 1J-1L illustrate three simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space. [Figure 1L] 1J-1L illustrate three simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space.

[0028] [Figure 1M] FIG. 1M illustrates an exemplary user physical environment and system architecture for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0029] [Figure 1N] FIG. 1N illustrates a system architecture for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0030] [Figure 1O] FIG. 10 illustrates an example of a prism according to one or more embodiments.

[0031] [Figure 1P] FIG. 1P illustrates an example of browser initialization to display a web page and reset the web page panel position and / or rotation data to default in one or more embodiments.

[0032] [Figure 1Q]FIG. 1Q illustrates an example of determining position and / or rotation data for a web page panel using a GET request in one or more embodiments.

[0033] [Figure 1R] FIG. 1R illustrates an example of determining position and / or rotation data for a web page panel using a SET request in one or more embodiments.

[0034] [Figure 1S] FIG. 1S illustrates another high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0035] [Figure 1T] FIG. 1T illustrates another high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0036] [Figure 1U] 1U-1W illustrate various detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. [Figure 1V] 1U-1W illustrate various detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. [Figure 1W]1U-1W illustrate various detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0037] [Figure 1X] FIG. 1X illustrates an example of determining orientation data for a web page panel using a SET request in one or more embodiments.

[0038] [Figure 1Y] FIG. 1Y illustrates an example flow for 3D orientation updates of a web page panel in virtual 3D space in some embodiments.

[0039] [Figure 1Z] FIG. 1Z illustrates an exemplary flow for web page translation updates from the universe browser engine to the browser engine.

[0040] [Figure 1AA] FIG. 1AA illustrates a portion of the universe browser engine (126Y) in an example flow for 3D orientation updates of web page panels in a virtual 3D space (eg, a prism) in some embodiments.

[0041] [Figure 1AB] 1AB-1AC illustrate an exemplary high-level flow diagram for event dispatch flow in some embodiments. [Figure 1AC] 1AB-1AC illustrate an exemplary high-level flow diagram for event dispatch flow in some embodiments.

[0042] [Figure 2A]FIG. 2A illustrates a high-level block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0043] [Figure 2B] FIG. 2B illustrates a more detailed block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0044] [Figure 2C] FIG. 2C illustrates a high-level block diagram for creating a Universe Prism, which can be utilized to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0045] [Figure 2D] FIG. 2D illustrates a high-level block diagram for opening and placing a web page within the Universe Browser Prism, according to some embodiments.

[0046] [Figure 2E] FIG. 2E illustrates a high-level block diagram for transforming software objects within the Universe Browser Prism in one or more embodiments.

[0047] [Figure 2F] FIG. 2F illustrates further details about a portion of the block diagram shown in FIG. 2E, according to some embodiments.

[0048] [Figure 2G] 2G-2I illustrate various transform trees and group trees in some embodiments. [Figure 2H] 2G-2I illustrate various transform trees and group trees in some embodiments. [Figure 2I] 2G-2I illustrate various transform trees and group trees in some embodiments.

[0049] [Figure 2J] FIG. 2J illustrates a high-level block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0050] [Figure 2K] FIG. 2K illustrates a more detailed block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments.

[0051] [Figure 2L] FIG. 2L illustrates a high-level block diagram for transforming software objects within the Universe Browser Prism in one or more embodiments.

[0052] [Figure 2M] FIG. 2M illustrates further details about a portion of the block diagram shown in FIG. 2L, according to some embodiments.

[0053] [Figure 2N] FIG. 2N illustrates a simplified high-level architecture diagram of an exemplary rendering process and an exemplary browser process in some embodiments.

[0054] [Figure 3] FIG. 3 illustrates a computerized system on which a method for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0055] In the following description, certain specific details are set forth to provide a thorough understanding of various disclosed embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks are not shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0056] It should be noted that throughout the following specification and claims, unless the context otherwise requires, the words "comprise" and variations thereof, such as "comprises" and "comprising," are intended to be interpreted in an open-ended, inclusive sense, as meaning "including, but not limited to."

[0057] Furthermore, it should be noted that throughout this specification, references to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Moreover, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Furthermore, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally employed in its sense, including "and / or," unless the context clearly dictates otherwise.

[0058] Various techniques, in various embodiments, are directed to managing and displaying web pages and web resources within a three-dimensional virtual space using virtual reality ("VR"), augmented reality ("AR"), mixed reality ("MR"), and / or extended reality ("XR") systems (collectively referred to as "XR systems").

[0059] In various embodiments, VR systems are designed and configured to provide immersive experiences within virtual reality or computer-simulated reality. Virtual reality systems employ computer technology to generate realistic sounds, images, and other sensations that replicate real environments or create imaginary worlds, albeit simulated, using, for example, a reality headset. AR systems are designed and configured to provide live, direct, or indirect views of physical, real-world environments, elements of which are augmented (or supplemented) by computer-generated sensory input, such as sound, video, graphics, or sensory data (e.g., GPS data, etc.). Because augmented reality exists on top of the physical, real-world environment, the AR systems described herein provide users with the same degree of freedom as is afforded in the user's normal life. MR systems described herein provide hybrid realities, merging real and virtual worlds to produce new environments and visualizations in which physical and digital objects coexist and interact in real time or near real time (e.g., to account for time differences in signal transmission, processing time, etc.). The MR systems provided herein provide new images in real space in a way that allows the new images to interact, at least to some extent, with what actually exists in the physical world. One of the characteristics of the described MR systems is that synthetic content and real-world content can react to each other in real time or near real time (e.g., to account for time lags in signal transmission, processing time, etc.). The extended reality or XR systems provided herein provide any or nearly any combined real and virtual environment and human-machine interaction generated by computer technology and devices (e.g., wearable electronics, etc.). The extended reality systems provided herein are conceived and / or configured to provide any or nearly any form of representation, such as AR systems, VR systems, and MR systems.In some embodiments, the term "XR" is used as an umbrella term that combines all three realities (AR, VR, MR) under one term.

[0060] In some embodiments, the system includes at least a mixed reality device, a browser engine, and a universe browser engine to interact with a website. In some of these embodiments, a website developer may incorporate a set of application programming interfaces (APIs) related to one or more embodiments described herein into the website's code, which set of APIs communicates with the browser to get and set the position and rotation of the web page. The developer may set position and rotation data (e.g., x-, y-, and z-coordinates) within the three-dimensional space of the web page that the developer would like the end user to perceive through this set of APIs.

[0061] In some other embodiments, some or all of the APIs in the aforementioned set of APIs may be hosted within or referenced locally by a browser engine and / or universe browser engine that is part of a mixed reality system that projects image data of a web page as virtual content to a user's eyes. This set of APIs may include, for example, a first API that sets and retrieves three-dimensional position data for a web page and a second API that sets and retrieves three-dimensional rotation data for the web page. It should be noted that while the terms "web page" and "web page panel" may be used interchangeably herein, strictly speaking, a web page panel includes a two- and / or three-dimensional area within a browser window in which a web page, or portion thereof, displays its content.

[0062] An example use case includes a scenario in which a user of a mixed reality system opens a browser and types in a website URL or search criteria in a designated field. Various embodiments described herein enable the universe browser engine to work in conjunction with the browser engine to adjust the rotation and / or position of individual web pages within the virtual three-dimensional space based on the HTML code for the particular web page.

[0063] More specifically, in the aforementioned examples, when a user launches a browser in their mixed reality system, the browser acts as an intermediary between the website and the universe engine. The browser engine, in some embodiments, updates the position and rotation data (e.g., x-, y-, and z-coordinates) for a requested web page by using the stored position and rotation data. In some of these embodiments, the universe browser engine may store the position and rotation data of a web page panel within the browser engine (for displaying the web page). In one example where Chromium's Blink is used as the browser engine, the universe browser engine may store the position and rotation data within a RenderWidget class.

[0064] One of the purposes of storing position and rotation data with the browser engine is so that the browser can quickly communicate this most recently cached position and rotation data to a website in response to a request to position and / or rotate a web page that forms the website, for example, without the additional delay from retrieving the position and / or rotation data from the universe browser. RenderWidget is a class within the Chromium project, WebKit, etc. that implements an abstract interface (e.g., a two-dimensional window or a three-dimensional volume in a virtual three-dimensional space) within a mixed reality display, receives input events, and paints or renders content therein.

[0065] In some embodiments where a web page developer has already set position and rotation data for a web page, the browser engine may set the position and rotation data for the web page to be rendered and pass the position and rotation data to a universe browser engine that is part of the mixed reality system and interfaces with the user. The browser engine and universe browser engine may synchronize the position and rotation data. The browser engine may then render the web page by invoking a function call and using the position and rotation data.

[0066] In some embodiments, the user may further manipulate the web page panel, e.g., by manipulating positioning and / or rotation handles associated with the web page or web page panel on the display, e.g., by repositioning and / or rotating the web page panel or even the web page on the display (e.g., a prism, which will be described in more detail later) within the three-dimensional virtual space. In these embodiments, the user effectively changes the position and / or rotation of the web page, and the universe browser engine may also synchronize the new position and / or rotation data with the browser engine.

[0067] In some embodiments, the mixed reality system may display a placeholder (e.g., a solid or semi-transparent or transparent bounding box with partial or full bounds) in the virtual three-dimensional space before rendering the desired web page with the browser engine. In some other embodiments, a mini-view of the desired web page may be first presented in the virtual three-dimensional space before the full rendering of the web page. In yet some other embodiments, the web page may be fully rendered and displayed, along with its three-dimensional coordinates, in the virtual three-dimensional space before any changes, by either the developer or the end user, are made to change the position and / or rotation of the web page.

[0068] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present invention to enable those skilled in the art to practice the present invention. It should be noted that the following figures and examples are not intended to limit the scope of the present invention. Where certain elements of the present invention can be implemented, partially or completely, using known components (or methods or processes), only those portions of such known components (or methods or processes) necessary for understanding the present invention will be described, and detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the present invention. Furthermore, various embodiments encompass present and future known equivalents of components referenced herein by way of example.

[0069] 1A illustrates a high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, a universe browser engine 106A may transmit web page position and rotation data to a browser engine 104A and store such data within the browser engine 104A. The universe browser engine (e.g., 106A) may act as a 3D window manager, similar to the 2D window managers that run on laptop or desktop computers to manage 2D windows displayed on the laptop or desktop computer's display screen.

[0070] A universe browser engine or application (e.g., 106A) working in conjunction with the universe browser engine also manages the generation, placement, and display of virtual content within the user's landscape via the mixed reality system. When browser engine 104A initializes to display a web page of website 102A or content therein (e.g., when a user launches a browser), browser 102A may set position and rotation data for the web page to be rendered by browser engine 104A when the website is coded to incorporate a set of APIs coded to both set position and rotation data to the client browser and retrieve position and rotation data from the client browser (108A).

[0071] In some embodiments, the universe browser engine provides the functionality of a 3D window manager, managing the virtual content to be displayed (e.g., how and where) and how and where prisms should be generated for the extended reality system, as will be described in more detail. For example, the universe browser engine may provide persistence functionality. As a non-limiting example, an entity (e.g., a software application or instance thereof, a virtual object, etc.) placed on a user's kitchen counter may appear on the kitchen counter unless and until the user changes the entity's placement. The user, in some embodiments, may not need to relaunch the entity every time the system is turned on / off or the user leaves and returns to the room. In these embodiments, the universe browser engine stores prism information or data within the passable world, and the universe browser engine may resume an entity placed on the user's kitchen every time the user uses the extended reality system and comes into close proximity to the entity application in the user's kitchen.

[0072] The universe browser engine may also maintain or change the state of a software application (or an instance thereof) for a user of an XR system that includes the universe browser engine. For example, an application may automatically start, suspend / pause, and / or resume without requiring any explicit user action. This contrasts with 2D window managers, which require user interaction (e.g., a user clicking a "close" button) to change the operational state of an application. This aspect also differs from other window managers, which require user interaction (e.g., a user clicking an application's "close" button) to change the application's state. Also, unlike other windows, a 3D prism (or even a 3D virtual space within which the prism resides) may be both interactive and private. For example, the universe browser engine may enable both the privacy and interactive features of a prism (or 3D virtual space). In contrast, a traditional window is either interactive or private, but cannot be both interactive and private.

[0073] A browser engine is a software component of a web browser that often performs tasks such as converting HTML and / or other resources of a web page into an interactive visual representation on a user's device (e.g., a mixed reality headset). While the following description may refer to Chromium or a Chromium-based browser, other browsers (e.g., Mozilla's Firefox, Apple's Safari, Microsoft's Edge and Internet Explorer, Google's Chrome, Opera, 3D Browser, etc.) are also contemplated, and techniques apply with full equal effect to different browsers corresponding to their respective browser engines (e.g., Gecko for Mozilla, WebKit for Safari, Blink for Chromium, Chromium-based browsers, Microsoft's Edge, and Opera, and Trident for Internet Explorer, etc.).

[0074] Browser engine 104A may include any publicly available browser, such as the browser mentioned immediately above or any custom browser (e.g., a 3D browser). One of the functions of browser engine 104A is to provide functionality including, for example, using position and rotation data for a web page to convert HTML documents and other resources of the web page into an interactive visual representation on the user's device, calculating graphical coordinates for the visual representation using position and rotation data provided by either the website or universe browser engine 106A, painting or rendering prisms (or portions thereof), etc.

[0075] This set of APIs for one or more embodiments for managing and displaying web pages described herein may include APIs that conform to the Portable Operating System Interface (POSIX) API standard and provide spatial computing functionality independent of the platform or operating system. This set of APIs may, in some embodiments, be incorporated into the code of a website, as previously described above, and / or in some other embodiments, may be integrated into part of the operating system or software application resident on the mixed reality system.

[0076] This set of APIs not only sets and gets position and rotation data between website 102A, browser engine 104A, and universe browser engine 106A, but also works in conjunction with a runtime layer between the operating system and software applications and one or more 3D engines (or more precisely, between the operating system services layer and software applications). This runtime layer includes libraries, applications, services, etc., that provide (either alone or in conjunction with a 3D engine) advanced graphical user interfaces for the underlying mixed reality system and various computing functionality such as 3D translation and rotation transformations, 3D models with material and skeletal 3D model animation, 2D sprite animation, high-fidelity spatialized text rendering for spatial computing, 2D and sound field audio, 2D and stereoscopic video, rigid-body collision physics, real-time particle FX, input events and tactile control, hand gestures, speech recognition and speech-to-text conversion, real-time spatial computing techniques for rendering objects, lighting, shading, and real-world occlusion.

[0077] For example, the runtime layer may include a set of three-dimensional translation and rotation transforms that may be applied, e.g., to a prism (to be described later), a virtual object, a browser window, a rendered web page, etc., displayed in a virtual three-dimensional space such that the virtual object, the browser window, the rendered web page, etc., or any of its attributes (e.g., display coordinates, size, shape, perspective, lighting effects, etc.) appears or is perceived by the user as real or nearly real. Further details about these transforms will be described later with reference to Figures 2E-2I below.

[0078] 1A , using a set of APIs, website 102A may send a position and / or rotation request from browser engine 104A (112A). In this case, browser engine 104A may transmit the most recent position and rotation data to the website (114A). As described above, universe browser engine 106A may store the position and rotation data within or with browser engine 104A (e.g., in a cache accessible by browser engine 104A) so that browser engine 104A can respond to such requests from website 102A, for example, without the additional delay from retrieving such position and rotation data from universe browser engine 106A.

[0079] In addition to returning the latest position and rotation data to website 102A (114A), browser engine 104A may transmit the position and rotation data, either alone or along with the rendered web page, to universe browser engine 106A (110A). Universe browser engine 106A creates or identifies a prism (or smaller virtual 3D volume) based, in part or in whole, on the position and rotation data received via 110A, and universe browser engine 106A synchronizes the position and rotation data (116A) with browser engine 104A. The universe browser engine 106A may call the browser engine 104A and invoke its native functions, libraries, and / or APIs (e.g., RenderWidget in Chromium or WebKit, OpenGL API, OpenGL ES2.0 API, etc., Almost Native Graphics Layer Engine, Direct3D API, WebGL, Gfx API, etc., or any combination thereof) to render the content of the web page within a prism or portion thereof based on the position and rotation data for display.

[0080] In some embodiments, the universe browser engine may invoke one or more 3D engines launched above the operating system core and operating system services in the operating system stack to render the 3D content. These one or more 3D engines may be any custom 3D engine for 3D and / or 2D graphics, a commercially or publicly available 3D engine (e.g., Unreal Engine 2.0) that provides an abstraction layer for a graphics processing unit (GPU), or any other 3D engine. The mixed reality system may include a 3D engine (e.g., Engine 4, Unreal Engine 3, CryEngine V, Unity 3D, Source Engine, Source Engine 2, etc.). In some of these embodiments, the mixed reality system need not incorporate an entire 3D engine. Rather, the mixed reality system may incorporate smaller portions, such as a rendering engine or rendering API, a physics engine for emulating the laws of physics, a scripting engine for parsing and executing scripts, such as JavaScript scripts, a memory management module, and / or a threading module. In some embodiments, the mixed reality system described herein may invoke a renderer for rendering 2D content and a separate renderer for rendering 3D graphics.

[0081] The user may further modify the position and / or rotation of the web page displayed within the virtual 3D space through the mixed reality system. For example, the user may freely move and / or rotate the displayed web page, a placeholder therefor, or a mini-preview version of the web page by grasping software handles associated with positioning and rotating the displayed web page, a placeholder therefor, or a mini-preview version of the web page, respectively. The position and / or rotation data of the web page is accordingly modified, and universe browser engine 106A may further synchronize the position and / or rotation data with browser engine 104A (116A). At that point, browser engine 104A then replaces the original position and / or rotation data with this modified position and / or rotation data.

[0082] FIG. 1B illustrates another high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, FIG. 1B illustrates the interaction between a browser panel 102B for a web page, a browser engine 104B, and a universe browser engine 106B, and the individual tasks and functions performed by the browser 102B, browser engine 104B, and universe browser engine 106B in some embodiments. A browser or web browser is a software application for accessing information about the World Wide Web, where web resources (e.g., individual web pages, images, videos, etc.) are identified by uniform resource locators, allowing the web browser to retrieve these web resources from web servers and display them on a user's device. The browser engine is the software component of a web browser that performs tasks such as translating HTML and / or other resources of a web page into an interactive visual representation on a user's device (e.g., a mixed reality headset). The various embodiments described herein leverage any publicly or commercially available browser and its respective engine to enable developers and users to manage and display Internet content and utilize web resources by using at least the aforementioned set of APIs, a mixed reality system, and software applications and libraries (e.g., the Universe Browser Engine).

[0083] The website may set position, rotation, and / or scale data for the web page of interest or a panel of the web page (102B) and may further request the current position, rotation, and / or scale of the current web page (102B). In some embodiments, where a web page developer may want the web page of interest to be presented in a certain manner that includes certain position, rotation, and / or scale data, the website may further send such position, rotation, and / or scale data to the browser (102B).

[0084] In some embodiments, when a browser is launched to display content from the Internet, the browser initializes and renders and displays the website's web page or content therein (104B). The browser may also send position and rotation data of the web page to be rendered by the browser engine (104B). For example, when a website is coded to incorporate the aforementioned set of APIs that are coded to both set position and rotation data in the client's browser and retrieve position and rotation data from the client browser, the browser may receive this position and rotation data from the website and set the position and rotation data for the web panel to display content from the website.

[0085] The browser may reset the transformation, position data, and / or rotation data for the web panel for the web page (104B). For example, the browser may reset the 3D transformations (e.g., 3D translation transformations, 3D rotation transformations, and / or 3D scaling transformations) for the web page panel to default values ​​or states. The default values ​​or states, in one embodiment, may be stored in non-transitory memory (e.g., cache memory) accessible by the renderer (e.g., RenderWidget for WebKit and Chromium). The browser may relay the position and / or rotation request from the website to its browser engine (104B). The browser engine acts as an intermediary between the website and the universe browser engine, which resides in the mixed reality system. In some embodiments where the position and rotation data of the web page panel is stored in the aforementioned non-transitory memory, the browser or its browser engine may respond quickly to the position and / or rotation request from the website without additional delay from retrieving such data from the universe browser engine, for example. This position and rotation data accessible by the browser engine may be synchronized when the website or the browser changes the data or portions thereof. Additionally, or alternatively, this position and rotation data accessible by the browser engine may be initialized and reset by the browser. In some embodiments, this position and rotation data may be updated by a website request (see 102B) for position and rotation data. In some embodiments where the browser engine receives position and / or rotation data from a website, the browser engine may also pass the data to the browser.

[0086] The browser may also send position and rotation data, either alone or along with the rendered content (e.g., a web page), to the universe browser engine (104B). The universe browser engine may create a 3D virtual volume (e.g., a prism) with virtual boundaries in the virtual 3D space, which may correspond to the real-world 3D space, to display the rendered web page, based at least in part on the position and rotation data from the browser engine. Additionally or alternatively, some embodiments may provide the user with the ability to manipulate the rendered web page (or content) within the virtual 3D space created by the mixed reality system.

[0087] The user may therefore move and / or rotate the rendered web page within the virtual 3D space, such that a new set of transformations needs to be applied to the rendered web page. The operating system of the mixed reality system, including a runtime layer and a 3D engine, may apply the new set of transformations to the rendered web page based, at least in part, on the position and rotation data received from the browser engine. After the new set of transformations is applied, the universe browser engine may also transmit the new position and rotation data back to the browser engine (106B) and update the previous position and rotation data therein.

[0088] When the browser initializes or when a user interacts with a rendered web page, thus changing the position and / or rotation data, the browser engine may receive the position and rotation data, if available, from the universe browser engine (104B). The browser engine may therefore update the position and / or rotation data stored in non-transitory memory (e.g., cache) with the refreshed position and / or rotation data received from the universe browser engine, e.g., stored in cache memory (104B). The browser engine may also set the position and rotation data for the web page panels (104B). In some embodiments where a web page developer has set the position and rotation of the web page (e.g., a developer has set the rotation of a web page displaying an online chess game), the browser may also set the position and rotation data for the web page panels according to the position and rotation data received from the website.

[0089] Additionally, the browser engine may send position and rotation data to the universe browser engine so that the universe browser engine may determine (e.g., by creating a new one or identifying one from an existing one) a virtual 3D space (e.g., a prism) for presenting the rendered web page via the mixed reality system (104B). In some embodiments in which the user is provided with the ability to further manipulate the rendered web page, the browser engine may provide the position and rotation data of the rendered web page to the universe browser engine, which in turn may provide a software handle for the user to manipulate the rendered web page, perform corresponding transformations, and respond to user manipulations.

[0090] The universe browser engine also includes or works in conjunction with an operating system, operating system services, a set of runtime applications and libraries, one or more 3D engines, and a suite of applications for the mixed reality system to provide an advanced graphical user interface and functionality for a user to manage and view web pages and other content and / or resources within a virtual 3D space (e.g., one or more prisms) at 106B. For example, one of the functions of the universe browser engine is to provide position and rotation data of web pages (or other virtual content) to the browser engine (104B). The universe browser engine also synchronizes recent position and / or rotation data with the browser engine (106B) so that recent position and / or rotation data can be pushed to (from) or pulled from (by) the browser engine.

[0091] 1C-1E illustrate further detailed block diagrams for an exemplary browser engine and universe browser engine that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, FIG. 1C illustrates simplified pseudocode for setting 3D position and / or rotation data of a web page panel in a virtual 3D space (e.g., a prism) in some embodiments. In these embodiments, the pseudocode is based on a similar set of code for Chromium, although code for other web browsers is also contemplated and can be applied to the techniques described herein with full equal effect.

[0092] For example, browser engine 100C may execute Window.VirtualWorld.3DPosition and / or Window.VirtualWorld.3DRotation at 102C to initialize a process for setting 3D position and / or rotation data of a web page panel within the universe browser engine. Window.VirtualWorld.3DPosition and Window.VirtualWorld.3DRotation may utilize, for example, OpenGL Mathematics (GLM) (or any other mathematical library) to set and retrieve position data of a web page panel based, in part or in whole, on 3D position data, 3D rotation data, scale, and one or more transformations for 2D and 3D entities in 2D or 3D space (e.g., the pseudocode "glm::vec3old_pos(0.0f)", "glm::quatrotation(glm::vec3(0.0f))", "glm::vec3scale(1.0f)", and "glm::vec3old_pos(0.0f)")). glm::mat4transform=web_container_->getCurrentVolumeTransform()") and rotation data (e.g., by the pseudocode "glm::vec3position(0.0f)", "glm::quatold_rot(glm::vec3(0.0f))", "glm::vec3scale(1.0f)", and "glm::mat4transform=web_container_->getCurrentVolumeTransform()"). It should be noted that the pseudocode, programming languages, and names of various classes, variables, etc. are used herein for purposes of illustration and explanation, and that any other suitable programming languages, names, etc. that serve the same or substantially similar purposes are also contemplated and may be used to achieve such purposes.

[0093] Browser engine 100C may further execute VirtualWorld::Set3DPosition at 104C to set and get position and / or rotation data (not shown) for the web page panel, where VirtualWorld is an in-process class in Blink (the browser engine for Chromium) for frames. Note that FIG. 1C illustrates an example of setting and determining position data but not rotation data; setting and determining rotation data may be done in the same or substantially similar manner using corresponding code. In some embodiments, the browser engine may also include a RemoteDomWindow (without a document object or widget) for the proxy that resides in another renderer process. The construct VirtualWorld::3DPosition may get and set position and / or rotation data using a low-level graphics abstraction layer such as Gfx (a simple graphics library for CSE20211) or Gfx-rs (e.g., via "gfx::Point3fpos(x-coordinate, y-coordinate, z-coordinate)", "gfx::Point3f rot(x-rotation, y-rotation, z-rotation)", "gfx::Point3F rot=local_frame_client->GetVirtualWorldRotation()", or "gfx::Point3Fflat_rot{-M_PI_2,0.0f, 0.0f}", etc.).

[0094] The browser engine 100C may further execute LocalFrameClientImpl::Set3DPosition (106C), RenderFrameImpl:Set3DPosition (108C), and send an inter-process communication (IPC) message—FrameHostMsg_Set3DPosition. The construct RenderFrameImpl:Set3DPosition may be used to call RenderWidget::Set3DPosition and send the IPC message. The construct LocalFrameClientImpl::Set3DPosition is derived from the C++ LocalFrameClientImpl.cpp and uses a low-level graphics abstraction layer, such as Gfx or Gfx-rs, and position and rotation data to set and get the position and / or rotation for the client frame. LocalFrameClientImpl::Set3DPosition may be implemented to get and set the position and rotation data of a web page panel using the following pseudocode: [ka]

[0095] The construct RenderFrameImpl::Set3DPosition is derived from C++ RenderFrameImpl.cpp and is configured to invoke a renderer (e.g., RenderWidget for a Chromium-based browser) and return a frame to the browser engine (e.g., Blink for a Chromium-based browser) based on Gfx and position and / or rotation data. The browser engine 100C may also invoke the renderer by executing RenderWidget::Set3DPosition at 112C to “paint” or render the content of the web page of interest according to the 3D position and rotation data (e.g., position and rotation data set by the web page developer). More specifically, the renderer process RenderWidget may use a glue interface (e.g., WebWidgetDelegate) to implement an abstract interface within the glue interface. This abstract interface may, for example, contain a 2D window or a 3D volume within the display space, receive input events, and render content within it. In some of these embodiments where auxiliary display items (e.g., selection boxes with up / down arrows indicating a list of options, tabs, pop-up windows, navigation commands, etc.) are to be rendered, browser engine 100C may also execute a separate rendering process (e.g., RenderView) for such display items.

[0096] In some embodiments where a Chromium or Chromium-based browser is used, browser engine 100C renders web pages using its rendering system (“views”), and the user interface is structured as a tree of components, called “views,” that are responsible for rendering, layout, and event handling. Each view in the tree of components has its own boundary and represents a different component of the user interface, with widgets (native 2D windows or 3D volumes) located at the root of such a tree. Some embodiments use position and location data to create a virtual 3D space (or 2D window) and pass the virtual 3D space (or 2D window) to the RootView, which then propagates events up the tree. Some display items may be rendered using operating system controls hosted in special types of views that know how to display and size native widgets. These display items include, for example, buttons, tables, radio buttons, check boxes, text fields, other controls, etc. Further details about such trees of components are described below with reference to Figures 2G-2I.

[0097] The browser engine also implements WebContentsImpl::OnSet3DPosition (114C), CefBrowserHostImpl::Set3DPosition (116C) using "delegate_->Set3DPosition(position)" and "delegate_->Set3DPosition(position)" based at least in part on the position and rotation data. The browser engine also implements CefUniverseBrowserEnginePrivateHandler::OnSet3DWebPagePosition (118C) using a universe browser engine specific interface that allows custom interaction between the Chromium Embedded Framework (CEF) and the universe browser engine. A CEF-based application may contain components such as an entry point for initializing CEF and starting the CEF message loop, a CEFApp-derived class for handling process-specific callbacks, a CEFClient-derived class for handling browser-instance specific callbacks (e.g., callbacks for browser lifespan, context menus, dialogs, display notifications, drag events, attention events, keyboard events, etc.), and one or more CEFBrowser instances created by CefBrowserHost::CreateBrowser().

[0098] Browser engine 100C may further implement the constructs CefBrowserHostImpl::Set3DPosition (116C) and CefBrowserHostImpl::Set3DRotation to determine the handler (e.g., "CefRefPtr <cefuniversebrowserengineprivatehandler>handler=client_->GetUniverseBrowserEnginePrivateHandler()" and "handler->OnSetVirtualWebPagePosition(this, position.x(), position.y(), position.z())", "handler->OnSetVirtualWebPageRotation(this, The browser engine may also implement the public functions CefUniverseBrowserEnginePrivateHandler::OnSet3DWebPagePosition and CefUniverseBrowserEnginePrivateHandler::OnSet3DWebPageRotation to set the 3D position and rotation of the web page panel.

[0099] The universe browser engine 126C may use the 3D position and rotation data received from the browser engine 100C to execute ClientHandler::OnSet3DWebPagePosition and ClientHandler::OnSet3DWebPageRotation (120C), Browser::OnSet3DWebPagePosition and Browser::OnSet3DWebPageRotation (122C), and BrowserWindow::OnSet3DWebPagePositionandBrowserWindow::OnSet3DWebPageRotation (124C). The ClientHandler includes event handlers that fire within the browser without requiring callbacks to the server to provide a means for defining rules (e.g., declarative conditions and actions that can be evaluated within the browser and reduce round-trip delay or latency). The ClientHandler may also delegate setting the web page position and rotation within a virtual 3D space (e.g., a prism) using GLM (OpenGL Mathematics).

[0100] 1C, illustrating how browser engine 100C obtains 3D position and / or rotation data within a virtual 3D space (e.g., a prism) from universe browser engine 126C. Downward arrows indicate a sequence of execution and / or a path of execution results, and upward arrows indicate a return of a value.

[0101] In these embodiments, browser engine 100C may execute Window.VirtualWorld.3DPosition and / or Window.VirtualWorld.3DRotation at 102C to initialize a process for setting 3D position and / or rotation data of a web page panel within the universe browser engine, similar to that described with reference to FIG. 1C above. Additionally, browser engine 100C may further execute VirtualWorld::Set3DPosition at 104C to set and retrieve position and / or rotation data (not shown) of a web page panel, where VirtualWorld is a class representing one of the platform's XR devices. Note that FIG. 1D illustrates an example of setting and determining position data rather than rotation data, and setting and determining rotation data may be done in the same or substantially similar manner using corresponding code.

[0102] Unlike the browser engine 100C illustrated in FIG. 1C, the browser engine 100C in FIG. 1D may further implement LocalFrameClientImpl::Get3DPosition (102D) and RenderFrameImpl:Set3DPosition (104D). Similar to the description of FIG. 1C, the construct RenderFrameImpl:Get3DPosition may be used to call RenderWidget::Get3DPosition, send an IPC message, and / or return a value for VirtualWorld::3DPosition to return position and / or orientation data. The construct LocalFrameClientImpl::Get3DPosition, like LocalFrameClientImpl::Get3DPosition in FIG. 1C, is derived from the C++ LocalFrameClientImpl.cpp and may use a low-level graphics abstraction layer, such as Gfx or Gfx-rs, as position and rotation data to set and get the position and / or rotation for the client frame.

[0103] Like RenderWidget::Set3DPosition in FIG. 1C , browser engine 100C may also invoke a renderer by executing RenderWidget::Get3DPosition at 106D to return position and / or orientation data (e.g., the position and / or orientation data of the web page) to RenderFrameImpl::Get3DPosition(104D), and further "paint" or render the content of the web page of interest according to the 3D position and rotation data (e.g., the position and rotation data set by the web page developer). In some embodiments, the renderer process RenderWidget may use a glue interface (e.g., WebWidgetDelegate) to implement an abstract interface within the glue interface. This abstract interface may, for example, contain a 2D window or a 3D volume within the display space, receive input events, and render content within it. In some of these embodiments where auxiliary display items (e.g., selection boxes with up / down arrows indicating a list of options, tabs, pop-up windows, navigation commands, etc.) are to be rendered, browser engine 100C may also execute a separate rendering process (e.g., RenderView) for such display items.

[0104] FIG. 1E illustrates a simplified block diagram with pseudocode for synchronizing 3D position and rotation data between a browser engine and a universe browser engine in some embodiments. For example, a web page in a virtual 3D space may be modified (e.g., by a user who has moved and rotated the web page in virtual 3D space). The block diagram in FIG. 1E illustrates how the web page's current 3D position and rotation data is synchronized between browser engine 100E and universe browser engine 126E. FIG. 1E may also illustrate an embodiment in which universe browser engine 126E sends position and rotation data to browser engine 100E upon browser initialization.

[0105] 1E, upon initialization (120E), or upon receiving modified 3D position and / or rotation data (not shown) for the web page, universe browser engine 126E may reset the transforms of the web page's volume, rendered browser, and / or prism (122E). The 3D position and rotation data is then passed to browser engine 100E, which executes CefBrowserView::UpdateCached3DPosition and CefBrowserView::UpdateCached3DRotation (102E). Browser engine 100E further executes RenderWidgetHostImpl::UpdateCached3DPosition and RenderWidgetHostImpl::UpdateCached3DRotation to update the position and / or rotation data with the most recent position and / or rotation data from universe browser engine 126E.

[0106] In some embodiments, universe browser engine 126E may also check whether the web content transformation has changed at 121E. Universe browser engine 126E may execute the following example code / pseudocode with a corresponding class definition in a header file—“void CheckIfWebContentTransformChanged();” to determine whether the web content transformation has changed: [ka]

[0107] In the above example, if "cef_window" does not exist, some embodiments retain the uninitialized value of the transform without performing any additional operations. Furthermore, the code segment "if(std::abs(pDiff[i])>0.0001f)..." shows a conditional statement for determining whether the associated transform has changed significantly.

[0108] The Universe Browser Engine determines whether the cursor position confirmation response by the system (e.g., AR system) is the same as or different from the actual cursor position, and then executes it in the class definition "virtual void UpdateLoop(float / * delta * The stage and 3D volumes in the virtual 3D space may be updated with [ka]

[0109] In some embodiments, a threshold may be determined in terms of a "delta" so that updates will not be performed when the difference between said two cursor positions is below the threshold. Some embodiments pool node transform changes and update the stage and separate volume positions using "content_view_->CheckIfWebContentTransformChanged();".

[0110] The browser engine 100E also sends IPC (inter-process communication) messages (e.g., ViewMsg_UpdateCached3DPosition, ViewMsg_UpdateCached3DRotation, etc.) and executes RenderWidget::OnUPdateCached3DPosition and RenderWidget::OnUPdateCached3DRotation (108E) and RenderWidget::Set3DPosition and RenderWidget::Set3DRotation (110E).

[0111] In some embodiments, universe browser engine 126E may store the 3D position and rotation data in a non-transitory computer-readable medium (e.g., cache memory) accessible by browser engine 100E. Universe browser engine 126E and browser engine 100E are therefore synchronized each time the position and / or rotation of a web page panel in virtual 3D space is modified. Each time a web page begins loading, the web page panel's transformation is reset to default, and the stored values ​​in RenderWidget will be updated as well, to initialize and reset the position and rotation variables in RenderWidget.

[0112] 1F illustrates a simplified example of a universe browser prism in one or more embodiments. In this example, two universe browser prisms (or simply, prisms) 100F and 102F are created in a virtual 3D space for a user 104F wearing a mixed reality device. While prisms 100F and 102F appear as rectangular prisms, it should be noted that the prisms may be any shape and size (e.g., cylinders, cubes, spheres, tetrahedrons, etc., or even irregular 3D volumes).

[0113] A prism is a three-dimensional volumetric space within which virtual content is rendered and displayed. Prisms exist within a virtual 3D space provided by an extended reality system, which in some embodiments may include more than one prism. In some embodiments, one or more prisms may be placed in the real world (e.g., a user's environment), thus providing one or more real-world locations for the prism. In some of these embodiments, one or more prisms may be placed in the real world relative to one or more objects (e.g., physical objects, virtual objects, etc.), one or more two-dimensional surfaces (e.g., surfaces of physical objects, surfaces of virtual objects, etc.), and / or one or more one-dimensional points (e.g., vertices of physical objects, surfaces of virtual objects, etc.). In some embodiments, a single software application may correspond to more than one prism. In some embodiments, a single application corresponds to a single prism.

[0114] In some embodiments, a prism may represent a subtree of a multi-application scene graph for a user's current location of an extended reality system in some embodiments. Retrieving one or more prisms previously deployed to the user's current location may include retrieving instance data for the one or more prisms from an external database, such as (e.g., a database that stores a passable world model in a cloud environment), and reconfiguring a local database (e.g., an internal passable world model database that comprises a smaller portion of the externally stored passable world model) with the instance data for the one or more prisms.

[0115] In some of these embodiments, the instance data for a prism includes a data structure of one or more prism properties that define the prism. The prism properties may comprise, for example, at least one of location, orientation, range width, range height, range depth, anchor type, and / or anchor position. Additionally or alternatively, the instance data for a prism may include key / value pairs of one or more application-specific properties, such as state information for virtual content previously rendered in the prism by the application. In some embodiments, the data may be stored entirely locally, such that an external database is not required.

[0116] Prisms, in some embodiments, include a 3D bounded space with fixed and / or adjustable boundaries depending on their creation, although degenerate 3D prisms with lower dimensions are also contemplated. Once generated, prisms may be positioned anywhere within the XR system's virtual 3D space and / or the user's environment or anywhere within the real world (e.g., by the universe browser engine or an instance thereof). The boundaries of a prism may be defined, at least in part, by the system (e.g., the universe browser engine), by the user, and / or by a web page developer based on the size or extent of the content to be rendered within the prism. In some embodiments, only the XR system (e.g., its universe browser engine) may create and / or adjust the boundaries of a prism on the XR system. The boundaries of a prism may be displayed (e.g., in a graphically de-emphasized manner) in some embodiments. In some other embodiments, the boundaries of a prism are not displayed.

[0117] The bounds of the prism define a space within which virtual content and / or rendered content may be created. The bounds of the prism may also, in some embodiments, constrain where and how much a web page panel may be moved and rotated. For example, when a web page panel would be positioned, rotated, and / or scaled such that at least a portion of the web page panel is outside the prism, the system (e.g., the Universe Browser Engine) may prevent such positioning, rotation, and / or scaling.

[0118] In some embodiments, the system may position, rotate, and / or scale the web page panel to the next possible position that is closest to or close to the original position, rotation, or scale in response to an original positioning, rotation, or scaling request in some embodiments. In some of these embodiments, the system may show a residual image or frame of the next possible position, rotation, or scale, and optionally display a message indicating that the original position, rotation, or scale may result in at least a portion of the web page panel being outside the prism.

[0119] Applications may render graphics into prisms, at least in part, through the universe browser engine. In some embodiments, the universe browser engine renders the scene graph and / or has full control over the positioning, rotation, scale, etc. of the prisms. Additionally, the universe browser engine may provide the ability to bind one or more prisms to physical objects such as walls, surfaces, etc., and register the prisms in a passable world that may be shared among multiple XR system users as described herein.

[0120] Additionally or alternatively, the universe browser engine may control the sharing of content among multiple XR system users. In some embodiments, the universe browser engine may also manage prisms. For example, the universe browser engine may create prisms, manage positioning and / or snapping rules for one or more physical objects, provide user interface controls (e.g., "close" buttons, action bars, navigation panels, etc.), and track prism records or data (e.g., which applications own or invoke which prisms, where prisms should be placed, how prisms are anchored—body-centered, world-anchored, etc.).

[0121] In some embodiments, prism behavior may be based, in part or in whole, on one or more anchors. In some embodiments, prism behavior may be based, in part, on positioning, rotation, and / or scaling (e.g., user placement of web page content or the prism itself through user interaction, developer positioning, rotation, and / or scaling of web page panels, etc.) and / or body motion (e.g., billboarding, body centering, delayed head fixation, etc.). Prisms may move within 3D virtual space in some embodiments. In some of these embodiments, the universe browser engine may track and manage prism movement (e.g., billboarding to user / body centering, delayed billboarding, rocking during movement, collision bounces, etc.).

[0122] Additionally or alternatively, prisms, including browsers, web page panels, and any other virtual content, may be transformed in many different ways by applying corresponding transformations to the prisms. For example, prisms can be moved, rotated, scaled, and / or morphed within the virtual 3D space. In some embodiments, a set of transformations is provided for transforming web pages, web page panels, browser windows, prisms, etc. In some embodiments, prisms having a set of functionality may be automatically created. The set of functionality may, in some embodiments, comprise, for example, minimum and / or maximum sizes allowed for the prism and / or aspect ratios for resizing the prism. The set of functionality may comprise associations between prisms and objects (e.g., virtual objects, physical objects, etc.) within the virtual or physical 3D spatial environment. Additional virtual content may be rendered into one or more additional prisms, each virtual content may be rendered into a separate prism in some embodiments, or two or more virtual contents may be rendered into the same prism in some other embodiments.

[0123] In some embodiments, the prisms may be completely transparent and therefore invisible to the user, or in some other embodiments, may be semi-transparent and therefore visible to the user. Unlike traditional web pages that are displayed within a browser window, the browser window may be configurable (e.g., via the Universe Browser Engine) to be shown within or hidden from the virtual 3D space. In some embodiments, the browser window may be hidden and therefore invisible to the user, but some browser controls (e.g., navigation, address bar, home icon, reload icon, bookmarks bar, status bar, etc.) may still be visible to the user within the virtual 3D space. These browser controls may be displayed to be translated, rotated, and translated along with the corresponding web page in some embodiments, or may be displayed independently of the corresponding web page in some other embodiments.

[0124] In some embodiments, the prisms may not overlap with other prisms in the virtual 3D space. The prisms may comprise one or more universal features to ensure that different software applications interact properly with each other and / or with one or more application-specific features selected from a list of options.

[0125] In some embodiments, the prism vertex (106F) may be displayed to the user in a de-emphasized manner (e.g., reduced brightness, etc.) so that the user perceives the boundaries of the prism within which the virtual object or rendered web page may be translated or rotated. For example, if a web page or web page panel is translated or rotated such that a portion of the web page or web page panel falls outside the boundaries defined by the prism, in some embodiments, the system may nevertheless still display the remaining portions of the web page or web page panel that are within the prism but not the portions of the web page that fall outside the boundaries of the prism. In some other embodiments, the mixed reality system limits the translation, rotation, and translation of the web page or web page panel so that the entire web page or web page panel may be freely translated, rotated, or translated but may obey the boundaries of the prism.

[0126] As illustrated in FIG. 1F , the virtual 3D space may include one or more prisms. Furthermore, a prism may also include one or more other prisms, such that, in some embodiments, a prism may be considered a parent of one or more other prisms. In some of these embodiments, a prism tree structure may be constructed, with each node representing a prism and an edge between two connected nodes representing a parent-child relationship between the two connected nodes. Two prisms may be moved to overlap one another or even to entirely contain one prism within the other. While an inclusive relationship between two prisms may or may not indicate that a parent-child relationship exists between the two prisms, the mixed reality system may be configured to allow a user to define a parent-child relationship between two prisms. Furthermore, a first prism may or may not need to be entirely contained within a second prism for a parent-child relationship to exist. In some embodiments, all child prisms inherit the transformations, translations, and rotations that have been or will be applied to the parent prism, such that the parent prism and its child prisms are transformed, translated, and rotated together.

[0127] 1G-1I illustrate projected views of several simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space, according to some embodiments. FIG. 1G illustrates a side view of a 3D rectangular prism 100G created in a physical user's environment 114G relative to a physical object (e.g., a desk) 118G above a floor 116G in the physical environment. The mixed reality system may apply various techniques described herein to display a web page 106G within the prism 100G.

[0128] As described above, the display of the web browser window is suppressed in FIG. 1G, and therefore the browser window is invisible to the user 114G. Browser controls (104G) may nevertheless be displayed to the user 114G for the user to control the behavior of the browser. Additionally, the name or other identification of the application (e.g., the name of the browser, the name or identification of the web page, etc.) 102G may also be rendered either within or outside of the prism 100G. In some embodiments, 102G may be contained within a separate prism with a relationship (e.g., a fixed relative location or a parent-child relationship, etc.) to the prism 100G.

[0129] Using mixed reality headset 115G, a user is enabled to view virtual content, such as web page 106G and browser control 104G, that is projected through mixed reality headset 115G to the eyes of user 114G. The mixed reality system or a component thereof (e.g., universe browser engine) may further provide translation handle 112G, rotation handles 108G and 110G, scale handle (not shown), or one or more other transformation handles (not shown) for the user to translate, rotate, scale, and transform prism 100G, browser window (which is obscured), web page 106G, control 104G, etc. within the virtual 3D space.

[0130] Figure 1H illustrates a scenario in which a user 114G moves and rotates the web page and controls in Figure 1G to a downward and first tilted orientation (102H and 104H). The web page and controls may be moved, rotated, and translated, separately or together, in a full 360-degree manner within prism 100G, and Figure 1H illustrates the web page and controls being moved and rotated together. For example, rather than moving the web page and controls downward and tilting them upward, the web page and controls may be moved upward and rotated downward, ending up in the configuration shown as 106H and 108H in Figure 1H.

[0131] Figure 1I illustrates a scenario in which a user 114G wearing a mixed reality headset 115G moves and rotates the web page and controls in Figure 1G facing downward and to a first orientation (102I and 104I) so that the web page 104I and control 102I appear stationary on top of a desk 118G. The user 114G may also move the web page and controls facing upward to a second orientation (106I and 108I) so that the web page 108I and control 106I appear seated above the user 114G. The web page and controls may be moved, rotated, and translated, separately or together, in a full 360-degree manner within the prism 100G, and Figure 1I illustrates that the web page (104I and 108I) and the controls may be moved and rotated together (106I and 102I). FIG. 1I further illustrates a configuration in which controls 104G can be independently operated by a user 114G.

[0132] 1J-1L illustrate three simplified examples of using a mixed reality system to manage and display web pages and web resources in a three-dimensional virtual space. FIG. 1J illustrates a working example of the application of various techniques described herein for managing and displaying web pages in a 3D virtual space. In this example, a developer 102J may code a web page to rotate and move a chessboard 104J so that, when projected by a mixed reality system to a user's eye, it appears seated on a desk or table (not shown) at the correct eye level within the developer's or user's 102J field of view 106J.

[0133] This example may further illustrate a scenario in which a planar web page view (or other 3D perspective view) of a chessboard is originally presented to the user 102J in virtual 3D space. The user 102J may use the techniques described herein to move and / or rotate the web page 104J to a desired position and / or orientation, for example, by manipulating positioning and rotation handles (not shown). Using the web page and web resources provided by that website (e.g., 16 chess pieces per player), the mixed reality system may enable the user 102J to place virtual 3D chess pieces on the chessboard 104J, positioned and / or rotated as desired by the user 102J, using either the user's hands, totem, or controller (108J). Note that in this example, the boundaries of the web page, browser window, and prism are suppressed (e.g., hidden), and these boundaries are invisible to the user. In this embodiment, a parent-child relationship may be created between the virtual 3D models of the chess board 104J (parent) and the chess pieces 110J (children) such that the virtual 3D chess pieces are transformed, translated, and rotated together with the chess board 104J displayed within the web page.

[0134] 1K illustrates another working example of the application of various techniques described herein for managing and displaying web pages within the present 3D virtual space. In this example, a developer 102J may code a web page to rotate and translate a map 104M showing a proposed development neighborhood so that the map 104J, when projected by a mixed reality system to a builder's eye, appears seated on a desk or table (not shown) within the builder's 102J field of view 106J at the correct eye level (e.g., horizontal).

[0135] Additionally, developer 102J may use the techniques described herein to move and / or rotate web page 104J (and thus the map displayed therein) to a desired position and / or orientation. A developer may position a virtual 3D model of office building 106K, built by the developer or provided from another resource, by using the developer's hand, totem, or controller (104K) to move and rotate the 3D virtual office building 106K onto the correct location in the map. A developer may further incorporate other virtual 3D models, such as trees 112K, office buildings 108K, vehicles 110K, etc., onto the map to build a virtual 3D plan or 3D map for plan development or 3D navigation.

[0136] Once at least some virtual objects have been imported onto the map, the developer may further rotate the web page showing the map. For example, when two office buildings 108K are imported onto the map, these two office buildings 108K may occlude the developer's view of portions of the map. In this example, the developer may further rotate the map such that the occluded portions are exposed to the developer so that the developer can import other virtual objects onto them. Similar to the example shown in FIG. 1J, parent-child relationships may also be created between map 102K (parent) and virtual 3D objects 106K, 108K, and 110K (children) such that the virtual objects can be translated, translated, and rotated along with map 102K displayed within the web page.

[0137] 1L illustrates another working example of the application of various techniques described herein for managing and displaying web pages within a 3D virtual space. In this example, a user 102L wearing a mixed reality headset (not shown) may be presented with a prism 100L that includes an optional array or launch pad (or launcher) of website icons 106L within the user's field of view 104L, which may be rendered, for example, from a browser's bookmarks toolbar or provided by the universe browser engine of the mixed reality system described herein. The user may use an individual icon to launch the home web page of the corresponding website, for example, via a hand gesture, voice command, etc.

[0138] This example further illustrates that a user may work on the virtual development plan or 3D map 108L primarily by populating 3D virtual objects (e.g., 124L) with a developer's hand, totem, or controller 122L, as shown in FIG. 1K and described above. User 102L may also open several other web pages (e.g., streaming video web page 110L, online dictionary web page 112L, social media web page 114L, online wiki web page 116L, email client 118L, etc.) and further manipulate and arrange these web pages to the user's desired position and orientation within prism 100L by using the techniques described herein.

[0139] As can be seen from the example illustrated in Figures 1J-1L, various virtual content may be displayed across multiple depth or focal planes due to translation and rotation of the corresponding web page within the virtual 3D space. The mixed reality systems described herein provide the ability to render these virtual contents by using multiple depth or focal planes in conjunction with their variable focus mechanisms. Further details regarding mixed reality systems are described in the U.S. patent applications referenced in the first paragraph of this application.

[0140] 1M illustrates an exemplary user physical environment and system architecture for managing and displaying web pages and web resources in virtual 3D using a mixed reality system, in one or more embodiments. The exemplary environment 100 includes a user's landscape 110 as viewed by a user 103 through a head-mounted system 160. The user's landscape 110 is a 3D view of the world in which user-placed content can be composited on top of the real world. The exemplary environment 100 further includes access to a universe application or universe browser engine 130 via a processor 170 operably coupled to a network (not shown).

[0141] Although processor 170 is shown as a separate and isolated component from head-mounted system 160, in alternative embodiments, processor 170 may be integrated with one or more components of head-mounted system 160 and / or may be integrated within other system components in representative environment 100, such as, for example, a computing network (not shown) and a network for accessing external storage device 150. In some embodiments, processor 170 may not be connected to a network. Processor 170 may be configured with software (e.g., universe application or universe browser engine 130) for receiving and processing information, such as video, audio, and / or other data (e.g., depth camera data) received from head-mounted system 160, local storage device 137, application 140, computing network, and / or external storage device 150.

[0142] The universe application or universe browser engine 130 may be, for example, a 3D window manager similar to a 2D window manager running on a desktop computer for managing 2D windows displayed on the desktop computer's display screen. However, the universe application or universe browser engine 130 (which may hereinafter, for simplicity, be referred to as a "universe") manages the creation, placement, and display of virtual content 115 within a 3D spatial environment and the interactions between multiple pieces of virtual content 115 displayed within the user's landscape 110. The virtual content 115 from applications 140 is presented to the user 103 inside one or more 3D window display management units, such as bounded volumes and / or 3D windows, which may hereinafter be referred to as prisms 113.

[0143] The bounded volume / 3D window / prism 113 may be a rectangular, cubic, cylindrical, or any other shaped volume of space that can be positioned and oriented in space. The prism 113 may be a stereoscopic display space with boundaries within which content (e.g., virtual content) is rendered / displayed; the boundaries are not displayed. In some embodiments, the boundaries may be displayed. The prism 113 may present a standard, foundational level of interaction and control of an application's content and its placement. The prism 113 may represent a subtree of a multi-application scene graph, which may be embedded inside the universe browser engine 130 or external to but accessed by the universe browser engine. A scene graph is a general data structure commonly used by vector-based graphics, editing applications, and modern gaming software that arranges the logic, and often (but not necessarily) spatial representation, of a graphical scene. A scene graph may be considered a data structure that defines how content is positioned and transformed relative to each other within that structure. An application 140 is given an instance of a prism 113 into which to place content. The application may render 2D / 3D content within the prism 113 using relative placement algorithms and arbitrary transformations, but the universe browser engine (130) may still ultimately be responsible for managing overall interaction patterns such as content extraction. Multiple applications may render to the universe browser engine (130) through prisms 113, with process boundaries separating the prisms 113. There may be n bounded volumes / prisms 113 per application process; this is an n:1 relationship, explicitly meaning that only one process per application may run per bounded volume / prism 113; however, there may be m processes running, each with its own bounded volume / prism 113.

[0144] The universe browser engine (130) operates using a prism / distributed scene graph approach for 2D and / or 3D content. A portion of the universe browser engine's scene graph is reserved per application for rendering. Each interaction with an application, such as a launcher menu, landscape, or body-centric application zone (all described in more detail below), may occur through a multi-application scene graph. Each application may be allocated one to N rectangular prisms, representing a subtree of the scene graph. Prisms are not allocated by client-side applications but instead are created through user interaction inside the universe browser engine (130), for example, when a user opens a new application in the landscape by clicking a button on a controller. In some embodiments, an application may request a prism from the universe browser engine (130), but the request may be denied. In some embodiments, if an application requests a new prism and is granted, the application may convert only the new prism to one of the other prisms.

[0145] The universe browser engine (130) organizes virtual content 115 from applications 140 into objects called prisms 113. Each application process or instance may render its virtual content into its own individual prism 113 or set of prisms. The universe browser engine (130) manages the world space, sometimes called a landscape, in which the prisms 113 are displayed. In some embodiments, the universe browser engine (130) provides the ability to connect applications to walls and surfaces, place prisms at arbitrary locations in space, register them with the mixed reality system's world database, and / or control the sharing of content among multiple users of the mixed reality system.

[0146] In some embodiments, the purpose of prisms 113 is to provide behavior and control over the rendering and display of content. Similar to 2D displays, where windows may be used to define the location, menu structure, and display of 2D content within a 2D window, in 3D virtual displays, prisms allow a mixed reality system (e.g., the universe browser engine (130)) to wrap controls around the display of 3D content related to, for example, content location, 3D window behavior, and / or menu structure. For example, controls may include at least placing virtual content in a specific location within the user's landscape 110, removing virtual content from the landscape 110, copying virtual content and / or placing copies in a different location, etc. In some embodiments, prisms may be created and destroyed by, and only by, the user. This may be done explicitly to help control abuse of the interface provided and to help the user maintain control of their content.

[0147] Additionally, in some embodiments, the application 140 does not know where its volumes are placed in the landscape, only that they exist. In some embodiments, an application may request one or more prisms, and the request may or may not be granted. After a new prism is created, the user may change its position, and / or the application may automatically position the new prism relative to the current existing prisms associated with the application. In some embodiments, each application 140 that utilizes the services of the universe browser engine to render 3D content (e.g., composited 3D content) into the universe browser engine process may be required to first register a listener with the universe browser engine. This listener may be used to inform the application 140 of the creation and destruction of rendering prisms based on user movement and user interaction with those prisms. A listener is an interface object that receives messages from an inter-process communication system. For example, in the Android operating system, a listener is an object that receives messages through the Android Binder interface. However, any IPC system may be used so that the Binder is not used all the time.

[0148] In some embodiments, a prism may be created from one of the following interactions: (1) a user extracts content from an extractable node (disclosed further below), (2) a user starts an application from a launcher, (3) a user downloads a nearby passable world map tile containing an installed instance of an application that the user has permission to view, (4) a user downloads a nearby passable world map tile containing an object for which the passable world object recognizer infrastructure has detected that a given application must render content, and / or (5) a user triggers a dispatch from another application that must be handled within a different application. In some embodiments, the passable world model allows a user to effectively pass pieces of the user's world (e.g., surroundings, interactions, etc.) to another user.

[0149] Extractable content is content inside a prism (including, but not limited to, icons, 3D icons, words in a text display, and / or images) that can be extracted from the prism and placed into the landscape using an input device. For example, a prism may display a web page showing running shoes for sale. To extract the running shoes, the shoes can be selected and "extracted" using an input device. A new prism is created using a 3D model representing the shoes, and that prism will move out from the original prism toward the user. Like any other prism, the user may use an input device to move, scale, shrink, or rotate the new prism containing the shoes within the 3D space of the landscape. Extractable nodes are nodes in the prism's scene graph that have been tagged as being capable of being extracted. In the Universe Browser Engine, extracting content means selecting an extractable node and using an input device to extract the content out of the prism. The input to initiate this extraction may be aiming a six-degree-of-freedom pointing device at the extractable content and pulling a trigger on the input device.

[0150] Each user's individual mixed reality system (e.g., mixed reality device) captures information as the user passes through or resides within the environment, which is the mixed reality system process to produce a passable world model. Further details regarding passable worlds are described in U.S. patent application Ser. No. 14 / 205,126, filed March 11, 2014, entitled "SYSTEM AND METHOD FOR AUGMENTED AND VIRTUAL REALITY," previously incorporated by reference. Individual mixed reality systems may communicate or pass the passable world model to a common or shared data collection, referred to as the cloud. Individual mixed reality systems may communicate or pass the passable world model to other users, either directly or via the cloud. The passable world model essentially provides the ability to efficiently communicate or pass information encompassing at least the user's field of view. In one embodiment, the system uses pose and orientation information and the collected 3D points described above to create the passable world.

[0151] In some embodiments, a passable world model enables users to integrate content (e.g., virtual and / or physical content) with the real world. A passable world system may include one or more mixed reality systems or mixed reality user devices that can connect to a cloud network, a passable world model, a set of object recognizers, and a database (e.g., external database 150). The passable world model may be configured to receive information from the mixed reality user devices and transmit data to them over the network. For example, based on input from the users, pieces of the passable world may be passed from one user to another. The passable world model may be viewed as a collection of images, points, and other information (e.g., real-world information) based on which a mixed reality system can build, update, and augment a virtual world on the cloud and effectively pass pieces of the virtual world to various users. For example, a set of real-world points collected from mixed reality user devices may be collected within the passable world model. Various object recognizers may crawl through the passable world model, recognize objects, tag images, etc., and associate semantic information with the objects. A passable world model may use a database to augment knowledge of the world, associate semantic information, and store data associated with the passable world.

[0152] For a Prism that is visible to the user but whose controlling application is not currently installed, the Universe Browser Engine may render a temporary placeholder for that application that, when interacted with, redirects the user to the application store page for that application. In some embodiments, a Prism may be destroyed upon similar interactions: (1) the user moves far enough away from the passable world map tile that the installed instance of the application is unloaded (i.e., removed) from volatile memory, (2) the user abandons the installed instance of the application, and / or (3) the application requests that the Prism be closed.

[0153] In some embodiments, if prisms for an application are not visible and / or not loaded, the processes associated with those prisms may be paused or terminated. Once the installed prisms for that application become visible again, the processes may resume. Prisms may also be hidden, although in some embodiments, this may occur only at the request of the universe browser engine and the user. In some embodiments, multiple prisms may be installed in the same exact location. In such embodiments, the universe browser engine may manage rendering by showing only one instance of an installed prism in a location at a time and hiding the visibility of the prism (and its associated content) until a user interaction is detected, such as the user "swiping" to the next visible element (e.g., a prism) in that location.

[0154] In some embodiments, each prism 113 may be exposed to the application 140 through a volume listener interface, with methods for accessing the prism's 113 properties and registering content within a scene graph subtree for shared resources such as meshes, textures, and videos. In some embodiments, because the application 140 does not know where a given prism 113 is placed in 3D space, the volume listener interface may provide accessor methods for a set of hints that help define where a given prism resides within the universe browser engine, such as hand-centered, fixed in landscape, or body-centered. These properties additionally specify the prism's expected behavior and may be controlled in a limited manner by either the user, the application 140, or the universe browser engine. A given prism can be positioned relative to other prisms owned by the application. An application can specify that prisms from that application should snap together (that is, both sides of their bounding volumes touch) while being placed. Additionally, prisms may provide an API for key / value data storage. Some of these key / value pairs are only writable by authorized applications.

[0155] In some embodiments, application 140 is a client software application that provides content to be displayed to user 103 in user landscape 110. For example, application 140 may be a video streaming application, where video data may be streamed to the user to be displayed on a 2D planar surface. As another example, application 140 may be the Halcyon application, which provides 3D imaging of physical objects that may indicate past time periods that were calm, happy, and peaceful for the user. Application 140 provides content that the user may want to include in user landscape 110. The universe browser engine, via prism 113, manages the placement and management of content generated by application 140.

[0156] When a non-immersive application is executed / launched within the user's landscape 110, its content (e.g., virtual content) is rendered inside the prism 113. A non-immersive application may be an application that is capable of launching and / or displaying content within a shared 3D environment simultaneously with one or more other applications. While the virtual content may be contained within the prism, the user may still interact with the virtual content, for example, by hovering over an object, clicking on it, etc. The prism 113 may also bound the displayed content of the applications 140, so that different applications 140 do not interfere with each other or other objects in the user's landscape 110. The prism 113 may also provide a useful abstraction for interrupting, pausing, and / or minimizing virtual content from applications 140 that are outside of or too far away from the user's field of view.

[0157] Prisms 113 may be anchored / tied / pinned to various objects within the user's landscape 110, including snapping or anchoring to another prism. For example, a prism 113a displaying virtual content 115 (e.g., video 115a from a video streaming application) may be anchored to a vertical wall 117a. As another example, a prism 113b displaying a 3D tree 115b from the Halcyon application is shown in FIG. 1 as being anchored to a table 117b. Additionally, prisms 113 may be anchored relative to the user 103 (e.g., body-centered), i.e., the prism 113 displaying virtual content 115 may be anchored to the user's body such that as the user's body moves, the prism 113 moves relative to the user's body movements. Body-centered content may be application content, such as a plane, mesh, etc., that follows the user and remains positionally aligned with the user. For example, a small dialog box that follows the user but exists relative to the user's spine rather than the landscape 110. In addition, the prism 113 may also be anchored to a virtual object, such as a virtual display monitor, that is displayed within the user's landscape 110. The prism 113 may be anchored in different ways, which are disclosed below.

[0158] The universe browser engine may include a local database 137 to store properties and characteristics of prisms 113 for a user. The stored prism information may include prisms activated by the user within the user's landscape 110. The local database 137 may be operatively coupled to an external database 150, which may reside in the cloud or an external storage facility. The external database 150 may be a persistent database that maintains information about the user's and other users' mixed reality environments.

[0159] For example, as a user launches new applications and displays virtual content within the user's physical environment, local database 137 may store information corresponding to prisms created by the universe browser engine and placed in specific locations, and application 140 may render the content into prism 113 for display within the user's landscape 110. Information corresponding to prisms 113, virtual content 115, and application 140 stored in local database 137 may be synchronized with external database 150 for persistent storage.

[0160] In some embodiments, persistent storage may be important because data stored in local database 137 may be erased, deleted, or become non-persistent when the mixed reality system is turned off. Thus, when a user turns on the mixed reality system, the universe browser engine may synchronize with external database 150 and retrieve an instance of local database 137 corresponding to user 103 and the user's landscape 110 prior to the mixed reality system being turned off. Local database 137 may be an instance of external database 150, which includes information related to user 103 and the user's current environment. External database 150 may additionally store local database instances for other users, multiple users, the same user over time, and / or other environments. External database 150 may contain information used to manage and share virtual content among multiple users of the mixed reality system, while local database 137 stores and maintains information corresponding to user 103.

[0161] The universe browser engine may create a prism 113 for an application 140 each time the application 140 needs to render virtual content 115 on the user's landscape 110. In some embodiments, the prism 113 created by the universe browser engine allows the application 140 to focus on rendering the virtual content for display, while the universe browser engine focuses on creating and managing the placement and display of the prism 113 with the virtual content 115 displayed within the bounds of the prism by the application 140.

[0162] Each piece of virtual content 115 rendered by application 140 to be displayed within user landscape 110 may be displayed within a single prism 113. For example, if application 140 needs to render two pieces of virtual content (e.g., 115a and 115b) to be displayed within user landscape 110, application 140 may render the two pieces of virtual content 115a and 115b. Because virtual content 115 includes only the rendered virtual content, the universe browser engine may create prisms 113a and 113b to correspond to each piece of virtual content 115a and 115b, respectively. Prism 113 may include 3D window management properties and characteristics for virtual content 115, allowing the universe browser engine to manage the virtual content 115 inside prism 113 and the placement and display of prism 113 within user landscape 110.

[0163] The universe browser engine may be the first application visible to the user 103 when the user 103 turns on the mixed reality device. The universe browser engine may be responsible for at least (1) rendering the user's world landscape, (2) 2D window management and 3D window (e.g., prism) management for planar applications, (3) displaying and executing application launcher menus, (4) allowing the user to place virtual content into the user's landscape 110, and / or (5) managing different states of the display of prisms 113 within the user's landscape 110.

[0164] Head-mounted system 160 may be a mixed reality head-mounted system that includes a display system (e.g., a user interface) positioned in front of the eyes of user 103, speakers coupled to the head-mounted system and positioned adjacent the user's ear canals, a user sensing system, an environmental sensing system, and a processor (all not shown). Head-mounted system 160 presents user 103 with the display system (e.g., a user interface) for interacting with and experiencing the digital world. Such interactions may involve the user and the digital world, one or more other users interfacing with representative environment 100, and objects within the digital and physical worlds.

[0165] The user interface may include viewing, selecting, positioning, and managing virtual content via user input through the user interface. The user interface may be at least one or a combination of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, an audio input device, a smartphone, a tablet, or a head-mounted system 160. A haptic interface device is a device that allows a human to interact with a computer through bodily sensations and movement. Haptic refers to a type of human-computer interaction technology that involves tactile feedback or other bodily sensations to perform an action or process on a computing device.

[0166] An example of a haptic controller may be a totem (not shown). In some embodiments, the totem is a handheld controller that tracks its position and orientation relative to the headset 160. In this example, the totem may be a six-degree-of-freedom (6DOF) controller that allows a user to move the prism in elevation and azimuth (on the spherical shell) by moving the totem up or down. In some embodiments, to move an object closer or farther away, a user may use a joystick on the totem to "push" or "pull" the prism, or may simply move the totem forward or backward. This may have the effect of changing the radius of the shell. In some embodiments, two buttons on the totem may expand or contract the prism. In some embodiments, rotating the totem itself may rotate the prism. Other totem operations and configurations may be used and should not be limited to the embodiments described above.

[0167] The user sensing system may include one or more sensors 162 operable to detect certain characteristics, properties, or information related to the user 103 wearing the head-mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical properties / measurements of the user 103, such as one or more of the following: pupil constriction / dilation, angulation / positioning of each pupil, sphericity, eye shape (as eye shape changes over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that can be used by the head-mounted system 160 to enhance the user's viewing experience.

[0168] The environmental sensing system may include one or more sensors 164 for acquiring data from the user's landscape 110. Objects or information detected by the sensors 164 may be provided as input to the head-mounted system 160. In some embodiments, this input may represent a user interaction with the virtual world. For example, a user (e.g., user 103) viewing a virtual keyboard on a desk (e.g., table 188) may gesture with their fingers as if they were typing on the virtual keyboard. Finger movement motions may be captured by the sensors 164 and provided as input to the head-mounted system 160, which may be used to change the virtual world or create new virtual objects.

[0169] Sensor 164 may, for example, generally include an outward-facing camera or scanner to capture and interpret scene information, e.g., through continuous and / or intermittently projected infrared structured light. The environmental sensing system may be used to map one or more elements of user landscape 110 around user 103 by detecting and registering one or more elements from the local environment, including static objects, dynamic objects, people, gestures, various lighting, atmospheric and acoustic conditions, etc. Thus, in some embodiments, the environmental sensing system may include image-based 3D reconstruction software embedded in a local computing system (e.g., processor 170) and operable to digitally reconstruct one or more objects or information detected by sensor 164.

[0170] In some embodiments, the environmental sensing system provides one or more of the following: motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, acoustic source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optically enhanced sensors. It should be understood that the environmental sensing system may include other components beyond those discussed above.

[0171] As noted above, processor 170, in some embodiments, may be integrated with other components of head-mounted system 160, integrated with other components of the system of representative environment 100, or may be a separate device (wearable or separate from user 103) as shown in FIG. 1 . Processor 170 may be connected to various components of head-mounted system 160 through a physical wired connection or through a wireless connection, such as, for example, a mobile network connection (including cellular telephone and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. Processor 170 may include memory modules, integrated and / or additional graphics processing units, wireless and / or wired Internet connectivity, and codecs and / or firmware capable of converting data from sources (e.g., computing networks and user and environmental sensing systems from head-mounted system 160) into image and audio data, which may be presented to user 103 via a user interface (not shown).

[0172] Processor 170 handles data processing for various components of head-mounted system 160 and data exchange between head-mounted system 160 and software applications such as the universe browser engine, external database 150, etc. For example, processor 170 may be used to buffer and process data streaming between user 103 and a computing network, including software applications, thereby providing a smooth, continuous, and high-fidelity user experience. Processor 170 may be configured to execute a set of program code instructions. Processor 170 may include a memory for retaining a set of program code instructions, wherein the set of program code instructions comprises program code for displaying virtual content within a subset of the available 3D viewable space by displaying the virtual content within a stereoscopic display space, where the boundaries of the stereoscopic display space are not displayed. In some embodiments, the processor may be two or more processors operatively coupled.

[0173] In some embodiments, the mixed reality system may be configured to assign prisms universal features and application-selected / application-specific features from a list of pre-approved options for application-specific display customization configuration. For example, universal features ensure that different applications interact well together. Some examples of universal features may include maximum / minimum size, non-overlapping prisms (excluding temporary overlaps from collision behavior), hiding content outside the prism's boundaries, and requiring an application to require permission from the user if the application wants to access sensors or senseable information. The application-selected / application-specific features result in an optimized application experience.

[0174] Application selection / application specific features may include maximum / minimum size (within system limits), default size (within system limits), type of body movement (e.g., non-fixed / world-fixed, billboarding, edge billboarding, following / passive head fixing, following based on external sensors, fading, discussed below), child prism spawn location, child head pose highlighting, child prism relationship behavior, surface behavior, independent transform control, resizing vs. scaling, idle timeout, collision behavior, permissions / password for accessing the application, etc. In another embodiment, the mixed reality system may be configured to display virtual content in one or more prisms, which in some embodiments do not overlap one another.

[0175] In some embodiments, one or more prisms may overlap to provide specific interactions. In some embodiments, one or more prisms may overlap, but only with other prisms from the same application. In another embodiment, the mixed reality system may be configured to change the state of a prism based, at least in part, on the relative position and location of the prism to the user. In another embodiment, the mixed reality system may be configured to manage content creation within an application and manage content display within a separate application. In another embodiment, the mixed reality system may be configured to provide content into a prism and simultaneously open an application that will place the prism within the mixed reality environment.

[0176] In some embodiments, the mixed reality system may be configured to assign location, orientation, and range data to prisms for displaying virtual content within the prisms, where the virtual content is 3D virtual content. In some embodiments, the mixed reality system may be configured to pin the launcher application to a real-world object within the mixed reality environment. In some embodiments, the mixed reality system may be configured to assign a behavior type to each prism, where the behavior type comprises at least one of world pinning, billboarding, edge billboarding, follow head pinning, follow based on an external sensor, or fading (described in more detail below). In some embodiments, the mixed reality system may be configured to identify the most used content or applications specific to the installed location of the launcher application and, as a result, sort the applications, for example, from most frequently used to least used. In another embodiment, the mixed reality system may be configured to display favorite applications in the installed launcher application, where the favorite applications are based, at least in part, on the context for the installed launcher location.

[0177] 1N illustrates a system architecture for managing and displaying web pages and web resources in a three-dimensional space using a mixed reality system, according to one or more embodiments. System 100N includes a universe or universe browser engine 130, an application 140, an icon grid application 160N, a status bar app 170N, a social panel app 180N, and a store panel app 190N. These applications may represent a base level of applications on system 100N; however, in some embodiments, more or fewer applications may be part of system 100N.

[0178] As discussed in FIG. 1M above, the universe browser engine may be considered a 3D window (e.g., Prism) manager, similar to a 2D window manager that manages 2D windows in a traditional computer desktop system or the like. FIG. 1N may provide further details of the universe browser engine from FIG. 1M. Here, universe browser engine 130 may also include universe server 105N, loader volume 110N, secondary UI volume 120N, universe client 125N, launcher application 130, and universe server 105N. Universe server 105N may be a processing thread of the universe browser engine within a multi-threaded processing environment for multi-parallel processing.

[0179] Loader volume 110N is a placeholder volume that is displayed to the user while the universe browser engine creates a prism for displaying virtual content within the user's landscape 110. For example, if the user selects an application for display in a particular location within the user's landscape 110, such as on a vertical wall of the user's landscape 110, while the universe browser engine sets up a prism and launches an application for rendering virtual content into the prism, the universe browser engine may display loader volume 110N with a default icon as a placeholder volume to indicate to the user that the universe browser engine is setting up the prism for display. Once the application has finished rendering the virtual content into the prism for display in the user's landscape, loader volume 110N is replaced with the actual prism containing the rendered virtual content.

[0180] In some embodiments, while the universe browser engine launches the application for displaying the virtual content, the user 103 may move the loader volume 110N to a different desired location. In some embodiments, the user may move the loader volume 110N to a location different from the location of the initially selected loader volume / prism. Once the universe browser engine finishes creating the prism and the application renders the virtual content into the prism, the universe browser engine may replace the loader volume 110N, and the prism may display the virtual content whenever the user places the loader volume 110N.

[0181] A secondary UI volume 120N is another prism that may be created when a prism (e.g., its “parent prism”) is created. The secondary UI volume 120N provides a universal interface for the prism for the user. For example, the secondary UI volume 120N may be considered a window decoration because the secondary UI volume 120N provides a mechanism for managing the prism (e.g., close / remove, share, follow, take screenshots of the prism's contents, etc.). When a prism is created, a secondary UI volume 120N may be created for the prism if the prism is not part of a launcher (a launcher application may not have a secondary UI volume). The secondary UI volume 120N provides a space / volume for displaying graphical user interface icons, such as close / remove, share, follow, screenshots, etc., for the user to interact with and manage the prism. The secondary UI volume 120N may be associated with and grouped with the parent prism. The secondary UI volume 120N lifetime ends when the parent prism lifetime with which it is associated ends.

[0182] In some embodiments, the secondary UI volume 120N may have at least three states: (1) displaying nothing when the parent prism is out of focus; (2) displaying the “visible name” of the component when the parent prism is in focus; and (3) displaying a “carousel” of application menu option icons upon detecting a specific user interaction, such as holding the home button on a handheld controller (e.g., a totem or other suitable user interaction controller) for a certain number of seconds. The carousel displays a collection of icons, one of which may be a large “X” icon for closing the prism. In some embodiments, the secondary UI volume 120N receives input through its parent prism. In other words, the parent prism may determine whether the secondary UI volume 120N displays its carousel, and therefore whether the parent prism redirects user input to the secondary UI. The carousel of the secondary UI volume 120N is disclosed below.

[0183] In some embodiments, the launcher may be the default "home" menu for the mixed reality system. The launcher may organize multiple panels of content along with a system status bar. Each panel may represent a different content type. Applications may be pulled from the launcher and pinned into the landscape for quick return. The launcher itself may also be placed into the landscape for location-specific customization and / or quick access.

[0184] The launcher 130N provides users with the ability to launch new applications into their landscape 110. The launcher 130N may be an application consisting of a series of body-centered prisms called panels. The panels may be vertically and horizontally scrollable, and the user may switch between panels using, for example, a swipe motion. In some embodiments, one panel (e.g., the center panel) may be visible at a time, with two neighboring panels visible on its sides as placeholder panels. When the user swipes to the next panel, the placeholder panel may expand to reveal the full panel. The panels may include an icon grid application 160N, a social panel 180N, and a store panel 190N. In some embodiments, when the user swipes to the next panel, the panels themselves do not move or change, but instead, content (e.g., icons) in different panels may be animated in and out of the center panel (e.g., the active panel). Additionally, applications may be pulled from the launcher 130N and pinned into the user's landscape 110 for location-specific customization, as discussed further below.

[0185] In some embodiments, applications 140 (FIG. 1M) may communicate with the universe browser engine through a centralized rendering service client 150N on each application 140. The centralized rendering service client 150N may communicate with a universe server 105N in the universe browser engine 130. The centralized rendering service client 150N may be a client service of the centralized rendering system that allows applications 140 and other applications that generate content for display in the user's landscape to communicate with the universe browser engine through the universe server 150N.

[0186] Universe server 150N may provide the services of a unified rendering system that allows the universe browser engine to communicate with applications that provide universe browser engine content for display within the user's landscape. In some embodiments, the communication may include more than rendering data, such as entering data, requesting security permissions, requesting the display or hiding of a virtual keyboard, etc.

[0187] In some embodiments, a unified rendering system may be a system dedicated to hardware and software resources for receiving graphical data from multiple applications for display on a single display (e.g., within a user's landscape in a mixed reality system). The unified rendering system combines graphical data from multiple applications 140 into a “unified” data structure, such as a scene graph, which can be used to render a scene on a display that reflects the graphical data from the multiple applications in a realistic and efficient manner. To achieve a unified rendering system, in some embodiments, applications may make changes to a local representation of a prism, called a client prism (e.g., client prism 115N from FIG. 1N). These changes may then be transmitted to universe server 105N and stored in the server prism. The unified rendering system may then render the updated data into the server prism. The unified rendering system may hereinafter be referred to as the “Cali” or “Kali” system. The universe browser engine may be considered an extended version of the Cali server, for example, because the universe browser engine may manage prisms in the real world.

[0188] In some embodiments, each application 140 that creates virtual content for the universe browser engine (e.g., 115a or 115b in FIG. 1M, 115 in FIG. 1O) communicates with the centralized rendering system and universe browser engine through a centralized rendering service client 150N (hereinafter, may be referred to as a “Cali client”) that is installed on each individual application 140. Further information may be disclosed in related U.S. Provisional Patent Application No. 62 / 479,134, entitled “CENTRALIZED RENDERING,” filed March 30, 2017, and incorporated herein by reference in its entirety. The centralized rendering system improves the user's experience by ensuring that virtual content from multiple different applications is properly analyzed and processed, and, if necessary, that the virtual content is displayed to the user in a realistic manner. In some embodiments, the universe browser engine is an instance of a Cali server, with additional functionality such as managing prisms. In some embodiments, a client prism is an instance of a client volume and a server prism is an instance of a server volume, with additional functionality such as providing an application options display, displaying a loader volume while a prism loads its content, colliding with other prisms, and the ability to be part of a transformation tree.

[0189] Client prisms 115N-a and 115N-b comprise virtual content generated by application 140 and sent by Cali client 115N-a to universe server 105N for display in the user's landscape. In some embodiments, as application 140 makes changes to virtual content 115N-a and 115N-b, the changes to the virtual content are communicated from client prism 115N to universe server 105N, and the information is stored inside the universe browser engine in a corresponding server prism data structure (e.g., 113a or 113b in Figures 1M-1N or 113 in Figure 1O). In some embodiments, application 140 does not know where in the user's landscape virtual content 115N-a is displayed. The universe browser engine may manage the display location of the virtual content 115N-a (e.g., the virtual content 115a after being processed by the centralized rendering system) through a corresponding server prism 113a associated with the client prism 115N-a.

[0190] An application 140 may request a new prism by accessing universe server 105N. In some embodiments, universe server 105N may be a software module within the universe browser engine that communicates with a centralized rendering service client from an application 150N, which provides virtual content for display within the user's landscape 110. For example, when a user launches an application and desires to display virtual content from the application within the user's landscape, the application may provide the virtual content to the universe browser engine via the centralized rendering service client, i.e., from the application to a universe browser engine centralized rendering service on the universe browser engine, for display within a prism that may be anchored within the user's landscape.

[0191] In some embodiments, the icon grid application 160N may include a recent applications section (not shown) and / or a general applications section (not shown). The general applications section includes an icon representing each application installed on the mixed reality system. The general applications section may first include a call to a package manager (not shown) to determine a list of installed packages. An icon is added for each application within each package. As the package manager notifies the universe browser engine of package installations and uninstallations, the icon grid application 160N adjusts its icons accordingly. The package manager service manages the installation of applications and maintains information about those applications, such as their names, icon graphics, security permissions, executable files, and data files.

[0192] The recent icon section may be first reconstructed from the log on disk and then updated by calls from other services. Package names may be logged to disk when the lifecycle service notifies the launcher of an application start event and when the package manager notifies the launcher of a package uninstall event. A user may interact with the icon grid application 260 by selecting and launching an icon or extracting an icon and placing it in the landscape.

[0193] The lifecycle service may be a centralized service that manages the processes of starting, stopping, putting to sleep, and waking up applications. The lifecycle service also knows when an application terminates unexpectedly (crashes). When any of these events occur, listeners for the service are notified, and the universe browser engine is one of the listeners. The universe browser engine accesses this service to start, stop, sleep, and wake up applications. In some embodiments, the lifecycle service provides an application programming interface (API) for controlling the lifecycle of application processes running within the mixed reality system. The lifecycle service may spawn new processes, launch the application binary with a set of permissions, and call APIs on predefined interfaces implemented by the application to control its lifecycle. The lifecycle service also provides a listener interface through which other modules may track applications that are started / stopped / paused / resume. The lifecycle service may be a program separate from the launcher or the universe browser engine. In some embodiments, the lifecycle service may be middleware.

[0194] In some embodiments, as shown in FIG. 1N , an icon grid application 160N comprises a unified rendering service client 250b and a client prism 115N-c. As discussed above, in some embodiments, an application that displays content within a user's landscape may send its content to the universe browser engine via the unified rendering service client 150N, which communicates with the universe server 105N. Here, the icon grid application 160N, which provides icons for applications installed on the mixed reality system for the launcher menu, is similar to any other application that provides content for display within the user's landscape. However, in some embodiments, an icon within the icon grid application, when selected by the user, may instruct the universe browser engine to launch and start a new application, at which point the new application may request the universe browser engine to create a new prism (e.g., through the universe server 105N) so that the application can provide content for display in the new prism. If the application is already running, the universe browser engine may request the application to open the new prism.

[0195] The status bar application 170N provides status indicators for the mixed reality system. The status indicators and the status bar application 170N may not be user-adjustable. The status indicators may be initially populated by querying a first service for operating and maintaining Wi-Fi service, a second service for maintaining Bluetooth service, and a third service for status. When these services notify the status bar application 170N of updated status, the status bar may adjust accordingly. The status bar provides users with quick glanceable information to which they can respond quickly and efficiently from anywhere in the system. In some embodiments, the status bar may be displayed above the launcher. Four main sections in the status bar may be (1) global search, (2) notifications, (3) quick settings, and (4) power. Additional temporary sections, such as music, phone, and sharing, may be added to the status bar as needed.

[0196] When the user is in the launcher menu, the status bar is reduced to glanceable icons. When the user swipes to the top, it may trigger animation and the status bar may expand. The status bar may stay above the launcher while the user may swipe left and right through the launcher panel. When the status bar is highlighted, it may expand and animate forward. A sub-selection highlight may appear by default, for example, to the left above global search. If there is another section with more urgent content (e.g., recent notifications, low battery, etc.), the sub-selection highlight may appear over that section instead.

[0197] The social panel application 180N may consist of a set of contacts with which a user may interact. The social panel may initially be populated with a call to a contacts service for available contacts. Each contact may be added to the social panel and displayed to the user as an icon. As the social panel application 280 receives new contact, updated contact, and removed contact events, the social panel application 180N may adjust its contact information accordingly. A user may interact with a contact icon by clicking on it to pop up an options menu with various available contact providers. When the user selects a provider, a launcher application may launch the associated application with the contact's information.

[0198] The store panel application 190N may allow a user to search for, download, and install applications 140 for the mixed reality system. When a user requests to download and install an application, the launcher application 130N may verify the user's identity using an identity verification service (not shown) and then install the application using a package manager. Lifecycle services may be invoked when a user starts an application from a panel. In some embodiments, each panel in the launcher may function as a separate application instead of as one launcher application.

[0199] In some embodiments, the universe client 125N renders content specific to the universe browser engine. The universe server 105N does not render third-party applications. This is because content within prisms can only be rendered by the universe client 125N, not the universe server 105N. Therefore, to render infinity prisms, loader volume / prisms, and / or secondary UI (user interface) prisms, the work to render those specific types of content for the server may need to be delegated to the universe client 125N. Infinity prisms may be used by the universe to render additional graphics around a prism, for example, when two prisms collide. Infinity prisms are discussed further below. For loader prisms and secondary UI prisms, specific communication may exist between the universe server 105N and the universe client 125N to coordinate certain functionality.

[0200] For example, the universe server 105N may be informed that an application has finished loading. The universe server 105N may then notify the client-side loader prism that the application is now loaded. The loader prism would need to react to the event that the application has finished loading by showing animation. Once the client-side loader prism has finished showing animation, the loader prism may notify the universe server 105N that it has finished animating. The universe server 105N may then react to the notification that the loader prism has finished animating by dismissing the loader prism, discarding the loader prism, and displaying the app prism with rendered animation instead of the loader prism. The disclosed subject matter is merely one example of how the universe client 125N may function. Those skilled in the art will understand that there may be other examples in which the universe client 125N may assist the universe browser engine 130.

[0201] 10 shows an example of a universe browser prism, according to one or more embodiments. Application content is presented to a user inside one or more bounding volumes, called prisms. As discussed above, when a non-immersive application runs within a mixed reality system, its content is rendered inside a prism. Prism properties and characteristics allow the universe browser engine to consistently manage prisms within the user's landscape.

[0202] The volumetric space of the prism 113 may have a clear and definitive boundary, as shown by the dashed line in FIG. 1O. The boundary provides a bounding volume for the virtual content 115 to be displayed only within the boundary of the prism 113. The boundary of the prism prevents content from an application displayed within the prism from overflowing, i.e., spilling out of the prism and into the user's landscape. The boundary of the prism 113 may not be displayed to the user when the user views the virtual content 115 displayed within the prism 113. This is an important feature because it is important not to show the prism boundary that bounds the virtual content 115 in order to maintain a realistic display of 3D content within the user's landscape. Those skilled in the art will understand the importance of not showing the prism boundary that wraps around the virtual content 115 so that the virtual content is displayed within the user's landscape in a more realistic manner. In contrast to 2D windows, the borders and boundaries of 2D windows are generally displayed, so that a user of a computer displaying 2D windows can clearly distinguish content within one 2D window from content from another 2D window. However, in some embodiments, it may be advantageous to at least temporarily display the boundaries of a prism, for example, to help troubleshoot problems with one or more applications.

[0203] An application is given an instance of Prism 113 by the Universe Browser Engine to place content into. An application may render 2D and / or 3D content into Prism 113 using relative placement algorithms and / or arbitrary transformations, but the Universe Browser Engine is still ultimately responsible for managing the overall interaction patterns, such as content extraction. Multiple applications may render process boundaries separating Prisms to the Universe Browser Engine through Prism 113.

[0204] Each prism distributed within the universe browser engine may have a set of associated key / value properties that can be adjusted to determine various bits of behavior or convey information about why a given prism exists. Some properties are read-only for normal applications, but for applications with private APIs, these properties are writable. Prisms 113 may include prism properties 110O, application-specific properties 120O, and virtual content 115. In addition, some prisms 113 include secondary UI volumes 130O to provide users with additional prism management options. However, in some embodiments, a prism may not have a secondary UI volume 130O, for example, because these other types of prisms (e.g., launcher menu prisms) may not require the features provided by the secondary UI volume 130O. Like prism boundaries, secondary UI volumes 130O may also not be displayed to the user. When a user desires to make changes to a prism, the user may initiate a request to display an application options menu, which displays the UI controls of the prism within the volume space of the secondary UI volume.

[0205] Depending on the application they hold, Prisms may require different properties to provide appropriate feedback and behavior for their content. Application developers may choose from several pre-programmed options for their Prisms when creating their applications, so their content can be properly presented based on their preferences. Below are some examples of these options:

[0206] Prism properties 110O, at least in part, define a prism and enable the universe browser engine to manage and maintain the prisms within a user's landscape. For example, prism properties 110O may include one or more of the following: default size, maximum size, minimum size, anchor / placement type (e.g., options for billboard display, etc.), behavior of a given prism for an anchor type, anchor location, child prism spawn location, child head pose highlight, surface behavior, independent transform control, resizing vs. rescaling indicator, idle timeout variable, etc. Prism properties 110O provide the universe browser engine with the ability to track and manage each and every prism within a user's landscape. Having a single application manage the virtual content displayed within a user's landscape ensures that content displayed within a user's landscape is displayed in a consistent and reliable manner. Some of prism properties 110O are further disclosed below.

[0207] Maximum, minimum, and default size: Applications may have upper and lower bounds defined by the application developer (optionally with additional limits from the universe browser engine). In addition, the application developer may have a default size when the application is first launched.

[0208] Option to billboard during movement sequences: It makes sense for certain objects (e.g., content on a flat surface) to billboard towards the user during movement sequences to facilitate visibility and less management. For example, certain content displayed on a flat surface may be positioned relative to a specific location and / or object, but its orientation is automatically calculated so that the content displayed on the flat surface always faces the direction of the user viewing the content displayed on the flat surface. Other optional body movement behaviors may be added to this as well.

[0209] Child Prism Spawn Locations: Prisms may spawn children to create flexible layouts. Application developers should be able to determine the responsive range of locations in which children may spawn relative to a parent prism. Child Head Pose Highlights: Applications may be able to choose whether head pose highlights on child prisms can be treated as separate highlights or whether to continue to highlight all child / parent prisms as one unit.

[0210] Child Prism Relationship Behavior: Prisms determine whether their child prisms can be anchored to them in translation, rotation, and scale, and may also choose whether child prisms will occlude the main prism.

[0211] Surface Behavior: Prisms may snap to a surface and query that surface to determine if they wish to change size / scale. If the surface has space, the prisms may resize to fit all or a percentage of the surface and incorporate it into the user's field of view (FOV).

[0212] Independent Transform Control: An application may request independent control of its translation, rotation, and scaling. This may allow the application to move and transform itself.

[0213] Resizing vs. Scaling: Some applications may choose to resize their bounds instead of scaling only their content. This can accommodate more content being displayed within its bounds. This can function like a traditional computer 2D window.

[0214] Idle timeout: An application may be able to choose how long it takes to become idle. This may handle situations where the application may want to continue playing content even when out of view. For example, an application that displays live video may want to continue displaying content and playing audio even if the user temporarily looks away.

[0215] Application-specific properties 120O may be a list of key / value pairs that store application-specific state information for each prism. The list of key / value pairs is specific to an application, and the key / value pairs provide state information for the application's content being displayed or rendered within the prism. The list of key / value pairs may be different for each prism, depending on the application rendering within the prism. For example, if the application is a video streaming application, some key / value pairs may include the video name, the maximum viewing time for the video, the aspect ratio for displaying the video, etc.

[0216] Both prism properties 110O and application-specific properties 120O may be stored for each prism in a data structure in local database 137. Prism data is constantly updated while a user operates the mixed reality system and interacts with the prisms. As discussed above, prism instance data in local database 137 may be persisted by synchronizing with external database 150 on a periodic basis. In some embodiments, local database 137 and external database 150 may be synchronized in near real time.

[0217] When a user launches an application within the Universe Browser Engine, the user may pull out a prism from a launcher menu and place the resulting volume in space. Other methods of launching an application, such as clicking an application icon, may also be used. In some embodiments, the user may move the prism in altitude and azimuth (on the spherical shell) by moving a controller / input device (e.g., a totem) up or down. To move an object closer or farther away, the user may use a joystick on the totem to "push" or "pull" the prism, or slide their finger across a touch-sensitive portion of the totem. This has the effect of changing the radius of the shell. In some embodiments, two buttons on the totem may expand or contract the prism. Finally, rotating the totem itself may rotate the prism. This assumes that the totem may have six degrees of freedom (DOF). This is consistent with the types of controls used in VR painting applications, for example, but the totem can be any suitable user input device.

[0218] In some embodiments, prisms may not allow themselves to be positioned such that they fully or partially intersect with other prisms. Prisms may either not intersect at all or not reside / actively display at the exact same location (anchor point), with the exception that prisms may only slightly overlap for physics purposes, as discussed below. If more than one prism is positioned at the exact same location, the active application may be displayed, and other applications anchored at the exact same location may be hidden. The user may know that multiple applications are present at a location, for example, by dots displayed within the volume. For example, if three prisms / applications are present at a particular spot, three dots may be present. If the user views application #2 of the three, the second dot may be illuminated while the other dots may be dimmed. The user may then swipe or scroll through different applications. Graphics may switch, and dots may be updated (e.g., by illuminating the active dot) to indicate the currently active application.

[0219] In some embodiments, several prisms may be co-located at the same anchor location. At first glance, this may seem odd. With all the 3D space available to place applications in a user's landscape, why place them in the same spot? For example, a user's favorite place to play virtual board games may be on the kitchen table. In the morning, the user may prefer to play "Ticket To Ride" while eating breakfast. However, when the user gets home from work, the user may prefer to play "Risk" against the computer. The user may have multiple board games located in the same spot and switch between them as needed.

[0220] In some embodiments, a prism may be placed at an arbitrary location in space. In this case, the prism may be anchored by the center point of the cubic / rectangular volume. However, if the prism is moved near a horizontal surface in the landscape (e.g., during placement), the prism may attempt to snap to the surface. The anchor point may then be the center of the prism's bottom plane. Similarly, if the prism is moved toward a vertical surface (e.g., a wall), it may attempt to snap to it, and the anchor point may be the side of the prism that is next to the vertical surface.

[0221] The purpose of the anchor point may be to position the prism so that it does not interpenetrate the surface to which it is anchored. The anchor point may also move with the object to which it is anchored. When multiple prisms share the same location, that location may be the anchor point, not the center point of their individual volumes. Applications do not know or need to know where they are located, but may query the individual prisms to see how they are anchored. Applications may also specify which anchor types are valid. For example, it does not make sense to anchor a Halcyon to a vertical surface.

[0222] All content (graphics) for an application may be contained within the volume of a prism. The universe browser engine may automatically mask out graphics that extend outside of the prism. Because applications have no knowledge of other applications in the world, the universe browser engine may manage the interactions that occur between different prisms of different applications.

[0223] The user interface design for placing a prism may require the prism to bob and flop in a physical manner (like an object on a string) while it is being moved in a placed state. Instead of trying to predict the type of physical behavior that different applications will desire, the prism may feed movement information to the application (through a binder interface) while it is placed. The application can then behave appropriately.

[0224] There may also be physics between the prisms as they are placed. This may override the application's physics implementation, and the application may stop receiving movement data. The prisms may initially resist intersection. If the user continues to push two prisms into the same place, the prisms may snap to the anchor locations of the intersecting prisms. This may be done in a bouncy (e.g., similar to soap bubbles interacting with each other) and loosely physics-based manner.

[0225] Audio emitters may be placed in the application's scene graph as child nodes. These nodes may be local to the root node transform. Thus, prisms may be moved, and moving a prism does not require the application to update the transform of the audio node. The universe browser engine may be responsible for the final transform from audio emitters to world space. Prisms may also be responsible for constraining audio nodes to their boundaries. An application may not emit audio from points outside its individual prism.

[0226] In some embodiments, it may not be desirable to spatialize the audio. For example, if a user places a virtual television (TV) on a wall and focuses on the TV image, the TV's audio may be presented to the user without modification. This is likely to provide a better audio experience for the user. In the case of ambient sound, the audio signal already has spatial information. The sound may be emitted from virtual speakers placed in optimal locations relative to the TV.

[0227] In some embodiments, in response to a user pressing a button to control audio intensity, the universe browser engine may check the head pose, determine which prism the user is looking at, and send a volume up or volume down event to the corresponding prism. The prism may forward that information to an application launched in the prism, and the application may decide how to interpret it. If no application is focused in the landscape, the volume button setting may adjust the global volume.

[0228] In some embodiments, one difference between a traditional 2D window and a prism 113 is that in a 2D window, the border defining the boundary of the 2D window is intended to be visible by the user to provide a concrete boundary for containing content within the 2D window and separate it from content outside the 2D window's border. However, in some embodiments, the border of a 3D window (e.g., a prism 113) is intended to be invisible. If the user could see the outline (e.g., border) of all prisms, it would destroy the illusion of "reality," and virtual content displayed within a prism whose border is displayed would appear like computing / digital / virtual content instead of being real. In some embodiments, the border may be displayed, for example, to enable user interaction, if desired.

[0229] Another difference is that 2D windows are generally intended to be controlled and / or interacted with by the user. For example, a close button may always appear in the upper right corner of a traditional 2D window, or a menu bar may be displayed on the top border of the 2D window. However, with Prism, the user generally does not interact with the prism and its borders. Instead, a secondary menu (e.g., an app options menu) may be temporarily pulled down for the user to control and manage / operate the prism from a list of options.

[0230] Furthermore, a 2D window is independent of its surroundings. For example, the content displayed on a computer screen does not automatically change when a user moves the screen. However, prisms must be placed in context with the real world. For example, each prism may be placed in the real world relative to (1) an object in the real environment, such as a wall, a table, etc.; (2) a virtual object created to provide a background or canvas for the prism to anchor; and / or (3) the user. In some embodiments, a prism may be placed in context with the passable world and the real world.

[0231] Still further, in some embodiments, prisms may not be allowed to overlap / interpenetrate each other, with the exception that prisms may only overlap slightly for physics purposes. For example, in some embodiments, when virtual content within two or more prisms collide, the virtual content may appear to exhibit a bounce between the two virtual contents as they appear to collide with each other. Here, the prisms may only overlap slightly to create the effect of a bounce between the two virtual contents. In some embodiments, when the bounding boxes for two or more prisms collide, the prisms, and therefore the content of the prisms, may appear to bounce.

[0232] However, 2D windows on a computer may overlap, and often 2D windows may cascade on top of each other, obscuring each other from the user's view. In some embodiments, when two prisms are co-anchored in the user's landscape 110, one of the prisms may be displayed while the other prism is minimized from the display, and an icon or text or image (or any other visual indicator) is displayed to indicate to the user that another prism is co-anchored in the exact same location. In some embodiments, infinity prisms may be implemented to render additional graphics around the prisms, for example, when they collide. In some embodiments, an infinity prism may be a prism whose boundary is set to infinity.

[0233] For example, if two prisms are close enough to collide, the universe browser engine may render a glow in the region of space between the two prisms. To handle these exceptions, the universe browser engine may create an infinity prism that may encompass all space around / surrounding the two prisms, the user's entire field of view (what the user can currently see), the user's entire kinesthetic field of view (what the user can see as they move around), etc. This may allow the universe browser engine to draw graphics anywhere between the two prisms. In some embodiments, the infinity prisms may not collide or interact in any way. In some embodiments, the infinity prisms have no secondary UI, etc. In some embodiments, only the universe browser engine may have access to the infinity prism. The infinity prism may be created at universe browser engine initialization time and may exist at all times until the universe browser engine shuts down. In a second example, the infinity prism may be useful for moving characters (e.g., avatars, personal assistants, butterflies, animals, etc.) between other landscape apps, for example, to explain to the user the content of each application and / or how to use the application.

[0234] 1P illustrates an example of browser initialization to display a web page and reset position and / or rotation data of web page panels to defaults, in one or more embodiments. In this example, a browser (104P) initializes and loads a web page of interest (102P). The browser (104P) works in conjunction with a universe browser engine (106P), which transmits default position and rotation data to the browser engine (104P) at 158P. The universe browser engine (106P) further resets position and rotation data of web page panels for the web page at 156P. The browser engine (104P) receives the default position and rotation data from the universe browser engine and stores the default position and rotation data in a non-transitory computer-readable medium (108P) at 160P.

[0235] FIG. 1Q illustrates an example of determining position and / or rotation data for a web page panel using a GET request, in accordance with one or more embodiments. In this example, a web page of interest (102Q) may be coded to issue a request to a browser or browser engine (104Q) at 108Q to obtain position and rotation data for the web page panel in response to loading the web page by a browser, e.g., in response to user input or triggered by an event. The browser or browser engine (104Q) may obtain the position and / or rotation data from a non-transitory computer-readable medium (106Q) at 110Q. The browser or browser engine (104Q) may then return this position and / or rotation data from the non-transitory computer-readable medium (106Q) to the web page at 112Q.

[0236] 1R illustrates an example of determining position and / or rotation data for a web page panel using a SET request, in accordance with one or more embodiments. In this example, a web page of interest (102R) may be coded to issue a request to a browser or browser engine (104R) to set position and rotation data for the web page panel in response to loading the web page by a browser at 152R. In response to receiving the position and / or rotation data, the browser or browser engine (104R) may update the position and / or rotation data previously stored in a non-transitory computer-readable storage medium 108R (e.g., a cache) at 154R.

[0237] The browser or browser engine (104R) may further pass the received position and / or rotation data along with the SET request to the universe browser engine (106R) at 156R. The browser or browser engine (104R) may then return this position and / or rotation data from the non-transitory computer-readable medium (108R) to the web page. To fulfill the SET request, the universe browser engine 106R may set the position and / or rotation data of the web page panel by using the position and / or rotation data received along with the SET request and transmitted to 158R at 110R.

[0238] At least some of the embodiments described with reference to Figures 1S-1X and 2J-2M provide techniques for allowing web page developers to manipulate (e.g., set, adjust, etc.) the orientation (e.g., rotation about one or more axes) of a web page, rather than its position. In some of these embodiments, the developer is limited to setting the web page in a horizontal or vertical orientation (and not any other angle therebetween) relative to the user's frame of reference. In some other embodiments, the developer may be limited to setting the web page in a horizontal or vertical orientation and any other predetermined angle therebetween. These manipulations of the web page's position may be provided, for example, by the universe browser engine described herein, but not to the web page developer. In these embodiments, the web page starts from a position determined, for example, by the universe browser engine described herein. By providing web page developers with the ability to manipulate the orientation of a web page, and with the ability, for example, by the universe browser engine, to manipulate both the position and orientation of a web page, these embodiments may also provide techniques for allowing users of the XR systems provided herein to manipulate both the position and orientation of a web page.

[0239] 1S illustrates a high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, the universe browser engine 106S may transmit web page orientation data to the browser engine 104S and store such data within the browser engine 104S. The universe browser engine (e.g., 106S) may act as a 3D window manager, similar to the 2D window managers that run on laptop or desktop computers to manage 2D windows displayed on the laptop or desktop computer's display screen. The universe browser application, working in conjunction with the universe browser engine, also manages the generation, placement, and display of virtual content within the user's landscape via the mixed reality system. When the browser engine 104S initializes to display a web page of the website 102S or content therein (e.g., when a user launches a browser or loads / visits a website), the browser 102S may set orientation data for the web page to be rendered by the browser engine 104S if the website is coded to incorporate a set of APIs that are coded to both set orientation data in the client browser and retrieve orientation data from the client browser (108S).

[0240] A browser engine is a software component of a web browser that often performs tasks such as converting HTML and / or other resources of a web page into an interactive visual representation on a user's device (e.g., a mixed reality headset). While the following description may refer to Chromium or a Chromium-based browser, other browsers (e.g., Mozilla's Firefox, Apple's Safari, Microsoft's Edge and Internet Explorer, Google's Chromium, Opera, 3D Browser, etc.) are also contemplated, and the techniques apply with full equal effect to different browsers corresponding to their respective browser engines (e.g., Gecko for Mozilla, WebKit for Safari, Blink for Chromium, Chromium-based browsers, Microsoft's Edge and Opera, and Trident for Internet Explorer, etc.).

[0241] Browser engine 104S may include any publicly available browser, such as the browsers mentioned immediately above or any custom browser (e.g., a 3D browser). One of the functions of browser engine 104S is to provide functionality including, for example, using orientation data for a web page to translate HTML documents and other resources of the web page into an interactive visual representation on the user's device, calculating graphical coordinates for the visual representation using orientation data provided by either the website or universe browser engine 106S, painting or rendering prisms (or portions thereof), etc.

[0242] This set of APIs for one or more embodiments for managing and displaying web pages described herein may include APIs that conform to the Portable Operating System Interface (POSIX) API standard and provide spatial computing functionality independent of the platform or operating system. This set of APIs may, in some embodiments, be incorporated into the code of a website, such as those previously described above, and / or, in some other embodiments, may be integrated into part of an operating system or software application resident on a mixed reality system.

[0243] This set of APIs not only sets and gets orientation data between the website 102S, the browser engine 104S, and the universe browser engine 106S, but also works in conjunction with a runtime layer between the operating system and software applications and one or more 3D engines (or more precisely, between the operating system services layer and software applications). This runtime layer includes libraries, applications, services, etc., that provide (either alone or in conjunction with the 3D engines) advanced graphical user interfaces for the underlying mixed reality system and various computing functionality such as 3D translation and rotation transformations, 3D models with material and skeletal 3D model animation, 2D sprite animation, high-fidelity spatialized text rendering for spatial computing, 2D and sound field audio, 2D and stereoscopic video, rigid-body collision physics, real-time particle FX, input events and tactile control, hand gestures, speech recognition and speech-to-text conversion, real-time spatial computing techniques for rendering objects, lighting, shading, and real-world occlusion.

[0244] For example, the runtime layer may include, for example, a set of three-dimensional translation and rotation transforms that may be applied to a prism (as will be described later), a virtual object, a browser window, a rendered web page, etc., displayed in a virtual three-dimensional space such that the virtual object, the browser window, the rendered web page, etc., or any of its attributes (e.g., display coordinates, size, shape, perspective, lighting effects, etc.) appears or is perceived by the user as real or nearly real.

[0245] 1S, using the set of APIs, website 102S may send a request for orientation from browser engine 104S (112S). In this case, browser engine 104S may transmit the most recent orientation data to the website (114S). As described above, universe browser engine 106S may store the orientation data within or with browser engine 104S (e.g., in a cache accessible by browser engine 104S) so that browser engine 104S may respond to such requests from website 102S without, for example, additional delay from retrieving such orientation data from universe browser engine 106S.

[0246] In addition to returning (114S) the latest orientation data to the website 102S, the browser engine 104S may transmit the orientation data (110S) to the universe browser engine 106S, either alone or together with the rendered web page. The universe browser engine 106S creates or identifies a prism (or a smaller volume thereof) based on the orientation data received via 110S, and the universe browser engine 106S synchronizes the orientation data (116S) with the browser engine (104S). The universe browser engine 106S may call the browser engine 104S and invoke its native functions, libraries, and / or APIs (e.g., RenderWidget in Chromium or WebKit, OpenGL API, OpenGL ES2.0 API, etc., Almost Native Graphics Layer Engine, Direct3D API, WebGL, Gfx API, etc., or any combination thereof) to render the content of the web page within the prism or a portion thereof for display based on the orientation data.

[0247] In some embodiments, the universe browser engine 106S may invoke one or more 3D engines launched above the operating system core and operating system services in the operating system stack to render 3D content. These one or more 3D engines may include commercially or publicly available 3D engines (e.g., Unreal Engine 4, Unreal Engine 3, CryEngine V, Unity 3D Source Engine, Source Engine 2, etc.) that provide an abstraction layer for a graphics processing unit (GPU), any custom 3D engine for 3D and / or 2D graphics. In some of these embodiments, the mixed reality system need not incorporate an entire 3D engine. Rather, the mixed reality system may incorporate smaller portions, such as a rendering engine or rendering API, a physics engine for emulating the laws of physics, a scripting engine for parsing and executing scripts such as JavaScript scripts, a memory management module, and / or a threading module. In some embodiments, the mixed reality system described herein may invoke a renderer for rendering 2D content and a separate renderer for rendering 3D graphics.

[0248] The user may further modify the orientation of the web page displayed within the virtual 3D space through the mixed reality system. For example, the user may freely move and / or rotate the displayed web page, a placeholder therefor, or a mini-preview version of the web page by grasping a software handle associated with orienting the displayed web page, a placeholder therefor, or a mini-preview version of the web page, respectively. The orientation data of the web page is accordingly modified, and the universe browser engine 106S may further synchronize the orientation data with the browser engine 104S (116S). At that point, the browser engine 104S then replaces the original orientation data with this modified orientation data.

[0249] 1S-1X and 2J-2M, orientation data for a web page or a panel thereof may include both positioning and orientation data in some embodiments, while orientation data may include only orientation and not positioning data in some other embodiments. Furthermore, orientation data may refer to either positioning data, orientation data, or both positioning and orientation data in those embodiments in which orientation data includes both positioning and orientation data. In some embodiments in which orientation data includes only orientation data (e.g., rotation about one or more axes), these embodiments provide web page developers with the ability to manipulate (e.g., set, change, etc.) the orientation of a web page but not its position. In some embodiments, a web page developer may get the current position and rotation of a web page panel using, for example, Window.VirtualWorld.Position and Window.VirtualWorld.rotation, and the developer may further set the position and rotation using, for example, Window.VirtualWorld.Position=newDomPoint(x,y,z) and Window.VirtualWorld.rotation=newDomPoint(x,y,z), respectively.

[0250] 1T illustrates another high-level block diagram of a simplified system for interacting with a website to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, FIG. 1T illustrates the interaction between a browser panel 102T for web pages, a browser engine 104T, and a universe browser engine 106T, and the individual tasks and functions performed by the browser 102T, browser engine 104T, and universe browser engine 106T in some embodiments. A browser or web browser is a software application for accessing information about the World Wide Web, where web resources (e.g., individual web pages, images, videos, etc.) are identified by uniform resource locators, and the web browser allows these web resources to be retrieved from web servers and displayed on a user's device.

[0251] A browser engine (e.g., 104T) is a software component of a web browser that performs tasks such as converting HTML and / or other resources of a web page into an interactive visual representation on a user's device (e.g., a mixed reality headset). Various embodiments described herein leverage any publicly or commercially available browser and its respective engine to enable developers and users to manage and display Internet content and utilize web resources by using at least the aforementioned set of APIs, a mixed reality system, and software applications and libraries (e.g., the Universe Browser Engine).

[0252] The website may set orientation data for the web page or panel of the web page of interest (102T) and may further request the current orientation of the current web page (102T). In some embodiments, where the web page developer may want the web page of interest to be presented in a certain manner that includes certain orientation data, the website may further send such orientation data to the browser (102T).

[0253] In some embodiments, when a browser is launched or initialized to display content from the Internet, the browser initializes and renders and displays a web page of a website or content therein (104T). The browser may also send orientation data for the web page to be rendered by the browser engine (104T). For example, when a website is coded to incorporate the aforementioned set of APIs that are coded to both set orientation data in a client's browser and retrieve orientation data from the client browser, the browser may receive this orientation data from the website and set the orientation data for a web panel to display content from the website.

[0254] The browser may reset transformation and orientation data for web panels for the web page (104T). For example, the browser may reset 3D transformations (e.g., 3D translation transformations, 3D rotation transformations, and / or 3D scaling transformations) for the web page panels to default values ​​or states. The default values ​​or states, in one embodiment, may be stored in non-transitory memory (e.g., cache memory) accessible by the renderer (e.g., RenderWidget for WebKit and Chromium). The browser may relay requests for orientation from the website to its browser engine (104T). The browser engine acts as an intermediary between the website and the universe browser engine, which resides in the extended reality (XR) system.

[0255] In some embodiments where the orientation data for a web page panel is stored in the aforementioned non-transient memory, the browser or its browser engine may respond quickly to a request for orientation data from a website without additional delay from, for example, retrieving such data from the universe browser engine. This orientation data accessible by the browser engine may be synchronized when the website or browser changes the data or portions thereof. Additionally, or alternatively, this orientation data accessible by the browser engine may be initialized and reset by the browser. In some embodiments, this orientation data may be updated by a website request (see 102T) for orientation data. In some embodiments where the browser engine receives orientation data from a website, the browser engine may further pass the data to the browser.

[0256] The browser may also send orientation data to the universe browser engine (104T), either alone or together with the rendered content (e.g., a web page). The universe browser engine may create a 3D virtual volume (e.g., a prism) to display the rendered web page based, at least in part, on the orientation data from the browser engine. Additionally or alternatively, some embodiments may provide a user with the ability to manipulate a rendered web page (or content) within a virtual 3D space created by an extended reality (XR) system.

[0257] The user may therefore move and / or rotate the rendered web page within the virtual 3D space such that a new set of transformations needs to be applied to the rendered web page. The operating system of the extended reality system includes a runtime layer and a 3D engine, and applies a new set of one or more transformations to the rendered web page based, at least in part, on the orientation data received from the browser engine. A new set of one or more transformations may be applied to the resulting web page. Afterwards, the universe browser engine may also return the new orientation data to the browser engine (106T) and update the previous orientation data therein.

[0258] When the browser initializes, or when a user interacts with a rendered web page, thus changing the orientation data, the browser engine may receive orientation data, if available, from the universe browser engine (104T). The browser engine may therefore update the orientation data stored in non-transitory memory (e.g., cache) with the refreshed orientation data received from the universe browser engine (104T), e.g., stored in cache memory. The browser engine may also set orientation data for web page panels (104T). In some embodiments where a web page developer has set the orientation of the web page (e.g., the developer has set the position and / or rotation of a web page displaying an online chess game), the browser may also set the orientation data of the web page panels according to the orientation data received from the website.

[0259] In some embodiments in which a user is allowed to modify the position of a web page panel displayed within a prism, the universe browser engine may update the position data stored in non-transient memory (e.g., cache) based on user manipulation of the position of the web page panel, e.g., with the position data stored in cache memory. The universe browser engine may also set the position data for the web page panel (104T).

[0260] Additionally, the browser engine may send orientation data to the universe browser engine so that the universe browser engine may determine (e.g., by creating a new one or identifying one from an existing one) a virtual 3D space (e.g., a prism) for presenting the rendered web page via the extended reality system (104T). In some embodiments in which the user is provided with the ability to further manipulate the rendered web page, the browser engine may provide the orientation data of the rendered web page to the universe browser engine, which in turn may provide a software handle for the user to manipulate the rendered web page, perform corresponding transformations, and respond to user manipulations.

[0261] The universe browser engine may also include or function in conjunction with an operating system, one or more operating system services, one or more sets of runtime applications and libraries, one or more 3D engines, and a suite of extended reality system applications to provide an advanced graphical user interface and functionality for a user to manage and view web pages and other content and / or resources within a virtual 3D space (e.g., one or more prisms) at 106T. For example, one of the functions of the universe browser engine is to provide orientation data (and / or position data) of web pages (or other virtual content) to the browser engine (104T). The universe browser engine also synchronizes recent orientation data (and / or position data) with the browser engine (106B) so that recent orientation data can be pushed to (from) or pulled from (by) the browser engine.

[0262] 1U-1W illustrate further detailed block diagrams of exemplary browser engines and universe browser engines that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, FIG. 1U illustrates simplified pseudocode for setting 3D orientation data of web page panels in a virtual 3D space (e.g., a prism) in some embodiments. In these embodiments, the pseudocode is based on a similar set of code for Chromium, although code for other web browsers is also contemplated and can be applied to the techniques described herein with full equal effect.

[0263] For example, browser engine 100U executes Window.VirtualWorld.Orientation at 102U to initialize a process for setting the orientation data of a web page panel within the universe browser engine. In some embodiments, Window.VirtualWorld.Orientation may set and get orientation data for a web page panel based, in part or in whole, on orientation data and one or more transformations for 2D and 3D entities in 2D or 3D space, utilizing, for example, OpenGL Mathematics (GLM) (or any other math library) or the Simple Graphics Library for CSE20211 (Gfx) (e.g., pseudocode "glm::vec3old_pos(0.0f)", "glm::quat rotation(glm::vec3(0.0f))", "glm::vec3 scale(1.0f)", and "glm::mat4 transform=web_container_->getCurrentVolumeTransform()" or "gfx::Point3F rot=local_frame_client->GetVirtualWorldRotation()" in GLM, or "gfx::Point3F flat_rot{-M_PI_2,0.0f, In some other embodiments, where a web page developer is allowed to manipulate (e.g., set, modify, etc.) the Window.VirtualWorld.Orientation parameter to select only the rotation between 0 and 90 degrees, and not the position of the web page panel, the Window.VirtualWorld.Orientation parameter may receive input (e.g., the strings "flat" or "upright") that is defined as an enumerated type ("enum") within the browser engine. In some of these embodiments, the universe browser engine may convert the enumerated type input to a GLM type and set the data for the web page panel. For example, in one case where the input indicates a flat web page panel, the rotation of the web page panel may be set to {-M_PI_2,0.0f, 0.0f}.As another example, in one case where the input indicates an upright web page panel, the rotation of the web page panel may be set to {0,0,0}. It should be noted that pseudocode, programming languages, and names of various classes, variables, etc. are used herein for purposes of illustration and explanation, and that any other suitable programming languages, names, etc. that serve the same or substantially similar purposes are also contemplated and may be used to achieve such purposes. The following is an example code segment for the foregoing discussion, implemented within a portion of a header file to contain function declarations, but it should be noted that other similar or equivalent implementations are also contemplated and may therefore be used. [ka]

[0264] Browser Engine 100U also uses LocalFrameClientImpl::SetOrientation(106U) (or LocalFrameClient * ), RenderFrameImpl:SetOrientation(108U), and VirtualWorld::setOrientation, RenderFrameHost, as described above or as described herein. * , WebLocalFrameImpl * The construct RenderFrameImpl::SetOrientation may be used to call RenderWidget::SetOrientation and send the IPC message - FrameHostMsg_SetOrientation. The construct RenderFrameImpl::SetOrientation may be used to call RenderWidget::SetOrientation and send the IPC message. The construct LocalFrameClientImpl::SetOrientation is derived from the C++ LocalFrameClientImpl.cpp and sets and gets the orientation data for the client frame using a low-level graphics abstraction layer, for example Gfx or Gfx-rs, and the orientation data.

[0265] The construct RenderFrameImpl::SetOrientation may be derived from C++ RenderFrameImpl.cpp and configured to invoke a renderer (e.g., RenderWidget for Chromium-based browsers) and return a frame to the browser engine (e.g., Blink for Chromium-based browsers) based on the Gfx and orientation data. The browser engine 100U may also invoke a renderer or rendering process at 112U by executing at least RenderWidget::SetOrientation to update the orientation data (e.g., orientation values ​​stored in a cache memory) stored in the RenderWidget class for the web page panel. A separate content rendering process may also be executed or triggered to run by one or more processes (e.g., one or more rendering functions) to “paint” or render the content of the web page of interest according to the 3D orientation data. More specifically, the renderer process RenderWidget may use a glue interface (e.g., WebWidgetDelegate) to implement an abstract interface within the glue interface. This abstract interface may, for example, contain a 2D window or a 3D volume within the display space, receive input events, and render content within it. In some of these embodiments where auxiliary display items (e.g., selection boxes with up / down arrows indicating a list of options, tabs, pop-up windows, navigation commands, etc.) are to be rendered, browser engine 100U may also execute a separate rendering process (e.g., RenderView) for such display items.

[0266] The aforementioned header file may be extended with some example code as listed below, although other similar or equivalent code is also contemplated and may therefore be used to achieve similar purposes. [ka] [ka] [ka] [ka]

[0267] The following sections include some example code for rendering frames, although other similar or equivalent code is also contemplated and may therefore be used to achieve similar purposes. [ka]

[0268] The following includes some example code for obtaining the orientation, but it should be noted that other similar or equivalent implementations are also contemplated and may therefore be used. [ka]

[0269] The following includes some example code for setting the orientation, but note that other similar or equivalent implementations are also contemplated and may therefore be used. [ka]

[0270] In some embodiments where a Chromium or Chromium-based browser is used, browser engine 100U renders web pages using its rendering system ("views"), and the user interface is structured as a tree of components, called "views," that are responsible for rendering, layout, and event handling. Each view in the tree of components has its own boundary and represents a different component of the user interface, with widgets (true 2D windows or 3D volumes) located at the root of such a tree.

[0271] Some embodiments use the orientation data to create a virtual 3D space (or 2D window) and pass the virtual 3D space (or 2D window) to the RootView, which then propagates events up the tree. Some display items may be rendered using operating system controls hosted within special kinds of views that know how to display and size native widgets. These display items include, for example, buttons, tables, radio buttons, check boxes, text fields, other controls, etc. Further details about such trees of components are described below with reference to Figures 2G-2I.

[0272] The browser engine may further implement WebContentsImpl::OnSetOrientation (114U), CefBrowserHostImpl::SetOrientation (116U) using "delegate_->SetOrientation(orientation)" and "delegate_->SetOrientation(orientation)" based at least in part on the orientation data. The browser engine also implements CefUniverseBrowserEnginePrivateHandler::OnSetPageOrientation (118U) using a universe browser engine specific interface that enables custom interaction between the Chromium Embedded Framework (CEF) and the universe browser engine. A CEF-based application may contain components such as an entry point for initializing CEF and starting the CEF message loop, a CEFApp-derived class for handling process-specific callbacks, a CEFClient-derived class for handling browser-instance specific callbacks (e.g., callbacks for browser lifespan, context menus, dialogs, display notifications, drag events, attention events, keyboard events, etc.), and one or more CEFBrowser instances created by CefBrowserHost::CreateBrowser().

[0273] Browser engine 100U may further implement the constructs CefBrowserHostImpl::SetOrientation(116U) and CefBrowserHostImpl::SetOrientation (e.g., within a browser host implementation) to determine the handler (e.g., "CefRefPtr <cefuniversebrowserengineprivatehandler>handler=client_->GetUniverseBrowserEnginePrivateHandler() and handler->OnSetPageOrientation(this, orientation)). Note that the orientation may or may not be a vector3 type. For example, the orientation may be an enumeration type ("enum"), such as "flat", "upright", etc. In some embodiments where the orientation is a vector3 type, handler->OnSetPageOrientation(this, orientation.x(), orientation.y(), orientation.z()) may be used. The browser engine may also execute the public function CefUniverseBrowserEnginePrivateHandler::OnSetPageOrientation to set the 3D orientation of the web page panel.

[0274] The universe browser engine 126U may execute ClientHandler::OnSetPageOrientation(120U), Browser::OnSetPageOrientation(122U), and BrowserWindow::OnSetPageOrientation and BrowserWindow::OnSetPageOrientation(124U) using the 3D orientation data received from the browser engine 100U. The ClientHandler includes event handlers that fire within the browser without requiring a callback to the server to provide a means for defining rules (e.g., declarative conditions and actions that can be evaluated within the browser and reduce round-trip delay or latency). The ClientHandler may also delegate setting the web page orientation within a virtual 3D space (e.g., a prism) using GLM (OpenGL Mathematics).

[0275] 1V illustrates communication between some of the aforementioned blocks in FIG. 1V and illustrates how browser engine 100V obtains 3D orientation data in a virtual 3D space (e.g., a prism) from universe browser engine 126C. Downward arrows indicate a sequence of execution and / or a path of execution results, and upward arrows indicate a return of a value.

[0276] In these embodiments, browser engine 100V may execute Window.VirtualWorld.Orientation at 102V to initialize a process for setting 3D orientation data for a web page panel within the universe browser engine, similar to that described with reference to Figure 1U above. Note that Figure 1V illustrates an example of setting and determining one type of orientation (e.g., rotational data rather than positional data) rather than another type of orientation data, and that setting and determining this other type of orientation data may be performed in the same or substantially similar manner using corresponding code.

[0277] Unlike browser engine 100U illustrated in FIG. 1U, browser engine 100C in FIG. 1V may further implement LocalFrameClientImpl::GetOrientation(106V) and RenderFrameImpl:GetOrientation(108V). Similar to the description of FIG. 1U, construct RenderFrameImpl:GetOrientation(108V) may be used to call RenderWidget::GetOrientation(112V) and return a value to Window.VirtualWorld::orientation to return orientation data. Construct LocalFrameClientImpl::GetOrientation, like LocalFrameClientImpl::GetOrientation in FIG. 1U, is derived from C++ LocalFrameClientImpl.cpp and may use a low-level graphics abstraction layer, such as Gfx or Gfx-rs, and orientation data to set and get the orientation for the client frame.

[0278] Like RenderWidget::SetOrientation in FIG. 1U, browser engine 100V may also query orientation data by executing RenderWidget::GetOrientation at 106V and return the orientation data (e.g., the orientation data of a web page) to RenderFrameImpl::GetOrientation (104V), which in turn returns the orientation data to the web page, triggering a Window.VirtualWorld.Orientation call. In some embodiments, renderer process RenderWidget may use a glue interface (e.g., WebWidgetDelegate) to implement an abstract interface within the glue interface. This abstract interface may, for example, contain a 2D window or 3D volume within the display space, receive input events, and render content within it. In some of these embodiments where auxiliary display items (e.g., a selection box with up / down arrows indicating a list of options, a tab, a pop-up window, a navigation command, etc.) are to be rendered, browser engine 100V may also execute a separate rendering process (e.g., RenderView) for such display items.

[0279] FIG. 1W illustrates a simplified block diagram with pseudocode for synchronizing 3D orientation data between a browser engine and a universe browser engine in some embodiments. For example, a web page in virtual 3D space may be modified (e.g., by a user who has moved and rotated the web page in virtual 3D space). The block diagram in FIG. 1W illustrates a manner in which the web page's current 3D orientation data is synchronized between browser engine 100W and universe browser engine 126W. FIG. 1W may also illustrate an embodiment in which universe browser engine 126W sends orientation data to browser engine 100W in response to browser initialization.

[0280] 1W, universe browser engine 126W may reset the transformations of the web page's volume, rendered browser, and / or prism (122W) upon initialization (120W) or upon receiving modified 3D orientation data for the web page (not shown). The 3D orientation data is then passed to browser engine 100W, which executes CefBrowserView::UpdateCachedOrientation (102W). Browser engine 100W further executes RenderWidgetHostImpl::UpdateCachedOrientation to update the orientation data with the most recent orientation data from universe browser engine 126W.

[0281] The browser engine 100W also sends IPC (Inter-Process Communication) messages (eg, ViewMsg_UpdateCachedOrientation, ViewMsg_UpdateCachedOrientation, etc.) and executes RenderWidget::OnUPdateCachedOrientation (108W) and RenderWidget::SetOrientation (110W).

[0282] In some embodiments, universe browser engine 126W may store 3D orientation data in a non-transitory computer-readable medium (e.g., cache memory) accessible by browser engine 100W. Universe browser engine 126W and browser engine 100W are therefore synchronized each time the orientation of a web page panel in virtual 3D space is modified. Each time a web page begins loading, the transformation of the web page panel will be reset to default, and the stored value in RenderWidget will be updated as well, to initialize and reset the orientation variable in RenderWidget.

[0283] 1X illustrates an example of determining orientation data for a web page panel using a SET request in one or more embodiments, where a web page of interest (102X) may be coded to issue a request to a browser or browser engine (104X) to set orientation data for the web page panel in response to loading the web page by a browser at 152X. In response to receiving the orientation data, the browser or browser engine (104X) may update orientation data previously stored in a non-transitory computer-readable storage medium 108X (e.g., a cache) at 154X.

[0284] The browser or browser engine (104X) may further pass the received orientation data along with the SET request to the universe browser engine (106X) at 156X. To fulfill the SET request, the universe browser engine 106X may set the orientation data of the web page panel by using the orientation data received along with the SET request and transmitted to 158X at 110X.

[0285] 1Y-1AA illustrate further detailed block diagrams of exemplary browser engines and universe browser engines that may be used in managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. More specifically, FIG. 1Y illustrates an exemplary flow for 3D orientation updates of web page panels in a virtual 3D space (e.g., Prism) in some embodiments. In these embodiments, the exemplary flow is based on a similar set of code for Chromium, but code for other web browsers can also be considered and applied to the techniques described herein with full equal effect. In the BrowserWindow, ContentView::OnPageOrientationChanged will be called after the web content plane has been updated with the input orientation value. Additionally, through CEF, an event will be dispatched to notify that the page orientation change has been completed. In some embodiments in which JavaScript is used, the universe browser engine 126Y may listen for page orientation events and call a callback.

[0286] FIG. 1Y illustrates an exemplary flow for a web page translation update from browser engine 100Y to universe browser engine 126Y. The exemplary flow is similar to that illustrated in FIG. 1C described above. More specifically, browser engine 100Y may invoke or execute a JavaScript function call "Window.VirtualWorld.Orientation" to initialize the process flow illustrated in FIG. 1Y. On the browser engine 100Y side,

[0287] Like Window.VirtualWorld.3DPosition and Window.VirtualWorld.3DRotation in FIG. 1C, processes such as “Window.VirtualWorld.Orientation” and / or “VirtualWorld” in FIG. 1Y may utilize, for example, OpenGL Mathematics (GLM) (or any other mathematical library) to set and retrieve position data for web page panels based, in part or in whole, on 3D position data, 3D rotation data, scale, and one or more transformations for 2D and 3D entities in 2D or 3D space (e.g., see the pseudocode “glm::vec3old”). _pos(0.0f), glm::quatrotation(glm::vec3(0.0f)), glm::vec3scale(1.0f), and glm::mat4transform=web_container_->getCurrentVolumeTransform()) and rotation data (e.g., by the pseudocode “glm::vec3position(0.0f)”, “glm::quatold_rot(glm::vec3(0.0f))”, “glm::vec3scale(1.0f), and glm::mat4transform=web_container_->getCurrentVolumeTransform()”).

[0288] It should be noted that the pseudocode, programming languages, and names of various classes, variables, etc. are used herein for purposes of illustration and explanation, and that any other suitable programming languages, names, etc. that serve the same or substantially similar purposes are also contemplated and may be used to achieve such purposes. Additionally, it should be further noted that while Figure 1Y illustrates setting only the orientation (e.g., rotation) of a web page panel, other web page panel transformations, such as position re-positioning, scaling, etc., may also be incorporated using similar techniques.

[0289] The browser engine 100C may further execute VirtualWorld::setOrientation in FIG. 1Y, similar to VirtualWorld::Set3DPosition in FIG. 1C, to set and retrieve position, rotation, and / or scale data (not shown) of the web page panel.

[0290] Browser engine 126Y may further implement LocalFrameClientImpl::SetPageOrientation, RenderFrameImpl::SetPageOrientation, and RenderWidget::SetRotation, which are illustrated in Figure 1Y. Browser engine 100Y may send messages (e.g., inter-process communication (IPC) messages) by, for example, implementing or calling SendIPCMessage::FrameHostMsg_SetPageOrientation, which is illustrated in Figure 1Y. The construct RenderFrameImpl:SetPageOrientation may be used to call the function RenderWidget::SetOrientation and send IPC messages via SendIPCMessage::FrameHostMsg_SetPageOrientation. The construct LocalFrameClientImpl::SetPageOrientation is derived from the C++ LocalFrameClientImpl.cpp and sets and gets the position and / or rotation for the client frame using a low-level graphics abstraction layer, e.g., Gfx or Gfx-rs, and the position and rotation data. LocalFrameClientImpl::SetPageOrientation may be implemented to get and set the orientation data of a web page panel using the following pseudocode: [ka]

[0291] The construct RenderFrameImpl::SetPageOrientation is derived from C++ RenderFrameImpl.cpp and may be configured to invoke a renderer (e.g., RenderWidget for Chromium-based browsers) and position and / or rotation data and send the frame back to the browser engine (e.g., Blink for Chromium-based browsers).

[0292] The browser engine 100Y may also invoke a renderer by executing RenderWidget::SetRotation to “paint” or render the web page of interest according to 3D position, rotation, and / or scale data (e.g., position and rotation data set by a web page panel developer). More specifically, the renderer process RenderWidget may use a glue interface (e.g., WebWidgetDelegate) to implement an abstract interface within the glue interface. This abstract interface may, for example, contain a 2D window or a 3D volume within the display space, receive input events, and render content within it. In some of these embodiments where auxiliary display items (e.g., a selection box with up / down arrows showing a list of options, a tab, a pop-up window, a navigation command, etc.) are to be rendered, the browser engine 100Y may also execute a separate rendering process (e.g., RenderView) for such display items.

[0293] In some embodiments where a Chromium or Chromium-based browser is used, the browser engine 100Y renders web pages using its rendering system (“views”), and the user interface is structured as a tree of components, called “views,” that are responsible for rendering, layout, and event handling. Each view in the tree of components has its own boundary and represents a different component of the user interface, with widgets (native 2D windows or 3D volumes) located at the root of such a tree. Some embodiments use position and location data to create a virtual 3D space (or 2D window) and pass the virtual 3D space (or 2D window) to the RootView, which then propagates events up the tree. Some display items may be rendered using operating system controls hosted in special types of views that know how to display and size native widgets. These display items include, for example, buttons, tables, radio buttons, check boxes, text fields, other controls, etc. Further details about such trees of components are described below with reference to Figures 2G-2I.

[0294] The browser engine also executes WebContentsImpl::OnSetPageOrientation and CefBrowserHostImpl::SetPageOrientation based at least in part on the position, rotation, and / or scale data.

[0295] The browser engine 100Y also implements CefPrivateHandler::OnSetPageOrientation and communicates with a universe browser engine (126Y)-specific interface that enables custom interaction between the Chromium Embedded Framework (CEF) and the universe browser engine. A CEF-based application may include components such as an entry point for initializing CEF and starting the CEF message loop, a CEFApp-derived class for handling process-specific callbacks, a CEFClient-derived class for handling browser-instance-specific callbacks (e.g., callbacks for browser lifespan, context menus, dialogs, display notifications, drag events, attention events, keyboard events, etc.), and one or more CEFBrowser instances created by CefBrowserHost::CreateBrowser().

[0296] 1Z illustrates an example flow for a web page transformation update from universe browser engine 126Y to browser engine 100Y. When universe browser engine 126Y determines that a web page transformation has occurred, browser engine 100Y may execute "CefBrowserView::OnPageOrientationChanged" using the following example code and the declaration "virtual void OnMLPageOrientationChanged()=0;" in the header file. When the browser engine recognizes the web page transformation, browser engine 100Y may also execute or call "CefBrowserViewImpl::OnPageOrientationChanged" using the declaration "void OnMLPageOrientationChanged() override; void OnMLPageOrientationChanged() override;" in the corresponding header file. [ka]

[0297] The browser engine 100Y may then execute "RenderBrowserViewImpl::OnPageOrientationChanged" and "RenderWidgetHostImpl::OnPageOrientationChanged" when it determines that a web page transformation has occurred, using the following example code in conjunction with "virtual void OnMLPageOrientationChanged()=0;" in the corresponding header file for host implementation: [ka]

[0298] The browser engine 100Y may then execute or call "SendMessage::ViewMsg_PageOrientationChanged" to send a message (eg, an inter-process message or IPC) by using the following example code: [ka]

[0299] Once it is determined that the web page orientation has changed, the browser engine will call the following example code and the corresponding header file: Alternatively, you can execute "RenderWidget::OnPageOrientationChanged" which has "RenderWidget::OnPageOrientationChanged" as its parameter. [ka]

[0300] Browser engine 100Y may further execute "Document::DispatchEvent" to dispatch events for web page transformation. JavaScript "document.addEventListener("pageorientation,callback)" initiates a callback indicating that the change in position, rotation, and / or scale (web page transformation) is completed on the browser engine 100Y side.

[0301] FIG. 1AA illustrates, in an exemplary flow, a portion of the universe browser engine (126Y) for updating the 3D orientation of a web page panel within a virtual 3D space (e.g., a prism) in some embodiments. In response to receiving a web page panel transformation (e.g., a change in the position, orientation, or scale of a web page panel) from the browser engine (e.g., 100Y), the universe browser engine 126Y may further execute the construct ClientHandler::OnSetPageOrientation (102AA) to determine the handler (e.g., "CefRefPtr <cefuniversebrowserengineprivatehandler>handler=client_->GetUniverseBrowserEnginePrivateHandler() and handler->OnSetVirtualWebPagePosition(this, position.x(), position.y(), position.z()), handler->OnSetVirtualWebPageRotation(this, rotation.x(), rotation.y(), rotation.z()).

[0302] The universe browser engine 126Y may further execute Browser::OnSetPageOrientation (104AA) and BrowserWindow::OnSetPageOrientation using the 3D position, rotation, and / or scale data received from the browser engine 100Y. The ClientHandler includes event handlers that fire within the browser without requiring a callback to the server to provide a means for defining rules (e.g., declarative conditions and actions that can be evaluated within the browser and reduce round-trip delays or latency). The ClientHandler may also delegate setting the web page position and rotation within a virtual 3D space (e.g., a prism) using GLM (OpenGL Mathematics).

[0303] 1AB-1AC illustrate an exemplary high-level flow diagram for event dispatch flow in some embodiments. More specifically, the event dispatch flow illustrated in FIGS. 1AB-1AC proceeds through SynchronizeVisualProperties. Additionally, BrowserWindowBase::RenderLoop periodically or repeatedly checks to determine whether the transformation (e.g., position, rotation, and / or scale, etc.) of the web page plane has changed. If BrowserWindowBase::RenderLoop determines that the transformation of the web page plane has changed, a changed transformation event will be propagated to the JavaScript document.

[0304] In some embodiments, the following example code implements the class definition "virtual void UpdateLoop(float / * delta * / );" may be used for BrowserWindowBase::UpdateLoop (102AB) for the Universe Browser Engine (126AB). [ka]

[0305] In some embodiments, the example code below also implements the class definition "virtual void UpdateLoop(float / * delta * / );" may be executed or called for BrowserWindowBase::UpdateLoop (102AB) for the Universe Browser Engine (126AB), which polls for web node transformation changes and keeps the stage and isolated volume positions updated. [ka]

[0306] In some embodiments, the following example code may be used for WebContentsView::CheckWEbContentTransformChanged (104AB) for the Universe Browser Engine (126AB), along with the class definition "void CheckIfWebContentTransformChanged();" in the corresponding header file: [ka]

[0307] The function CEFWindow (106AB) may correspond to "virtual void WebContentTransformChanged()=0;" in the corresponding header file. Additionally, the browser engine 100AB may further execute the function RenderWidgetHostViewAura::OnWebContentTransformChanged() (114B) when it is determined that a web content transform (e.g., position, rotation, and / or scale) has changed, together with "void OnWebContentTransformChanged() override;" in the corresponding header file, using the following example code: [ka]

[0308] Additionally, the browser engine 100AB may further execute a Widget function (108AB) when it is determined that a web content transformation (e.g., position, rotation, and / or scale) has changed, using the following example code, along with "void WebContentTransformChangedWebContent();" in the corresponding header file: [ka]

[0309] Additionally, the browser engine 100AB may further execute the function DesktopNativeWidgetAura::WebContentTransformChanged() (110AB) when it is determined that the web content transform (e.g., position, rotation, and / or scale) has changed, using the following example code, along with "void WebContentTransformChanged() override;" in the corresponding header file: [ka]

[0310] Additionally, the browser engine 100AB may further execute the function DesktopWindowTreeHostVirtualWorld::WebContentTransformChanged() (112AB) when it is determined that a web content transform (e.g., position, rotation, and / or scale) has changed, using the following example code, along with "void WebContentTransformChanged() override;" in the corresponding header file: [ka]

[0311] The browser engine (100AB) may, in some embodiments, execute the RenderWidget function for widgets that support a dominant frame. In some other embodiments, the browser engine (100AB) may invoke the RenderView function for widgets that support a remote dominant frame. The following example code may be used, for example, for the function RenderWidget in 116AB and 118AB. Note that "SetIsFullscreen(params.is_fullscreen_granted);" in the example code below enables full-screen mode without changing the viewport size. [ka] [ka] [ka] [ka]

[0312] Additionally, the browser engine 100AB may further execute the function WebLocalFrameImpl(120AB) when it is determined that the web content transformation (e.g., position, rotation, and / or scale) has changed, using the following example code, along with "void SendPageOrientationChangeEvent() override;" in the corresponding header file: [ka]

[0313] Browser engine 100AB may execute a "document" function (122AB) to dispatch an event and notify that the page orientation change has been completed. Additionally, or alternatively, browser engine 100AB may execute a JavaScript function "Document.addEventListener" to listen for page orientation change events (124AB).

[0314] FIG. 2A illustrates a high-level block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system, in one or more embodiments. In these embodiments, a browser engine may receive input at 202A for three-dimensional (3D) transformations (e.g., changes in position, rotation, or scale) for a web page. In some embodiments, the input may include input by a user of a mixed reality system requesting a presentation of a web page of interest, or by a web page developer who prefers the web page to be presented in a certain manner (e.g., position, orientation / rotation, or scale) and codes accordingly. In some other embodiments, the input may include a request for position and rotation data. For example, a request for position and rotation data (or even, in some embodiments, scale data) may be issued by the website of the web page of interest in some embodiments. For example, a web page developer may incorporate application programming interfaces (e.g., the Window.VirtualWorld.3DPosition API and Window.VirtualWorld.3DRotation API described above) into the code of the web page of interest.

[0315] As described immediately above, in some embodiments, input may come from a developer who, for example, prefers to represent a web page at a certain position and / or with a certain rotation when presented within a virtual 3D space. In some of these or other embodiments, input may come from a user's manipulation of a web page, a previously visited identical web page, or other web pages from the same website, or optionally from other websites, within the virtual 3D space created by the mixed reality system. For example, a web page may be opened and displayed for the user within the virtual 3D space, and the user may adjust the position and / or rotation of the web page within the virtual 3D space. In some of these embodiments, similar techniques may also be applied to manipulate a web browser for the web page or a prism within which the web page is rendered. The user may freely position and rotate the web page representation within the virtual 3D space, the only constraints on such positioning and rotation including the bounds of the prism. In some embodiments, such position and rotation may also optionally be constrained by collision and / or overlap dynamics between the web page representation and one or more other physical and / or virtual objects in the virtual 3D space, such that the web page representation and the virtual object appear or are perceived as real.

[0316] A browser engine (e.g., Blink, Gecko, WebKit, Trident, etc.) is not a stand-alone computer program. Rather, a browser engine is a software component of a web browser that converts web page resources (e.g., HTML documents, etc.) into interactive visual representations. Other commonly used terms for a browser engine include a layout engine, a rendering engine, etc., for example, by calculating graphical coordinates for the visual representation based on rules in a Cascading Style Sheet (CSS).

[0317] As its name suggests, the browser engine performs layout and rendering for web pages, enforces security policies, and implements the DOM (Document Object Model) data structure that is exposed to page scripts. The browser engine may also handle hyperlinks and web forms. In some embodiments, the browser engine utilizes a separate, dedicated script engine for the execution of JavaScript® code. In addition to performing at least the aforementioned functions for a web browser, the browser engine may also perform the same or substantially similar functions for other software programs, such as email clients, other frameworks for software applications (e.g., Google's Electron Framework), etc.

[0318] The browser engine may determine the position and rotation data and respond to the 3D position and / or rotation request at 204A. In some embodiments, the browser engine may store the most recent position and rotation data in a non-transitory computer-readable medium (e.g., cache memory) accessible by the browser engine. Such position and rotation data may be transmitted by the universe browser engine of the mixed reality system each time the position and / or rotation data is modified (e.g., by a user further moving, rotating, or translating a web page displayed in virtual 3D space).

[0319] When a web browser initializes to render and display a web page, its browser engine may respond to such position and / or rotation requests with position and rotation data that is stored, for example, in a cache memory accessible by the browser engine (e.g., in the RenderWidget class in Chromium-based browsers). Once the position and / or rotation data is modified, the browser engine may refresh its stored position and / or rotation data with the updated position and / or rotation data or update with position and / or rotation data from the universe browser engine. Using the most recent position and rotation data, the browser engine may render the content of the web page (e.g., by calling the RenderWidget class) and, for example, calculate graphical coordinates for the rendered content using the most recent position and rotation data and rules that govern the layout of the rendered content (e.g., cascading style sheets).

[0320] The universe browser engine of the mixed reality system may invoke a browser engine at 206A, which in turn invokes one or more mixed reality software modules and / or hardware mechanisms to display the web page within the virtual 3D space. For example, depending on the rotation and positioning of the web page within the virtual 3D space (e.g., within a prism), the virtual 3D representation of the web page may span multiple depth planes or multiple focal planes. The mixed reality system may, for example, utilize its switchable focal plane or variable focus mechanism to render the virtual 3D representation of the web page based on the 3D position and rotation data. Further details about mixed reality systems are described in the U.S. patent applications referenced in the first paragraph of this application. In some embodiments, this invocation of the mixed reality mechanisms may occur via a browser (e.g., Chromium) and / or a browser engine (e.g., Blink for Chromium). For example, the browser (or its browser engine) may apply the position and / or rotation data (or, optionally, scale data) to the web page panel, invoke the mixed reality mechanisms, and display the virtual 3D content.

[0321] 2B illustrates a further detailed block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. In response to an instruction to open a web page (e.g., by a user clicking a link, by a user entering search criteria into a search field, by a user entering a URL for a website, etc.), a web browser or web page panel is launched at 202B. The universe browser engine may send the position and rotation data to the browser engine of the web browser so that the position and rotation data may be stored in a non-transitory computer-readable medium (e.g., a cache) or the position and rotation data in the non-transitory computer-readable medium may be refreshed. The browser engine may then cache or store the received position and rotation data, for example, in a cache memory accessible by the browser engine. In some embodiments where the position and rotation data is stored in a non-transitory computer-readable memory (e.g., a cache) accessible by the browser engine, the stored position and rotation data stored in the non-transitory computer-readable memory may be initialized with the position and rotation data from the universe browser engine.

[0322] The web page may optionally send position and rotation requests to the browser engine at 204B. For example, a web page developer may incorporate a set of application programming interfaces (APIs) (e.g., the Window.VirtualWorld.3DPosition API and the Window.VirtualWorld.3DRotation API described above) into the web page's code so that the web page sends position and / or rotation requests to the browser engine upon initialization or shortly thereafter. In some embodiments in which a web page developer may desire to present the web page at a certain position and / or rotation, the web page may also set the position and rotation data by utilizing the set of APIs to communicate the position and rotation data to the browser engine. In some of these embodiments, the web page may obtain the position and / or rotation data of the web page pane from the browser engine in response to user input (e.g., a user clicking a link, a user typing search criteria into a search field, a user typing a URL for a website, etc.).

[0323] In response to an optional position and / or rotation request from a web page, the browser engine may respond to the request at 206B with recent position and rotation data stored in a non-transitory computer-readable medium accessible by the browser engine, for example, without additional delay from obtaining such position and rotation data from the mixed reality system's universe browser engine. If recent position and rotation data is unavailable (e.g., for a mixed reality system being initialized for the first time or for other reasons), the browser engine may respond to such a request from a web page with default position and rotation data.

[0324] The browser engine may also transmit the position and rotation data to the universe browser engine at 208B. For example, some embodiments may provide a user with the ability to position and / or rotate a representation of a web page within a virtual 3D space. In some embodiments in which a GET request is received, the browser engine returns the position and rotation data, stored in a non-transitory computer-readable medium (e.g., a cache), to the web page. The universe browser engine receives the position and rotation data from the browser engine of the web browser and invokes other modules of the mixed reality system (e.g., the runtime layer, the operating system services layer, one or more 3D engines, or one or more software applications, etc.) and other hardware and software components thereof to enable the user to freely position and / or rotate the representation of the web page within the virtual 3D space (e.g., a prism). In some embodiments, 206B is executed when the web page obtains the position and / or rotation data (“GET”), and 208B is executed when the web page determines the position and / or rotation data (“SET”).

[0325] The universe browser engine, in some embodiments, updates the position and rotation data for the web page in response to receiving the position and rotation data from the browser engine at 210B. In some other embodiments in which a SET request is received for new position and / or rotation data, the new position and / or rotation data may be stored in a non-transitory computer-readable medium (e.g., a cache) for the browser engine and further passed to the universe browser engine. The universe browser engine further determines whether the position and / or rotation data has been updated. For example, any positioning and rotation of the web page representation by the user may modify the position and / or rotation data. If the universe browser engine determines that the position and / or rotation data has changed, the universe browser engine may update the position and / or rotation data and send the latest version of the position and / or rotation data to the browser engine at 212B. The browser engine may then, in turn, store the position and / or rotation data stored in a non-transitory computer-readable medium (e.g., a cache) along with the latest version of the position and / or rotation data.

[0326] In response to receiving the latest version of the position and / or rotation data, the browser engine, in some embodiments, may further calculate the graphical coordinates of the web page content and render the web page content. The web page panels may then be translated, rotated, scaled, or otherwise transformed based, at least in part, on the latest version of the position and / or rotation data. The browser engine, in these embodiments, may then pass the rendered content to the universe browser engine at 214B, which in turn invokes various software and hardware modules of the mixed reality system to generate a representation of the rendered content in virtual 3D space and display the representation of the rendered content to the user within the translated, rotated, scaled, or otherwise transformed web page panels in virtual 3D space.

[0327] In some other embodiments, the browser engine renders the content of the web page using the position and rotation data stored in the browser engine's cache and transmits the rendered content and the position and rotation data to the universe browser engine. The universe browser engine treats the rendered content as a software object and allows the user to position and / or rotate the rendered content of the web page. Once the user determines the position and / or rotation of the web page representation, the universe browser engine updates the position and rotation data, invokes, for example, a transform and render module, generates an updated representation of the web page for the software object in the virtual 3D space, and presents the updated representation of the web page to the user. In some of these embodiments, the mixed reality system may display the web page in the virtual 3D space in near real-time fashion while the user changes the position and / or rotation of the web page in the virtual 3D space, with no perceptible delay between the user's manipulation and the display of the web page (214B).

[0328] FIG. 2C illustrates a high-level block diagram for identifying or creating a Universe Prism, which can be utilized to manage and display web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. As described above, a prism comprises a three-dimensional space with a boundary that can be translated, rotated, and / or transformed with one or more three-dimensional matrices. The three-dimensional space defining a prism may also be reduced in dimension. For example, a prism may comprise a two-dimensional (or even one-dimensional) area reduced from the three-dimensional space. In these embodiments illustrated in FIG. 2C, a prism may be identified (if existing) or created in 202C, in part or in whole, based on one or more behavior types of the user, the prism, or the content to be rendered within the prism.

[0329] In some embodiments, these one or more behavior types include, for example, world-fixed, billboarding, edge billboarding, following head-fixed, following based on external sensors, or fading. A billboarding-type behavior includes a rule, constraint, requirement, or preference that is useful for displaying an object in a manner that billboards toward (e.g., facing) the user during a user, head, and / or eye movement sequence, improving the visibility of the virtual object and reducing the user's effort to manage the virtual object and / or prism in the movement sequence. For example, a planar virtual object (e.g., a web page displaying some information) may be positioned and oriented by automatically translating, rotating, and / or translating the planar virtual object within a prism relative to the physical environment so that the planar virtual object faces the user during the user's movement sequence. A prism may therefore be created or identified at 202C to accommodate this type of behavior.

[0330] Edge billboard display type behavior begins to translate, rotate, or translate toward the user of the mixed reality system if or when the user would view the virtual content at some predefined oblique angle, but the virtual content remains stationary before such translation, rotation, or translation is triggered by the oblique viewing angle. World fixed type behavior fixes the prism without adapting to any dynamic aspects of the user or virtual object.

[0331] In head-fixed following type behavior, one or more virtual objects (e.g., web page panels, controls for a web browser, etc.) may be translated, rotated, and / or transformed in such a way that the one or more virtual objects appear fixed relative to the user's head and therefore move with the user's head movements. Similarly, body-fixed following type behavior, eye-fixed following type behavior, and following based on one or more external sensors may also exist. Prisms for displaying such one or more virtual objects may be identified (if existing) or newly created (if non-existing) to accommodate these types of behavior.

[0332] In fading-type behavior, the virtual object may begin to break down or disappear (e.g., the mixed reality system stops rendering the virtual object) at a specified distance. In some embodiments, the virtual object fades out smoothly, while in some other embodiments, the virtual object disappears abruptly. Another type of behavior includes the behavior of a prism relative to another object, such as the anchor location of another prism in virtual space. For example, a prism may be generated or placed on top of the surface of a physical object (e.g., a wall, desk, floor, ceiling, etc.).

[0333] Relationship-type behavior includes the behavior of a prism relative to another prism. For example, a second prism may be created or installed to maintain a child-parent relationship with a first prism, where the first prism is the parent of the second prism. These two prisms may be translated, rotated, and / or transformed using the same transform. An example of such a parent-child prism may include a parent prism for a web page and a child prism for a web browser control, such that when the parent prism is translated, rotated, or transformed, the child prism also translates, rotates, or transforms along with the parent prism, although these two prisms may also be translated, rotated, and / or transformed independently, as described above.

[0334] Another type of behavior includes collision and overlap behavior. This type of behavior governs how two prisms behave, such as whether two prisms can overlap one another, depending on the occurrence of a collision. For example, this type may allow or prevent collisions between two prisms and, if allowed, determine the amount of overlap allowed between these two prisms. In some embodiments where collisions and overlaps may be prevented by this type of behavior, a user may override such rules by continuing to translate, rotate, or translate one or more prisms (e.g., by continuing to manipulate the prisms, by ignoring messages about the collision / overlap, etc.), and the mixed reality system may therefore consider such rule overrides and allow the prisms to collide or overlap one another. Prisms may therefore be identified or generated based, at least in part, on the aforementioned types of behavior.

[0335] The web browser and its browser engine may be started at 204C. The web browser may be started in response to a user action or interaction with the universe browser engine, such as opening a new application, typing search criteria, typing or clicking a URL (using hand gestures, using a controller, using voice commands, etc.), interacting with a launcher, etc., and the web browser and web engine may be started with a service at 204C.

[0336] The universe browser engine operates using Prism and, optionally, a distributed scene graph for 2D and / or 3D content, and the browser engine (and optionally other services, such as a runtime layer of a mixed reality system, one or more 3D engines, libraries, etc.) to render the content of a web page. The distributed scene graph is a data structure that can be used by vector-based graphics, editing applications, gaming software, etc., to arrange a logical representation (and optionally a spatial representation) of a graphical scene using, for example, relative placement algorithms and / or arbitrary transformations. The scene graph may define how content is arranged and transformed relative to each other within that structure.

[0337] Additionally, more than one software application (e.g., a web browser, an email client, etc.) may be rendered in the virtual 3D space, also managed by the universe browser engine. One or more prisms may be allocated per software application, such that these one or more prisms for a software application may constitute a subtree of the distributed scene graph. Furthermore, prisms may be created through user action or interaction or by an application. For example, a software application may request the creation of a prism, but the universe browser engine may deny such a request. The universe browser engine, rather than any other specific software component, is responsible for the management and creation of prisms. A unique identification of the web browser (or its browser engine) may be determined at 206C, for example, by a package manager service. Similarly, the universe browser engine may also determine a unique identification of the web browser (or its browser engine).

[0338] The web browser (or its browser engine), for which a prism is to be created or installed, may register a listener with the universe browser engine at 208C. The listener includes an interface object that receives messages or communications. For example, the listener may receive IPC (inter-process communication) messages or, in the case of an Android® system, from the Android Binder interface. The listener may be used to notify the web browser (or its browser engine) of the creation and destruction of prisms, in part or in whole, based on user movement or interaction with the prism.

[0339] The universe browser engine may then associate the prism with the listener at 210C and assign the prism to the web browser (or its browser engine) by using the unique identification at 212C. The universe browser engine may then place the prism in the 3D virtual or 3D viewable space of the mixed reality system by using the scene graph at 214C. In some embodiments, the universe browser engine may place the prism in 3D space based in part on position and rotation data of the web page (or web page panel) to be rendered in the 3D virtual or viewable space.

[0340] FIG. 2D illustrates a high-level block diagram for opening a web page and placing it within the Universe Browser prism, according to some embodiments. In these embodiments, the mixed reality system may receive a first user input at 202D indicating a user's interest in content from the internet. The first user input may include, for example, a user action to extract content from an extractable node on the internet (e.g., a website that provides extractable content within its web page). The first user input may include a user action or interaction (e.g., by clicking the controller, by hand gestures, via voice commands, etc.) that initiates a software application (e.g., a web browser, an email client, etc.). The first user input may also include a user triggering a dispatch from another software application, where the dispatch must be handled by a different application. For example, a user may click a link in a social media page (using the controller, via hand gestures, or using voice commands), and the link will be handled by the web browser.

[0341] In response to the first user input, a software application (e.g., a web browser) may be launched at 204D and generate the content of interest. For example, the universe browser engine may launch the web browser (and its browser engine) in response to the user typing search criteria into a search field generated by the mixed reality system and also provided in the virtual 3D space. A prism may be determined at 206D to display the content of interest. For example, the universe browser engine may determine an existing prism or create a new prism to display the content of interest in response to the first user input.

[0342] A mini-preview of the content of interest may optionally be generated in 208D within the virtual 3D space with little or no discernible / readable detail. In some embodiments, a mini-preview prism may be generated and displayed to the user in 208D, which may or may not include a mini-preview of the content of interest. The mini-preview of the content and / or the mini-preview prism may be generated and presented to the user before the content of interest is rendered so that the user may decide whether to reposition, rotate, and / or translate the mini-preview content or the mini-preview prism.

[0343] A second user input may be received at 210D. The second user input indicates whether the prism, mini-preview prism, content, or mini-preview content should be translated, rotated, and / or translated, and, if applicable, the amount by which the prism, mini-preview prism, content, or mini-preview content should be translated, rotated, and / or translated. For example, the universe browser engine may determine whether the user will translate, rotate, and / or translate the prism or content of interest. If the determination is affirmative, the universe browser engine may further determine the extent of such translation, rotation, and / or translation based, at least in part, on the second user input (e.g., the user moves and rotates the content or web page to a new position and a new orientation).

[0344] As described above, the universe browser engine works in conjunction with the browser engine to initially display a mini-preview prism or a mini-preview of the web page to the user before the content of interest is fully rendered. Once the user finishes translating, rotating, and / or transforming the mini-preview content or mini-preview prism, new position and rotation data may be returned to the browser engine, which in turn calculates graphical coordinates and renders the content of interest using at least the calculated graphical coordinates. In some other embodiments, the universe browser engine and the browser engine render the content of interest and display it to the user, who may subsequently translate, rotate, and / or transform the rendered content of interest. The universe browser engine similarly returns new position and / or rotation data resulting from the translation, rotation, and / or transformation to the browser engine, which in turn updates its stored position and / or rotation data. The content of interest may then be displayed within the prism at 212D based, at least in part, on the second user input.

[0345] 2E illustrates a high-level block diagram for transforming software objects within a universe browser prism in one or more embodiments. A web page or web page panel can be identified from a transformation tree structure at 202E. The transformation tree structure may include a scene graph having multiple nodes, each providing a movable local space corresponding to a prism. A prism may also be a node within the transformation tree structure that is to undergo one or more transformations, partially or entirely, based on a hierarchical structure relative to the prism within the transformation tree structure.

[0346] The transform tree structure may also include nodes that are not prisms. Rather, these nodes may be used to group content or may be used as intermediate transformations (e.g., scaling, translation, rotation, etc.). For example, a group of nodes may undergo a set of identical transformations such that when a set of transformations is applied to one node, the remaining nodes in the group are also transformed along with the node.

[0347] One or more parents of the web page identified in 202E may be identified from the transform tree structure in 204E. A node in the transform tree structure may have zero or more child nodes, but at most one parent. In some embodiments, a prism may be initialized with a null parent and zero children in the corresponding transform tree structure. This transform tree structure may then be populated with more nodes based on the content, which is then rendered into the prism. Child nodes of a parent node in the transform tree structure inherit the transformations that the parent node receives.

[0348] In some embodiments, a root node in a transform tree structure corresponds to a root node transform, which, when applied to a prism's transformation, does not require updating the transforms of nodes local to the root node transform. For example, a node representing audio content may be local to the root node transform so that the audio signal is constrained within the prism and is not perceived as emanating from outside the prism, even when the prism is transformed. In some embodiments, a child node may change its parent node, but the child node does not change its current position or orientation after the change unless the child node is desired or required to undergo a different set of transformations to initiate the change in its parent node. For example, a child prism facing the user may change its parent prism to a new parent prism. Nevertheless, the change does not change the position and rotation of the child prism so that the child prism continues to face the user after the change.

[0349] The universe browser engine may receive position and / or rotation data of the web page or web page panel from the browser engine at 206E. Such position and / or rotation data may be received directly from the browser engine, in some embodiments alone, or in some other embodiments as part of other data (e.g., as part of a web page rendered by the browser engine). The universe browser engine receives the web page position and rotation data from the browser engine to enable a user to manipulate (e.g., translate, rotate, and / or translate) the web page or a panel thereof.

[0350] As described above, the universe browser engine manages the creation, placement, and display of virtual content within the 3D spatial environment, but the universe browser engine, in some embodiments, may store the position and rotation data within the browser engine (e.g., in a cache memory accessible by the browser engine) so that the browser engine may quickly respond to position and / or rotation requests from a web page without further delay by requesting such position and / or rotation data from the universe browser engine. In some other embodiments, the universe browser engine may also store the position and rotation data in another non-transitory computer-readable medium accessible by the universe browser engine, and the two copies of the position and rotation data are synchronized each time the position and / or rotation data changes.

[0351] The universe browser engine may further receive position and / or rotation input for the web page at 208E. For example, a user may translate, rotate, and / or translate the web page or a panel thereof, thus providing input for position and / or rotation data for the web page. The universe browser engine may determine position and / or rotation data for the web page or a portion thereof (e.g., a control for the web page, a sub-panel, etc.) at 210E based, at least in part, on the position and / or rotation input received at 208E. Note that terms web page or a portion thereof (e.g., a smaller portion of the web page) may be used synonymously herein unless explicitly described otherwise. As the position and / or rotation input modifies the position and / or rotation data of the web page (or a panel thereof), the universe browser engine may synchronize that position and / or rotation data with corresponding data stored within the browser engine at 212E. In some embodiments, the universe browser engine synchronizes the entire position and rotation data with the browser engine. In some other embodiments, the universe browser engine tracks changes to its position and rotation data and synchronizes only the deltas between its own position and rotation data and the corresponding data in the browser engine.

[0352] The browser engine receives the updated position and / or rotation data from the universe browser engine, calculates the graphical coordinates of the content to be rendered, and renders the content by invoking a renderer process (e.g., Blink or RenderWidget for WebKit). The universe browser engine may then work in conjunction with the browser engine and the mixed reality system at 214E to display the rendered content within the prism using at least the position and rotation data of the web page.

[0353] FIG. 2F illustrates further details about portions of the block diagram shown in FIG. 2E , according to some embodiments. More specifically, FIG. 2F illustrates further details about determining, at 210E, rotation and / or rotation data for a web page based, in part or in whole, on the position and / or rotation input received at 208E of FIG. 2E . In these embodiments, a set of transformations may optionally be identified, at 202F, with respect to the parent of the web page, if applicable. In some other embodiments, the browser engine may identify the position and / or rotation data, and the universe browser engine may identify the position and / or rotation data, at 202F, directly from the browser engine, either in the form of the position and / or rotation data alone or as part of other data (e.g., a rendered web page), without identifying a set of transformations for the parent (or portion) of the web page. Transformation characteristics of a set of transformations may be identified, at 204F, for the web page based, in part or in whole, on the position and / or rotation input received at 208E. Transformation characteristics may include, for example, a desired or required transformation for the web page (e.g., a desired or required transformation envisioned by the developer of the web page), a parent of the node representing the web page, a type of behavior of the web page or a prism in which the web page is rendered, characteristics of the prism in which the web page is rendered (e.g., range, collision characteristics, overlap / intersection characteristics, etc.), or any combination thereof.

[0354] In some embodiments, the position and / or rotation input may include a change in the parent-child relationship for a node representing a web page within a transform tree structure. For example, the position and / or rotation input may change the parent of a node for a web page. Generally, a set of transforms applied to a parent node also applies to the parent node's child nodes. In some embodiments, changing the parent of a child node does not necessarily change the transforms applied to the child node. For example, if a web page is presented with a head-following (or eye-following) type of behavior, changing the parent of a child node representing a web page may not change the set of transforms applied to the web page because the parent set of transforms applied to the parent node may be overridden by this type of behavior.

[0355] Position and / or rotation inputs for the web page may be identified at 206F. For example, a user may translate, rotate, and / or transform the web page or a panel thereof, thus providing position and / or rotation inputs for the web page, at 206F. As described above, the position and / or rotation inputs may include changes to the node structure of the transformation tree structure. For example, a node representing the web page may be added to or removed from one or more nodes as its parent and / or child.

[0356] Changing the parent of a node representing a web page may or may not change the set of transforms applied to the node representing the web page. For example, in some embodiments where world transforms are applied to all nodes in a transform tree structure, changing the parent of a node representing a web page may nevertheless not change the world transforms applied to the node and other nodes in the transform tree structure. As another example, if a web page is to be presented facing the user according to a Head-Stay-Follow behavior type, this behavior type may override one or more other transforms that may interfere with the Head-Stay-Follow behavior type, such that these one or more other transforms are not applied to the node.

[0357] A set of transforms for the web page may be determined at 208F based, at least in part, on, for example, one or more of the position and / or rotation input, the set of transforms for the parent, or the transform characteristics for the web page identified at 204F. In some embodiments, a node representing a web page in the transform tree structure may include or correspond to one or more of the extents of the web page (or a panel thereof) in virtual 3D space, one or more local transforms for the web page, a 3D rotation matrix governing the rotation of the web page, a scaling factor corresponding to the local transforms, a position vector of the web page, one or more local-to-world transforms, or a local-to-world scale.

[0358] The local transform includes a transform that is applied to a parent node of a node representing a web page in the transform tree structure. In some embodiments where the node representing a web page does not have a parent node, the world transform (from the general or default space, the local prism into which the web page is rendered) may be assigned to the local transform. In some embodiments where the node includes or corresponds to multiple elements (e.g., multiple transforms) described above, the local transform may be determined by successively applying these multiple elements. For example, if the web page node undergoes a translation transform, a scaling transform, and a rotation transform, the local transform may be symbolically represented as follows: local transform = translation * Rotation matrix * This local transform, when applied to a web page, scales the web page according to a scaling factor (e.g., non-uniform scaling across three axes may be supported), rotates the scaled web page according to a rotation matrix, and then translates the scaled and rotated web page according to a translation matrix.

[0359] In some embodiments, OpenGL Mathematics (GLM) may be used to perform the transformation for the web page, although other mathematical libraries may also be used to perform it. The following example code demonstrates the use of GLM to perform the transformation of a web page: [ka]

[0360] In the above code, "TransformNode::getCurrentVolumeTransform()" may be used to get the transform of the web page panel, and other code segments may be used to set the transform of the web page panel. Additionally, while the following example code may be used to decompose the transform of the web page panel into a position and rotation, it should be noted that other similar or equivalent implementations are also contemplated and may be used to achieve the same or substantially similar purpose. [ka]

[0361] Using the set of transforms for the web page determined at 208F, position and rotation data for the web page (or a panel thereof) may be determined by applying the set of transforms to the web page at 210F. Using the position and rotation data, the browser engine may calculate the graphical coordinates of the content to be rendered and invoke a renderer (e.g., a RenderWidget or other renderer) to render the content. The rendered content may then be displayed in whole or in part within a prism based on the position and rotation data at 212F.

[0362] 2G-2I illustrate embodiments in which any node may be grouped in any manner, and the corresponding transformation may depend on the parent-child relationships resulting from such grouping of nodes. These examples illustrated in FIGS. 2G-2I may, in some embodiments, include various transform trees, grouping trees, and / or creating new groups. For example, with respect to transform tree 200G in FIG. 2G, this embodiment illustrates grouping together node A (202G) and node E (210G), which are currently separate transform trees. A determination may first be made as to whether node A (202G) or node E (210G) are already in a group. Currently, neither node A (202G) nor node E (210G) has a group-parent, and therefore neither is in a group. Thus, a group root may be created.

[0363] In FIG. 2H, a group tree 1022 having a group root 202H is created. The group parents of node A (202G) and node E (210G) are assigned to the group root 202H, as depicted in FIG. 2H. Further, a determination is made to determine whether node E (210G) is an ancestor (or parent) of node A (202G). Node E (210G) is not an ancestor of node A (202G) because node A (202G) has no parent. Additionally, another determination may be made to determine whether node A (202G) is an ancestor (or parent) of node E (210G). Similarly, node A (202G) is not an ancestor of node E (210G) because node E (210E) has no parent.

[0364] FIG. 2I illustrates a sample computation according to some embodiments of the present disclosure. In some embodiments, any transformation applied directly to a node that belongs within a group may transform the group along with it. An indirect transformation performed on a group node may not cause the group to be transformed. For example, if a parent node is not within a group, but its children are within a group, and the parent is transformed, the nodes within the child's group will not transform along with it, even if the child's transformation changes. If moving an entire group is desired, the parent should be added to the group. To compute a transformation to apply to a group, the transformation to use must be applied to each immediate child of the group root.

[0365] For a given transform tree 200G and group tree 200H as depicted in FIG. 2I, the present embodiment applies a transform to node A (202G). Because node A (202G) belongs to the group tree 200H, nodes C (206G), D (208G), and E (210G) are expected to be transformed along with node A (202G). In addition, node B (204G), which does not belong to the group, must also be transformed along with the group because it is a child of node A (202G). The new world matrix of node A (202G) may be denoted as Maw′, where Maw′ stands for “matrix a world new.” The old world matrix of node A (202G) may be denoted as Maw, where Maw stands for “matrix a world old.” The matrix to apply to node A (202G) to obtain Maw′ may be denoted as Mgw, where Mgw stands for “matrix a world old.” represents "group world". One of the goals is to determine a matrix to set on the group root (Mgw) that, when applied to node A's world matrix, will be equal to node A's new matrix. This same matrix can then be applied to other direct children of the group root.

[0366] FIG. 2J illustrates a high-level block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using an extended reality system, in one or more embodiments. In these embodiments, the browser engine may receive input for a three-dimensional (3D) orientation for the web page at 202J. In some embodiments, the input may include input by a user of the mixed reality system requesting presentation of a web page of interest, or by a web page developer who prefers the web page to be presented in a certain manner and codes it accordingly. In some other embodiments, the input may include a request for orientation data. For example, a request for orientation data (e.g., in some embodiments, positioning data, rotation data, and / or even scale data) may be issued by the website of the web page of interest in some embodiments. For example, the web page developer may incorporate an application programming interface (e.g., the Window.VirtualWorld.Orientation API described above) into the code of the web page of interest.

[0367] As explained immediately above, in some embodiments, the input may come from, for example, a developer who prefers to represent a web page in a certain orientation (e.g., a certain rotation) when presented in the virtual 3D space. In some of these or other embodiments, the input may come from a user's manipulation of a web page, a previously visited same web page, or other web pages from the same website, or optionally from other websites, etc., within the virtual 3D space created by the mixed reality system.

[0368] For example, a web page may be opened and displayed in a virtual 3D space for a user, and the user may adjust the orientation of the web page within the virtual 3D space. In some of these embodiments, similar techniques may also be applied to manipulate a web browser for the web page or a prism within which the web page is rendered. The user may freely adjust the orientation of the web page representation within the virtual 3D space, and constraints on such orientation manipulation may include, for example, prism boundaries, allowable orientation values, allowable orientation ranges, etc., as provided, for example, by the universe browser engine and / or the web page developer. In some embodiments, such orientation may also optionally be constrained by collision and / or overlap dynamics between the web page representation and one or more other physical and / or virtual objects within the virtual 3D space, such that the web page representation and the virtual objects appear or are perceived as real.

[0369] A browser engine (e.g., Blink, Gecko, WebKit, Trident, etc.) is not a stand-alone computer program. Rather, a browser engine is a software component of a web browser that converts web page resources (e.g., HTML documents, etc.) into interactive visual representations. Other commonly used terms for a browser engine include a layout engine, a rendering engine, etc., for example, by calculating graphical coordinates for the visual representation based on rules in a Cascading Style Sheet (CSS).

[0370] As its name suggests, the browser engine performs layout and rendering for web pages, enforces security policies, and implements the DOM (Document Object Model) data structure that is exposed to page scripts. The browser engine may also handle hyperlinks and web forms. In some embodiments, the browser engine utilizes a separate, dedicated script engine for the execution of JavaScript® code. In addition to performing at least the aforementioned functions for a web browser, the browser engine may also perform the same or substantially similar functions for other software programs, such as email clients, other frameworks for software applications (e.g., Google's Electron Framework), etc.

[0371] The browser engine may determine orientation data and respond to requests for 3D orientation or 3D orientation manipulation at 204J. In some embodiments, the browser engine may store recent orientation data in a non-transitory computer-readable medium (e.g., cache memory) accessible by the browser engine. Such orientation data may be transmitted by the universe browser engine of the mixed reality system each time the orientation data is modified (e.g., by a user further moving, rotating, or translating a web page displayed in virtual 3D space).

[0372] When a web browser initializes to render and display a web page, its browser engine may respond to such a request for orientation with orientation data that is stored, for example, in a cache memory accessible by the browser engine (e.g., in the RenderWidget class in Chromium-based browsers). Once the orientation data is modified, the browser engine may refresh its stored orientation data with the updated orientation data or update with orientation data from the universe browser engine. Using the most recent orientation data, the browser engine may render the content of the web page (e.g., by calling the RenderWidget class) and, for example, calculate graphical coordinates for the rendered content using the most recent orientation data and rules that govern the layout of the rendered content (e.g., cascading style sheets).

[0373] The universe browser engine of the mixed reality system may invoke the browser engine at 206J, which in turn invokes one or more mixed reality software modules and / or hardware mechanisms to display the web page within the virtual 3D space. For example, depending on the orientation of the web page within the virtual 3D space (e.g., within a prism), the virtual 3D representation of the web page may span multiple depth planes or multiple focal planes. The extended reality system may, for example, utilize its switchable focal plane or variable focusing mechanism to render the virtual 3D representation of the web page based on the 3D orientation data.

[0374] Further details about extended reality systems are described in the U.S. patent applications referenced in the first paragraph of this application. In some embodiments, this invocation of the mixed reality mechanism may occur via a browser (e.g., Chromium) and / or a browser engine (e.g., Blink for Chromium). For example, the browser (or its browser engine) may apply orientation data (or, optionally, scale data) to a web page panel, invoke the mixed reality mechanism, and display the virtual 3D content.

[0375] 2K illustrates a further detailed block diagram for a process for managing and displaying web pages and web resources in a three-dimensional virtual space using a mixed reality system in one or more embodiments. In response to an instruction to open a web page (e.g., by a user clicking a link, by a user entering search criteria into a search field, by a user entering a URL for a website, etc.), a web browser or web page panel is launched at 202K.

[0376] The universe browser engine may send the orientation data to the browser engine of the web browser so that the orientation data may be stored in a non-transitory computer-readable medium (e.g., a cache) or so that the orientation data in the non-transitory computer-readable medium may be refreshed. The browser engine may then cache or store the received orientation data, for example, in a cache memory accessible by the browser engine. In some embodiments where the orientation data is stored in a non-transitory computer-readable memory (e.g., a cache) accessible by the browser engine, the stored orientation data stored in the non-transitory computer-readable memory may be initialized with the orientation data from the universe browser engine.

[0377] The web page may optionally send an orientation request to the browser engine at 204K. For example, the web page developer may incorporate a set of application programming interfaces (APIs) (e.g., the Window.VirtualWorld.Orientation API described above) into the web page's code so that the web page sends an orientation data request to the browser engine upon initialization or shortly thereafter. In some embodiments in which the web page developer may desire the web page to be presented in a certain orientation, the web page may also set the orientation data by utilizing the set of APIs to communicate the orientation data to the browser engine. In some of these embodiments, the web page may obtain the orientation data of the web page pane from the browser engine in response to user input (e.g., the user clicking a link, the user typing search criteria into a search field, the user typing a URL for a website, etc.).

[0378] In response to an optional orientation request from a web page displaying the web page, the browser engine may respond to the request at 206K with recent orientation data stored in a non-transitory computer-readable medium accessible by the browser engine, for example, without further delay from obtaining such orientation data from the universe browser engine of the mixed reality system. If recent orientation data is unavailable (e.g., for a mixed reality system being initialized for the first time or for other reasons), the browser engine may respond to such a request from the web page with default orientation data.

[0379] The browser engine may also transmit orientation data to the universe browser engine at 208K. For example, some embodiments may provide a user with the ability to orient a representation of a web page within a virtual 3D space. In some embodiments in which a GET request is received, the browser engine returns orientation data to the web page, which is stored in a non-transitory computer-readable medium (e.g., a cache).

[0380] The universe browser engine receives orientation data from the browser engine of the web browser and invokes other modules of the mixed reality system (e.g., the runtime layer, the operating system services layer, one or more 3D engines, or one or more software applications described above) and other hardware and software components thereof to allow the user to freely adjust the 3D orientation for the representation of the web page within the virtual 3D space (e.g., a prism). In some embodiments, 206K is executed when the web page obtains the orientation data ("GET"), and 208K is executed when the web page determines the orientation data ("SET").

[0381] The universe browser engine, in some embodiments, updates the orientation data for the web page in response to receiving the orientation data from the browser engine at 210K. In some other embodiments where a SET request is received for new orientation data, the new orientation data may be stored in a non-transitory computer-readable medium (e.g., a cache) for the browser engine and may be further passed to the universe browser engine. The universe browser engine further determines whether the orientation data has been updated.

[0382] For example, any orientation of a web page representation by a user may modify the orientation data. If the universe browser engine determines that the orientation data has changed, the universe browser engine may update the orientation data and send the latest version of the orientation data to the browser engine at 212K. The browser engine may then, in turn, store the orientation data stored in a non-transitory computer-readable medium (e.g., a cache) along with the latest version of the orientation data.

[0383] In response to receiving the latest version of the orientation data, the browser engine, in some embodiments, may further calculate the graphical coordinates of the web page content and render the web page content. The web page panels may then be translated, rotated, scaled, or otherwise transformed based, at least in part, on the latest version of the orientation data. The browser engine, in these embodiments, may then pass the rendered content to the universe browser engine at 214K, which in turn invokes various software and hardware modules of the mixed reality system to generate a representation of the rendered content in virtual 3D space and display the representation of the rendered content to the user within the translated, rotated, scaled, or otherwise transformed web page panels in virtual 3D space.

[0384] In some other embodiments, the browser engine renders the content of the web page using orientation data stored in the browser engine's cache and transmits the rendered content and orientation data to the universe browser engine, which treats the rendered content as a software object and allows a user to adjust the 3D orientation of the rendered content of the web page.

[0385] Once the user determines the orientation of the web page representation, the universe browser engine updates the orientation data, invokes, for example, a transformation and rendering module, generates an updated representation of the web page for the software object in the virtual 3D space, and presents the updated representation of the web page to the user. In some of these embodiments, the mixed reality system may display the web page in the virtual 3D space in near real-time fashion while the user changes the position and / or rotation of the web page in the virtual 3D space, with no perceptible delay between the user's manipulation and the display of the web page (214K).

[0386] FIG. 2L illustrates a high-level block diagram for transforming software objects within a universe browser prism in one or more embodiments. A web page or web page panel can be identified from a transformation tree structure at 202L. The transformation tree structure may include a scene graph having multiple nodes, each providing a movable local space corresponding to a prism. A prism may also be a node within the transformation tree structure that is to undergo one or more transformations, partially or entirely, based on a hierarchical structure relative to the prism within the transformation tree structure.

[0387] The transform tree structure may also include nodes that are not prisms. Rather, these nodes may be used to group content or may be used as intermediate transformations (e.g., scaling, translation, rotation, etc.). For example, a group of nodes may undergo a set of identical transformations such that when a set of transformations is applied to one node, the remaining nodes in the group are also transformed along with the node.

[0388] One or more parents of the web page identified in 202L may be identified from the transform tree structure at 204L. A node in the transform tree structure may have zero or more child nodes, but at most one parent. In some embodiments, a prism may be initialized with a null parent and zero children in the corresponding transform tree structure. This transform tree structure may then be populated with more nodes based on the content, which is then rendered into the prism. Child nodes of a parent node in the transform tree structure inherit the transformations that the parent node receives.

[0389] In some embodiments, the root node in the transform tree structure corresponds to a root node transform, which, when applied to the transformation of a prism, does not require updating the transforms of nodes local to the root node transform. For example, a node representing audio content may be local to the root node transform so that the audio signal is constrained within the prism and is not perceived as emanating from outside the prism, even when the prism is transformed.

[0390] In some embodiments, a child node may change its parent node, but the child node does not change its current position or orientation after the change unless the child node is desired or required to undergo a different set of transformations to initiate the change in its parent node. For example, a child prism facing the user may change its parent prism to a new parent prism. Nevertheless, the change does not change the position and rotation of the child prism so that the child prism continues to face the user after the change.

[0391] The universe browser engine may receive orientation data for a web page or web page panel from the browser engine at 206L. Such orientation data may be received as orientation data directly from the browser engine, in some embodiments alone, or in some other embodiments as part of other data (e.g., as part of a web page rendered by the browser engine). The universe browser engine receives the orientation data for the web page from the browser engine to enable a user to manipulate (e.g., translate, rotate, and / or translate) the web page or a panel thereof.

[0392] As described above, the universe browser engine manages the creation, placement, and display of virtual content within the 3D spatial environment, but the universe browser engine, in some embodiments, may store orientation data within the browser engine (e.g., in a cache memory accessible by the browser engine) so that the browser engine can quickly respond to orientation requests from web pages without further delay by requesting such orientation data from the universe browser engine. In some other embodiments, the universe browser engine may also store the orientation data in another non-transitory computer-readable medium accessible by the universe browser engine, and the two copies of the orientation data are synchronized each time the orientation data changes.

[0393] The universe browser engine may further receive orientation input for the web page at 208L. For example, a user may translate, rotate, and / or translate the web page or a panel thereof, thus providing input for orientation data for the web page. The universe browser engine may determine orientation data for the web page or a portion thereof (e.g., a control for the web page, a sub-panel, etc.) at 210L based, at least in part, on the orientation input received at 208L. Note that a web page or a portion thereof (e.g., a smaller portion of a web page) may be used synonymously herein unless explicitly described otherwise.

[0394] As the orientation input modifies the orientation data of a web page (or a panel thereof), the universe browser engine may synchronize that orientation data with the corresponding data stored in the browser engine at 212L. In some embodiments, the universe browser engine synchronizes the entire orientation data with the browser engine. In some other embodiments, the universe browser engine tracks changes to its orientation data and synchronizes only the delta between its own orientation data and the corresponding data in the browser engine.

[0395] The browser engine receives the updated orientation data from the universe browser engine, calculates the graphical coordinates of the content to be rendered, and renders the content by invoking a renderer process (e.g., Blink or RenderWidget for WebKit). The universe browser engine may then work in conjunction with the browser engine and the mixed reality system at 214L to display the rendered content within the prism using at least the orientation data of the web page.

[0396] FIG. 2M illustrates further details about a portion of the block diagram shown in FIG. 2L, according to some embodiments. More specifically, FIG. 2M illustrates further details about determining orientation data for a web page at 210L, partially or wholly, based on the orientation input received at 208L of FIG. 2L. In these embodiments, a set of transformations may optionally be identified at 202M with respect to the parent of the web page, if applicable. In some other embodiments, the browser engine may identify the orientation data, and the universe browser engine may identify the orientation data at 202M from the browser engine directly, either in the form of orientation data alone or as part of other data (e.g., a rendered web page), without identifying a set of transformations for the parent (or portion) of the web page.

[0397] Transformation characteristics of a set of transformations may be identified at 204M for the web page based, in part or in whole, on the orientation input received at 208L. The transformation characteristics may include, for example, a desired or required transformation for the web page (e.g., a desired or required transformation envisioned by a developer of the web page), a parent of the node representing the web page, a type of behavior of the web page or a prism within which the web page is rendered, characteristics of the prism within which the web page is rendered (e.g., extent, collision characteristics, overlap / intersection characteristics, etc.), or any combination thereof.

[0398] In some embodiments, the orientation input may include a change in parent-child relationships for a node representing a web page within a transform tree structure. For example, the orientation input may change the parent of a node for a web page. Generally, a set of transforms applied to a parent node also applies to the parent node's child nodes. In some embodiments, changing the parent of a child node does not necessarily change the transforms applied to the child node. For example, if a web page is presented with a head-following (or eye-following) type of behavior, changing the parent of a child node representing a web page may not change the set of transforms applied to the web page because the parent set of transforms applied to the parent node may be overridden by this type of behavior.

[0399] Orientation input for the web page may be identified at 206M. For example, a user may translate, rotate, and / or transform the web page or a panel thereof at 206M, thus providing orientation input for the web page. As described above, the orientation input may include changes to the node structure of the transformation tree structure. For example, a node representing the web page may be added to or removed from one or more nodes as its parent and / or child.

[0400] Changing the parent of a node representing a web page may or may not change the set of transforms applied to the node representing the web page. For example, in some embodiments where world transforms are applied to all nodes in a transform tree structure, changing the parent of a node representing a web page may nevertheless not change the world transforms applied to the node and other nodes in the transform tree structure. As another example, if a web page is to be presented facing the user according to a Head-Stay-Follow behavior type, this behavior type may override one or more other transforms that may interfere with the Head-Stay-Follow behavior type, such that these one or more other transforms are not applied to the node.

[0401] A set of transforms for the web page may be determined at 208M based, at least in part, on, for example, one or more of the orientation input, the set of transforms for the parent, or the transform characteristics for the web page identified at 204M. In some embodiments, a node representing a web page in the transform tree structure may include or correspond to one or more of the extents of the web page (or a panel thereof) in virtual 3D space, one or more local transforms for the web page, a 3D rotation matrix that governs the rotation of the web page, a scaling factor corresponding to the local transforms, an orientation vector for the web page (an orientation vector that indicates the positioned and / or orientation of the web page), one or more local / world transforms, or a local / world scale.

[0402] A local transform includes a transform that is applied to a parent node of a node representing a web page in the transform tree structure. In some embodiments where a node representing a web page does not have a parent node, a world transform (from a general or default space, the local prism into which the web page is rendered) may be assigned to the local transform. In some embodiments where a node includes or corresponds to multiple elements (e.g., multiple transforms) described above, the local transform may be determined by sequentially applying the...

Claims

1. A computer-implemented method comprising: receiving a web page or web page panel from a website, the web page or web page panel including first three-dimensional virtual content; generating and anchoring a first bounded prism at a first location within the mixed reality landscape; rendering the first three-dimensional virtual content of the web page or web page panel within the anchored first bounded prism for display to a user by a display device; generating a second bounded prism and anchoring the second bounded prism to the first location within the mixed reality landscape; updating the rendering of the first three-dimensional virtual content to the user within the anchored first bounded prism to second three-dimensional content to the user within the anchored second bounded prism in response to input from the user; 10. A computer-implemented method comprising:

2. The computer-implemented method of claim 1, wherein the first location at which the first bounded prism is anchored is on a virtual object within the user's mixed reality landscape.

3. The computer-implemented method of claim 1, wherein the first location at which the first bounded prism is anchored is on a physical object within the user's mixed reality landscape.

4. The computer-implemented method of claim 1, wherein the first bounded prism is a prism.

5. The computer-implemented method of claim 1, further comprising anchoring the first bounded prism at the first location within the mixed reality landscape in response to input from the user to translate the first bounded prism within the mixed reality landscape.

6. The computer-implemented method of claim 5, further comprising snapping the first bounded prism to the first location within the mixed reality landscape in response to input from the user to translate the first bounded prism near the first location within the mixed reality landscape.

7. The computer-implemented method of claim 1, further comprising selecting the first location within the mixed reality landscape such that the first three-dimensional virtual content does not interpenetrate the mixed reality landscape.

8. The computer-implemented method of claim 1, further comprising translating, rotating, scaling, or translating the first bounded prism within the mixed reality landscape in response to input from the user.

9. The computer-implemented method of claim 1, wherein the first bounded prism comprises one or more handles, and wherein input from the user includes interacting with the one or more handles.

10. The computer-implemented method of claim 1, wherein the first three-dimensional virtual content includes a parent virtual object and multiple child virtual objects, the first bounded prism is a bounded parent prism, and the at least one processor is configured to generate multiple bounded child prisms, anchor the bounded parent prism at the first location within the mixed reality landscape, render the parent virtual object within the bounded parent prism for display to the user by the display device, and render the child virtual objects each within the bounded child prism for display to the user by the display device.

11. The computer-implemented method of claim 10, further comprising translating, rotating, scaling, or translating each of the plurality of child virtual objects relative to the parent virtual object in response to input from the user.

12. The computer-implemented method of claim 10, further comprising translating, rotating, scaling, or translating the parent virtual object and the plurality of child virtual objects together in response to input from the user.

13. The computer-implemented method of claim 1, further comprising generating and anchoring a first bounded prism at the first location within the mixed reality landscape in response to receiving a request from an application configured to render the first three-dimensional virtual content of the web page within the anchored first bounded prism for display to the user by the display device.

14. An extended reality system, comprising: At least one processor wherein the at least one processor 1. An extended reality system configured to receive a web page or web page panel including first three-dimensional virtual content from a website, generate and anchor a first bounded prism at a first location within a mixed reality landscape, and render the first three-dimensional virtual content of the web page or web page panel within the anchored first bounded prism for display to a user by a display device, wherein the at least one processor is configured to generate a second bounded prism, anchor the second bounded prism at the first location within the mixed reality landscape, and update the rendering of the first three-dimensional virtual content for the user within the anchored first bounded prism to second three-dimensional content for the user within the anchored second bounded prism in response to input from the user.

15. A computer program product embodied on a computer-readable medium having stored thereon a sequence of instructions that, when executed by a processor, performs the following steps: receiving a web page or web page panel from a website, the web page or web page panel including first three-dimensional virtual content; generating and anchoring a first bounded prism at a first location within the mixed reality landscape; rendering the first three-dimensional virtual content of the web page or web page panel within the anchored first bounded prism for display to a user by a display device; generating a second bounded prism and anchoring the second bounded prism to the first location within the mixed reality landscape; updating the rendering of the first three-dimensional virtual content to the user within the anchored first bounded prism to second three-dimensional content to the user within the anchored second bounded prism in response to input from the user; a computer program product for causing the processor to execute a method comprising:

Images