Browsers for mixed reality systems

Abstract

An improved system and method for navigating and manipulating browser windows in a 3D mixed reality environment is disclosed. An improved method is provided for viewing a user's windows regardless of the user's current position relative to one or more previously opened windows. A method for displaying windows in a computing environment includes receiving an instruction for selecting a plurality of open windows. The method also includes retrieving information for the plurality of open windows, wherein the plurality of open windows are associated with different physical locations. The method further includes displaying representations of the plurality of open windows in a single user interface. In addition, the method includes upon receiving a selection of a selected window from the plurality of open windows, loading the selected window into a foreground for a field of view for the user.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for implementing browsing techniques in a spatial 3D environment. Background Art

[0002] The typical way to view a web page is to open it on a monitor of a computer, smartphone, tablet, etc. The user will scroll the web page to view different content displayed on the web page. Usually, whether the user is looking at a computer monitor, smartphone, or tablet, there is a fixed format for how the content is displayed on the monitor.

[0003] Through virtual reality, augmented reality and / or mixed reality systems (hereinafter collectively referred to as "mixed reality" systems), users are provided with a three-dimensional environment for the display of content. The problem is that conventional methods of displaying 2D content within a browser do not work very well when used in a 3D environment. One of the reasons for this is because, in conventional 2D web browsers, the display area of ​​the display device is limited to the screen area of ​​the monitor where the content is being displayed. As a result, conventional browsers are configured to only know how to organize and display content within the monitor display area. In contrast, a 3D environment is not limited to the strict limitations of the monitor display area. Therefore, when used in a 3D environment, conventional 2D browsers perform suboptimally because conventional browsing technology simply does not have the function or ability to utilize a 3D environment to display content.

[0004] For example, consider a situation when a user is using a mixed reality device and has placed multiple browser windows associated with different physical locations. For example, the user may have opened a first browser window in a first room and a second browser window in a second room. Since conventional 2D-based browsers are limited to display in a given monitor area, this means that conventional browsers do not even have the technology to grasp the concept of physically remote windows, let alone the ability to handle situations where multiple windows are opened in multiple physical locations, making it impossible for users to effectively view, navigate, and use these multiple windows.

[0005] Therefore, there is a need for an improved method to implement browsing technology in a 3D environment. Summary of the invention

[0006] Improved systems and methods are provided for navigation and manipulation of browser windows in a 3D mixed reality environment.Some embodiments are directed to an improved method for viewing a user's windows regardless of the user's current position relative to one or more previously opened windows.

[0007] Improved systems and methods for navigation and manipulation of browser windows can be applied in the context of 2D content that is deconstructed and displayed in a spatially organized 3D environment. This can include identifying the 2D content, identifying elements in the 2D content, identifying surrounding surfaces, mapping the identified elements to the identified surrounding surfaces, and displaying the elements as virtual content on the surrounding surfaces.

[0008] In one embodiment, a method for displaying windows in a computing environment includes receiving an instruction for selecting a plurality of open windows. The method also includes retrieving information for the plurality of open windows, wherein the plurality of open windows are associated with different physical locations. The method further includes displaying representations of the plurality of open windows in a single user interface. In addition, the method includes upon receiving a selection of a selected window from the plurality of open windows, loading the selected window into a foreground of a field of view for a user.

[0009] In one or more embodiments, a representation of multiple open windows is displayed in a single user interface by changing the position parameters for multiple open windows to positions within the current physical environment for the user. Multiple open windows are rendered and displayed to the user at the coordinates assigned to the multiple open windows within the single user interface. Multiple open windows are rendered in at least one of a preview form, a thumbnail form, or a complete form. All open windows are selected for display in a single user interface. Each of the multiple open windows is rendered into a separate prism for placement of virtual content. The window being hovered can be moved into the foreground, while other windows visually recede. Upon receiving a selection of the selected window, the unselected window is closed.

[0010] In another embodiment, a system for manipulating windows in a computing environment includes a mixed reality display device that displays three-dimensional content. The system also includes a processor. The system also includes a memory for holding programmable code executable by the processor. The programmable code includes instructions to: receive an instruction to select a plurality of open windows; retrieve information for the plurality of open windows, wherein the plurality of open windows are associated with different physical locations; display representations of the plurality of open windows in a single user interface of the mixed reality display device; and upon receiving a selection of a selected window of the plurality of open windows, load the selected window into a foreground for a user's field of view.

[0011] In one or more embodiments, a representation of multiple open windows is displayed in a single user interface by changing the position parameters for the multiple open windows to positions within the current physical environment for the user. The multiple open windows are rendered and displayed to the user at the coordinates assigned to the multiple open windows within the single user interface. The multiple open windows are rendered in at least one of a preview form, a thumbnail form, or a full form. All open windows are selected for display in a single user interface. Each of the multiple open windows is rendered into a bounded volume for placement of virtual content. A hover state is implemented in which the window being hovered moves into the foreground and other windows visually recede. Upon receiving a selection of the selected window, the unselected windows are closed.

[0012] In yet another embodiment, a computer program product is embodied on a computer-readable medium storing thereon a sequence of instructions which, when executed by a processor, causes the processor to perform a method comprising: receiving an instruction to select a plurality of open windows; retrieving information for the plurality of open windows, wherein the plurality of open windows are associated with different physical locations; displaying representations of the plurality of open windows in a single user interface; and upon receiving a selection of a selected window from the plurality of open windows, loading the selected window into a foreground for a user's field of view.

[0013] In one or more embodiments, a representation of multiple open windows is displayed in a single user interface by changing the position parameters for the multiple open windows to positions within the current physical environment for the user. The multiple open windows are rendered and displayed to the user at the coordinates assigned to the multiple open windows within the single user interface. The multiple open windows are rendered in at least one of a preview form, a thumbnail form, or a full form. All open windows are selected for display in a single user interface. Each of the multiple open windows is rendered into a prism for placement of virtual content. A hover state is implemented in which the window being hovered moves into the foreground and other windows visually recede. Upon receiving a selection of the selected window, the unselected windows are closed.

[0014] In yet another embodiment, a method for displaying virtual content in a computing environment includes receiving an instruction to select a plurality of open applications. The method also includes retrieving information for the plurality of open applications, wherein the plurality of open applications are associated with different physical locations. The method further includes displaying representations of the plurality of open applications in a single user interface. Additionally, the method includes upon receiving a selection of a selected application from the plurality of open applications, loading the selected application into a foreground for a user's field of view.

[0015] In one or more embodiments, representations of multiple open applications are displayed in a single user interface by changing position parameters for the multiple open applications to positions within a current physical environment for the user. The multiple open applications are rendered and displayed to the user at coordinates assigned to the multiple open applications within the single user interface. The multiple open applications are rendered in at least one of a preview form, a thumbnail form, or a full form. All open applications are selected for display in a single user interface. Each of the multiple open applications is rendered as a separate prism for placement of virtual content. A hover state is implemented in which the application being hovered moves into the foreground while other applications visually recede. Upon receiving a selection of the selected application, the unselected applications are closed.

[0016] Further details of aspects, objects, and advantages of the embodiments are described below in the detailed description, drawings, and claims.The foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the scope of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how the above and other advantages and objects of various embodiments of the present invention are obtained, the invention briefly described above will be described in more detail with reference to specific embodiments of the invention, which are shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered as limiting the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0018] Figure 1 An augmented reality environment for deconstructing 2D content to be displayed in a user's 3D environment is shown in accordance with some embodiments.

[0019] Figure 2 An example mapping of elements in 2D content to a user's 3D environment is shown in accordance with some embodiments.

[0020] Figure 3 is a flow chart illustrating a method for deconstructing 2D content to be displayed in a 3D environment according to some embodiments.

[0021] Figure 4 is a flow chart illustrating a method for identifying elements in 2D content according to some embodiments.

[0022] Figure 5 An example of a table for storing elements deconstructed from 2D content is shown in accordance with some embodiments.

[0023] Figure 6 is a flow chart illustrating a method for identifying a surface from a user's local environment according to some embodiments.

[0024] Figure 7 An example of a table for storing an inventory of surfaces identified from a user's local environment is shown in accordance with some embodiments.

[0025] Figure 8 is a flow chart illustrating a method for mapping elements from 2D content to an available surface according to some embodiments.

[0026] Fig. 9 An example of a table for storing a mapping of elements from 2D content to surfaces in a user's local environment is shown in accordance with some embodiments.

[0027] Fig.10 A flow chart of a method for implementing viewing of a user window is shown.

[0028] Fig.11A -B shows the process of displaying a window to the user without regard to the window's previous physical location.

[0029] Figure 12-13 Provides examples of possible methods for displaying multiple windows within a mixed reality interface.

[0030] Fig.14 A possible method of displaying multiple prisms within a mixed reality system is shown.

[0031] Fig.15 is a block diagram of an exemplary computing system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples of the present disclosure so that those skilled in the art can practice the present disclosure. It is noteworthy that the following figures and examples are not meant to limit the scope of the present disclosure. In the case where known components (or methods or processes) can be used to partially or completely implement certain elements of the present disclosure, only those parts of these known components (or methods or processes) necessary for understanding the present disclosure will be described, and the detailed description of other parts of these known components (or methods or processes) will be omitted, so as not to obscure the present disclosure. In addition, various embodiments encompass the current and future known equivalents of the components referred to herein by way of example.

[0033] Although the systems and methods described below are primarily described in the context of browser applications, it will be understood by those of ordinary skill in the art that the systems and methods described herein may also be applied in the context of one or more other applications. In some embodiments, an application for managing a user's photos and / or videos may utilize the systems and methods described below. In some embodiments, an application for playing card games may utilize the systems and methods described below. In some embodiments, a weather application may utilize the systems and methods described below. In some embodiments, any other application that may be installed and / or run on a device and / or system capable of displaying 3D virtual content to a user may utilize the systems and methods described below. In some embodiments, a single application may utilize the systems and methods described below. In some embodiments, more than one application may utilize the systems and methods described below. In some embodiments, all applications installed and / or run on a device and / or system capable of displaying 3D virtual content to a user may utilize the systems and methods described below.

[0034] Although the systems and methods described below are primarily described in the context of browser applications, it will be understood by those of ordinary skill in the art that the systems and methods described herein may also be applied in the context of one or more other applications. In some embodiments, an application for managing a user's photos and / or videos may utilize the systems and methods described below. In some embodiments, an application for playing card games may utilize the systems and methods described below. In some embodiments, a weather application may utilize the systems and methods described below. In some embodiments, any other application that may be installed and / or run on a device and / or system capable of displaying 3D virtual content to a user may utilize the systems and methods described below. In some embodiments, a single application may utilize the systems and methods described below. In some embodiments, more than one application may utilize the systems and methods described below. In some embodiments, all applications installed and / or run on a device and / or system capable of displaying 3D virtual content to a user may utilize the systems and methods described below. In some embodiments, multiple instances of an application may utilize the systems and methods described below.

[0035] Web page deconstruction

[0036] Embodiments of the present disclosure will deconstruct a 2D web page to be displayed in a spatially organized 3D environment. The 2D web page may originate from a web browser of a head mounted system, a mobile device (e.g., a cell phone), a tablet, a television, an application, etc. In some embodiments, the 2D web page may be received from another application or device such as a laptop, a desktop computer, an email application with a link to the 2D web page, an electronic message that references or includes a link to the 2D network, etc.

[0037] Refer to the figure ( Figure 1 ), environment 100 is representative of a physical environment and system for implementing the processes described below (e.g., deconstructing 2D content from a web page to be displayed on a 3D surface in the user's physical environment 105). The representative physical environment and system of environment 100 includes the user's physical environment 105 viewed by user 108 through head mounted system 160. The representative system of environment 100 also includes access to 2D content (e.g., a web page) via a web browser 110 operably coupled to a network 120. Network 120 can be the Internet, an intranet, a private cloud network, a public cloud network, etc. Web browser 110 is also operably coupled to processor 170 via network 120. Although processor 170 is shown as an isolated component separate from head mounted system 160, in alternative embodiments, processor 170 may be integrated with one or more components of head mounted system 160 and / or may be integrated into other system components within environment 100, such as network 120 accessing computing network 125 and storage device 130. The processor 170 may be configured with software 150 for receiving and processing information such as video, audio, and content received from the head mounted system 160, the local storage device 140, the web browser 110, the computing network 125, and the storage device 130. The software 150 may communicate with the computing network 125 and the storage device 130 via the network 120. The software 150 may be installed on the processor 170, or in another embodiment; the features and functions of the software may be integrated into the processor 170. The processor 170 may also be configured with the local storage device 140 for storing information used by the processor 170 for quick access without relying on information stored remotely on an external storage device near the user 108. In other embodiments, the processor 170 may be integrated within the head mounted system 160.

[0038] The user's physical environment 105 is the physical environment of the user 108 as the user moves around and views the user's physical environment 105 through the head mounted system 160. For example, referring to Figure 1, the user's physical environment 105 shows a room with two walls (e.g., a main wall 180 and a side wall 184, the main wall and the side wall are related to the user's viewing) and a table 188. On the main wall 180, there is a rectangular surface 182 depicted by a black solid line to show a physical surface with a physical boundary (e.g., a painting hanging or attached to a wall or window, etc.), which can be a candidate surface for projecting some 2D content onto it. On the side wall 184, there is a second rectangular surface 186 depicted by a black solid line to show a physical surface with a physical boundary (e.g., a painting hanging or attached to a wall or window, etc.). On the table 188, there can be different objects. 1) A virtual Rolodex 190 in which certain 2D content can be stored and displayed; 2) A horizontal surface 192 depicted by solid black lines to represent a physical surface with physical boundaries to project certain 2D content onto; 3) Multiple stacks of virtual square surfaces 194 depicted by dashed black lines to represent, for example, stacked virtual newspapers in which certain 2D content can be stored and displayed.

[0039] The web browser 110 may also display blog pages from the Internet or an intranet / private network. In addition, the web browser 110 may also be any technology that displays digital 2D content. The 2D content may include, for example, web pages, blogs, digital pictures, videos, news articles, newsletters, or music. The 2D content may be stored in a storage device 130 that the user 108 can access via the network 120. In some embodiments, the 2D content may also be streaming content, such as a live video feed or a live audio feed. The storage device 130 may include, for example, a database, a file system, a permanent storage device, a flash drive, a cache, etc. In some embodiments, a web browser 110 containing 2D content (e.g., a web page) is displayed via a computing network 125.

[0040] The computing network 125 accesses the storage device 130 to retrieve and store 2D content for display in a web page on the web browser 110. In some embodiments, the local storage device 140 can provide the 2D content of interest to the user 108. The local storage device 140 can include, for example, a flash drive, a cache, a hard drive, a database, a file system, etc. The information stored in the local storage device 140 can include recently accessed 2D content or content recently displayed in a 3D space. The local storage device 140 allows for improvements in the performance of the system of the environment 100 by locally providing certain content to the software 150 to help deconstruct the 2D content to display the 2D content on a 3D space environment (e.g., a 3D surface in the user's physical environment 105).

[0041] The software 150 includes a software program stored in a non-transitory computer-readable medium to perform the functions of deconstructing 2D content to be displayed in the user's physical environment 105. The software 150 may be executed on a processor 170, which may be locally attached to the user 108, or in some other embodiments, the software 150 and the processor 170 may be included in the head mounted system 160. In some embodiments, portions of the features and functions of the software 150 may be stored and executed on a computing network 125 remote from the user 108. For example, in some embodiments, deconstructing 2D content may occur on the computing network 125, and the results of the deconstruction may be stored in the storage device 130, wherein an inventory of surfaces of the user's local environment for presenting the deconstructed 2D content may occur in the processor 170, wherein the inventory of surfaces and mappings is stored in the local storage device 140. In one embodiment, the processes of deconstructing 2D content, inventorying local surfaces, mapping elements of the 2D content to local surfaces, and displaying elements of the 2D content may all occur locally in the processor 170 and the software 150.

[0042] The head mounted system 160 may be a virtual reality (VR) or augmented reality (AR) head mounted system that includes a user interface, a user sensing system, an environment sensing system, and a processor (all not shown). The head mounted system 160 presents an interface for interacting with and experiencing the digital world to the user 108. Such interactions may involve the user and the digital world, one or more other users connected to the environment 100, and objects located within the digital and physical worlds.

[0043] The user interface may include receiving 2D content through the user interface and selecting elements within the 2D content. The user interface may be at least one or a combination of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. A tactile interface device is a device that allows a person to interact with a computer through physical sensation and movement. Haptics refers to a type of human-computer interaction technology that includes tactile feedback or other physical sensations to perform actions or processes on a computing device. In some embodiments, the control interface may be a user interface so that a user can interact with the MR display system, for example, by providing user input to the system and the system responding by executing corresponding commands.

[0044] The user sensing system may include one or more sensors 162 operable to detect certain features, characteristics, or information related to the user 108 wearing the head mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical features / measurements of the user 108, such as one or more of pupil constriction / dilation, angle measurement / positioning of each pupil, sphericity, eye shape (change in eye shape over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that may be used by the head mounted system 160 to enhance the user's viewing experience.

[0045] The environment sensing system may include one or more sensors 164 for acquiring data from the user's physical environment 105. Objects or information detected by the sensors 164 may be provided as input to the head mounted system 160. In some embodiments, the input may represent the user's interaction with the virtual world. For example, a user (e.g., user 108) viewing a virtual keyboard on a table (e.g., table 188) may gesture with their fingers as if the user were typing on the virtual keyboard. The motion of the finger movement may be captured by the sensors 164 and provided as input to the head mounted system 160, where the input may be used to change the virtual world or create new virtual objects.

[0046] The sensor 164 may include, for example, a generally outward-facing camera or scanner for interpreting scene information, for example, by continuously and / or intermittently projecting infrared structured light. The environment sensing system may be used to map one or more elements of the user's physical environment 105 around the user 108 by detecting and registering the local environment (including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions). Thus, in some embodiments, the environment sensing system may include image-based 3D reconstruction software embedded in a local computing system (e.g., processor 170) and operable to digitally reconstruct one or more objects or information detected by the sensor 164.

[0047] In an exemplary embodiment, the environment sensing system provides one or more of the following: motion capture data (including gesture recognition), depth sensing, facial recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors. It should be understood that the environment sensing system may include other components in addition to those discussed above.

[0048] As described above, in some embodiments, processor 170 may be integrated with other components of head mounted system 160, integrated with other components of systems of environment 100, or may be a processor such as a computer. Figure 1 The processor 170 may be connected to the various components of the head mounted system 160 via a physical, wired connection or via a wireless connection, such as a mobile network connection (including cellular telephone and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. The processor 170 may include a storage module, an integrated and / or additional graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of transforming data from a source (e.g., a computing network 125 and from a user sensing system and an environmental sensing system of the head mounted system 160) into image and audio data, where the image / video and audio can be presented to the user 108 via a user interface (not shown).

[0049] Processor 170 handles data processing for various components of head mounted system 160 and data exchange between head mounted system 160 and 2D content from web pages displayed or accessed by web browser 110 and computing network 125. Processor 170 may be used to buffer and process the data stream between user 108 and computing network 125, thereby achieving a smooth, continuous, and high-fidelity user experience.

[0050] Deconstructing 2D content from a web page into elements and mapping the elements to be displayed on surfaces in a 3D environment may be accomplished in an intelligent and logical manner. A predetermined set of rules may be useful for recommending, suggesting, or indicating placement locations for certain types of elements / content identified within the 2D content / web page. For example, certain types of 2D content elements may have one or more content elements that may need to be mapped to a physical or virtual object surface suitable for storing and displaying the one or more elements, while other types of 2D content elements may be a single object, such as a main video or main article in a web page, in which case the single object may be mapped to the most meaningful surface to display the single object to the user.

[0051] Figure 2 An example mapping of elements of 2D content to a user's 3D environment is shown in accordance with some embodiments. Environment 200 depicts 2D content (e.g., a web page) displayed or accessed by a web browser 110 and a user's physical environment 105. Dashed lines with arrows depict elements (e.g., certain types of content) from 2D content (e.g., a web page) that are mapped to and displayed on the user's physical environment 105. Certain elements from the 2D content are mapped to certain physical or virtual objects in the user's physical environment 105 based on web designer hints or predefined browser rules.

[0052] As an example, the 2D content accessed or displayed by the web browser 110 may be a web page with multiple tabs, wherein the current active tab 260 is displayed and the secondary tab 250 is currently hidden until selected to be displayed on the web browser 110. Displayed within the active tab 260 is typically a web page. In this particular example, the active tab 260 is displaying a YOUTUBE page including a main video 220, user comments 230, and suggested videos 240. As shown in this exemplary Figure 2 , the primary video 220 may be mapped to be displayed on the vertical surface 182, the user comments 230 may be mapped to be displayed on the horizontal surface 192, and the suggested video 240 may be mapped to be displayed on a vertical surface 186 different from the vertical surface 182. In addition, the secondary tab 250 may be mapped to be displayed on the virtual Rolodex 190 and / or the multi-stacked virtual object 194. In some embodiments, specific content within the secondary tab 250 may be stored in the multi-stacked virtual object 194. In other embodiments, the entire content residing within the secondary tab 250 may be stored and / or displayed on the multi-stacked virtual object 194. Likewise, the virtual Rolodex 190 may contain specific content from the secondary tab 250, or the virtual Rolodex 190 may contain the entire content residing within the secondary tab 250.

[0053] The vertical surface 182 may be any type of structure that is already located on the main wall 180 of the room (depicted as the user's physical environment 105), such as a window pane or a picture frame. In some embodiments, the vertical surface 182 may be an empty wall on which the head mounted system 160 determines the optimal size of the frame of the vertical surface 182 suitable for the user 108 to view the main video 220. This determination of the size of the vertical surface 182 may be based at least in part on the distance of the user 108 from the main wall 180, the size and dimensions of the main video 220, the quality of the main video 220, the amount of uncovered wall space, and / or the posture of the user when viewing the main wall 180. For example, if the quality of the main video 220 is high definition, the size of the vertical surface 182 may be larger because the quality of the main video 220 will not be adversely affected by the vertical surface 182. However, if the video quality of the main video 220 is of poor quality, having a larger vertical surface 182 may greatly hinder the video quality, in which case the methods and systems of the present disclosure may resize / redefine the vertical surface 182 to be smaller to minimize the poor video quality from pixelation.

[0054] Like vertical surface 182, vertical surface 186 is a vertical surface on an adjacent wall (e.g., side wall 184) in the user's physical environment 105. In some embodiments, side wall 184 and vertical surface 186 may appear to be tilted, skewed surfaces based on the orientation of user 108. In addition to vertical and horizontal surfaces, tilted, skewed surfaces can also be a type of orientation of surfaces. In this example, a suggested video 240 from a YOUTUBE webpage can be placed on vertical surface 186 of side wall 184 to allow user 108 to view the suggested video by simply moving his head slightly to the right.

[0055] The virtual Rolodex 190 is a virtual object created by the head mounted system 160 and displayed to the user 108. The virtual Rolodex 190 may have the ability to bidirectionally cycle through a set of virtual pages for the user 108. The virtual Rolodex 190 may contain an entire web page, or may contain individual articles or videos or audio. As shown in this example, the virtual Rolodex 190 may contain a portion of the content from the secondary tab 250, or in some embodiments, the virtual Rolodex 190 may contain the entire page of the secondary tab 250. The user 108 may bidirectionally cycle through the content within the virtual Rolodex 190 by simply focusing on a specific tab within the virtual Rolodex 190, and one or more sensors (e.g., sensor 162) within the head mounted system 160 will detect the user 108 eye focus and cycle through the tabs in the virtual Rolodex 190 to obtain relevant information for the user 108 accordingly. In some embodiments, the user 108 can select relevant information from the virtual Rolodex 190 and instruct the head mounted system 160 to display the relevant information on an available surrounding surface or on another virtual object, such as a virtual display in close proximity to the user 108 (not shown).

[0056] Similar to the virtual Rolodex 190, the multi-stack virtual object 194 can contain the following: the complete content of one or more tabs or specific content from various web pages or tabs that the user 108 has marked, saved for future viewing, or has opened (i.e., inactive tabs). The multi-stack virtual object 194 is also similar to a real-world newspaper stack. Each stack within the multi-stack virtual object 194 can belong to a specific newspaper article, page, magazine issue, recipe, etc. It can be understood by those of ordinary skill in the art that there can be multiple types of virtual objects to achieve the same purpose, to provide a surface to place 2D content elements or content from a 2D content source.

[0057] It will be appreciated by those of ordinary skill in the art that the 2D content accessed or displayed by the web browser 110 may be more than just a web page. In some embodiments, the 2D content may be pictures from an album, videos from a movie, a TV show, a YOUTUBE video, an interactive form, and the like. However, in other embodiments, the 2D content may be an e-book or any electronic way of displaying a book. Finally, in other embodiments, since 2D content is typically the current way of presenting information, the 2D content may be other types of content that have not yet been described. If the electronic device can consume 2D content, the head-mounted system 160 can use the 2D content to deconstruct and display the 2D content in a 3D setting (e.g., AR).

[0058] In some embodiments, mapping the accessed 2D content may include extracting the 2D content (e.g., from a browser) and placing it on the surface (so that the content is no longer in the browser and is only on the surface), and in some embodiments, mapping may include copying the content (e.g., from a browser) and placing it on the surface (so that the content is both in the browser and on the surface).

[0059] Deconstructing 2D content is a technical problem existing in the field of Internet and computer-related technologies. 2D content such as web pages is constructed using certain types of programming languages ​​such as HTML to instruct computer processors and technical components where and how to display elements within the web page on the screen for the user. As described above, web page designers usually work within the limitations of a 2D canvas (e.g., a screen) to place and display elements (e.g., content) within the 2D canvas. HTML tags are used to determine how to format an HTML document or a portion within an HTML document. In some embodiments, the (extracted or copied) 2D content can maintain HTML tag references, and in some embodiments, the HTML tag references can be redefined.

[0060] Figure 3 3 is a flow chart illustrating a method for deconstructing 2D content to be displayed in a 3D environment according to some embodiments. The method includes: identifying 2D content at 310; identifying elements in the 2D content at 320; identifying surrounding surfaces at 330; mapping the identified elements in the identified 2D content to the identified surfaces from the identified surrounding surfaces at 340; and displaying the elements as virtual content on a selected surface at 350, wherein the selected surface is selected from the mapping of the elements to the identified surfaces.

[0061] Identifying 2D content at 310 may involve searching for digital content using the head mounted system 160. Identifying 2D content at 310 may also include accessing digital content on a server (e.g., storage device 130) connected to the network 120. Identifying 2D content at 310 may include browsing the Internet for web pages of interest to the user 108. In some embodiments, identifying 2D content at 310 may include a voice activated command given by the user 108 to search for content on the Internet. For example, the user 108 may be interacting with a device (e.g., head mounted system 160) where the user 108 requests the device to search for a specific video by speaking a command to search for videos followed by the name of the video and a brief description of the video, thereby searching for a specific video on the Internet. The device may then search the Internet and pull up the video on a 2D browser to allow the user 108 to view the video displayed on the 2D browser of the device. The user 108 may then confirm that the video is a video that the user 108 wants to view in the spatial 3D environment.

[0062] Once the 2D content is identified, the method then identifies elements within the 2D content at 320 to obtain a list of available elements in the 2D content for display to the user 108. For example, the elements within the 2D content may include videos, articles and newsletters posted on web pages, comments and posts on social media sites, blog posts, pictures posted on various websites, audiobooks, etc. These elements within the 2D content (e.g., web pages) may contain HTML tags with attributes associated with HTML tags provided by the content designer to define where specific elements are placed on the web page and in some cases when and how the elements are displayed on the web page. In some embodiments, the methods and systems of the present disclosure will utilize these HTML tags and attributes as hints and suggestions provided by the content designer to assist in the mapping process at 340 in order to determine where and how to display elements in the 3D setting. For example, the following is an example HTML web page code provided by a web page developer.

[0063] Sample HTML page code provided by the web developer

[0064]

[0065]

[0066] The example HTML web page code provided by the web page developer includes a preference for how to display the main video on the web page and a preference for how to display the recommendation (or suggested video). In particular, the HTML web page code uses the "style" tag to specify how to display the main video using the "vertical" type value to specify the vertical surface on which the video is displayed. In addition, within the "style" tag, other hints provided by the web page developer may include a "priority" preference for the matching algorithm, which is used to prioritize which HTML element / content in the web page (e.g., the main video) should be mapped to which potential surface area. In the example HTML web page code, for videos with a vertical plane layout, the priority is set to a value of 100, where, in this example, a higher priority value indicates a higher priority. In addition, in this example, a preference is indicated by the web page developer to place the suggested video in a stack with a "horizontal" type value in a stacked layout, where the distance between stacked objects (e.g., in this case, a suggested video related to another suggested video) should be 20 cm.

[0067] Figure 4 is a flow chart illustrating a method for identifying elements in 2D content according to some embodiments. Figure 4 According to some embodiments disclosed in Figure 3 Detailed process of identifying elements in 2D content at 320 . Figure 4 Beginning with identifying elements within 2D content at 410, similar to Figure 3 At 320, elements within the 2D content are identified. The method proceeds to the next step of identifying attributes from the tags related to the placement of the content at 420. As described above, when designing and configuring a web page, a web page designer can associate elements within the web page with HTML tags to define where and how each element is displayed. These HTML tags can also include attributes related to placing the elements on specific parts of the web page. The head mounted system 160 will detect these HTML tags and their attributes and coordinate with other components of the system to use as input regarding the location where a specific element can be displayed.

[0068] Extraction of hints or tags from each element is performed at 430. Hints or tags are typically formatting hints or formatting tags provided by a content designer of the 2D content / web page and / or a web page developer. As described above, the content designer may provide instructions or hints, for example in the form of HTML tags as shown in the "Sample HTML web page code provided by the web page developer", to instruct the web browser 110 to display elements in the 2D content in a specific portion of the page or screen. In some embodiments, the web page designer may define additional formatting rules using additional HTML tag attributes. For example, if the user has a reduced sensitivity to a particular color (e.g., red), red is not displayed but other colors are used, or as described above, if a video with a preference to be displayed on a vertical surface cannot be displayed on a vertical surface, the video is instead displayed on another (physical) surface or a virtual surface is created and displayed on the virtual surface. The following is an example HTML page parser implemented in a browser for parsing an HTML page to extract hints / tags from each element within the HTML page.

[0069] Example HTML page parser implemented in a browser

[0070]

[0071]

[0072]

[0073]

[0074] The example HTML page parser shows how to parse and identify and / or extract / copy an HTML page containing HTML tags that are used to provide display preferences for specific elements / objects within 2D content (e.g., a web page). As disclosed in the example HTML page parser, the disclosed example code can be used to parse elements within 2D content (e.g., a web page). The HTML page parser (e.g., ML.layout, ML.container, etc.) can identify / extract certain HTML tags that use various element names and values ​​to determine how to display specific elements to a user in a 3D environment (e.g., by mapping the element to a specific surface).

[0075] A lookup / search is performed at 440 for alternative display forms for one or more elements. Certain formatting rules may be specified for images on a web page. For example, if the web browser 110 is capable of displaying a 3D version of an image, the web page designer may place additional tags or define certain attributes of a particular tag to allow the web browser 110 to identify images that may have alternative versions of the image (e.g., a 3D version of the image). The web browser 110 may then access the alternative version of the image (e.g., a 3D version of the image) to be displayed in the 3D-enabled browser.

[0076] Storing the identified elements within the 2D content is performed at 450. The method may store the identified elements in a non-transitory storage medium for use by a mapping routine (e.g., in Figure 3 The non-transitory storage medium may include a data storage device such as storage device 130 or local storage device 140. The element may be stored in a storage device such as described below. Figure 5 In some embodiments, the identified elements within the 2D content may be stored in a temporary storage medium.

[0077] Figure 5 5 shows an example of a table for storing elements deconstructed from 2D content according to some embodiments. Element table 500 is an exemplary table that can store Figure 4The result of identifying the element within the 2D content at 410 is stored in the database. The element table 500 includes, for example, information about one or more elements within the 2D content, including an element identification (ID) 510, a preference indicator 520 for a location where the element can be placed on a 3D surface, a parent element ID 530 if the particular element is included within a parent element, a child element ID 540 if the element can contain a child element, and a plurality of entity indicators 550 to indicate whether the element contains multiple implementations that can ensure that the surface or virtual object used to display the element is compatible with multiple versions of the display element. The parent element is an element / object within the 2D content that can contain child elements (e.g., child elements). For example, an element ID having a value of 220 (e.g., main video 220) has a parent element ID value of 260 (e.g., active tab 260), which indicates that the main video 220 is a child element of the active tab 260. Or stated differently, the main video 220 is included within the active tab 260. Continuing with the same example, the main video 220 has a sub-element ID 230 (e.g., user comment 230) indicating that the user comment 230 is associated with the main video 220. It will be appreciated by those skilled in the art that the element table 500 may be a table in a relational database or any type of database. In addition, the element table 500 may be an array in a computer memory (e.g., a cache) containing Figure 4 At 410 , the result of identifying elements within the 2D content.

[0078] Each of the rows 560 of the element table 500 corresponds to an element within the web page. Element ID 510 is a column that contains a unique identifier (e.g., element ID) for each element. In some embodiments, the uniqueness of an element may be defined as a combination of the element ID 510 column and another column within the table (e.g., the preference 520 column if the content designer has identified more than one preference). Preference 520 is a column whose values ​​may be based at least in part on a number of preferences defined by the content designer / developer (e.g., web page designer) and defined by the content designer / developer. Figure 4 At 430, hints or tags are extracted from each element to determine the HTML tags and attributes identified in the disclosed systems and methods. In other embodiments, the preferences 520 column can be determined based at least in part on predefined browser rules to specify where certain types of elements within a web page should be displayed in the 3D environment. These predefined rules can provide suggestions to the system and method to determine where to best place the elements in the 3D environment.

[0079] Parent Element ID 530 is a column that contains the element ID of the parent element in which the particular element in the current row is displayed or is related to the parent element. A particular element within a web page may be embedded, placed within, or related to another element on the page. For example, in the current embodiment, the first entry in the Element ID 510 column stores the ID of the parent element. Figure 2 The preference value in the preference 520 column corresponding to the primary video 220 is determined based on the HTML tags and / or attributes, and in the current embodiment, is the "primary" location where the element should be placed in the user's physical environment 105. Depending on the current location of the user 108, that primary location could be a wall in a living room, or a stove top hood in a kitchen that the user 108 is currently looking at, or if in a wide open space, a virtual object projected in front of the user's field line where the primary video 220 can be projected. More information about how elements of 2D content are displayed to the user 108 will be disclosed in a later section. Continuing with the current example, the parent element ID 530 column stores the value of the element ID 220 corresponding to the primary video 220. Figure 2 The value of the element ID 260 corresponding to the active tab 260. Therefore, the main video 220 is a child of the active tab 260.

[0080] Subelement ID 540 is a column that contains the element ID of the subelement within which the particular element in the current row has been displayed or is related to. A particular element within a web page may be embedded, placed within, or related to another element on the page. Continuing with the current example, the Subelement ID 540 column stores the element ID of the subelement within which the particular element in the current row has been displayed or is related to. Figure 2 The value of the element ID 230 corresponding to the user comment 230 .

[0081] The multiple entity indicator 550 is a column that indicates whether the element contains multiple entities, which may ensure that the surface or virtual object used to display the element is compatible with multiple versions of the display element (e.g., the element may be a user comment 230, where more than one comment may be available for the primary video 220). Continuing with the current example, the multiple entity indicator 550 column stores a value of "no" to indicate that the primary video 220 does not have or does not correspond to multiple primary videos in the active tab 260 (e.g., "no" multiple versions of the primary video 220).

[0082] Continuing with the current example, the second entry in the element ID 510 column stores the Figure 2The preference value corresponding to the user comment 230 in the preference 520 column shows a preference of "horizontal" to indicate that the user comment 230 should be placed on a "horizontal" surface somewhere in the user's physical environment 105. As described above, the horizontal surface will be determined based on the available horizontal surfaces in the user's physical environment 105. In some embodiments, the user's physical environment 105 may not have a horizontal surface, in which case the systems and methods of the present disclosure may identify / create a virtual object with a horizontal surface to display the user comment 230. Continuing with the current example, the parent element ID 530 column stores the value of the element ID 230 corresponding to the user comment 230. Figure 2 The value element ID 220 corresponding to the main video 220, and the multiple entity indicator 550 column stores a value of "yes" to indicate that the user comment 230 can contain more than one value (eg, more than one user comment).

[0083] The remaining rows within the element table 500 contain information for the remaining elements that are of interest to the user 108. One of ordinary skill in the art will appreciate that storing the results of identifying elements within the 2D content at 410 can improve the functionality of the computer itself because once the analysis has been performed on the 2D content, the system and method can retain that information for future analysis of the 2D content if another user is interested in the same 2D content. The system and method for deconstructing that particular 2D content can be avoided because it has already been done before.

[0084] In some embodiments, the element table 500 may be stored in the storage device 130. In other embodiments, the element table 500 may be stored in the local storage device 140 for quick access to recently viewed 2D content or for possible re-access to recently viewed 2D content. In other embodiments, the element table 500 may be stored in both the storage device 130 remote from the user 108 and the local storage device 140 local to the user 108.

[0085] return Figure 3, the method continues at 330 to identify surrounding surfaces. The user 108 may view the user's physical environment 105 through the head mounted system 160 to allow the head mounted system 160 to capture and identify surrounding surfaces, such as walls, tables, paintings, window frames, stoves, refrigerators, televisions, etc. The head mounted system 160 is aware of real objects within the user's physical environment 105 due to sensors and cameras located on the head mounted system 160 or on any other type of similar device. In some embodiments, the head mounted system 160 may match real objects observed within the user's physical environment 105 with virtual objects stored in the storage device 130 or the local storage device 140 to identify surfaces that can be used for these virtual objects. Real objects are objects recognized within the user's physical environment 105. Virtual objects are physical objects that do not exist within the user's physical environment, but can be displayed to the user to make it appear as if the virtual objects exist in the user's physical environment. For example, the head mounted system 160 may detect an image of a table within the user's physical environment 105. The image of the table can be simplified into a 3D point cloud object for fast and efficient comparison and matching at the storage device 130 or the local storage device 140. If a match between a real object and a 3D point cloud object (e.g., of a table) is detected, the system and method will recognize the table as having a horizontal surface because the 3D point cloud object representing the table is defined as having a horizontal surface. Figure 6 A more detailed description of identifying surrounding surfaces is disclosed in .

[0086] In some embodiments, the virtual object may be an extracted object, where the extracted object may be a physical object identified within the user's physical environment 105, but displayed to the user as a virtual object located at the location of the physical object so that additional processing and associations may be performed on the extracted object that would not be possible on the physical object itself (e.g., to change the color of the physical object to highlight a particular feature of the physical object, etc.). Additionally, the extracted object may be a virtual object extracted from 2D content (e.g., a web page from a browser) and displayed to the user 108. For example, the user 108 may select an object such as a sofa from a web page that is displayed on the 2D content / web page to be displayed in the user's physical environment 105. The system may identify the selected object (e.g., sofa) and display the extracted object (e.g., sofa) to the user 108 as if the extracted object (e.g., sofa) physically existed in the user's physical environment 105. In addition, virtual objects may also include objects having surfaces for displaying content (for example, a transparent display screen close to the user for viewing certain content). These surfaces not only do not physically exist in the user's physical environment 105, but also can be ideal display surfaces for presenting certain content to the user from the perspective of the displayed content in the 2D content.

[0087] Figure 6 is a flow chart illustrating a method for identifying a surface from a user's local environment according to some embodiments. Figure 6 It is publicly available Figure 3 Detailed process of identifying surrounding surfaces at 330 . Figure 6 Beginning with identifying the user's current surroundings at 610, similar to Figure 3 The surrounding surface is identified at 330. The method proceeds to the next step of determining the user's posture at 620.

[0088] Determining the user's posture at 620 is an important step in identifying the user's current surroundings because the user's posture will provide the user 108 with a perspective on objects within the user's physical environment 105. For example, referring to Figure 1 , a user 108 using the head mounted system 160 is viewing the user's physical environment 105. Determining the user's pose (i.e., vector and / or origin position information relative to the world) at 620 helps the head mounted system 160 understand, for example: (1) how high the user 108 is relative to the ground, (2) the angle at which the user 108 must rotate their head to move around and capture an image of the room, and (3) the distance between the user 108 and the table 188, main wall 180, and side wall 184. Additionally, the user's 108 pose helps determine the angle at which the head mounted system 160 is viewing vertical surfaces 182 and 186, as well as other surfaces within the user's physical environment 105.

[0089] At 630, the method identifies the size of the surrounding surfaces. Each candidate surface within the user's physical environment 105 is labeled and categorized by a corresponding size. In some embodiments, each candidate surface within the user's physical environment 105 is also labeled and categorized by a corresponding orientation. This information will help identify which elements should be mapped to which surfaces based at least in part on the size of the surface, the orientation of the surface, the distance of the user 108 from a particular surface, and the type of information that needs to be displayed for the element. For example, a video may be displayed farther away than a blog or article, which may contain a large amount of information, where the text size of the article may be too small for the user to see if it is displayed on a distant wall with a small size.

[0090] At 640, the method stores the list of surrounding surfaces in a non-transitory storage medium for use by a mapping routine (e.g., mapping elements to Figure 3 The non-transitory storage medium may include a data storage device such as storage device 130 or local storage device 140. The identified surface may be stored in a storage device such as described below. Figure 7In some embodiments, the identified surfaces may be stored in a temporary storage medium.

[0091] Figure 7 An example of a table for storing a list of surfaces identified from a user's local environment is shown in accordance with some embodiments. Surface table 700 is an exemplary table in which the results of the process of identifying surrounding surfaces may be stored in a database. Surface table 700 includes, for example, information about surfaces within the user's physical environment 105, having data columns including surface ID 710, width 720, height 730, orientation 740, real or virtual indicator 750, quantity (multiple) 760, and location 770. It will be appreciated by those of ordinary skill in the art that surface table 700 may be a table in a relational database or any type of database. Additionally, surface table 700 may be a table stored in a database. Figure 3 At 330 , the results of identifying surrounding surfaces are stored in an array in a computer memory (eg, a cache).

[0092] Each of the rows 780 of the surface table 700 may correspond to a surface from the user's physical environment 105 or a virtual surface that may be displayed to the user 108 within the user's physical environment 105. The surface ID 710 is a column that contains a unique identifier (e.g., a surface ID) that uniquely identifies a particular surface. The dimensions of the particular surface are stored in the width 720 and height 730 columns.

[0093] Orientation 740 is a column that indicates the orientation of the surface relative to the user 108 (e.g., vertical, horizontal, etc.). Real / virtual 750 is a column that indicates whether a particular surface is located on a real object within the user's physical environment 105, as perceived by the user 108 using the head mounted system 160, or whether the particular surface is located on a virtual object generated by the head mounted system 160 and displayed in the user's physical environment 105. The head mounted system 160 may have to generate virtual objects for situations where the user's physical environment 105 may not contain enough surfaces to display a certain amount of content that the user 108 wishes to display. In these embodiments, the head mounted system 160 may search from a database of existing virtual objects that may have appropriate surface sizes to display certain types of elements identified for display. The database may be from the storage device 130 or the local storage device 140.

[0094] Quantity 760 is a column that indicates whether the surface / object is compatible with multiple versions of the display element (e.g., the element may be Figure 2250, where there may be more than one secondary (i.e., inactive) tab for a particular web browser 110 (e.g., one web page per tab). If the quantity 760 column has a value of "multiple" (such as the case where the fourth entry of the surface ID column stores the same Figure 2 The virtual Rolodex 190 corresponds to the value of 109, and the fifth entry of the surface ID column stores the value Figure 2 ), the systems and methods will know that if there is an element (which can have multiple versions of that element), such as is the case for an inactive tab, then these are types of surfaces that can accommodate multiple versions.

[0095] Position 770 is a column that indicates the location of the physical surface relative to a reference frame or reference point. Figure 7 As shown in the column heading of position 770 in , the position of the physical surface can be predetermined as the center of the surface. In other embodiments, the position can be predetermined as another reference point of the surface (e.g., the front, back, top, or bottom of the surface). The position information can be represented as a vector and / or position information from the center of the physical surface relative to a reference frame or reference point. There can be several ways to represent the position in surface table 700. For example, the value of the position of surface ID 194 in surface table 700 is represented in an abstract form to illustrate the vector information and reference frame information (e.g., a "frame" subscript). x, y, z are 3D coordinates in each spatial dimension, and frame indicates which reference frame the 3D coordinates are relative to.

[0096] For example, surface ID 186 shows that the position of the center of surface 186 is (1.3, 2.3, 1.3) relative to the real world origin. As another example, surface ID 192 shows that the position of the center of surface 192 is (x, y, z) relative to the user reference system, and surface ID 190 shows that the position of the center of surface 190 is (x, y, z) relative to another surface 182. The reference system is important for eliminating the ambiguity of which reference system is currently being used. In the case where the origin of the real world is used as the reference system, it is usually a static reference system. However, in other embodiments, when the reference system is a user reference system, the user may be a mobile reference system, in which case, if the user is moving and the user reference system is used as the reference system, the plane (or vector information) may move with the user and change. In some embodiments, the reference system for each surface may be the same (e.g., the user reference system). In other embodiments, the reference system for the surface stored in the surface table 700 may be different depending on the surface (e.g., the user reference system, the world reference system, another surface or object in the room, etc.).

[0097] In the current example, the values ​​stored in the surface table 700 are contained in Figure 2 Physical surfaces (e.g., vertical surfaces 182 and 186 and horizontal surface 192) and virtual surfaces (e.g., virtual Rolodex 190 and multi-stacked virtual objects 194) identified within the user's physical environment 105. For example, in the current embodiment, the first entry of the surface ID 710 column stores the Figure 2 The value of surface ID 182 corresponding to the vertical surface 182. The width value in the width 720 column and the height value in the height 730 column, which correspond to the width and height of the vertical surface 182, respectively, indicate that the vertical surface 182 has a size of 48" (width) by 36" (height). Similarly, the orientation value in the orientation 740 column indicates that the vertical surface 182 has a "vertical" orientation. In addition, the real / virtual value in the real / virtual 750 column indicates that the vertical surface 182 is an "R" (e.g., real) surface. The quantity value in the quantity 760 column indicates that the vertical surface 182 is "single" (e.g., can only accommodate a single content). Finally, the position 770 column indicates the position of the vertical surface 182 relative to the user 108 using vector information of (2.5, 2.3, 1.2).

[0098] The remaining rows in the surface table 700 contain information about the remaining surfaces in the user's physical environment 105. It will be appreciated by those skilled in the art that the information stored in the surface table 700 is not necessarily the same as that stored in the surface table 700. Figure 3 The result of identifying the surrounding surfaces at 330 improves the functionality of the computer itself because once this analysis has been performed on the surrounding surfaces, if another user or the same user 108 is located in the same physical environment 105 but is interested in different 2D content, the analysis can be retained by the head mounted system 160 for future analysis of the user's surrounding surfaces. The processing steps of identifying the surrounding surfaces at 330 can be avoided because these processing steps have been completed previously. The only difference can include that the element table 500 based at least in part on identifying elements within different 2D content can be used to identify additional or different virtual objects.

[0099] In some embodiments, the surface table 700 is stored in the storage device 130. In other embodiments, the surface table 700 is stored in the local storage device 140 of the user 108 for quick access to recently viewed 2D content or possible re-access to recently viewed 2D content. In other embodiments, the surface table 700 may be stored in both the storage device 130 remote from the user 108 and the local storage device 140 local to the user 108.

[0100] return Figure 3, the method continues by mapping the element to the identified surface at 340 using a combination of the identified element from identifying the element in the 2D content at 320 and the identified surrounding surface from identifying the surrounding surface at 330 and, in some embodiments, using the virtual object as an additional surface. Mapping the identified element to the identified surface may involve multiple factors, some of which may include analyzing hints provided by the 2D content designer / author via HTML tag elements defined by the 2D content designer / author using an HTML page parser such as the example HTML page parser discussed above. Other factors may include selecting from a set of predefined rules provided by the AR browser, AR interface, and / or cloud storage on how and where to map certain 2D content. Figure 8 A detailed flow of a mapping process for mapping one or more elements from 2D content to an identified surface is provided.

[0101] Figure 8 Depicted is a flow chart illustrating a method for mapping elements from 2D content to a surface in accordance with some embodiments. Figure 8 It is publicly available Figure 3 Detailed process of mapping elements to the identified surface at 340 .

[0102] At 810, the method determines whether the identified element contains a hint provided by a 2D content designer. When the 2D content designer initially designs the 2D content, the 2D content designer may provide hints about where to best display a particular element. For example, Figure 2The main video 220 can be a YOUTUBE video displayed on a web page within the active tab 260. The 2D content designer (e.g., a web page designer) can provide a hint to indicate that the main video 220 is best displayed on a vertical plane in the direct view of the user 108. In some embodiments, this can be achieved by using existing HTML tag elements originally designed for 2D web page content to further define the way in which specific elements within the 2D content are displayed when a 3D display environment is available. As another example, a 2D content designer can provide a hint that indicates that for a specific web page, 3D images are available instead of 2D images. For example, in the case of a 2D image, in addition to providing basic HTML tags to identify the source of the 2D content, the 2D content designer can also provide other less commonly used HTML tags to identify the source of the 3D version of the 2D image, and in addition, a hint is provided: if a 3D version of the image is used, it is highlighted in front of the user's view (e.g., in the main frame of the 3D layout). In some embodiments, the 2D content designer may provide this additional "hint" to the 3D image position of the 2D image only if the web browser 110 that renders the 2D content may have 3D display capabilities to take advantage of the enhanced 3D image. It will be appreciated by those skilled in the art that, in addition to what has been disclosed herein, the 2D content designer may provide many other ways to provide hints about where specific content elements should be placed in the 2D layout, and these are some examples of different ways in which the 2D content designer may provide hints to optimally display some or all elements within the 2D content.

[0103] In another embodiment, the HTML tag standard may include the creation of new HTML tags or similar markup languages ​​for providing hints for placement of 3D objects in the user's surrounding environment for an AR / VR specific type of browser, such as the example HTML web pages provided by the web page developers discussed above. As of this writing, these new HTML tags have not yet been created and / or adopted as standard tags within the HTML language. However, once the HTML standard includes these types of additional tags, certain embodiments of the current methods and systems will utilize these new tags to further provide a mapping of identified elements to identified surfaces. It will be appreciated by those skilled in the art that there are many other languages ​​other than HTML tags that may be modified or adopted to further provide hints as to how content elements should be best displayed in a 3D environment, and that the new HTML tag standard is merely one method to achieve such a goal.

[0104] At 820, the method determines whether to use hints provided by the 2D content designer or to use a predefined set of rules to map one or more content elements from the 2D content to certain types of 3D surfaces. In some embodiments, in the absence of hints for a particular content element provided by the 2D content designer, the system and method may determine the best way to map the content element to the surface using a predefined set of rules. In other embodiments, even when there may be hints for the placement of content elements provided by the 2D content designer, the system and method may determine that the content elements may be best mapped to the surface using a predefined set of rules. However, in other embodiments, the system and method may determine that the hints provided by the 2D content designer are sufficient, and therefore use these hints to map the content elements to the surface. Finally, the final decision of the AR browser is to determine whether to use the hints provided by the 2D content designer or to use predefined rules to map the content elements to the surface.

[0105] At 830, assuming that it is determined that using a hint provided by a 2D content designer is the method of choice, the method analyzes the hint and searches a list of identified surrounding surfaces that can be used to display a particular content element based at least in part on the hint (e.g., querying the surface table 700). At 840, the method runs a best fit algorithm to select a best fit surface for the particular content element based on the hint provided. For example, the best fit algorithm may make a hint of "primary content" for a particular content element within a particular web page and attempt to identify a 3D surface from the available identified surrounding surfaces that is front and center relative to the user 108 in the 3D environment. For example, Figure 2 The main video 220 is mapped to the vertical surface 182 because the main video 220 has Figure 5 The preference value of “Primary” in the preference 520 column of the element table 500 , and the vertical surface 182 is the surface that the user 108 looks directly at and has an optimal size to display the main video 220 .

[0106] At 850, the method stores the mapping results for the content elements in the mapping of the element to the surface table in a non-transitory storage medium for use by the display algorithm to display the content elements on their respective mapped surfaces, whether these surfaces are recognized surrounding surfaces or virtual objects displayed in the user's surrounding environment. The non-transitory storage medium may include a data storage device such as the storage device 130 or the local storage device 140. The mapping results may be stored in a storage device such as described below. Fig. 9 In a specific table of the tables disclosed in.

[0107] Fig. 9An example of a table for storing mappings of content elements from 2D content to surfaces according to some embodiments is shown. Mapping table 900 is an exemplary table that stores the results of a process in which content elements are mapped to surfaces in a database. Mapping table 900 includes, for example, information about content elements (e.g., element IDs) and surfaces to which the content elements are mapped (e.g., surface IDs). It will be appreciated by those of ordinary skill in the art that mapping table 900 may be a table stored in a relational database or any type of database or storage medium. In addition, mapping table 900 may be an array in a computer memory (e.g., a cache) that contains information about the content elements (e.g., element IDs) and the surfaces to which the content elements are mapped (e.g., surface IDs). Figure 3 The result of mapping the 340 elements to the identified surrounding surfaces.

[0108] Each row in mapping table 900 corresponds to a content element that is mapped from the 2D content to a surface in the user's physical environment 105 or a virtual object displayed to user 108, where the virtual object appears to be an object located in the user's physical environment 105. For example, in the current embodiment, the first entry in the element ID column stores the value of element ID 220 corresponding to primary video 220. The surface ID value corresponding to primary video 220 in the surface ID column is 182, which corresponds to vertical surface 182. In this manner, primary video 220 is mapped to vertical surface 182. Similarly, user comments 230 are mapped to horizontal surface 192, suggested videos 240 are mapped to vertical surface 186, and secondary tabs 250 are mapped to virtual Rolodex 190. The element IDs in mapping table 900 may be mapped to the elements stored in the element ID column. Figure 5 The surface ID in the mapping table 900 can be associated with the element ID in the element table 500. Figure 7 The surface ID in the surface table 700 is associated with the surface.

[0109] Back to Figure 8 At 860, assuming that using predefined rules is the method of choice, the method queries a database containing content element to surface mapping rules and determines which type of surface should be considered for mapping the content element for a particular content element within the web page. Figure 2The rules returned by the primary video 220 may indicate that the primary video 220 should be mapped to a vertical surface, and thus after searching the surface table 700, multiple candidate surfaces are displayed (e.g., vertical surfaces 182 and 186 and the virtual Rolodex 190). At 870, the predefined rule group can run a best fit algorithm to select which surface is the best fit for the primary video 220 from the available candidate surfaces. Based at least in part on the best fit algorithm, due to the totality of the candidate surfaces, it is determined that the primary video 220 should be mapped to the vertical surface 182, the vertical surface 182 is a surface within the direct line of sight of the user 108, and the vertical surface 182 has an optimal size for displaying the video. Once the mapping of one or more elements is determined, at 850, the method stores the mapping results for the content elements in the mapping of the element to the surface table in a non-transitory storage medium as described above.

[0110] Return to Figure 3 , the method continues with displaying one or more elements as virtual content onto the mapped surface at 350. The head mounted system 160 may include one or more display devices, such as a micro projector (not shown) within the head mounted system 160 to display information. As mapped at 340, the one or more elements are displayed on the corresponding mapped surface. Using the head mounted system 160, the user 108 will see the content on the corresponding mapped surface. It will be appreciated by those of ordinary skill in the art that the content elements are displayed to appear to be physically attached to various surfaces (physical or virtual), but in reality, as perceived by the user 108, the content elements are actually projected onto the physical surface, and in the case of virtual objects, the virtual objects are displayed to appear to be attached to the respective surfaces of the virtual objects. It will be appreciated by those of ordinary skill in the art that when the user 108 turns their head or looks up or down, the display devices within the head mounted system 160 may continue to attach the content elements to their respective surfaces to further provide the user 108 with the perception that the content is attached to the mapped surface. In other embodiments, the user 108 may change the content of the user's physical environment 105 through movements made by the user's 108 head, hands, eyes, or voice.

[0111] Improved browser / app implementation

[0112] In a mixed reality system, the user's workspace is not limited by the size of the display screen. Therefore, unlike a conventional browser, a browser window in a mixed reality system can be placed and retained anywhere in the user's environment. The problem is that conventional browser technology is configured to assume that the displayable browser position must be constrained within the boundaries of the display screen.

[0113] The following portion of the disclosure is directed to an improved method for viewing windows in a mixed reality environment. Using a mixed reality device, a user can have multiple browser windows that are associated with and placed in the user's physical space. For example, a user can open a first browser window in a first room and a second browser window in a second room. The problem solved by this portion of the disclosure relates to a situation where a browser window is opened so that it is anchored to a location in a first location so that the browser window is no longer visible when the user goes to the second location. The problem is that as the user changes environments (e.g., moves between rooms or goes to a different geographic location), the user may still need to access his / her previous session in a previous geographic location.

[0114] Fig.10 A flow chart of a method for implementing viewing of a user's window is shown, regardless of the user's current position relative to the position of one or more previously opened windows. In some embodiments, a control interface is provided to select the display of all and / or multiple windows associated with the user. In some embodiments, the control interface may be a user interface so that the user can interact with the MR display system, for example, by providing user input to the system and the system responding by executing corresponding commands. In some embodiments, the user can interact with the visual, auditory, tactile or other aspects of the MR system. In some embodiments, the user interface may include a browser hub, which in some embodiments may be a visual representation of one or more aspects of one or more browser applications. For example, an "all windows" icon may be presented within the browser hub, where the "all windows" icon may be selected to initiate the display of multiple windows associated with the user, regardless of the user's position relative to the current window position (e.g., the position where the window is opened). Fig.10 Beginning at step 1702, when the system receives a command to display all or multiple windows (1702). In some embodiments, step 1702 may occur when a user selects an all windows icon that may be located within a browser hub user interface. In some embodiments, the system receives a selection for more than one window. In some embodiments, the system may receive user input indicating that the user wants to view more than one window associated with the user's system.

[0115] At 1704, information is retrieved for multiple windows associated with the user. In some embodiments, a user may have one or more windows associated with the user. The windows for which information is collected may be located in different physical locations. According to some embodiments, instead of managing browser windows in a VR / AR environment independently on a one-to-one basis by each application, windows may instead be rendered as bounded volumes, which may be referred to as "prisms" hereinafter. Each prism may have properties and attributes that allow a universe application to manage and display the prism in a VR / AR environment, so that the universe application can manage the placement and display of virtual content in the VR / AR environment by managing the prism itself. Information about windows can be collected by accessing a database of prisms associated with a user, where a prism can display one or more windows at a specified location.

[0116] In some embodiments, an "all windows" view is loaded, showing all open windows and tabbed windows, each represented by a preview, icon, domain name and / or page title or any other suitable visual representation of the window (1706). In some embodiments, examples of open windows include windows actively interacted with by one or more users. Other examples include placed applications / windows / browsers, whether they have an open / active state, a paused state, a stopped state, a closed state, etc. In addition, as long as an instance of an application exists / is placed and has one or more tags with content, in some embodiments, it can be remotely accessed using the current inventive method. As another example, an open window can correspond to some or all prisms associated with a given application (e.g., a browser), regardless of its state (active, paused, closed, etc.), which can be remotely accessed through the "all windows" view in the current embodiment. In some embodiments, the "all windows" view may include all browser windows contained in one or more prisms at one or more physical locations in the real world. Figure 12-14 Examples of "all windows" and similar "all applications" views are shown and described below. Although "all windows" is used as an example, any other single application may be used instead. Although "all applications" is used as an example, any subset of all applications may be used instead. The individual windows identified in step 1704 can be displayed at the user's current location in this manner. This can be done by changing the position parameters of the identified window to the position in the user's current physical environment, in effect summoning the window to the user. In some embodiments, this can be achieved by creating a copy of the window information and instead associating a new position with the information, such as a position at or near the user's current position. The window is then rendered (in preview form, thumbnail form, and / or full form) and displayed to the user at the coordinates assigned to the individual windows and / or window prisms.

[0117] At 1708 (optional in this method), a hover state can be identified and the hover state can act relative to one or more windows. For example, when hovering, the window being hovered can be moved into the foreground, and other windows can optionally retreat slightly. A window with multiple tabs can be slightly expanded to display a background tab. In some embodiments, instead of a window, the object being hovered can be any visual representation of a browser window, such as a preview, a full screen, or a reduced screen. At 1710, the user selects one or more of the windows. In some embodiments, the user can select a window by clicking a button on a controller (e.g., a totem), or by performing a specific gesture, or by looking at the window for a predetermined period of time. If the user selects a window, a copy of the original window will be loaded in the foreground of the user's FOV, and all window views will be closed. In some embodiments, depending on the preferences selected by the user, the copy repeatedly updates the original, the copy updates all or some other copies, and / or the copy is independent of the original. In some embodiments, the content loaded in the foreground corresponds to an existing prism that is moved (e.g., not pinned and moved as a whole). In some embodiments, the content loaded in the foreground corresponds to an existing prism that has been copied with new associated position information. If the user activates the context menu, a user menu may be presented to the user that includes options for closing the window, adding it to a collection, and / or minimizing the window. The context menu may be a user interface with predetermined user interface options that tell the system to perform a specific function when selected. In some embodiments, the context menu may be activated by applying pressure to the center of the touch pad on the totem while hovering over a selectable object such as a window. In some embodiments, the context window may be similar to a right click on a desktop computer, in that the action enables the user to perform actions such as moving, closing, etc., on the selected object.

[0118] Fig.11A -B shows this process of displaying a window to a user, regardless of the window's previous physical location. In a mixed reality embodiment, a window can be associated with a device and / or a physical space. A user can place content throughout their home or at different geographic locations throughout the day. Fig.11A In FIG. 1 , it can be seen that a first browser window 1 has been placed at a first physical location, and a second browser window 2 has been placed at a second physical location. Since windows are associated with specific physical locations / coordinate spaces in mixed reality embodiments, this means that window 1 is generally only visible when user 108 is at physical location 1, and is not visible when user 108 is at physical location 2. Similarly, window 2 is generally only visible when user 108 is at physical location 2, and is not visible when user 108 is at physical location 1.

[0119] like Fig. 11B As shown, the "all windows" view 1804 allows the user 108 to view, reopen and close open windows, regardless of physical location (for examples of "open" windows, see the previous paragraph). Therefore, view 1804 can display manipulable versions (e.g., visual representations) of window 1 and window 2, although these windows are associated with different physical locations. When accessed from the control hub of the browser, viewing all windows (or optionally "all windows") allows the user to view all open windows, regardless of their physical location or geographic location. The windows can be located in the same room, different rooms, or completely in another space. Screenshots, website icons, domains, and / or page titles are used to identify (e.g., visually represent) each window. In some embodiments, windows with multiple tabs display a stacked preview of the underlying tabs when hovering. Using the context menu, the user can open a new instance of the window, close the window, minimize the window, mark the window, and add the window to the collection - regardless of the location. A global button can also be provided that can be used to close or minimize all open windows.

[0120] Figure 12-13 Examples of possible methods for displaying multiple windows within a mixed reality interface are provided. These figures illustrate example methods for implementing an interface in which multiple windows are displayed and presented to a user. Any window in the browser window may be selected for further viewing by the user via a suitable user input device, such as a pointing device. If there are too many windows to fit on the interface, then in some embodiments, the additional windows may be visually "ghosted" (e.g., Fig.12 and Fig.13 ), and provides scroll controls to scroll to the additional window.

[0121] Thus, an improved method of viewing windows in a mixed reality environment has been described, wherein a view is provided having a user's windows regardless of the user's current location relative to one or more previously opened windows. This addresses and solves situations where a user may want to access one or more browser windows associated with one or more different physical locations when using a mixed reality device.

[0122] Although the above embodiments have been described in terms of browser applications, the scope of the claims also covers any other application or group of applications. In some embodiments, all applications in the operating system can be selected and displayed according to the claims. Such an embodiment will have applications in a prism, rather than parsed browser content in a window.

[0123] exist Fig.14Such an embodiment is depicted in , which displays multiple applications in multiple prisms. The "All" button is an exemplary drop-down filter to help categorize application options for display and selection (e.g., by category). Although other suitable selection or filtering methods and / or interfaces may be used, an exemplary slider in the range of 9m to 30m selects applications to be included in the all applications / landscape manager display based on the distance from the user. In some embodiments, the user can set the slider to a smaller distance corresponding to a room to display all applications available in the room. In some embodiments, the user can set the slider to a larger distance corresponding to a house to display all applications available in the entire house. In some embodiments, the slider can be set to the far right corresponding to all applications, regardless of location. The "Close All" button is an exemplary user interface element for controlling and / or manipulating applications. As described above, other user interface elements can be all opened, moved, etc. Fig.14 Depicted are two different instances of the "HELLO" application and the "Collection" application among the open applications. Thus, the "All" button can display multiple instances of one application as well as different applications.

[0124] Additional Embodiments

[0125] Other embodiments of the present disclosure are described below. These additional embodiments may combine elements from the embodiments disclosed above, and elements of these additional embodiments may be combined into the embodiments disclosed above.

[0126] 1. A computer program product embodied on a computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, cause the processor to perform a method comprising:

[0127] Receive an instruction to select multiple open windows;

[0128] retrieving information for the plurality of open windows, wherein the plurality of open windows are associated with different physical locations;

[0129] displaying representations of the plurality of open windows in a single user interface; and

[0130] Upon receiving a selection of a selected window from the plurality of open windows, the selected window is loaded into a foreground for a user's field of view.

[0131] 2. The computer program product of embodiment 1, wherein the representations of the multiple open windows are displayed in the single user interface by changing position parameters for the multiple open windows to positions within a current physical environment for the user.

[0132] 3. The computer program product of embodiment 2, wherein the plurality of open windows are rendered and displayed to the user at coordinates assigned to the plurality of open windows within the single user interface.

[0133] 4. The computer program product of embodiment 3, wherein the plurality of open windows are rendered in at least one of a preview form, a thumbnail form, or a full form.

[0134] 5. The computer program product of embodiment 1, wherein all open windows are selected for display in the single user interface.

[0135] 6. The computer program product of embodiment 1, wherein each of the plurality of open windows is rendered as a bounded volume for placement of virtual content.

[0136] 7. The computer program product of embodiment 1, wherein a hover state is implemented wherein the window being hovered moves into the foreground while other windows visually recede.

[0137] 8. The computer program product of embodiment 1, wherein upon receiving the selection of the selected window, unselected windows are closed.

[0138] 9. A method for displaying a window in a computing environment, comprising:

[0139] receiving an instruction to select a plurality of open applications;

[0140] retrieving information for the plurality of open applications, wherein the plurality of open applications are associated with different physical locations;

[0141] displaying representations of the plurality of open applications in a single user interface; and

[0142] Upon receiving a selection of a selected application from the plurality of open applications, the selected application is loaded into a foreground for a user's field of view.

[0143] 10. The method of embodiment 9, wherein the representations of the multiple open applications are displayed in the single user interface by changing location parameters for the multiple open applications to locations within a current physical environment for the user.

[0144] 11. The method of embodiment 10, wherein the plurality of open applications are rendered and displayed to the user at coordinates assigned to the plurality of open applications within the single user interface.

[0145] 12. The method of embodiment 11, wherein the plurality of open applications are rendered in at least one of a preview form, a thumbnail form, or a full form.

[0146] 13. The method of embodiment 9, wherein all open applications are selected for display in the single user interface.

[0147] 14. The method of embodiment 9, wherein each of the plurality of open applications is rendered as a separate prism for placement of virtual content.

[0148] 15. The method of embodiment 9, wherein a hover state is implemented wherein the application being hovered moves into the foreground while other applications visually recede.

[0149] 16. The method of embodiment 9, wherein upon receiving the selection of the selected application, unselected applications are closed.

[0150] System Architecture Overview

[0151] Fig.15 14 is a block diagram of an exemplary computing system 1400 suitable for implementing embodiments of the present disclosure. The computing system 1400 includes a bus 1406 or other communication mechanism for communicating information, which interconnects subsystems and devices such as a processor 1407, a system memory 1408 (e.g., RAM), a static storage device 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., a modem or Ethernet card), a display 1411 (e.g., a CRT or LCD), and an input device 1412 (e.g., a keyboard and mouse).

[0152] According to some embodiments, the computing system 1400 performs specific operations by executing one or more sequences of one or more instructions contained in the system memory 1408 by the processor 1407. Such instructions may be read into the system memory 1408 from another computer-readable / usable medium (e.g., static storage device 1409 or disk drive 1410). In alternative embodiments, hard-wired circuits may be used instead of software instructions or in combination with software instructions to implement the present disclosure. Therefore, the embodiments of the present disclosure are not limited to any specific combination of hardware circuits and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or part of the present disclosure.

[0153] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1407 for execution. Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 1410. Volatile media include dynamic memory, such as system memory 1408.

[0154] Common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0155] In an embodiment of the present disclosure, the execution of the sequence of instructions for practicing the present disclosure is performed by a single computing system 1400. According to other embodiments of the present disclosure, two or more computing systems 1400 coupled by a communication link 1415 (e.g., a LAN, PTSN, or wireless network) can cooperate with each other to execute the sequence of instructions required to practice the present disclosure.

[0156] The computing system 1400 can send and receive messages, data, and instructions, including programs (i.e., application code), via the communication interface 1414 over the communication link 1415. The received program code can be executed by the processor 1407 as it is received and / or stored in the disk drive 1410 or other non-volatile storage for later execution. The computing system 1400 can communicate with a database 1432 on an external storage device 1431 via a data interface 1433.

[0157] In the foregoing description, the present disclosure has been described with reference to specific embodiments of the present disclosure. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above-described process flow is described with reference to a specific order of processing actions. However, the order of many of the described processing actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings should be considered to be exemplary and not restrictive.

Claims

1. A method for displaying a window in a computing environment, comprising: receiving, at a mixed reality device, from a single user interface, an instruction to select a plurality of open windows including a first open window and a second open window, wherein the first open window is rendered as a first bounding volume and includes a first space for placement of first virtual content, and the second open window is rendered as a second bounding volume and includes a second space for placement of second virtual content; retrieving information for the first and second open windows, wherein the first and second open windows are rendered and associated with first and second physical locations, respectively, in a physical environment in which a user of the mixed reality device is located at a user location; presenting, to the user at the user location, in the single user interface, a view that includes both the first virtual content of the first open window rendered at the first physical location and the second virtual content of the second open window rendered at the second physical location, wherein the first open window rendered at the first physical location is visible within a field of view of the user at the user location and the second open window rendered at the second physical location is not visible within the field of view of the user at the user location; and Upon receiving a selection of the second open window from the view in the single user interface, the second virtual content placed in the second open window is loaded into the field of view of the user by at least placing the second virtual content in the second bounded volume into the field of view of the user at the user position.

2. The method according to claim 1, wherein: The view having the first and second open windows is displayed in the single user interface by at least associating corresponding position parameters associated with corresponding positions of open windows having the first open window or the second open window with separate locations within the physical environment in which the user is located.

3. The method according to claim 2, wherein: The multiple open windows are rendered to the user at corresponding coordinates within the single user interface that are respectively assigned to the multiple open windows, and the view of the multiple open windows is rendered in at least one of a preview form, a thumbnail form, or a full form.

4. The method according to claim 1, further comprising: identifying a plurality of web pages for a browser application or corresponding links to the plurality of web pages, wherein the corresponding links include bookmarks for web pages, and the plurality of web pages include a first web page, a second web page, and a third web page; creating a multi-stacked virtual object having a first virtual tab and a second virtual tab in the single user interface; creating the first virtual tab for the multiple stacked virtual object by at least associating a first portion of the first web page with the first virtual tab of the multiple stacked virtual object; as well as The second virtual tab for the multiple stacked virtual object is created by at least associating a second portion of the first web page with the second virtual tab of the multiple stacked virtual object.

5. The method according to claim 4, further comprising: creating a third virtual tab for the multiple stacked virtual objects by at least associating the second webpage with the third virtual tab of the multiple stacked virtual objects; as well as The fourth virtual tab for the multi-stacked virtual object is created at least by associating the third web page with a fourth virtual tab of the multi-stacked virtual object, wherein, when the user focuses on a specific tab, the multi-stacked virtual object automatically advances to the specific tab to display corresponding virtual content associated with the specific tab.

6. The method according to claim 1, wherein: The corresponding content in the first and second open windows is rendered into the first bounded volume and the second bounded volume for corresponding placement of virtual content, respectively, instead of being rendered into a browser, the first or the second bounded volume comprising a three-dimensional volume having a specified boundary and managed by an application executed on the mixed reality device, and loading the second virtual content in the second open window into the field of view includes: identifying a bounded volume that can be selected from the field of view, wherein the bounded volume is identified in the following manner: by copying the second bounded volume with new associated position information in the field of view or by moving an existing bounded volume that can be selected from the field of view based at least in part on the new associated position information.

7. The method according to claim 1, wherein: Presenting the view includes: displaying multiple views of multiple tabs stacked and corresponding to the second open window respectively in the view, wherein a hover state is implemented, the window being hovered moves into the foreground within the user's field of view, and one or more other windows visually recedes or is closed, and the second open window includes the multiple tabs with an active tab displayed in the second open window.

8. The method according to claim 1, wherein: Loading the second virtual content placed in the second open window into the field of view includes: Deconstructing two-dimensional (2D) content rendered in the second open window into a specific bounded volume within the field of view, the specific bounded volume including one or more first virtual objects, deconstructing the two-dimensional content includes: identifying one or more real or virtual objects presented in the field of view; mapping the one or more first virtual objects to the one or more real or virtual objects in the field of view; and The one or more first virtual objects are rendered on the one or more real or virtual objects respectively to complete loading the second virtual content in the second opened window into the field of view.

9. A system for manipulating windows in a computing environment, comprising: A mixed reality display device that displays three-dimensional content; processor; a memory for holding programmable code executable by the processor, wherein the programmable code comprises instructions that, when executed by the processor, cause the processor to: receiving, at a mixed reality device, from a single user interface, an instruction to select a plurality of open windows including a first open window and a second open window, wherein the first open window is rendered as a first bounding volume and includes a first space for placement of first virtual content, and the second open window is rendered as a second bounding volume and includes a second space for placement of second virtual content; retrieving information for the first and second open windows, wherein the first and second open windows are rendered and associated with first and second physical locations, respectively, in a physical environment in which a user of the mixed reality device is located at a user location; presenting to the user at the user location in the single user interface presented by the mixed reality display device a view that includes both the first open window rendered at the first physical location and the second open window rendered at the second physical location, wherein the first open window rendered at the first physical location is visible within a field of view of the user at the user location and the second open window rendered at the second physical location is not visible within the field of view of the user at the user location; and Upon receiving a selection of the second open window in the single user interface, the second virtual content rendered in the second open window is loaded into the field of view of the user by at least placing the second virtual content in the second bounded volume into the field of view of the user at the user's position.

10. The system according to claim 9, wherein: The view having the first and second open windows is displayed in the single user interface by at least associating corresponding position parameters associated with corresponding positions of open windows having the first open window or the second open window with separate locations within the physical environment in which the user is located.

11. The system according to claim 10, wherein: The multiple open windows are rendered and displayed to the user at coordinates respectively assigned to the multiple open windows within the single user interface, and the view of the multiple open windows is rendered in at least one of a preview form, a thumbnail form, or a full form.

12. The system according to claim 11, wherein: The programmable code also includes the instructions, which when executed by the processor further cause the processor to: identifying a plurality of web pages for a browser application or corresponding links to the plurality of web pages, wherein the corresponding links include bookmarks for web pages, and the plurality of web pages include a first web page, a second web page, and a third web page; creating a multi-stacked virtual object having a first virtual tab and a second virtual tab in the single user interface; creating the first virtual tab for the multiple stacked virtual object by at least associating a first portion of the first web page with the first virtual tab of the multiple stacked virtual object; as well as The second virtual tab for the multiple stacked virtual object is created by at least associating a second portion of the first web page with the second virtual tab of the multiple stacked virtual object.

13. The system according to claim 12, wherein: The programmable code also includes the instructions, which when executed by the processor further cause the processor to: creating a third virtual tab for the multiple stacked virtual objects by at least associating the second webpage with the third virtual tab of the multiple stacked virtual objects; as well as The fourth virtual tab for the multi-stacked virtual object is created at least by associating the third web page with a fourth virtual tab of the multi-stacked virtual object, wherein, when the user focuses on a specific tab, the multi-stacked virtual object automatically advances to the specific tab to display corresponding virtual content associated with the specific tab.

14. The system according to claim 9, wherein: The corresponding content in the first and second open windows is rendered into the first bounded volume and the second bounded volume for corresponding placement of virtual content, respectively, instead of being rendered into a browser, the first or the second bounded volume comprising a three-dimensional volume having a specified boundary and managed by an application executed on the mixed reality device, and loading the second virtual content in the second open window into the field of view includes: identifying a bounded volume that can be selected from the field of view, wherein the bounded volume is identified in the following manner: by copying the second bounded volume with new associated position information in the field of view or by moving an existing bounded volume that can be selected from the field of view based at least in part on the new associated position information.

15. The system according to claim 9, wherein: The instructions, when executed by the processor, further cause the processor to: display multiple views of multiple tabs stacked and corresponding to the second open window, respectively, in the view, wherein a hover state is implemented, wherein the window being hovered moves into the foreground within the field of view of the user, while one or more other windows visually recede or are closed, and the second open window includes the multiple tabs with one active tab displayed in the second open window.

16. The system of claim 9, wherein: The programmable code executed to cause the processor to load the virtual content further includes the instructions, which when executed by the processor further cause the processor to: Deconstructing two-dimensional (2D) content rendered in the second open window into a specific bounded volume within the field of view, the bounded volume including one or more virtual objects, at least by: identifying one or more real or virtual objects presented in the field of view; mapping the one or more virtual objects to the one or more real or virtual objects in the field of view; as well as The one or more virtual objects are rendered on the one or more real or virtual objects respectively to complete loading the second virtual content in the second opened window into the field of view.

17. A computer program product embodied on a non-transitory computer readable medium storing thereon a sequence of instructions which, when executed by a processor, cause the processor to perform a set of actions comprising: An instruction to select a plurality of open windows including a first open window and a second open window is received from a single user interface at a mixed reality device, wherein: The first open window is rendered as a first bounding volume and includes a first space for placement of first virtual content, and the second open window is rendered as a second bounding volume and includes a second space for placement of second virtual content; retrieving information for the plurality of open windows, wherein the first and second open windows are rendered and associated with a first physical location and a second physical location, respectively, in a physical environment in which a user of the mixed reality device is located at a user location; presenting to the user at the user location a view in the single user interface presented to the user by the mixed reality device that includes both the first open window rendered at the first physical location and the second open window rendered at the second physical location, wherein the first open window rendered at the first physical location is visible within a field of view of the user at the user location and the second open window rendered at the second physical location is not visible within the field of view of the user at the user location; as well as Upon receiving a selection of the second open window in the single user interface, the second virtual content placed in the second open window is loaded into the field of view of the user by at least placing the second virtual content in the second bounded volume into the field of view of the user at the user position.

18. A method for displaying virtual content in a computing environment, comprising: receiving, at a mixed reality device, from a single user interface, an instruction to select a plurality of open applications including a first open application and a second open application, wherein the first open application is rendered as a first bounded volume and includes a first space for placement of first virtual content, and the second open application is rendered as a second bounded volume and includes a second space for placement of second virtual content; retrieving information for the first and second open applications, wherein the first and second open applications are rendered and associated with first and second physical locations, respectively, in a physical environment in which a user of the mixed reality device is located at a user location; presenting, to the user at the user location, in the single user interface, a view that includes both the first virtual content of the first open application rendered at the first physical location and the second virtual content of the second open application rendered at the second physical location, wherein the first open application rendered at the first physical location is visible within a field of view of the user at the user location and the second open application rendered at the second physical location is not visible within the field of view of the user at the user location; and Upon receiving a selection of the second open application from the view in the single user interface, the second virtual content placed in the second open application is loaded into the field of view of the user by at least placing the second virtual content in the second bounded volume into the field of view of the user at the user's location.

Images