Browser for mixed reality system

Abstract

An improved system and method for navigation and manipulation of browser windows in a 3D mixed reality environment is disclosed. An improved approach is provided to view a user's window irrespective of the user's current position with respect to one or more previously opened windows. A method for displaying windows in a computing environment includes receiving an instruction to select a plurality of opened windows. The method also includes retrieving information for a plurality of opened windows, wherein the plurality of opened windows are associated with different physical locations. The method further includes displaying a representation of the plurality of opened windows in a single user interface. Further, the method includes, upon receiving a selection of a selected window of the plurality of opened windows, loading the selected window into a foreground of the field of view for the user.

Description

[0001] This application is a divisional application of the application with the filing date of February 21, 2019, PCT international application number PCT / US2019 / 018932, Chinese national phase application number 201980014719.1, and invention title "Browser for Mixed Reality Systems". Technical Field

[0002] The present disclosure relates to systems and methods for implementing browsing techniques in a spatial 3D environment. Background Art

[0003] A typical way to view a web page is to open the web page on a monitor of a computer, smartphone, tablet, etc. The user will scroll the web page to view different content displayed on the web page. Generally, regardless of whether the user is looking at a computer monitor, smartphone or tablet, the content is displayed on the monitor in a fixed format.

[0004] Through virtual reality, augmented reality, and / or mixed reality systems (collectively referred to hereinafter as "mixed reality" systems), a three-dimensional environment for the display of content is provided to the user. The problem is that when used in a 3D environment, the conventional methods for displaying 2D content within a browser do not work very well. One of the reasons is that in a conventional 2D web browser, the display area of the display device is limited to the screen area of the monitor that is displaying content. As a result, conventional browsers are configured to only know how to organize and display content within that monitor display area. In contrast, a 3D environment is not limited by the strict constraints of the monitor display area. Therefore, when used in a 3D environment, conventional 2D browsers perform sub-optimally because conventional browsing techniques simply do not have the function or ability to utilize the 3D environment to display content.

[0005] For example, consider the situation when a user is using a mixed reality device and has placed multiple browser windows associated with different physical locations. For example, the user may have opened a first browser window in a first room and a second browser window in a second room. Since conventional 2D-based browsers are limited to the display within a given monitor area, this means that conventional browsers do not even have the technology to grasp the concept of physically remote windows, let alone the ability to handle such a situation where multiple windows are opened at multiple physical locations, making it impossible for the user to effectively view, navigate, and use these multiple windows.

[0006] Therefore, there is a need for an improved method to implement browsing techniques in a 3D environment. Summary of the Invention

[0007] Improved systems and methods are provided for navigating and manipulating browser windows in a 3D mixed reality environment. Some embodiments are directed to an improved method for viewing a user window regardless of the user's current position relative to one or more previously opened windows.

[0008] The improved systems and methods for navigating and manipulating browser windows can be applied in the context of 2D content that is deconstructed and displayed in a spatially organized 3D environment. This can include identifying 2D content, identifying elements in the 2D content, identifying surrounding surfaces, mapping the identified elements to the identified surrounding surfaces, and displaying the elements as virtual content on the surrounding surfaces.

[0009] In one embodiment, a method for displaying windows in a computing environment includes receiving an instruction to select a plurality of open windows. The method further includes retrieving information for the plurality of open windows, where the plurality of open windows are associated with different physical locations. The method further includes displaying a representation of the plurality of open windows in a single user interface. Additionally, the method includes loading the selected window into the foreground for the user's field of view upon receiving a selection of the selected window among the plurality of open windows.

[0010] In one or more embodiments, a representation of a plurality of open windows is displayed in a single user interface by changing the position parameters for the plurality of open windows to a position within the user's current physical environment. The plurality of open windows are rendered and displayed to the user at coordinates assigned to the plurality of open windows within the single user interface. The plurality of open windows are rendered in at least one of a preview form, a thumbnail form, or a full form. All open windows are selected for display in the single user interface. Each of the plurality of open windows is rendered as a separate prism for the placement of virtual content. The window being hovered over can be moved into the foreground while the other windows visually recede. Unselected windows are closed upon receiving a selection of the selected window.

[0011] In another embodiment, a system for manipulating windows in a computing environment includes a mixed reality display device that displays three-dimensional content. The system further includes a processor. The system further includes a memory that holds programmable code executable by the processor. The programmable code includes instructions to: receive an instruction to select a plurality of open windows; retrieve information for the plurality of open windows, where the plurality of open windows are associated with different physical locations; display a representation of the plurality of open windows in a single user interface of the mixed reality display device; and load the selected window into the foreground for the user's field of view upon receiving a selection of the selected window among the plurality of open windows.

[0012] In one or more embodiments, representations of multiple open windows are displayed in a single user interface by changing the location parameters for the multiple open windows to locations within the user's current physical environment. The multiple open windows are rendered and displayed to the user at coordinates within the single user interface that are assigned to the multiple open windows. The multiple open windows are rendered in at least one of a preview form, a thumbnail form, or a full form. All open windows are selected for display in the single user interface. Each of the multiple open windows is rendered as a bounded volume for the placement of virtual content. A hover state is implemented where the window being hovered over moves to the foreground while the other windows visually recede. Upon receiving a selection of a selected window, the unselected windows are closed.

[0013] In yet another embodiment, a computer program product embodied on a computer-readable medium has a sequence of instructions stored thereon that, when executed by a processor, cause the processor to perform a method including the following aspects: receiving instructions to select multiple open windows; retrieving information for the multiple open windows, where the multiple open windows are associated with different physical locations; displaying representations of the multiple open windows in a single user interface; and upon receiving a selection of a selected window among the multiple open windows, loading the selected window into the foreground for the user's field of view.

[0014] In one or more embodiments, representations of multiple open windows are displayed in a single user interface by changing the location parameters for the multiple open windows to locations within the user's current physical environment. The multiple open windows are rendered and displayed to the user at coordinates within the single user interface that are assigned to the multiple open windows. The multiple open windows are rendered in at least one of a preview form, a thumbnail form, or a full form. All open windows are selected for display in the single user interface. Each of the multiple open windows is rendered as a prism for the placement of virtual content. A hover state is implemented where the window being hovered over moves to the foreground while the other windows visually recede. Upon receiving a selection of a selected window, the unselected windows are closed.

[0015] In yet another embodiment, a method for displaying virtual content in a computing environment includes receiving instructions to select multiple open applications. The method further includes retrieving information for the multiple open applications, where the multiple open applications are associated with different physical locations. The method further includes displaying representations of the multiple open applications in a single user interface. Additionally, the method includes loading the selected application into the foreground for the user's field of view upon receiving a selection of the selected application among the multiple open applications.

[0016] In one or more embodiments, representations of multiple open applications are displayed in a single user interface by changing location parameters for the multiple open applications to locations within the user's current physical environment. The multiple open applications are rendered and displayed to the user at coordinates assigned to the multiple open applications within the single user interface. The multiple open applications are rendered in at least one of a preview form, a thumbnail form, or a full form. All open applications are selected for display in the single user interface. Each of the multiple open applications is rendered as a separate prism for placement of virtual content. A hover state is implemented where the application being hovered over moves into the foreground while the other applications visually recede. Upon receiving a selection of the selected application, the unselected applications are closed.

[0017] Further details of aspects, objects, and advantages of the embodiments are described in the detailed description, the drawings, and the claims below. The foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the scope of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale and, throughout the drawings, elements of similar structure or function are represented by like reference numerals. To better understand how to obtain the above and other advantages and objects of the various embodiments of the present invention, the present invention briefly described above will be described in more detail with reference to specific embodiments of the present invention, which are illustrated in the drawings. It is understood that these drawings only depict typical embodiments of the present invention and should not be considered as limiting the scope of the present invention. The present invention will be described and explained with additional features and details by using the drawings, wherein:

[0019] Figure 1 An augmented reality environment for deconstructing 2D content to be displayed in a user's 3D environment is shown, according to some embodiments.

[0020] Figure 2 An example mapping of elements in 2D content to a user's 3D environment is shown, according to some embodiments.

[0021] Figure 3 A flowchart illustrating a method for deconstructing 2D content to be displayed in a 3D environment, according to some embodiments.

[0022] Figure 4 A flowchart illustrating a method for identifying elements in 2D content, according to some embodiments.

[0023] Figure 5 An example table for storing elements deconstructed from 2D content is shown, according to some embodiments.

[0024] Figure 6 is a flowchart showing a method for identifying surfaces from a user's local environment according to some embodiments.

[0025] Figure 7 An example of a table for storing a list of surfaces identified from a user's local environment according to some embodiments is shown.

[0026] Figure 8 is a flowchart showing a method for mapping elements from 2D content to available surfaces according to some embodiments.

[0027] Figure 9 An example of a table for storing the mapping of elements from 2D content to surfaces in a user's local environment according to some embodiments is shown.

[0028] Figure 10 A flowchart showing a method for implementing the viewing of a user window is shown.

[0029] Figure 11A -B shows the process of displaying a window for a user without considering the previous physical location of the window.

[0030] Figures 12 - 13 An example of a possible method for displaying multiple windows within a mixed reality interface is provided.

[0031] Figure 14 A possible method for displaying multiple prisms within a mixed reality system is shown.

[0032] Figure 15 is a block diagram of an exemplary computing system suitable for implementing embodiments of the present disclosure. Detailed Description

[0033] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples of the present disclosure so that those skilled in the art can practice the present disclosure. It should be noted that the following drawings and examples are not meant to limit the scope of the present disclosure. In cases where certain elements of the present disclosure can be implemented partially or fully using known components (or methods or processes), only those parts of these known components (or methods or processes) that are necessary for understanding the present disclosure will be described, and detailed descriptions of other parts of these known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Additionally, various embodiments cover current and future known equivalents of the components referred to herein by way of example.

[0034] Although the systems and methods described below are primarily described in the context of browser applications, those of ordinary skill in the art will understand that the systems and methods described herein can also be applied in the context of one or more other applications. In some embodiments, an application for managing a user's photos and / or videos can utilize the systems and methods described below. In some embodiments, an application for playing card games can utilize the systems and methods described below. In some embodiments, a weather application can utilize the systems and methods described below. In some embodiments, any other application that can be installed and / or run on a device and / or system capable of displaying 3D virtual content to a user can utilize the systems and methods described below. In some embodiments, a single application can utilize the systems and methods described below. In some embodiments, more than one application can utilize the systems and methods described below. In some embodiments, all applications installed and / or run on a device and / or system capable of displaying 3D virtual content to a user can utilize the systems and methods described below.

[0035] Although the systems and methods described below are primarily described in the context of browser applications, those of ordinary skill in the art will understand that the systems and methods described herein can also be applied in the context of one or more other applications. In some embodiments, an application for managing a user's photos and / or videos can utilize the systems and methods described below. In some embodiments, an application for playing card games can utilize the systems and methods described below. In some embodiments, a weather application can utilize the systems and methods described below. In some embodiments, any other application that can be installed and / or run on a device and / or system capable of displaying 3D virtual content to a user can utilize the systems and methods described below. In some embodiments, a single application can utilize the systems and methods described below. In some embodiments, more than one application can utilize the systems and methods described below. In some embodiments, all applications installed and / or run on a device and / or system capable of displaying 3D virtual content to a user can utilize the systems and methods described below. In some embodiments, multiple instances of an application can utilize the systems and methods described below.

[0036] Web Page Deconstruction

[0037] Embodiments of the present disclosure will deconstruct a 2D web page to be displayed in a 3D environment that is to be spatially organized. The 2D web page can originate from a web browser of a head-mounted system, a mobile device (e.g., a phone), a tablet, a television, an application, etc. In some embodiments, the 2D web page can be received from another application or device such as a laptop computer, a desktop computer, an email application having a link to the 2D web page, an electronic message that references or includes a link to the 2D web page.

[0038] Referring to FIG.( Figure 1 ), environment 100 is representative of a physical environment and system for implementing the processes described below (e.g., deconstructing 2D content from a web page to be displayed on a 3D surface in a user's physical environment 105). Representative physical environments and systems of environment 100 include the user's physical environment 105 viewed by user 108 through a head-mounted system 160. Representative systems of environment 100 also include accessing 2D content (e.g., a web page) via a web browser 110 operably coupled to a network 120. Network 120 can be the Internet, an intranet, a private cloud network, a public cloud network, etc. Web browser 110 is also operably coupled to a processor 170 via network 120. Although processor 170 is shown as a separate, isolated component from head-mounted system 160, in alternative embodiments, processor 170 can be integrated with one or more components of head-mounted system 160 and / or integrated into other system components within environment 100, e.g., network 120 accessing a computing network 125 and a storage device 130. Processor 170 can be configured with software 150 for receiving and processing information such as video, audio, and content received from head-mounted system 160, local storage device 140, web browser 110, computing network 125, and storage device 130. Software 150 can communicate with computing network 125 and storage device 130 via network 120. Software 150 can be installed on processor 170, or in another embodiment; the features and functions of the software can be integrated into processor 170. Processor 170 can also be configured with a local storage device 140 for storing information used by processor 170 for quick access without relying on information stored remotely on an external storage device near user 108. In other embodiments, processor 170 can be integrated within head-mounted system 160.

[0039] The user's physical environment 105 is the physical environment of user 108 when moving around the user and viewing the user's physical environment 105 through the head-mounted system 160. For example, referring to Figure 1, the user's physical environment 105 shows a room with two walls (e.g., a main wall 180 and a side wall 184, which are relevant to the user's viewing) and a table 188. On the main wall 180, there is a rectangular surface 182 depicted by a solid black line to show a physical surface with physical boundaries (e.g., a painting hanging or attached to a wall or window, etc.), which can be a candidate surface onto which certain 2D content can be projected. On the side wall 184, there is a second rectangular surface 186 depicted by a solid black line to show a physical surface with physical boundaries (e.g., a painting hanging or attached to a wall or window, etc.). On the table 188, there can be different objects. 1) A virtual Rolodex 190, in which certain 2D content can be stored and displayed; 2) A horizontal surface 192 depicted by a solid black line to represent a physical surface with physical boundaries onto which certain 2D content can be projected; 3) A stack of virtual square surfaces 194 depicted by a dashed black line to represent, for example, a stack of virtual newspapers in which certain 2D content can be stored and displayed.

[0040] The web browser 110 can also display blog pages from the Internet or within an intranet / private network. Additionally, the web browser 110 can also be any technology that displays digital 2D content. The 2D content can include, for example, web pages, blogs, digital pictures, videos, news articles, newsletters, or music. The 2D content can be stored in a storage device 130 that the user 108 can access via the network 120. In some embodiments, the 2D content can also be streaming content, e.g., a live video feed or a live audio feed. The storage device 130 can include, for example, a database, a file system, a permanent storage device, a flash drive, a cache, etc. In some embodiments, the web browser 110 that displays 2D content (e.g., a web page) via the computing network 125.

[0041] The computing network 125 accesses the storage device 130 to retrieve and store 2D content for display in the web pages on the web browser 110. In some embodiments, the local storage device 140 can provide the user 108 with 2D content of interest. The local storage device 140 can include, for example, a flash drive, a cache, a hard drive, a database, a file system, etc. The information stored in the local storage device 140 can include recently accessed 2D content or content recently displayed in the 3D space. The local storage device 140 helps improve the performance of the system of the environment 100 by locally providing certain content to the software 150 to help deconstruct the 2D content for display on a 3D surface (e.g., in the user's physical environment 105) in the 3D space environment.

[0042] Software 150 includes software programs stored in a non-transitory computer-readable medium to perform the function of deconstructing 2D content to be displayed in the user's physical environment 105. Software 150 can run on processor 170, where processor 170 can be locally attached to user 108, or in some other embodiments, software 150 and processor 170 can be included within a head-mounted system 160. In some embodiments, portions of the features and functions of software 150 can be stored and executed on a computing network 125 remote from user 108. For example, in some embodiments, deconstructing the 2D content can occur on computing network 125, and the results of the deconstruction can be stored within storage device 130, where an inventory of the surfaces of the user's local environment for presenting the deconstructed 2D content can occur within processor 170, and the list of surfaces and mappings is stored in local storage device 140. In one embodiment, the processes of deconstructing the 2D content, inventorying local surfaces, mapping elements of the 2D content to local surfaces, and displaying the elements of the 2D content can all occur locally within processor 170 and software 150.

[0043] The head-mounted system 160 can be a virtual reality (VR) or augmented reality (AR) head-mounted system that includes a user interface, a user sensing system, an environmental sensing system, and a processor (all not shown). The head-mounted system 160 presents an interface to user 108 for interacting with and experiencing the digital world. Such interactions can involve the user and the digital world, one or more other users connected to environment 100, and objects located within the digital and physical worlds.

[0044] The user interface can include receiving 2D content through the user interface and selecting elements within the 2D content. The user interface can be at least one or a combination of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. A haptic interface device is a device that allows a person to interact with a computer through physical sensations and movements. Haptics refers to a type of human-computer interaction technology that includes haptic feedback or other physical sensations to perform actions or processes on a computing device. In some embodiments, the control interface can be the user interface such that the user can interact with the MR display system, for example, by providing user input to the system and the system responding by executing corresponding commands.

[0045] The user sensing system may include one or more sensors 162 that are operable to detect certain characteristics, properties, or information related to a user 108 wearing the head-mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry that is capable of detecting real-time optical characteristics / measurements of the user 108, such as one or more of the following: pupil constriction / dilation, angular measurement / location of each pupil, sphericity, eye shape (which changes over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that can be used by the head-mounted system 160 to enhance the user's viewing experience.

[0046] The environment sensing system may include one or more sensors 164 for obtaining data from the user's physical environment 105. The objects or information detected by the sensors 164 may be provided as an input to the head-mounted system 160. In some embodiments, this input may represent the user's interaction with the virtual world. For example, a user (e.g., user 108) viewing a virtual keyboard on a table (e.g., table 188) may make gestures with their fingers as if the user were typing on the virtual keyboard. The movement of the fingers may be captured by the sensors 164 and provided as an input to the head-mounted system 160, where the input may be used to change the virtual world or create new virtual objects.

[0047] The sensor 164 may include, for example, an outward-facing camera or scanner that is used to interpret scene information, for example, by continuously and / or intermittently projected infrared structured light. By detecting and registering the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions, the environment sensing system may be used to map one or more elements in the user's physical environment 105 around the user 108. Thus, in some embodiments, the environment sensing system may include image-based 3D reconstruction software embedded in the local computing system (e.g., processor 170) that is operable to digitally reconstruct one or more objects or information detected by the sensors 164.

[0048] In one exemplary embodiment, the environment sensing system provides one or more of the following: motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors. It should be understood that the environment sensing system may include other components in addition to those discussed above.

[0049] As described above, in some embodiments, the processor 170 may be integrated with other components of the head-mounted system 160, integrated with other components of the system of the environment 100, or may be an isolation device (wearable or separated from the user 108) as shown in Figure 1 . The processor 170 may be connected to various components of the head-mounted system 160 through a physical, wired connection or through a wireless connection, such as a mobile network connection (including cellular phones and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. The processor 170 may include a storage module, an integrated and / or additional graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of transforming data from sources (such as the computing network 125 and the user sensing system and the environment sensing system of the head-mounted system 160) into image and audio data, where the image / video and audio may be presented to the user 108 via a user interface (not shown).

[0050] The processor 170 processes data processing for various components of the head-mounted system 160 and data exchange between the head-mounted system 160 and 2D content from web pages displayed or accessed by the web browser 110 and the computing network 125. The processor 170 may be used to buffer and process the data stream between the user 108 and the computing network 125, thereby achieving a smooth, continuous, and high-fidelity user experience.

[0051] The deconstruction of 2D content from a web page into elements and the mapping of the elements to be displayed on the surface in a 3D environment can be done in an intelligent and logical manner. A predefined set of rules may be used to recommend, suggest, or indicate the placement of certain types of elements / content identified within the 2D content / web page. For example, certain types of 2D content elements may have one or more content elements that may need to be mapped to the surface of a physical or virtual object suitable for storing and displaying the one or more elements, while other types of 2D content elements may be a single object, such as the main video or main article in a web page. In this case, the single object can be mapped to the most meaningful surface to display the single object to the user.

[0052] Figure 2 An example mapping of elements of 2D content to the user's 3D environment according to some embodiments is shown. The environment 200 depicts 2D content (such as a web page) displayed or accessed by the web browser 110 and the user's physical environment 105. The dashed lines with arrows depict elements (such as a specific type of content) from the 2D content (such as a web page) that are mapped to and displayed on the user's physical environment 105. Based on web designer cues or predefined browser rules, certain elements from the 2D content are mapped to certain physical or virtual objects in the user's physical environment 105.

[0053] As an example, the 2D content accessed or displayed by the web browser 110 can be a web page with multiple tabs, where the current active tab 260 is displayed and the secondary tab 250 is currently hidden until selected to be displayed on the web browser 110. What is typically displayed within the active tab 260 is a web page. In this particular example, the active tab 260 is displaying a YOUTUBE page including a main video 220, user comments 230, and suggested videos 240. As depicted in this exemplary Figure 2 example, the main video 220 can be mapped to be displayed on the vertical surface 182, the user comments 230 can be mapped to be displayed on the horizontal surface 192, and the suggested videos 240 can be mapped to be displayed on a vertical surface 186 different from the vertical surface 182. Additionally, the secondary tab 250 can be mapped to be displayed on the virtual Rolodex 190 and / or the multi-stack virtual object 194. In some embodiments, the specific content within the secondary tab 250 can be stored in the multi-stack virtual object 194. In other embodiments, the entire content residing within the secondary tab 250 can be stored and / or displayed on the multi-stack virtual object 194. Similarly, the virtual Rolodex 190 can contain specific content from the secondary tab 250, or the virtual Rolodex 190 can contain the entire content residing within the secondary tab 250.

[0054] The vertical surface 182 can be any type of structure that is already located on the main wall 180 of a room (depicted as the user's physical environment 105), such as a window glass or a photo frame. In some embodiments, the vertical surface 182 can be an empty wall on which the head-mounted system 160 determines the optimal size of the frame of the vertical surface 182 suitable for the user 108 to view the main video 220. This determination of the size of the vertical surface 182 can be at least partially based on the distance of the user 108 from the main wall 180, the size and dimensions of the main video 220, the quality of the main video 220, the amount of uncovered wall space, and / or the posture of the user while viewing the main wall 180. For example, if the quality of the main video 220 is high definition, the size of the vertical surface 182 may be larger because the quality of the main video 220 will not be adversely affected by the vertical surface 182. However, if the video quality of the main video 220 is of poor quality, having a larger vertical surface 182 may significantly impede the video quality, in which case the methods and systems of the present disclosure can resize / redefine the vertical surface 182 to be smaller to minimize the poor video quality from pixelation.

[0055] Similar to the vertical surface 182, the vertical surface 186 is a vertical surface on an adjacent wall (e.g., the side wall 184) in the user's physical environment 105. In some embodiments, based on the orientation of the user 108, the side wall 184 and the vertical surface 186 may appear as inclined and skewed surfaces. In addition to vertical and horizontal surfaces, inclined and skewed surfaces can also be a type of surface orientation. In this example, the recommended video 240 from the YOUTUBE web page can be placed on the vertical surface 186 of the side wall 184 to allow the user 108 to view the recommended video by simply moving their head slightly to the right.

[0056] The virtual Rolodex 190 is a virtual object created and displayed to the user 108 by the head-mounted system 160. The virtual Rolodex 190 can have the ability to enable the user 108 to cycle through a set of virtual pages bidirectionally. The virtual Rolodex 190 can contain entire web pages, or can contain individual articles or videos or audio. As shown in this example, the virtual Rolodex 190 can contain a portion of the content from the sub-tab 250, or in some embodiments, the virtual Rolodex 190 can contain the entire page of the sub-tab 250. The user 108 can cycle through the content within the virtual Rolodex 190 bidirectionally by simply focusing on a specific tab within the virtual Rolodex 190, and one or more sensors (e.g., the sensor 162) within the head-mounted system 160 will detect the user 108's eye focus and cycle through the tabs in the virtual Rolodex 190 to obtain relevant information about the user 108 accordingly. In some embodiments, the user 108 can select relevant information from the virtual Rolodex 190 and instruct the head-mounted system 160 to display the relevant information on an available surrounding surface or on another virtual object, such as a virtual display adjacent to the user 108 (not shown).

[0057] Similar to the virtual Rolodex 190, the multi-stack virtual object 194 can contain the following: the complete content of one or more tabs or specific content from various web pages or tabs marked, saved for future viewing, or already opened (i.e., inactive tabs) by the user 108. The multi-stack virtual object 194 is also similar to a stack of real-world newspapers. Each stack within the multi-stack virtual object 194 can belong to a specific newspaper article, page, magazine issue, recipe, etc. One of ordinary skill in the art can understand that there can be multiple types of virtual objects to achieve the same purpose of providing surfaces to place 2D content elements or content from 2D content sources.

[0058] One of ordinary skill in the art can understand that the 2D content accessed or displayed by the web browser 110 may not be just a web page. In some embodiments, the 2D content can be a picture from an album, a video from a movie, a TV program, a YOUTUBE video, an interactive form, etc. However, in other embodiments, the 2D content can be an e - book or any electronic way of displaying a book. Finally, in other embodiments, since the 2D content is generally the current way of presenting information, the 2D content can be other types of content not yet described. If the electronic device can consume 2D content, the head - mounted system 160 can use the 2D content to deconstruct and display the 2D content in a 3D setting (e.g., AR).

[0059] In some embodiments, mapping the accessed 2D content can include extracting the 2D content (e.g., from the browser) and placing it on a surface (so that the content is no longer in the browser and is only on the surface), and in some embodiments, mapping can include copying the content (e.g., from the browser) and placing it on a surface (so that the content is both in the browser and on the surface).

[0060] Deconstructing 2D content is a technical problem existing in the fields of the Internet and computer - related technologies. Certain types of programming languages such as HTML are used to construct 2D content such as web pages to instruct computer processors and technical components where and how to display elements within the web page on the screen for the user. As described above, web designers generally work within the limitations of a 2D canvas (e.g., the screen) to place and display elements (e.g., content) within the 2D canvas. HTML tags are used to determine how to format an HTML document or a part within the HTML document. In some embodiments, the (extracted or copied) 2D content can maintain HTML tag references, and in some embodiments, the HTML tag references can be re - defined.

[0061] Figure 3 is a flowchart showing a method for deconstructing 2D content to be displayed in a 3D environment according to some embodiments. The method includes: identifying 2D content at 310; identifying elements in the 2D content at 320; identifying surrounding surfaces at 330; mapping the identified elements in the identified 2D content to the identified surfaces from the identified surrounding surfaces at 340; and displaying the elements as virtual content on the selected surface at 350, where the selected surface is selected from the mapping of the elements to the identified surfaces.

[0062] Identifying 2D content at 310 may involve using the head-mounted system 160 to search for digital content. Identifying 2D content at 310 may also include accessing digital content on a server (e.g., storage device 130) connected to the network 120. Identifying 2D content at 310 may include browsing the Internet for web pages of interest to the user 108. In some embodiments, identifying 2D content at 310 may include a voice-activated command given by the user 108 to search for content on the Internet. For example, the user 108 may be interacting with a device (e.g., the head-mounted system 160), where the user 108 requests the device to search for a specific video by saying a command to search for a video and then saying the video name and a brief description of the video, thereby searching for a specific video on the Internet. Then, the device may search the Internet and pull up the video on a 2D browser to allow the user 108 to view the video displayed on the 2D browser of the device. Then, the user 108 may confirm that the video is the video that the user 108 wants to view in the 3D spatial environment.

[0063] Once the 2D content is identified, the method identifies elements within the 2D content at 320 to obtain a list of available elements in the 2D content for display to the user 108. For example, elements within the 2D content may include videos, articles and newsletters published on web pages, comments and posts on social media sites, blog posts, pictures published on various websites, audiobooks, etc. These elements within the 2D content (e.g., web page) may contain HTML tags with attributes associated with the HTML tags provided by the content designer to define the placement of a specific element on the web page and in some cases when and how to display the element on the web page. In some embodiments, the methods and systems of the present disclosure will utilize these HTML tags and attributes as hints and suggestions provided by the content designer to assist in the mapping process at 340 to determine where and how to display the elements in the 3D setting. For example, the following is an example HTML web page code provided by a web developer.

[0064] Sample HTML Web Page Code Provided by Web Developers

[0065]

[0066]

[0067] The example HTML web page code provided by the web developer includes preferences on how to display the main video on the web page and preferences on how to display recommended (or suggested videos). In particular, the HTML web page code uses the "style" tag to specify how to display the main video using the type value of "vertical" to specify the vertical surface for displaying the video. Additionally, within the "style" tag, other hints provided by the web developer may include the "priority" preference for the matching algorithm, which is used to prioritize which HTML element / content in the web page (e.g., the main video) should be mapped to which potential surface area. In the example HTML web page code, for a video with a vertical plane layout, the priority is set to a value of 100, where in this example, a higher priority value indicates a higher priority. Additionally, in this example, the web developer indicates a preference to place the suggested videos in a stack with a type value of "horizontal" in the stack layout, where the distance between the stacked objects (e.g., in this case, the suggested video related to another suggested video) should be 20 cm.

[0068] Figure 4 is a flowchart showing a method for identifying elements in 2D content according to some embodiments. Figure 4 is disclosed according to some embodiments in Figure 3 the detailed process of identifying elements in 2D content at 320. Figure 4 Starts with identifying elements within the 2D content at 410, similar to identifying elements within the 2D content at Figure 3 320. The method proceeds to the next step of identifying attributes from tags related to the placement of the content at 420. As described above, when designing and configuring a web page, the web designer can associate elements within the web page with HTML tags to define where and how each element is displayed. These HTML tags can also include attributes related to placing the element on a specific part of the web page. The head-mounted system 160 will detect these HTML tags and their attributes and coordinate with other components of the system to be used as input regarding where a specific element can be displayed.

[0069] At 430, extract cues or tags from each element. The cues or tags are typically formatting cues or formatting tags provided by the content designer and / or the web developer of the 2D content / web page. As described above, the content designer can provide instructions or cues, for example, in the form of HTML tags as shown in the "example HTML web page code provided by the web developer", to indicate to the web browser 110 the elements in the 2D content to be displayed in a specific portion of the page or screen. In some embodiments, the web designer can use additional HTML tag attributes to define additional formatting rules. For example, if the user has reduced sensitivity to a particular color (e.g., red), instead of displaying red, another color is used, or as described above, if a video with a preference to be displayed on a vertical surface cannot be displayed on a vertical surface, instead the video is displayed on another (physical) surface or a virtual surface is created and the video is displayed on that virtual surface. The following is an example HTML page parser implemented in a browser for parsing an HTML page to extract cues / tags from each element within the HTML page.

[0070] Sample HTML Page Parser Implemented in Browser

[0071]

[0072]

[0073]

[0074] The example HTML page parser shows how to parse and identify and / or extract / copy an HTML page containing HTML tags that are used to provide display preferences for specific elements / objects within 2D content (e.g., a web page). As disclosed in the example HTML page parser, the disclosed example code can be used to parse the elements within 2D content (e.g., a web page). The HTML page parser (e.g., ML.layout, ML.container, etc.) can identify / extract certain HTML tags using various element names and values to determine how to display a specific element to the user in a 3D environment (e.g., by mapping the element to a specific surface).

[0075] At 440, perform a lookup / search for alternative display forms for one or more elements. Certain formatting rules can be specified for an image on a web page. For example, if the web browser 110 is capable of displaying a 3D version of an image, the web designer can place additional tags or define certain attributes of a specific tag to allow the web browser 110 to identify an image that may have an alternative version (e.g., a 3D version of the image). The web browser 110 can then access the alternative version (e.g., the 3D version of the image) of the image to be displayed in a 3D-enabled browser.

[0076] Perform a store of the identified elements within the 2D content at 450. The method can store the identified elements in a non-transitory storage medium for use by a mapping routine (e.g., map the elements to the identified surface at 340 of Figure 3 ), in order to map the elements to a specific surface. The non-transitory storage medium can include a data storage device such as storage device 130 or local storage device 140. The elements can be stored in a specific table such as the table disclosed in Figure 5 below. In some embodiments, the identified elements within the 2D content can be stored in a transitory storage medium.

[0077] Figure 5 An example of a table for storing elements deconstructed from 2D content according to some embodiments is shown. Element table 500 is an exemplary table that can store the results of identifying elements within the 2D content at 410 of Figure 4 in a database. Element table 500 includes, for example, information about one or more elements within the 2D content, which includes an element identification (ID) 510, a preference indicator 520 for a location where the element can be placed on a 3D surface, a parent element ID 530 if a particular element is included within a parent element, a sub-element ID 540 if the element can contain sub-elements, and a plurality of entity indicators 550 to indicate whether the element contains a plurality of embodiments that can ensure compatibility between the surface or virtual object for displaying the element and multiple versions of the displayed element. A parent element is an element / object within the 2D content that can contain sub-elements (e.g., child elements). For example, an element ID with a value of 220 (e.g., main video 220) has a parent element ID value of 260 (e.g., active tab 260), which indicates that the main video 220 is a child element of the active tab 260. Or stated differently, the main video 220 is included within the active tab 260. Continuing with the same example, the main video 220 has a sub-element ID 230 (e.g., user comment 230) indicating that the user comment 230 is associated with the main video 220. Those of ordinary skill in the art will understand that element table 500 can be a table in a relational database or any type of database. Additionally, element table 500 can be an array in a computer memory (e.g., cache) that contains the results of identifying elements within the 2D content at 410 of Figure 4 ).

[0078] Each row in row 560 of the element table 500 corresponds to an element within a web page. The element ID 510 is a column that contains a unique identifier for each element (e.g., the element ID). In some embodiments, the uniqueness of an element can be defined as a combination of the element ID 510 column and another column within the table (e.g., the preference 520 column if the content designer identifies more than one preference). The preference 520 is a column whose value can be determined at least in part based on HTML tags and attributes defined by the content designer / developer (e.g., the web page designer) and identified by the system and method for extracting cues or tags from each element at Figure 4 430. In other embodiments, the preference 520 column can be determined at least in part based on predefined browser rules to specify where certain types of elements within a web page should be displayed in a 3D environment. These predefined rules can provide suggestions to the system and method for determining the optimal placement of elements within the 3D environment.

[0079] The parent element ID 530 is a column that contains the element ID of the parent element in which the particular element in the current row is displayed or is related to. A particular element within a web page can be embedded, placed within another element of the page, or related to another element on the page. For example, in the current embodiment, the first entry in the element ID 510 column stores the value of the element ID 220 corresponding to Figure 2 the main video 220. The preference value in the preference 520 column corresponding to the main video 220 is determined based on HTML tags and / or attributes and, in the current embodiment, is that the element should be placed in the "main" position of the user's physical environment 105. Depending on the current location of the user 108, this main position can be a wall in the living room, or the range hood in the kitchen that the user 108 is currently looking at, or if in a wide open space, it can be a virtual object projected in front of the user's field line where the main video 220 can be projected onto the field. More information on how elements of 2D content are displayed to the user 108 will be disclosed in a later section. Continuing with the current example, the parent element ID 530 column stores the value of the element ID 260 corresponding to Figure 2 the active tab 260. Thus, the main video 220 is a child of the active tab 260.

[0080] The child element ID 540 is a column that contains the element ID of the child element in which the particular element in the current row is already displayed or is related to. A particular element within a web page can be embedded, placed within another element of the page, or related to another element on the page. Continuing with the current example, the child element ID 540 column stores the value of the element ID 230 corresponding to Figure 2 the user comment 230.

[0081] The multiple entity indicator 550 is a column that indicates whether an element contains multiple entities that can satisfy the need to make the surface or virtual object for displaying the element compatible with multiple versions of the display element (e.g., the element can be a user comment 230, where for the main video 220, there may be more than one comment available). Continuing with the current example, the multiple entity indicator 550 column stores a value of "No" to indicate that the main video 220 does not have or does not correspond to multiple main videos located in the active tab 260 (e.g., multiple versions of the "No" main video 220).

[0082] Continuing with the current example, the second entry in the element ID 510 column stores the value of element ID 230 corresponding to the Figure 2 user comment 230. The preference value corresponding to the user comment 230 in the preferences 520 column shows a preference of "Horizontal" to indicate that the user comment 230 should be placed on a "Horizontal" surface somewhere in the user's physical environment 105. As described above, the horizontal surface will be determined based on the available horizontal surfaces in the user's physical environment 105. In some embodiments, the user's physical environment 105 may not have a horizontal surface, in which case the systems and methods of the present disclosure can identify / create a virtual object with a horizontal surface to display the user comment 230. Continuing with the current example, the parent element ID 530 column stores the value element ID 220 corresponding to the Figure 2 main video 220, and the multiple entity indicator 550 column stores a value of "Yes" to indicate that the user comment 230 can contain more than one value (e.g., more than one user comment).

[0083] The remaining rows within the element table 500 contain information about the remaining elements that user 108 is interested in. Those of ordinary skill in the art will understand that storing the results of identifying elements within the 2D content at 410 can improve the functionality of the computer itself because once this analysis has been performed on the 2D content, if another user is interested in the same 2D content, the systems and methods can retain this information for future analysis of the 2D content. The systems and methods for deconstructing that particular 2D content can be avoided because it has already been completed previously.

[0084] In some embodiments, the element table 500 can be stored in the storage device 130. In other embodiments, the element table 500 can be stored in the local storage device 140 for quick access to recently viewed 2D content or for possible re-access to recently viewed 2D content. In other embodiments, the element table 500 can be stored at both the storage device 130 that is remote from the user 108 and the local storage device 140 that is local to the user 108.

[0085] ReturnFigure 3 , the method continues to identify the surrounding surfaces at 330. The user 108 can view the user's physical environment 105 through the head-mounted system 160 to allow the head-mounted system 160 to capture and identify the surrounding surfaces, such as walls, tables, paintings, window frames, stoves, refrigerators, televisions, etc. The head-mounted system 160 is aware of the real objects within the user's physical environment 105 due to sensors and cameras located on the head-mounted system 160 or on any other type of similar device. In some embodiments, the head-mounted system 160 can match the real objects observed within the user's physical environment 105 with the virtual objects stored in the storage device 130 or the local storage device 140 to identify the surfaces available for these virtual objects. A real object is an object identified within the user's physical environment 105. A virtual object is a physical object that does not exist within the user's physical environment but can be displayed to the user to make it appear as if the virtual object exists within the user's physical environment. For example, the head-mounted system 160 can detect an image of a table within the user's physical environment 105. The image of the table can be reduced to a 3D point cloud object for fast and effective comparison and matching at the storage device 130 or the local storage device 140. If a match between the real object and the 3D point cloud object (e.g., of the table) is detected, the system and method will identify the table as having a horizontal surface since the 3D point cloud object representing the table is defined as having a horizontal surface. A more detailed description of identifying the surrounding surfaces is disclosed below in Figure 6 for a more detailed description of identifying the surrounding surfaces.

[0086] In some embodiments, the virtual object can be an extracted object, where the extracted object can be a physical object identified within the user's physical environment 105 but is displayed to the user as a virtual object located at the position of the physical object so that additional processing and association can be performed on the extracted object, which would not be possible on the physical object itself (e.g., changing the color of the physical object to highlight certain features of the physical object, etc.). Additionally, the extracted object can be a virtual object extracted from 2D content (e.g., a web page from a browser) and displayed to the user 108. For example, the user 108 can select an object such as a sofa from a web page, which is displayed on the 2D content / web page to be displayed within the user's physical environment 105. The system can identify the selected object (e.g., the sofa) and display the extracted object (e.g., the sofa) to the user 108 as if the extracted object (e.g., the sofa) physically exists within the user's physical environment 105. Additionally, the virtual object can also include an object having a surface for displaying content (e.g., a transparent display screen near the user for viewing certain content), which is not only not a physical presence within the user's physical environment 105 but can also be an ideal display surface for presenting certain content to the user from the perspective of the displayed content in the 2D content.

[0087] Figure 6 is a flowchart showing a method for identifying surfaces from a user's local environment according to some embodiments. Figure 6 is to disclose the detailed process of identifying surrounding surfaces at Figure 3 330. Figure 6 It starts by identifying the user's current surrounding environment at 610, similar to identifying the surrounding surfaces at Figure 3 330. The method proceeds to the next step of determining the user's pose at 620.

[0088] Determining the user's pose at 620 is an important step in identifying the user's current surrounding environment because the user's pose will provide a perspective on the objects within the user's physical environment 105 for the user 108. For example, referring to Figure 1 , the user 108 using the head-mounted system 160 is observing the user's physical environment 105. Determining the user's pose (i.e., the vector and / or origin position information relative to the world) at 620 will help the head-mounted system 160 understand, for example: (1) how high the user 108 is relative to the ground, (2) the angle by which the user 108 has to rotate their head to move around and capture images of the room, and (3) the distances between the user 108 and the table 188, the main wall 180, and the side wall 184. Additionally, the pose of the user 108 also helps determine the angles at which the head-mounted system 160 observes the vertical surfaces 182 and 186 and other surfaces within the user's physical environment 105.

[0089] At 630, the method identifies the dimensions of the surrounding surfaces. Each candidate surface within the user's physical environment 105 is labeled and classified according to the corresponding dimensions. In some embodiments, each candidate surface within the user's physical environment 105 is also labeled and classified according to the corresponding orientation. This information will help identify which elements should be mapped to which surfaces, at least in part, based on the size of the surface, the orientation of the surface, the distance of the user 108 from a particular surface, and the type of information that needs to be displayed for that element. For example, in a situation where the text size of an article may be too small for the user to see if it is displayed on a distant wall with a small size, a video can be displayed further away than a blog or an article, where the blog or article may contain a large amount of information.

[0090] At 640, the method stores the list of surrounding surfaces in a non-transitory storage medium for use by a mapping routine (e.g., mapping elements to the surfaces identified at Figure 3 340) to map elements to specific surfaces. The non-transitory storage medium can include a data storage device such as the storage device 130 or the local storage device 140. The identified surfaces can be stored in a manner such as described below Figure 7in a specific table of the tables disclosed in. In some embodiments, the identified surface can be stored in a temporary storage medium.

[0091] Figure 7 FIG. shows an example of a table for storing a list of surfaces identified from a user's local environment. The surface table 700 is an exemplary table that can store the results of the process of identifying surrounding surfaces in a database. The surface table 700 includes, for example, information about the surfaces within the user's physical environment 105, which has data columns including surface ID 710, width 720, height 730, orientation 740, real or virtual indicator 750, multiple 760, and location 770. Those of ordinary skill in the art will understand that the surface table 700 can be a table in a relational database or any type of database. Additionally, the surface table 700 can be an array stored in Figure 3 a computer memory (e.g., cache) at 330 that stores the results of identifying surrounding surfaces.

[0092] Each row 780 in the surface table 700 can correspond to a surface from the user's physical environment 105 or a virtual surface that can be displayed to the user 108 within the user's physical environment 105. The surface ID 710 is a column that contains a unique identifier (e.g., surface ID) for uniquely identifying a particular surface. The dimensions of the particular surface are stored in the width 720 and height 730 columns.

[0093] The orientation 740 is a column that indicates the orientation of the surface relative to the user 108 (e.g., vertical, horizontal, etc.). Real / Virtual 750 is a column that indicates whether the particular surface is on a real object within the user's physical environment 105, as perceived by the user 108 using the head-mounted system 160, or whether the particular surface is on a virtual object generated by the head-mounted system 160 and displayed in the user's physical environment 105. The head-mounted system 160 may have to generate virtual objects for situations where the user's physical environment 105 may not contain enough surfaces to display a certain amount of content that the user 108 wishes to display. In these embodiments, the head-mounted system 160 can search a database of existing virtual objects, which may have appropriate surface dimensions to display certain types of elements identified for display. The database can be from the storage device 130 or the local storage device 140.

[0094] The multiple 760 is a column that indicates whether the surface / object is compatible with multiple versions of a display element (e.g., the element can be Figure 2secondary tab 250, where for a particular web browser 110, there can be more than one secondary (i.e., inactive) tab (e.g., one web page per tab). If the quantity 760 column has a value of "multiple" (such as in the following case: the fourth entry in the surface ID column stores the value 109 corresponding to the Figure 2 virtual Rolodex 190, and the fifth entry in the surface ID column stores the value 194 corresponding to the Figure 2 multi-stack virtual object 194), the system and method will know that if there is an element (which can have multiple versions of that element), such as is the case for inactive tabs, then these are the types of surfaces that can accommodate multiple versions.

[0095] Location 770 is a column that indicates the position of the physical surface relative to a reference frame or reference point. As Figure 7 shown in the column header of location 770 in, the position of the physical surface can be predetermined as the center of the surface. In other embodiments, the position can be predetermined as another reference point of the surface (e.g., the front, back, top, or bottom of the surface). The position information can be represented as a vector and / or position information from the center of the physical surface relative to a certain reference frame or reference point. There can be several ways to represent the position in the surface table 700. For example, the position value for surface ID 194 in the surface table 700 is represented in an abstract form to illustrate the vector information and reference frame information (e.g., the "system" subscript). x, y, z are the 3D coordinates in each spatial dimension, and the system indicates which reference frame the 3D coordinates are relative to.

[0096] For example, surface ID 186 shows that the position of the center of surface 186 is (1.3, 2.3, 1.3) relative to the real-world origin. As another example, surface ID 192 shows that the position of the center of surface 192 is (x, y, z) relative to the user reference frame, and surface ID 190 shows that the position of the center of surface 190 is (x, y, z) relative to another surface 182. The reference frame is important for eliminating ambiguity about which reference frame is currently being used. In the case of using the real-world origin as the reference frame, it is usually a static reference frame. However, in other embodiments, when the reference frame is the user reference frame, the user can be a moving reference frame, in which case, if the user is moving and the user reference frame is used as the reference frame, the plane (or vector information) may move and change with the user. In some embodiments, the reference frame used for each surface can be the same (e.g., the user reference frame). In other embodiments, depending on the surface (e.g., the user reference frame, the world reference frame, another surface or object in the room, etc.), the reference frame used for the surfaces stored in the surface table 700 can be different.

[0097] In the current example, the values stored in the surface table 700 include physical surfaces (e.g., vertical surfaces 182 and 186 and horizontal surface 192) and virtual surfaces (e.g., virtual Rolodex 190 and multi-stack virtual object 194) identified within the user's physical environment 105 of Figure 2 For example, in the current embodiment, the first entry in the surface ID 710 column stores the value of surface ID 182 corresponding to the vertical surface 182 of Figure 2 The width value in the width 720 column and the height value in the height 730 column, corresponding to the width and height of the vertical surface 182 respectively, indicate that the vertical surface 182 has dimensions of 48” (wide) by 36” (high). Similarly, the orientation value in the orientation 740 column indicates that the vertical surface 182 has an “upright” orientation. Additionally, the real / virtual value in the real / virtual 750 column indicates that the vertical surface 182 is a “R” (e.g., real) surface. The quantity value in the quantity 760 column represents that the vertical surface 182 is “single” (e.g., can only accommodate a single piece of content). Finally, the position 770 column uses vector information of (2.5, 2.3, 1.2) to indicate the position of the vertical surface 182 relative to the user 108.

[0098] The remaining rows within the surface table 700 contain information about the remaining surfaces within the user's physical environment 105. Those of ordinary skill in the art will understand that the results of identifying the surrounding surfaces at Figure 3 330 improve the functionality of the computer itself because once the analysis has been performed on the surrounding surfaces, if another user or the same user 108 is located in the same physical environment 105 but is interested in different 2D content, the analysis can be retained by the head-mounted system 160 for future analysis of the user's surrounding surfaces. The processing steps of identifying the surrounding surfaces at 330 can be avoided because these processing steps have been completed previously. The only difference may include that the element table 500, which is at least partially based on identifying elements within different 2D content, can be used to identify additional or different virtual objects.

[0099] In some embodiments, the surface table 700 is stored in the storage device 130. In other embodiments, the surface table 700 is stored in the local storage device 140 of the user 108 for quick access to or possible re-access of recently viewed 2D content. In other embodiments, the surface table 700 can be stored at both the storage device 130 remote from the user 108 and the local storage device 140 local to the user 108.

[0100] Return Figure 3, the method continues by using a combination of the identified elements from identifying elements in the 2D content at 320 and the identified surrounding surfaces from identifying the surrounding surfaces at 330 and, in some embodiments, using virtual objects as additional surfaces to map the elements to the identified surfaces at 340. Mapping the identified elements to the identified surfaces can involve multiple factors, some of which can include analyzing cues provided by the 2D content designer / author via HTML tag elements defined by the 2D content designer / author using an HTML page parser such as the example HTML page parser discussed above. Other factors can include selecting from a predefined set of rules provided by the AR browser, the AR interface, and / or cloud storage on how and where to map certain 2D content. Figure 8 A detailed flow of the mapping process for mapping one or more elements from 2D content to the identified surfaces is provided.

[0101] Figure 8 A flowchart depicting a method for mapping elements from 2D content to a surface according to some embodiments is shown. Figure 8 is disclosed in Figure 3 the detailed process of mapping elements to the identified surfaces at 340 of

[0102] At 810, the method determines whether the identified elements contain cues provided by the 2D content designer. When initially designing the 2D content, the 2D content designer can provide cues regarding the optimal location for displaying a particular element. For example, Figure 2The main video 220 can be a YOUTUBE video displayed on a web page within the activity tab 260. A 2D content designer (e.g., a web page designer) can provide cues to indicate that the main video 220 is best displayed on a vertical plane within the direct view of the user 108. In some embodiments, this can be achieved by using existing HTML tag elements originally designed for 2D web page content to further define the manner in which specific elements within 2D content are to be displayed in the event that a 3D display environment is available. As another example, the 2D content designer can provide cues that indicate that for a particular web page, a 3D image is available instead of a 2D image. For example, in the case of a 2D image, in addition to providing basic HTML tags to identify the source of the 2D content, the 2D content designer can also provide other, less commonly used HTML tags to identify the source of the 3D version of the 2D image, and additionally, provide a cue that if the 3D version of the image is used, it is to be highlighted in front of the user's view (e.g., within the main frame of a 3D layout). In some embodiments, the 2D content designer can provide this additional "cue" to the 3D image location of the 2D image only if the web browser 110 presenting the 2D content can have 3D display capabilities to take advantage of the enhanced 3D image. Those skilled in the art will understand that in addition to what has been disclosed herein, the 2D content designer can provide many other ways to provide cues regarding where specific content elements should be placed within a 2D layout, and these are some examples of the different ways in which the 2D content designer can provide cues to best display some or all of the elements within 2D content.

[0103] In another embodiment, the HTML tag standard can include the creation of new HTML tags or a similar markup language for providing cues for the placement of 3D objects within the user's surrounding environment for AR / VR specific types of browsers, such as the example HTML web page provided by the web developer discussed above. As of this writing, these new HTML tags have not been created and / or adopted as standard tags within the HTML language. However, once the HTML standard includes these types of additional tags, certain embodiments of the current methods and systems will utilize these new tags to further provide a mapping of the identified elements to the identified surfaces. Those skilled in the art will understand that in addition to HTML tags, there are many other languages that can be modified or adopted to further provide cues regarding how content elements should be best displayed within a 3D environment, and the new HTML tag standard is merely one way to achieve such a goal.

[0104] At 820, the method determines whether to use the cues provided by the 2D content designer or a predefined set of rules to map one or more content elements from the 2D content to certain types of 3D surfaces. In some embodiments, in the absence of cues provided by the 2D content designer for a particular content element, the system and method may use a predefined set of rules to determine the best way to map the content element to a surface. In other embodiments, even when there may be cues for the placement of content elements provided by the 2D content designer, the system and method may determine that a predefined set of rules can be optimally used to map the content elements to a surface. However, in other embodiments, the system and method may determine that the cues provided by the 2D content designer are sufficient and thus use these cues to map the content elements to a surface. Finally, the ultimate decision of the AR browser is to determine whether to use the cues provided by the 2D content designer or a predefined rule to map the content elements to a surface.

[0105] At 830, assuming that it is determined that the method is to use the cues provided by the 2D content designer, the method analyzes the cues and searches for a list of the identified surrounding surfaces available for displaying a particular content element based at least in part on the cues (e.g., querying the surface table 700). At 840, the method runs an optimal fit algorithm to select an optimal fit surface for a particular content element based on the provided cues. For example, the optimal fit algorithm may make a "main content" cue for a particular content element within a particular web page and attempt to identify a 3D surface from the available identified surrounding surfaces that is in front of and in the middle of the user 108 in the 3D environment. For example, Figure 2 the main video 220 is mapped to the vertical surface 182 because the main video 220 has a preference value of "main" in the preference 520 column of the element table 500 of Figure 5 and the vertical surface 182 is the surface that the user 108 is looking directly at and has the optimal size to display the main video 220.

[0106] At 850, the method stores the mapping result regarding the content element in the element-to-surface table in a non-transitory storage medium for use by the display algorithm to display the content elements on their respective mapped surfaces, whether those surfaces are the identified surrounding surfaces or virtual objects displayed in the user's surrounding environment. The non-transitory storage medium may include a data storage device such as the storage device 130 or the local storage device 140. The mapping result may be stored in a particular table such as the table Figure 9 disclosed in the following description.

[0107] Figure 9Shows an example of a table for storing the mapping of content elements from 2D content to surfaces according to some embodiments. The mapping table 900 is an exemplary table that stores the results of the process of mapping content elements to surfaces in a database. The mapping table 900 includes information about, for example, the content elements (e.g., element IDs) and the surfaces to which the content elements are mapped (e.g., surface IDs). Those of ordinary skill in the art will understand that the mapping table 900 can be a table stored in a relational database or any type of database or storage medium. Additionally, the mapping table 900 can be an array in a computer memory (e.g., cache) that contains the results of the mapping of elements to the identified surrounding surfaces at Figure 3 340.

[0108] Each row in the mapping table 900 corresponds to a content element that is mapped from 2D content to a surface in the user's physical environment 105 or a virtual object presented to the user 108, where the virtual object appears to be an object located in the user's physical environment 105. For example, in the current embodiment, the first entry in the element ID column stores the value of the element ID 220 corresponding to the main video 220. The surface ID value corresponding to the main video 220 in the surface ID column is 182, which corresponds to the vertical surface 182. In this way, the main video 220 is mapped to the vertical surface 182. Similarly, the user comment 230 is mapped to the horizontal surface 192, the recommended video 240 is mapped to the vertical surface 186, and the secondary tab 250 is mapped to the virtual Rolodex 190. The element IDs in the mapping table 900 can be associated with the element IDs in the Figure 5 element table 500. The surface IDs in the mapping table 900 can be associated with the surface IDs in the Figure 7 surface table 700.

[0109] Returning to Figure 8 , at 860, assuming that it is determined that the method is performed using predefined rules, the method queries a database containing the mapping rules for content elements to surfaces and determines which type of surface should be considered for mapping a specific content element within a web page. For example, for content from Figure 2The rules returned by the main video 220 may indicate that the main video 220 should be mapped to a vertical surface, and thus after searching the surface table 700, multiple candidate surfaces are shown (e.g., vertical surfaces 182 and 186 and the virtual Rolodex 190). At 870, a predefined set of rules may run a best-fit algorithm to select which surface from the available candidate surfaces is the best fit for the main video 220. At least in part based on the best-fit algorithm, and due to all of the candidate surfaces, it is determined that the main video 220 should be mapped to vertical surface 182, which is a surface within the direct line of sight of the user 108 and which has the best dimensions for displaying the video. Once the mapping of one or more elements is determined, at 850, the method stores the mapping results regarding the content elements in the mapping of elements to surfaces in a non-transitory storage medium as described above.

[0110] Return to Figure 3 , the method continues to display one or more elements as virtual content on the mapped surface at 350. The head-mounted system 160 may include one or more display devices within the head-mounted system 160, such as a micro-projector (not shown) to display information. As mapped at 340, the one or more elements are displayed on the respective mapped surfaces. Using the head-mounted system 160, the user 108 will see the content on the respective mapped surfaces. One of ordinary skill in the art will understand that the content elements are displayed as appearing physically attached to various surfaces (physical or virtual), but in fact, as perceived by the user 108, the content elements are actually projected onto physical surfaces, and in the case of virtual objects, the virtual objects are displayed as appearing attached to the respective surfaces of the virtual object. One of ordinary skill in the art will understand that when the user 108 turns their head or looks up or down, the display devices within the head-mounted system 160 can continue to attach the content elements to their respective surfaces to further give the user 108 the perception that the content is attached to the mapped surface. In other embodiments, the user 108 may change the content of the user's physical environment 105 by movements made by the user 108's head, hands, eyes, or voice.

[0111] Improved Browser / Application Implementation

[0112] In a mixed reality system, the user's workspace is not limited by the size of a display screen. Thus, unlike a conventional browser, browser windows in a mixed reality system can be placed and retained anywhere in the user's environment. The problem is that conventional browser technologies are configured to assume that the displayable browser locations must be restricted within the bounds of the display screen.

[0113] The following part of the present disclosure is directed to an improved method of viewing windows in a mixed reality environment. Using a mixed reality device, a user can have multiple browser windows associated with and placed within the user's physical space. For example, the user can open a first browser window in a first room and a second browser window in a second room. The problem addressed by this part of the present disclosure relates to a situation in which a browser window is opened such that it is anchored to a location in a first position, such that the browser window is no longer visible when the user moves to a second position. The problem is that as the user changes the environment (e.g., moves between rooms or goes to different geographical locations), the user may still need to access his / her previous sessions in the previous geographical locations.

[0114] Figure 10 A flowchart of a method for implementing viewing of a user's windows is shown, regardless of the current position of the user relative to one or more previously opened windows. In some embodiments, a control interface is provided to select the display of all and / or multiple windows associated with the user. In some embodiments, the control interface can be a user interface such that the user can interact with the MR display system, for example, by providing user input to the system and the system responding by executing a corresponding command. In some embodiments, the user can interact with the visual, auditory, tactile, or other aspects of the MR system. In some embodiments, the user interface can include a browser hub, which in some embodiments can be a visual representation of one or more aspects of one or more browser applications. For example, an "all windows" icon can be presented within the browser hub, where selecting the "all windows" icon can initiate the display of multiple windows associated with the user, regardless of the user's position relative to the current window location (e.g., the location where the window is opened). Figure 10 Starting at step 1702 when the system receives a command to display all or multiple windows (1702). In some embodiments, step 1702 can occur when the user selects an all windows icon that can be located within the browser hub user interface. In some embodiments, the system receives a selection for more than one window. In some embodiments, the system can receive user input that indicates that the user wants to view more than one window associated with the user's system.

[0115] At 1704, information is retrieved for a plurality of windows associated with a user. In some embodiments, a user may have one or more windows associated with the user. The windows for which information is collected may be located at disparate physical locations. According to some embodiments, instead of each application independently managing browser windows on a one-to-one basis in a VR / AR environment, the windows may alternatively be rendered as bounded volumes, which may hereinafter be referred to as "prisms". Each prism may have characteristics and attributes that allow a universe application to manage and display the prism in a VR / AR environment such that the universe application can manage the placement and display of virtual content in the VR / AR environment by managing the prism itself. Information about the windows may be collected by accessing a database of prisms associated with the user, where the prism may display one or more windows at a specified location.

[0116] In some embodiments, an "all windows" view is loaded that displays all open windows and tabbed windows, each represented by a preview, icon, domain name, and / or page title or any other suitable visual representation of the window (1706). In some embodiments, examples of open windows include windows with which one or more users are actively interacting. Other examples include placed applications / windows / browsers, whether they have an open / active state, paused state, stopped state, closed state, etc. Additionally, as long as an instance of an application exists / is placed and has one or more tabs with content, in some embodiments, it can be remotely accessed using the current inventive method. As another example, open windows may correspond to some or all of the prisms associated with a given application (e.g., a browser), regardless of its state (active, paused, closed, etc.), and can be remotely accessed via the "all windows" view in the current embodiment. In some embodiments, the "all windows" view may include all browser windows contained within one or more prisms at one or more physical locations within the real world. Figures 12 - 14 Examples of the "all windows" and similar "all applications" views are shown and described below. Although the "all windows" is used as an example, any other single application may alternatively be used. Although the "all applications" is used as an example, any subset of all applications may alternatively be used. The individual windows identified in step 1704 may be displayed in the user's current location in this manner. This may be done by changing the location parameter of the identified windows to the location in the user's current physical environment, effectively summoning the windows to the user. In some embodiments, this may be achieved by creating a copy of the window information and alternatively associating the new location with the information, such as a location at or near the user's current location. The windows are then rendered (in preview form, thumbnail form, and / or full form) and displayed to the user at the coordinates assigned to the individual windows and / or window prisms.

[0117] At 1708 (which is optional in the method), a hover state can be recognized and the hover state can act relative to one or more windows. For example, on hover, the window being hovered over can be moved into the foreground and other windows can optionally be moved slightly back. A window with multiple tabs can be slightly expanded to show the background tabs. In some embodiments, instead of a window, the object being hovered over can be any visual representation of a browser window, such as a preview, full screen, or scaled-down screen. At 1710, the user selects one or more of the windows. In some embodiments, the user can select a window by clicking a button on a controller (e.g., a totem), or by performing a specific gesture, or by looking at the window for a predetermined period of time. If the user selects a window, a replica of the original window will be loaded in the foreground of the user's FOV and all window views will be closed. In some embodiments, depending on the user's selection preference, the replica updates the original, the replica updates all or some other copies, and / or the replica is independent of the original. In some embodiments, the content loaded in the foreground corresponds to an existing prism that has been moved (e.g., not pinned and moved as a whole). In some embodiments, the content loaded in the foreground corresponds to a replicated existing prism with new associated location information. If the user activates a context menu, a user menu can be presented to the user that includes options for closing the window, adding it to a collection, and / or minimizing the window. The context menu can be a user interface with predetermined user interface options that tell the system to perform a specific function when selected. In some embodiments, the context menu can be activated by pressing on the center of a touchpad on the totem while hovering over a selectable object such as a window. In some embodiments, the context window can be similar to a right click on a desktop computer in that the action enables the user to perform actions such as moving, closing, etc. on the selected object.

[0118] Figure 11A -B shows this process of displaying windows for the user regardless of the previous physical location of the windows. In a mixed reality embodiment, the windows can be associated with the device and / or the physical space. The user can place content throughout the home or at different geographical locations throughout the day. In Figure 11A it can be seen that the first browser window 1 has been placed at a first physical location while the second browser window 2 has been placed at a second physical location. Since in a mixed reality embodiment the windows are associated with a specific physical location / coordinate space, this means that window 1 is typically only visible when the user 108 is at physical location 1 and not visible when the user 108 is at physical location 2. Similarly, window 2 is typically only visible when the user 108 is at physical location 2 and not visible when the user 108 is at physical location 1.

[0119] As Figure 11B shown, the "All Windows" view 1804 allows user 108 to view, reopen, and close open windows regardless of physical location (for examples of "open" windows, see the previous paragraph). Thus, view 1804 can display manipulable versions (e.g., visual representations) of window 1 and window 2 even though these windows are associated with different physical locations. When accessed from the control hub of the browser, viewing all windows (or optionally "All Windows") allows the user to view all open windows regardless of their physical or geographical location. The windows can be in the same room, different rooms, or entirely in another space. Screenshots, website icons, domains, and / or page titles are used to identify (e.g., visually represent) each window. In some embodiments, a window with multiple tabs displays a stacked preview of the underlying tabs when hovered over. Using the context menu, the user can open a new instance of the window, close the window, minimize the window, flag the window, and add the window to a collection—regardless of location. Global buttons can also be provided and used to close or minimize all open windows.

[0120] Figures 12 - 13 Examples of possible ways to display multiple windows within a mixed reality interface are provided. These illustrate example methods of implementing an interface in which multiple windows are displayed and presented to the user. Any window in the browser window can be selected via a suitable user input device such as a pointing device for further viewing by the user. If there are too many windows to fit on the interface, in some embodiments, additional windows may "ghost" visually (as shown to the right of Figure 12 and Figure 13 ), and scroll controls are provided to scroll to the additional windows.

[0121] Thus, an improved method of viewing windows in a mixed reality environment has been described, in which a view is provided with the user's windows regardless of the user's current position relative to one or more previously opened windows. This answers and solves the situation where a user may want to access one or more browser windows associated with one or more different physical locations when using a mixed reality device.

[0122] Although the above embodiments have been described in terms of browser applications, the scope of the claims also encompasses any other application or group of applications. In some embodiments, all applications in the operating system can be selected and displayed according to the claims. Such embodiments would have applications in a prism rather than parsed browser content in windows.

[0123] In Figure 14An embodiment is depicted that shows multiple applications in multiple prisms. The "All" button is an exemplary drop-down filter to help categorize application options for display and selection (e.g., by category). Although other suitable selection or filtering methods and / or interfaces can be used, an exemplary slider bar in the range of 9m to 30m selects the applications to be included in all application / landscape manager displays based on the distance from the user. In some embodiments, the user can set the slider bar to a smaller distance corresponding to a room to display all the applications available in that room. In some embodiments, the user can set the slider bar to a larger distance corresponding to the house to display all the applications available in the entire house. In some embodiments, the slider bar can be set to the far right corresponding to all applications regardless of location. The "Close All" button is an exemplary user interface element for controlling and / or manipulating applications. As described above, other user interface elements can be all opened, moved, etc. Figure 14 Two different instances of the "HELLO" application and the "Collection" application among the opened applications are depicted. Thus, the "All" button can display an application as well as multiple instances of different applications.

[0124] Additional Embodiments

[0125] Other embodiments of the present disclosure are described below. These additional embodiments can incorporate elements from the embodiments disclosed above, and the elements of these additional embodiments can be incorporated into the embodiments disclosed above.

[0126] 1. A computer program product embodied on a computer-readable medium, the computer-readable medium storing a sequence of instructions thereon, the sequence of instructions, when executed by a processor, causing the processor to perform a method comprising the following aspects:

[0127] Receive an instruction to select multiple opened windows;

[0128] Retrieve information for the multiple opened windows, wherein the multiple opened windows are associated with different physical locations;

[0129] Display a representation of the multiple opened windows in a single user interface; and

[0130] When a selection of a selected window among the multiple opened windows is received, load the selected window into the foreground for the user's field of view..

[0131] 2. The computer program product according to embodiment 1, wherein the representations of the plurality of open windows are displayed in the single user interface by changing the position parameters for the plurality of open windows to a position within the current physical environment of the user.

[0132] 3. The computer program product according to embodiment 2, wherein the plurality of open windows are rendered and displayed to the user at coordinates within the single user interface that are assigned to the plurality of open windows.

[0133] 4. The computer program product according to embodiment 3, wherein the plurality of open windows are rendered in at least one of a preview form, a thumbnail form, or a full form.

[0134] 5. The computer program product according to embodiment 1, wherein all open windows are selected for display in the single user interface.

[0135] 6. The computer program product according to embodiment 1, wherein each of the plurality of open windows is rendered as a bounded volume for the placement of virtual content.

[0136] 7. The computer program product according to embodiment 1, wherein a hover state is implemented, wherein the window being hovered over is moved to the foreground while the other windows visually recede.

[0137] 8. The computer program product according to embodiment 1, wherein when the selection of the selected window is received, the unselected windows are closed.

[0138] 9. A method for displaying windows in a computing environment, comprising:

[0139] Receiving an instruction to select a plurality of open applications;

[0140] Retrieving information for the plurality of open applications, wherein the plurality of open applications are associated with different physical locations;

[0141] Displaying representations of the plurality of open applications in a single user interface; and

[0142] When a selection of a selected application among the plurality of open applications is received, loading the selected application into the foreground for the user's field of view.

[0143] 10. The method according to embodiment 9, wherein the representations of the plurality of open applications are displayed in the single user interface by changing the position parameters for the plurality of open applications to a position within the current physical environment of the user.

[0144] 11. The method according to embodiment 10, wherein the plurality of open applications are rendered and displayed to the user at coordinates assigned to the plurality of open applications within the single user interface.

[0145] 12. The method according to embodiment 11, wherein the plurality of open applications are rendered in at least one of a preview form, a thumbnail form, or a full form.

[0146] 13. The method according to embodiment 9, wherein all open applications are selected for display in the single user interface.

[0147] 14. The method according to embodiment 9, wherein each of the plurality of open applications is rendered as a separate prism for placement of virtual content.

[0148] 15. The method according to embodiment 9, wherein a hover state is implemented, wherein the application being hovered over moves into the foreground while other applications visually recede.

[0149] 16. The method according to embodiment 9, wherein upon receiving the selection of the selected application, unselected applications are closed.

[0150] Overview of System Architecture

[0151] Figure 15 is a block diagram of an exemplary computing system 1400 suitable for implementing embodiments of the present disclosure. Computing system 1400 includes a bus 1406 or other communication mechanism for conveying information that interconnects subsystems and devices, such as a processor 1407, a system memory 1408 (e.g., RAM), a static storage device 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., a modem or an Ethernet card), a display 1411 (e.g., a CRT or an LCD), and an input device 1412 (e.g., a keyboard and a mouse).

[0152] According to some embodiments, computing system 1400 performs particular operations by processor 1407 executing one or more sequences of one or more instructions contained in system memory 1408. Such instructions can be read into system memory 1408 from another computer-readable / usable medium (e.g., static storage device 1409 or disk drive 1410). In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Accordingly, embodiments of the present disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware for implementing all or part of the present disclosure.

[0153] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1407 for execution. Such a medium may take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive 1410. Volatile media includes dynamic memory, such as system memory 1408.

[0154] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0155] In an embodiment of the present disclosure, the execution of a sequence of instructions for practicing the present disclosure is performed by a single computing system 1400. According to other embodiments of the present disclosure, two or more computing systems 1400 coupled by a communication link 1415 (e.g., LAN, PTSN, or wireless network) may cooperate to execute a sequence of instructions required to practice the present disclosure.

[0156] Computing system 1400 may send and receive messages, data, and instructions, including programs (i.e., application code), via communication interface 1414 over communication link 1415. The received program code may be executed by processor 1407 as received and / or stored on disk drive 1410 or other non-volatile storage for later execution. Computing system 1400 may communicate with database 1432 on external storage device 1431 via data interface 1433.

[0157] In the foregoing specification, the present disclosure has been described with reference to specific embodiments of the present disclosure. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above-described processing flow is described with reference to a specific order of processing actions. However, the order of many of the described processing actions can be changed without affecting the scope or operation of the present disclosure. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method for displaying windows in a computing environment, comprising: Receiving, at a mixed reality device, a first instruction from a user interface to select a first open window and a second open window from representations of a plurality of open windows, wherein The first open window includes first content displayed at a first physical location and is visible within the field of view, which is perceived by the user when the user is at a user location in the physical environment and has a current pose relative to the physical environment, and The second open window includes second content displayed at a second physical location, and when the first open window including the first content is rendered visible to the user, the second open window is not visible within the field of view of the user at the user location and having the current pose; and When a second instruction is received from the user interface to display the second content placed in the second open window to the user, displaying the second content as virtual content on a surface within the field of view of the user by placing the second content with respect to the field of view of the user, such that the first open window and the second open window are visible within the field of view of the user while maintaining the current position and the current pose relative to the physical environment.

2. The method according to claim 1, wherein, Displaying the second content as the virtual content includes: Accessing a surface inventory list of one or more surfaces visible to the user; and Accessing an element inventory list of elements for identifying one or more elements displayed as the second content in the second open window.

3. The method according to claim 2, wherein, Displaying the second content as the virtual content includes: Determining at least one surface for the one or more elements from the surface inventory list at least partially based on a first characteristic of the one or more elements and a second characteristic of at least one surface; Mapping the one or more elements to the at least one surface; and Rendering the one or more elements on the at least one surface.

4. The method according to claim 1, wherein, Displaying the second content as the virtual content includes: Executing a third instruction, wherein the execution of the third instruction includes: Modifying a position parameter corresponding to the second open window to a modified position parameter; Rendering, at least partially based on the modified position parameter, the second open window including the virtual content regarding the second content within the field of view so as to display both the first open window and the second open window within the field of view of the user; or Executing a fourth instruction, wherein the execution of the fourth instruction includes: Creating a copy of window information regarding a third open window; Associate a separate position parameter with the third open window, where the separate position parameter is determined in the physical environment such that when the third open window is rendered with the separate position parameter, the third open window is visible within the field of view of the user who maintains the user position and the current pose together with the first open window; and Render the third open window including the virtual content regarding the second content within the field of view at least partially based on the separate position parameter, so as to display both the first open window and the third open window within the field of view of the user.

5. The method according to claim 4, wherein Displaying the second content as the virtual content includes:[[]] Copy the second content as a copied content; and Display the copied content on a surface in a three-dimensional environment within the field of view of the user and in a two-dimensional display; Mark or classify the surface using at least one of the size or orientation of the surface, the distance between the surface and the user, or the type of the copied content or two-dimensional content to be displayed; Store information related to the surface in a data structure, the data structure including a plurality of columns, the plurality of columns being respectively allocated for one or more surface identifiers of one or more surfaces, an indicator regarding the real or virtual type of the surface, a plurality of entity indicators, the one or more positions, heights, sizes or orientations of the one or more surfaces.

6. The method according to claim 5, wherein, Displaying the second content as the virtual content includes:[[]] Update the virtual content displayed to the user and the second content displayed in the second open window at the second physical position that is invisible to the user to reflect changes in the second content.

7. The method according to claim 5, wherein Displaying the second content as the virtual content includes:[[]] Update the virtual content displayed to the user to reflect changes in the second content, while the second content displayed in the second open window remains unchanged with the changes, where the second content displayed in the second open window includes a two-dimensional display of the second content, and the virtual content displayed to the user includes a three-dimensional display of the second content.

8. A system for displaying windows in a computing environment, comprising:[[]] A mixed reality display device for displaying three-dimensional content; A processor; A memory for holding programmable code executable by the processor, where the programmable code includes instructions that, when executed by the processor, cause the processor to perform a set of actions, the set of actions including:[[]] Receive, at the mixed reality device, a first instruction selected from a representation of a plurality of open windows including a first open window and a second open window from a user interface, where the first open window includes first content displayed at a first physical position and is visible within the field of view, where the field of view is perceived by the user when the user is at a user position in the physical environment and has a current pose relative to the physical environment, and The second opened window includes second content displayed at a second physical location, and when the first opened window including the first content is rendered visible to the user, the second opened window is not visible within the field of view of the user at the user's location and having the current pose; and When a second instruction for displaying the second content placed in the second opened window to the user is received from the user interface, the second content is displayed as virtual content on a surface located within the field of view of the user by placing the second content within the field of view of the user, so that the first opened window and the second opened window are visible within the field of view of the user while maintaining the current position and the current pose relative to the physical environment.

9. A wearable mixed reality system, comprising: A processor; And A non-transitory computer-readable storage medium having a sequence of instructions stored thereon, the sequence of instructions, when executed by the processor, causes the processor to perform a set of actions, the set of actions including: Identifying a plurality of elements from two-dimensional (2D) content; Using one or more sensors in the wearable mixed reality system to identify one or more surfaces in the physical environment where the user wearing the wearable mixed reality system is located; Mapping the elements identified from the 2D content to a surface among the one or more surfaces; and Displaying the elements as virtual content onto the surface for the user, wherein identifying the one or more surfaces includes: Identifying one or more prisms rendered in the physical environment, and Determining from a database one or more prisms associated with the user wearing the wearable mixed reality system, wherein the database stores therein physical location data of the one or more prisms and data representing the association of the one or more prisms with one or more applications executable by the wearable mixed reality system.

10. A method, comprising: Identifying a plurality of elements from two-dimensional (2D) content; Using one or more sensors in a wearable mixed reality system to identify one or more surfaces in the physical environment where the user wearing the wearable mixed reality system is located; Mapping the elements identified from the 2D content to a surface among the one or more surfaces; And Displaying the elements as virtual content onto the surface for the user, wherein identifying the one or more surfaces includes: Identifying one or more prisms rendered in the physical environment, and Determining from a database one or more prisms associated with the user wearing the wearable mixed reality system, wherein the database stores therein physical location data of the one or more prisms and data representing the association of the one or more prisms with one or more applications executable by the wearable mixed reality system.

Images