Object creation system and method using physical manipulation

Abstract

Improved systems and methods for navigating and manipulating interactable objects in a 3D mixed reality environment are described. Improved systems and methods are provided to enable physical manipulation of creating and placing interactable objects such as browser windows and wall hangings. The method includes receiving data indicating a selection of an interactable object contained within a first prism at the beginning of a user interaction. The method also includes receiving data indicating an end of a user interaction with the interactable object. The method further includes receiving data indicating a physical movement of a user corresponding to removal of the interactable object from the first prism between the beginning and the end of the user interaction. In addition, the method includes creating a second prism to contain data associated with the interactable object at the end of the user interaction with the interactable object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to U.S. Provisional Application No. 62 / 634,007, filed on February 22, 2018, entitled “Window Creation With Physical Manipulation”; and U.S. Provisional Application No. 62 / 665,251, filed on May 1, 2018, entitled “Object Creation With Physical Manipulation”. This disclosure is related to U.S. Provisional Application No. 62 / 492,292, filed on May 1, 2017, entitled “Spatially Deconstructed Browser”. The contents of the above-mentioned patent applications are hereby expressly and completely incorporated by reference in their entirety, as if fully set forth. Technical Field

[0003] The present disclosure relates to systems and methods for implementing browsing techniques in a spatial 3D environment. Background Art

[0004] The typical way to view a web page is to open the web page on a display of a computer, smart phone, tablet computer, etc. The user will scroll through the web page to view different content displayed on the web page. Usually, whether the user is looking at a computer display, a smart phone or a tablet computer, there is a fixed format for how to display content on the display.

[0005] Using virtual reality, augmented reality and / or mixed reality systems (hereinafter collectively referred to as "mixed reality" systems), a three-dimensional environment is provided for displaying content to users. The problem is that traditional methods of displaying and manipulating 2D content in a browser do not work well when used in a 3D environment.

[0006] For example, consider the process of creating a new window in a computing environment. Typically, a mouse or similar device is used to manipulate a menu structure to create a window, and then the mouse is used to drag the new window to the desired location. However, for traditional 2D web browsers, since the display area of ​​the display device is limited to the screen area of ​​the display that is displaying the content, this means that traditional browsers are configured to only know how to create and place in the display display area. Therefore, traditional window creation methods are greatly limited when creating new windows in non-2D environments (such as mixed reality systems) because 3D environments are not limited to the strict limitations of the display display area. In this way, traditional 2D browsers perform suboptimally when used in a 3D environment because traditional browsing technology simply does not have the function or ability to create and display content using a 3D environment.

[0007] This problem also exists for other types of objects in computing environments where conventional methods of interacting with computing environments do not allow for efficient methods for manipulating objects in 3D environments. For example, conventional computing systems simply do not provide an efficient mechanism to enable placement of newly created objects (such as hanging a digital picture on a wall).

[0008] Therefore, there is a need for improved methods of enabling creation and manipulation of objects in 3D environments. Summary of the invention

[0009] Improved systems and methods are provided to enable physical manipulation for creating and placing interactable objects. In some embodiments, a user of a mixed reality system is provided with the ability to place a newly created virtual object (i.e., data associated with the interactable object) into any eligible location in the user's environment, wherein placement of the interactable object is achieved through physical manipulation activities of the user.

[0010] Improved systems and methods for physical manipulation of creating and placing browser windows are provided. In some embodiments, a user of a mixed reality system is provided with the ability to place a browser window to any eligible location in the user's environment, wherein the placement of the window is achieved through physical manipulation activities of the user. The improved systems and methods for navigating and manipulating browser windows can be applied in the context of 2D content that is deconstructed and displayed in a spatially organized 3D environment. This can include identifying 2D content, identifying elements in the 2D content, identifying surrounding surfaces, mapping the identified elements to the identified surrounding surfaces, and displaying the elements as virtual content on the surrounding surfaces.

[0011] Some embodiments relate to improved systems and methods for implementing physical manipulation for creating and placing wall hangings in a single user interaction. In some embodiments, a user of a mixed reality system is provided with the ability to place a wall hanging at any eligible location in the user's environment, wherein the placement of the wall hanging is achieved through physical manipulation activities of the user.

[0012] Further details of aspects, objects and advantages of the invention are described below in the detailed description, drawings and claims.Both the foregoing summary and the following detailed description are exemplary and explanatory and are not intended to limit the scope of the invention.

[0013] In one embodiment, the method includes receiving data indicating a selection of an interactable object contained within a first prism at the beginning of a user interaction. The method also includes receiving data indicating an end of a user interaction with the interactable object. The method further includes receiving data indicating a body movement of a user corresponding to a removal of the interactable object from the first prism between the beginning and the end of the user interaction. Additionally, the method includes creating a second prism to contain data associated with the interactable object at the end of the user interaction with the interactable object.

[0014] In one or more embodiments, the method further includes: when the interactive object is outside the boundaries of the first prism, displaying a visual preview of data associated with the interactive object at the location of the interactive object between the start and end of the user interaction. The method may also include: the user input device provides data indicating the selection of the interactive object and data indicating the end of the user interaction with the interactive object. The user input device may be a tactile controller. The tactile controller may correspond to a totem device having at least six degrees of freedom. At a processing system associated with the tactile controller, the user's body movement may be converted into an event object, wherein the event object is associated with data for the movement or positioning of the tactile controller. By identifying the location where the trigger for the tactile controller is released after the selection of the interactive object, the location of the interactive object at the end of the user interaction can be determined. The user input device may be a camera for detecting user gestures. The data indicating the selection of the interactive object may correspond to a selection gesture. The data indicating the end of the user interaction with the interactive object may correspond to a release gesture.

[0015] In one or more embodiments, the data indicating the selection of the interactive object corresponds to at least a portion of the user's body that interacts with the link. A preview of the link can be generated by accessing the content of a URL associated with the link. The method may also include: receiving data indicating the user's body movement, the user's body movement corresponding to the physical movement of the preview. The interactive object can be opened and placed with the help of user interaction. The orientation of the interactive object can be determined by the following operations: determining the surface normal of the position of the interactive object at the end of the user interaction; determining the orientation of the data associated with the interactive object; and aligning the surface normal of the coordinate position with the orientation of the data associated with the interactive object. The interactive object may include a wall hanging.

[0016] In another embodiment, the method includes identifying an interactable object. The method also includes identifying a body movement of a user corresponding to the interactable object. The method further includes determining a coordinate location of data associated with the interactable object based on the body movement of the user. In addition, the method includes placing the data associated with the interactable object at the coordinate location corresponding to the body movement of the user.

[0017] In one or more embodiments, a user input device is used to detect the user's body movement. The user input device can be a tactile controller. The tactile controller can correspond to a totem device having at least six degrees of freedom. At a processing system associated with the tactile controller, the user's body movement can be converted into an event object, wherein the event object is associated with data indicating the movement or positioning of the tactile controller. The coordinate position of the interactive object can be determined by identifying the position where the trigger for the tactile controller is released after the user's body movement.

[0018] In one or more embodiments, the user input device is a camera for detecting user gestures. The interactive object to be opened can be identified by identifying a link that interacts with a part of the user's body. A preview of the link can be generated by accessing the content of a URL associated with the link. The user's body movement can correspond to the physical movement of the preview. The interactive object can be opened and placed with the help of a single user interaction of the computing device. The orientation of the interactive object can be determined by determining the surface normal of the coordinate location, determining the orientation of the data associated with the interactive object, and aligning the surface normal of the coordinate location with the orientation of the data associated with the interactive object. The interactive object may include a wall hanging.

[0019] In another embodiment, a system includes a mixed reality display device configured to display three-dimensional content. The system also includes a user input device configured to detect a user's body movement. The system further includes: a processor that receives and processes data from the user input device; a memory that stores programmable code that can be executed by the processor. The programmable code includes instructions for identifying an interactive object, identifying a user's body movement corresponding to the interactive object, determining a coordinate position of data associated with the interactive object based on the user's body movement, and placing the data associated with the interactive object at a coordinate position corresponding to the user's body movement.

[0020] In one or more embodiments, the user input device is a tactile controller. The tactile controller may correspond to a totem device having at least six degrees of freedom. At the processor, the user's body movement may be converted into an event object, wherein the event object is associated with data indicating the movement or positioning of the tactile controller. The coordinate position of the interactive object may be determined by identifying the position at which a trigger for the tactile controller is released after the user's body movement.

[0021] In one or more embodiments, the user input device is a camera for detecting user gestures. The interactive object to be opened can be identified by identifying a link that interacts with a part of the user's body. A preview of the link can be generated by accessing the content of the URL associated with the link. The user's body movement can correspond to the physical movement of the preview. The interactive object can include a wall hanging. The programmable code may also include instructions for determining the orientation of the interactive object by: determining the surface normal of the coordinate location; determining the orientation of the data associated with the interactive object; and aligning the surface normal of the coordinate location with the orientation of the data associated with the interactive object.

[0022] In another embodiment, a computer program product embodied on a computer readable medium having stored thereon a sequence of instructions that, when executed by a processor, causes the processor to perform a method, the method comprising identifying an interactable object. The method further comprises identifying a body movement of a user corresponding to the interactable object. The method further comprises determining a coordinate location of data associated with the interactable object based on the body movement of the user. Furthermore, the method comprises placing the data associated with the interactable object at a coordinate location corresponding to the body movement of the user.

[0023] In one or more embodiments, a user input device is used to detect the user's body movement corresponding to the window. The user input device can be a tactile controller. The tactile controller can correspond to a totem device having at least six degrees of freedom. At a processing system associated with the tactile controller, the user's body movement can be converted into an event object, wherein the event object is associated with data indicating the movement or positioning of the tactile controller. The coordinate position of the interactive object can be determined by identifying the position where the trigger for the tactile controller is released after the user's body movement.

[0024] In one or more embodiments, the user input device is a camera for detecting user gestures. The window to be opened can be identified by identifying a link that interacts with a part of the user's body. A preview of the link can be generated by accessing the content of the URL associated with the link. The user's body movement can correspond to the physical movement of the preview. The interactive object can be opened and placed with the help of a single user interaction of the computing device. The single user interaction can include a single press of a button on the computing device. The instruction sequence further determines the orientation of the interactive object by determining the surface normal of the coordinate position, determining the orientation of the data associated with the interactive object, and aligning the surface normal of the coordinate position with the orientation of the data associated with the interactive object.

[0025] In another embodiment, the method includes receiving data indicating a selection of an interactable object at the beginning of a user interaction. The method also includes receiving data indicating an end of a user interaction with the interactable object. The method further includes determining a position of the interactable object at the end of the user interaction. In addition, the method includes placing data associated with the interactable object at the position of the interactable object at the end of the user interaction.

[0026] In one or more embodiments, the method further includes: a user input device provides data indicating a selection of an interactive object and data indicating an end of a user interaction with the interactive object. The user input device may be a tactile controller. The tactile controller may correspond to a totem device having at least six degrees of freedom. At a processing system associated with the tactile controller, a user's body movement may be converted into an event object, wherein the event object is associated with data for movement or positioning of the tactile controller. The position of the interactive object at the end of the user interaction may be determined by identifying a position at which a trigger for the tactile controller is released after selection of the interactive object.

[0027] In one or more embodiments, the user input device is a camera for detecting user gestures. Data indicating the selection of an interactive object may correspond to a selection gesture. Data indicating the end of a user interaction with an interactive object may correspond to a release gesture. Data indicating the selection of an interactive object may correspond to at least a portion of the user's body interacting with the link. A preview of the link may be generated by accessing the content of a URL associated with the link. The method may further include: receiving data indicating a user's body movement, wherein the user's body movement corresponds to a physical movement of the preview. The interactive object may be opened and placed with the aid of user interaction. The orientation of the interactive object may be determined by determining a surface normal of the position of the interactive object at the end of the user interaction, determining an orientation of the data associated with the interactive object, and aligning the surface normal of the coordinate position with the orientation of the data associated with the interactive object. The interactive object may include a wall hanging.

[0028] In another embodiment, the method includes receiving data indicating a selection of an interactable object at the beginning of a user interaction. The method also includes receiving data indicating an end of a user interaction with the interactable object. The method further includes determining a position of the interactable object at the end of the user interaction. In addition, the method includes creating a new prism for data associated with the interactable object at the end of the user interaction when the position of the interactable object at the end of the user interaction is outside of an initial prism for the interactable object.

[0029] In one or more embodiments, the method further includes displaying a visual preview of data associated with the interactive object at the location of the interactive object between the start and end of the user interaction when the interactive object is outside the boundaries of the initial prism. The method may also include a user input device providing data indicating a selection of the interactive object and data indicating the end of the user interaction with the interactive object. The user input device is a tactile controller. The tactile controller may correspond to a totem device having at least six degrees of freedom. At a processing system associated with the tactile controller, the user's body movement may be converted into an event object, wherein the event object is associated with data for movement or positioning of the tactile controller. The position of the interactive object at the end of the user interaction may be determined by identifying the location at which a trigger for the tactile controller is released after selection of the interactive object.

[0030] In one or more embodiments, the user input device is a camera for detecting user gestures. Data indicating the selection of an interactive object may correspond to a selection gesture. Data indicating the end of a user interaction with an interactive object may correspond to a release gesture. Data indicating the selection of an interactive object may correspond to at least a portion of the user's body interacting with the link. A preview of the link may be generated by accessing the content of a URL associated with the link. The method may further include: receiving data indicating a user's body movement, wherein the user's body movement corresponds to a physical movement of the preview. The interactive object may be opened and placed with the aid of user interaction. The orientation of the interactive object may be determined by determining a surface normal of the position of the interactive object at the end of the user interaction, determining the orientation of the data associated with the interactive object, and aligning the surface normal of the coordinate position with the orientation of the data associated with the interactive object. The interactive object may include a wall hanging.

[0031] In another embodiment, a method includes receiving data indicating a selection of an interactable object contained within a first prism at the beginning of a user interaction. The method also includes creating a second prism at the location of the interactable object at the end of the user interaction to contain data associated with the interactable object when data representing a body movement of the user corresponding to the user interaction between the beginning and the end of the user interaction indicates that the interactable object is outside of the first prism at the end of the user interaction.

[0032] In one or more embodiments, the method further includes: a user input device provides data indicating a selection of an interactive object and data indicating a body movement of the user corresponding to the user interaction between the start and the end of the user. The user input device may be a tactile controller. The tactile controller may correspond to a totem device having at least six degrees of freedom. The body movement of the user may be converted into an event object at a processing system associated with the tactile controller, wherein the event object is associated with data for movement or positioning of the tactile controller. The position of the interactive object at the end of the user interaction may be determined by identifying the position at which a trigger for the tactile controller is released after selection of the interactive object.

[0033] In one or more embodiments, the user input device is a camera for detecting user gestures. Data indicating the selection of an interactive object may correspond to a selection gesture. Data indicating the end of a user interaction with an interactive object may correspond to a release gesture. Data indicating the selection of an interactive object may correspond to at least a portion of the user's body interacting with a link. A preview of a link may be generated by accessing the content of a URL associated with the link. The user's body movement may correspond to the physical movement of the preview. Interactive objects may be opened and placed with the aid of user interaction. The orientation of the interactive object may be determined by determining a surface normal of the position of the interactive object at the end of the user interaction, determining the orientation of the data associated with the interactive object, and aligning the surface normal of the coordinate position with the orientation of the data associated with the interactive object. The interactive object may include a wall hanging.

[0034] In another embodiment, the system includes a mixed reality display device configured to display three-dimensional content. The system also includes a user input device configured to detect a user's body movement. The system further includes a processor that receives and processes data from the user input device. In addition, the system includes a memory for storing programmable code executable by the processor, wherein the programmable code includes instructions for: receiving data indicating a selection of an interactive object contained within a first prism at the beginning of a user interaction, receiving data indicating an end of a user interaction with the interactive object, receiving data indicating a user's body movement corresponding to the removal of the interactive object from the first prism between the beginning and the end of the user interaction, and creating a second prism to contain data associated with the interactive object at the end of the user interaction with the interactive object.

[0035] In another embodiment, the system includes a mixed reality display device configured to display three-dimensional content. The system also includes a user input device configured to detect body movements of a user. The system further includes a processor that receives and processes data from the user input device. In addition, the system includes a memory for storing programmable code executable by the processor, wherein the programmable code includes instructions for: receiving data indicating a selection of an interactive object at the beginning of a user interaction, receiving data indicating the end of a user interaction with the interactive object, determining a position of the interactive object at the end of the user interaction, and when the position of the interactive object at the end of the user interaction is outside the initial prism of the interactive object, creating a new prism at the end of the user interaction for data associated with the interactive object.

[0036] In another embodiment, the system includes a mixed reality display device configured to display three-dimensional content. The system also includes a user input device configured to detect a user's body movement. The system further includes a processor that receives and processes data from the user input device. In addition, the system includes a memory for storing programmable code executable by the processor, wherein the programmable code includes instructions for: receiving data indicating a selection of an interactive object contained within a first prism at the beginning of a user interaction, and when data representing a user's body movement corresponding to the user interaction between the beginning and the end of the user interaction indicates that the interactive object is outside the first prism at the end of the user interaction, creating a second prism at the location of the interactive object at the end of the user interaction to contain data associated with the interactive object. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how the above and other advantages and objects of the various embodiments of the present disclosure are obtained, a more detailed description of the present disclosure briefly described above will be given by reference to specific embodiments of the invention shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present disclosure and, therefore, should not be considered limiting of the scope thereof, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0038] Figure 1 An augmented reality environment for deconstructing 2D content to be displayed in a user's 3D environment is shown in accordance with some embodiments.

[0039] Figure 2 An example mapping of elements of 2D content to a user's 3D environment is shown in accordance with some embodiments.

[0040] Figure 3 is a flow chart illustrating a method for deconstructing 2D content to be displayed in a 3D environment according to some embodiments.

[0041] Figure 4 is a flow chart illustrating a method for identifying elements in 2D content according to some embodiments.

[0042] Figure 5 An example of a table for storing elements deconstructed from 2D content is shown according to some embodiments.

[0043] Figure 6 is a flow chart illustrating a method for identifying a surface from a user's local environment according to some embodiments.

[0044] Figure 7An example of a table for storing an inventory of surfaces identified from a user's local environment is shown in accordance with some embodiments.

[0045] Figure 8 is a flow chart illustrating a method for mapping elements from 2D content to an available surface according to some embodiments.

[0046] Fig. 9 An example of a table for storing a mapping from elements of 2D content to surfaces of a user's local environment is shown in accordance with some embodiments.

[0047] Fig. 10A A flow chart illustrating a method of enabling physical placement and manipulation of data associated with an interactable object.

[0048] Fig. 10B A flow chart illustrating a method of implementing physical placement and manipulation of browser windows.

[0049] Fig. 10C A flow chart illustrating a method of achieving physical placement and manipulation of a wall mount.

[0050] Fig.11A -F provides schematic examples of methods for physically placing windows in a mixed reality environment.

[0051] Fig. 12A -F provides an illustrative example of a method for physically placing data associated with an interactable object in a mixed reality environment.

[0052] Fig.13 is a block diagram of an illustrative computing system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0053] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples of the present disclosure so that those skilled in the art can practice the present disclosure. It should be noted that the following drawings and examples are not meant to limit the scope of the present disclosure. In the case where known components (or methods or processes) can be used to partially or completely implement certain elements of the present disclosure, only those parts necessary for the present disclosure to understand the known components (or methods or processes) will be described, and the detailed description of other parts of the known components (or methods or processes) will be omitted, so as not to obscure the present disclosure. In addition, various embodiments encompass the current and future known equivalents of the components referred to herein by way of illustration.

[0054] Web page deconstruction

[0055] Embodiments of the present disclosure will deconstruct a 2D web page to be displayed in a spatially organized 3D environment. The 2D web page may originate from a web browser of a head mounted system, a mobile device (e.g., a phone), a tablet computer, a television, an application, etc. In some embodiments, the 2D web page may be received from another application or device such as a laptop computer, a desktop computer, an email application with a link to the 2D web page, an electronic message that references or includes a link to the 2D web page, etc.

[0056] Reference Figure( Figure 1 ), environment 100 represents a physical environment and systems for implementing the processes described below (e.g., deconstructing 2D content from a web page to display on a 3D surface in the user's physical environment 105). The representative physical environment and systems of environment 100 include the user's physical environment 105 as viewed by user 108 through head mounted system 160. The representative system of environment 100 further includes accessing 2D content (e.g., web pages) via a web browser 110 operably coupled to a network 120. Network 120 can be the Internet, an intranet, a private cloud network, a public cloud network, etc. Web browser 110 is also operably coupled to processor 170 via network 120. Although processor 170 is shown as an isolated component separate from head mounted system 160, in alternative embodiments, processor 170 can be integrated with one or more components of head mounted system 160, and / or can be integrated into other system components within environment 100 (such as, for example, network 120) to access computing network 125 and storage device 130. The processor 170 may be configured with software 150 to receive and process information, such as video, audio, and content, received from the head mounted system 160, the local storage device 140, the web browser 110, the computing network 125, and the storage device 130. The software 150 may communicate with the computing network 125 and the storage device 130 via the network 120. The software 150 may be installed on the processor 170, or in another embodiment, the features and functions of the software may be integrated into the processor 170. The processor 170 may also be configured with a local storage device 140 for storing information for the processor 170 to quickly access without having to rely on information stored remotely on an external storage device near the user 108. In other embodiments, the processor 170 may be integrated within the head mounted system 160.

[0057] The user's physical environment 105 is the physical surroundings of the user 108 as the user moves and views the user's physical environment 105 through the head mounted system 160. For example, referring to Figure 1, the user's physical environment 105 shows a room with two walls (e.g., a main wall 180 and a side wall 184, the main wall and the side wall are relative to the user's field of view) and a table 188. On the main wall 180, there is a rectangular surface 182 depicted by a solid black line to illustrate a physical surface with a physical boundary (e.g., a painting hanging or attached to a wall or window, etc.), which may be a candidate surface for projecting some 2D content onto it. On the side wall 184, there is a second rectangular surface 186 depicted by a solid black line to illustrate a physical surface with a physical boundary (e.g., a painting hanging or attached to a wall or window, etc.). On the table 188, there may be different objects. 1) a virtual Rolodex 190 in which certain 2D content can be stored and displayed; 2) a horizontal surface 192 depicted by solid black lines, which represents a physical surface with physical boundaries for projecting certain 2D content onto it; and 3) a plurality of stacked virtual square surfaces 194 depicted by dashed black lines, which represent, for example, stacked virtual newspapers in which certain 2D content can be stored and displayed.

[0058] The web browser 110 may also display blog pages from the Internet or within an intranet / private network. In addition, the web browser 110 may also be any technology that displays digital 2D content. The 2D content may include, for example, web pages, blogs, digital pictures, videos, news articles, newsletters, or music. The 2D content may be stored in a storage device 130 that the user 108 may access via the network 120. In some embodiments, the 2D content may also be streaming content, for example, a real-time video feed or a real-time audio feed. The storage device 130 may include, for example, a database, a file system, a permanent storage device, a flash drive, a cache, etc. In some embodiments, a web browser 110 containing 2D content (e.g., a web page) is displayed via a computing network 125.

[0059] The computing network 125 accesses the storage device 130 to retrieve and store 2D content for display in a web page on the web browser 110. In some embodiments, the local storage device 140 can provide the 2D content of interest to the user 108. The local storage device 140 may include, for example, a flash drive, a cache, a hard drive, a database, a file system, etc. The information stored in the local storage device 140 may include recently accessed 2D content or content in a 3D space that was recently displayed. The local storage device 140 allows the performance of the system of the environment 100 to be improved by locally providing certain content to the software 150 for assisting in deconstructing the 2D content to display the 2D content on a 3D space environment (e.g., a 3D surface in the user's physical environment 105).

[0060] The software 150 includes a software program stored in a non-transitory computer readable medium to perform the functions of deconstructing 2D content to be displayed in the user's physical environment 105. The software 150 may be executed on a processor 170, where the processor 170 may be locally attached to the user 108, or in some other embodiments, the software 150 and the processor 170 may be included within the head mounted system 160. In some embodiments, the features and functions of portions of the software 150 may be stored and executed on a computing network 125 remote from the user 108. For example, in some embodiments, deconstructing 2D content may occur on the computing network 125, and the results of the deconstruction may be stored within the storage device 130, where an inventory of surfaces of the user's local environment for presenting the deconstructed 2D content may occur within the processor 170, where the mapping and the inventory of surfaces are stored within the local storage device 140. In one embodiment, the process of deconstructing 2D content, inventorying local surfaces, mapping elements of the 2D content to local surfaces, and displaying elements of the 2D content may all occur locally within the processor 170 and the software 150.

[0061] The head mounted system 160 may be a virtual reality (VR) or augmented reality (AR) head mounted system that includes a user interface, a user sensing system, an environment sensing system, and a processor (all not shown). The head mounted system 160 presents an interface to the user 108 for interacting with and experiencing the digital world. The interaction may involve the user and the digital world, one or more other users interacting with the environment 100, and objects within the digital and physical worlds.

[0062] The user interface may include receiving 2D content and selecting elements within the 2D content by user input through the user interface. The user interface may be at least one or a combination of a tactile interface device, a keyboard, a mouse, a joystick, a motion acquisition controller, an optical tracking device, and an audio input device. A tactile interface device is a device that allows a person to interact with a computer through physical sensations and movements. Haptics refers to a type of human-computer interaction technology that includes tactile feedback or other physical sensations to perform actions or processes on a computing device.

[0063] The user sensing system may include one or more sensors 162 operable to detect certain features, characteristics, or information about the user 108 wearing the head mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical characteristics / measurements of the user 108, such as, for example, one or more of the following: pupil constriction / dilation, angle measurement / positioning of each pupil, sphericity, eye shape (changes in eye shape over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that may be used by the head mounted system 160 to enhance the user's viewing experience.

[0064] The environment sensing system may include one or more sensors 164 for obtaining data from the user's physical environment 105. Objects or information detected by the sensors 164 may be provided as input to the head mounted system 160. In some embodiments, the input may represent the user's interaction with the virtual world. For example, a user (e.g., user 108) viewing a virtual keyboard on a table (e.g., table 188) may gesture with their fingers as if the user were typing on the virtual keyboard. The motion of the finger movement may be collected by the sensors 164 and provided as input to the head mounted system 160, where the input may be used to change the virtual world or create new virtual objects.

[0065] Sensor 164 may include, for example, a generally outward-facing camera or scanner for interpreting scene information, for example, by continuously and / or intermittently projected infrared structured light. The environment sensing system may be used to map one or more elements of the user's physical environment 105 surrounding the user 108 by detecting and registering the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Thus, in some embodiments, the environment sensing system may include image-based 3D reconstruction software embedded in a local computing system (e.g., processor 170) and operable to digitally reconstruct one or more objects or information detected by sensor 164.

[0066] In an exemplary embodiment, the environment sensing system provides one or more of the following: motion acquisition data (including gesture recognition), depth sensing, facial recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors. It should be understood that the environment sensing system may include other components in addition to those discussed above.

[0067] As described above, in some embodiments, processor 170 can be integrated with other components of head mounted system 160, integrated with other components of systems of environment 100, or can be Figure 1 108). The processor 170 may be connected to the various components of the head mounted system 160 via a physical, wired connection or a wireless connection, such as, for example, a mobile network connection (including cellular telephone and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. The processor 170 may include a memory module, an integrated and / or attached graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of converting data from a source (e.g., the computing network 125 and the user sensing system and environment sensing system of the head mounted system 160) into image and audio data, where the image / video and audio may be presented to the user 108 via a user interface (not shown).

[0068] Processor 170 handles data processing for various components of head mounted system 160 and data exchange between head mounted system 160 and 2D content from web pages displayed or accessed by web browser 110 and computing network 125. For example, processor 170 may be used to buffer and process data streams between user 108 and computing network 125, thereby enabling a smooth, continuous, and high-fidelity user experience.

[0069] 2D content from a web page can be deconstructed into elements and mapped in an intelligent and logical manner for display on a surface in a 3D environment. A predetermined set of rules can be used to recommend, suggest, or indicate the placement of certain types of elements / content identified within the 2D content / web page. For example, certain types of 2D content elements may have one or more content elements that may need to be mapped to a physical or virtual object surface suitable for storing and displaying the one or more elements, while other types of 2D content elements may be a single object, such as a main video or main article within a web page, in which case the single object can be mapped to the most meaningful surface to display the single object to the user.

[0070] Figure 2 An example mapping of elements of 2D content to a user's 3D environment is shown in accordance with some embodiments. Environment 200 depicts 2D content (e.g., a web page) displayed or accessed by a web browser 110 and the user's physical environment 105. Dashed lines with arrows depict elements (e.g., a particular type of content) from the 2D content (e.g., a web page) mapped to and displayed on the user's physical environment 105. Certain elements in the 2D content are mapped to certain physical or virtual objects in the user's physical environment 105 based on web designer hints or predefined browser rules.

[0071] As an example, the 2D content accessed or displayed by the web browser 110 may be a web page with multiple tabs, in which the current active tab 260 is displayed, and the auxiliary tab 250 is currently hidden until the auxiliary tab 250 is selected for display on the web browser 110. Displayed within the active tab 260 is generally a web page. In this particular example, the active tab 260 is displaying a YOUTUBE page including a main video 220, user comments 230, and suggested videos 240. As shown in this exemplary example, the main video 220 may be mapped to be displayed on a vertical surface 182, the user comments 230 may be mapped to be displayed on a horizontal surface 192, and the suggested video 240 may be mapped to be displayed on a vertical surface 186 different from the vertical surface 182. In addition, the auxiliary tab 250 may be mapped to be displayed on a virtual Rolodex 190 and / or on a plurality of stacked virtual objects 194. In some embodiments, specific content within the auxiliary tab 250 may be stored in a plurality of stacked virtual objects 194. In other embodiments, the entire content within the auxiliary tab 250 may be stored and / or displayed on multiple stacked virtual objects 194. Similarly, the virtual Rolodex 190 may contain specific content from the auxiliary tab 250, or the virtual Rolodex 190 may contain the entire content located within the auxiliary tab 250.

[0072] The vertical surface 182 may be any type of structure that is already on the main wall 180 of the room (depicted as the user's physical environment 105), such as a window pane or a picture frame. In some embodiments, the vertical surface 182 may be an empty wall on which the head mounted system 160 determines the optimal size of the frame of the vertical surface 182 suitable for the user 108 to view the main video 220. Determining the size of the vertical surface 182 may be based at least in part on the distance of the user 108 from the main wall 180, the size and dimensions of the main video 220, the quality of the main video 220, the amount of uncovered wall space, and / or the posture of the user when looking at the main wall 180. For example, if the quality of the main video 220 is high resolution, the size of the vertical surface 182 may be larger because the quality of the main video 220 is not adversely affected by the vertical surface 182. However, if the video quality of the main video 220 is of poor quality, having a larger vertical surface 182 may significantly hinder the video quality. In this case, the method and system of the present disclosure can resize / redefine the vertical surface 182 to be smaller to minimize the poor video quality caused by pixelation.

[0073] Like vertical surface 182, vertical surface 186 is a vertical surface on an adjacent wall (e.g., side wall 184) in user's physical environment 105. In some embodiments, side wall 184 and vertical surface 186 may appear to be inclined surfaces on a slope based on the orientation of user 108. An inclined surface on a slope may also be a type of surface orientation in addition to vertical and horizontal surfaces. In this example, a suggested video 240 from a YOUTUBE webpage may be placed on vertical surface 186 of side wall 184 to allow user 108 to view the suggested video by simply moving his head slightly to the right.

[0074] The virtual Rolodex 190 may be a virtual object created by the head mounted system 160 and displayed to the user 108. The virtual Rolodex 190 may have the ability for the user 108 to bidirectionally cycle through a set of virtual pages. The virtual Rolodex 190 may contain an entire web page, or may contain individual articles or videos or audio. As shown in this example, the virtual Rolodex 190 may contain a portion of the content from the auxiliary tab 250, or in some embodiments, the virtual Rolodex 190 may contain the entire page of the auxiliary tab 250. In some embodiments, the virtual Rolodex 190 may also contain all or part of the content of the first tab. The user 108 may bidirectionally cycle through the content within the virtual Rolodex 190 by simply annotating a specific tab within the virtual Rolodex 190, and one or more sensors (e.g., sensor 162) within the head mounted system 160 will detect the eye focus of the user 108 and cycle through the tabs within the virtual Rolodex 190 accordingly to obtain relevant information for the user 108. In some embodiments, the user 108 can select relevant information from the virtual Rolodex 190 and instruct the head mounted system 160 to display the relevant information on an available surrounding surface or on another virtual object, such as a virtual display (not shown) in close proximity to the user 108 .

[0075] Similar to the virtual Rolodex 190, the plurality of stacked virtual objects 194 may contain content from the entire content of one or more tabs or from a specific content range of various web pages or tabs that the user 108 has marked, saved for future viewing, or has opened (i.e., inactive tabs). The plurality of stacked virtual objects 194 are also similar to a real-world stack of newspapers. Each stack within the plurality of stacked virtual objects 194 may belong to a specific newspaper article, page, magazine issue, recipe, etc. It will be appreciated by those of ordinary skill in the art that there may be multiple types of virtual objects to achieve the same purpose of providing a surface for placing 2D content elements or content in a 2D content source.

[0076] It will be appreciated by those of ordinary skill in the art that the 2D content accessed or displayed by the web browser 110 may be more than just a web page. In some embodiments, the 2D content may be pictures from an album, videos from a movie, a TV show, a YOUTUBE video, an interactive form, etc. However, in other embodiments, the 2D content may be an e-book or any electronic way of displaying a book. Finally, in other embodiments, since 2D content is typically how information is currently presented, the 2D content may also be other types of content not described. If the electronic device can consume 2D content, the head-mounted system 160 can use the 2D content to deconstruct and display the 2D content in a 3D setting (e.g., AR).

[0077] In some embodiments, mapping the accessed 2D content may include extracting the 2D content (e.g., from a browser) and placing it on the surface (so that the content is no longer in the browser and is only on the surface), and in some embodiments, mapping may include copying the content (e.g., from a browser) and placing it on the surface (so that the content is both in the browser and on the surface).

[0078] Deconstructing 2D content is a technical problem existing in the field of Internet and computer-related technologies. 2D content such as web pages is constructed using certain types of programming languages ​​such as HTML to instruct computer processors and technical components where and how to display elements within the web page on the screen for the user. As described above, web page designers usually work within the limitations of a 2D canvas (e.g., a screen) to place and display elements (e.g., content) within the 2D canvas. HTML tags are used to determine how to format an HTML document or a portion within an HTML document. In some embodiments, the (extracted or copied) 2D content can retain HTML tag references, and in some embodiments, HTML tag references can be redefined.

[0079] Figure 3 3 is a flow chart illustrating a method for deconstructing 2D content to be displayed in a 3D environment according to some embodiments. The method includes: identifying 2D content at 310; identifying elements in the 2D content at 320; identifying surrounding surfaces at 330; mapping the identified elements in the identified 2D content to the identified surfaces from the identified surrounding surfaces at 340; and displaying the elements as virtual content to the selected surface at 350, wherein the selected surface is selected from the mapping of the elements to the identified surfaces.

[0080] Identifying 2D content at 310 may involve searching for digital content using the head mounted system 160. Identifying 2D content at 310 may also include accessing digital content on a server (e.g., storage device 130) connected to the network 120. Identifying 2D content at 310 may include browsing a web page of interest to the user 108 on the Internet. In some embodiments, identifying 2D content at 310 may include a voice activated command given by the user 108 to search for content on the Internet. For example, the user 108 may be interacting with a device (e.g., head mounted system 160) where the user 108 requests the device to search for a specific video by speaking a command to search for the video and then speaking the name of the video and a brief description of the video, thereby searching for a specific video on the Internet. The device may then search the Internet and pull up the video on a 2D browser to allow the user 108 to view the video as displayed on the 2D browser of the device. The user 108 may then confirm that the video is a video that the user 108 wants to view in a spatial 3D environment.

[0081] After identifying the 2D content, the method identifies elements in the 2D content at 320 to inventory the available elements within the 2D content for display to the user 108. For example, the elements within the 2D content may include videos, articles and newsletters posted on a web page, comments and posts on social media sites, blog posts, pictures posted on various websites, audio books, etc. These elements within the 2D content (e.g., web page) may contain HTML tags with attributes associated with the HTML tags provided by the content designer to define where specific elements are placed on the web page, and in some cases when and how the elements will be displayed on the web page. In some embodiments, the methods and systems of the present disclosure will utilize these HTML tags and attributes as hints and suggestions provided by the content designer to assist the mapping process at 340 in determining where and how to display elements in the 3D setting. For example, the following is an example HTML web page code provided by a web page developer. Web developer Sample HTML page code provided

[0082]

[0083]

[0084] The example HTML web page code provided by the web page developer includes preferences about how to display the main video on the web page, and preferences about how to display recommendations (or suggested videos). In particular, the HTML web page code uses the "style" tag to specify how to use the "vertical" type value to specify a vertical surface for displaying the video to display the main video. In addition, within the "style" tag, additional hints provided by the web page developer may include a "priority" preference for the matching algorithm, which is used to prioritize which HTML element / content (e.g., main video) within the web page is mapped to which potential surface area. In the example HTML web page code, the priority of the video with a vertical plane layout is set to a value of 100, wherein, in this example, a higher priority value indicates a higher priority. In addition, in this example, the web page developer indicates a preference to place the suggested video in a stack with a type value of "horizontal" in a stacked layout, wherein the distance between the stacked objects (e.g., in this case, the suggested video relative to another suggested video) should be 20cm.

[0085] Figure 4 is a flow chart illustrating a method for identifying elements in 2D content according to some embodiments. Figure 4 According to some embodiments disclosed in Figure 3 Detailed process of identifying elements in 2D content at 320 . Figure 4 Beginning at 410, elements within 2D content are identified, similar to Figure 3 Elements in the 2D content are identified at 320. The method proceeds to the next step of identifying attributes of the tags related to the placement of the content at 420. As discussed above, when designing and configuring a web page, a web designer can associate elements within the web page with HTML tags to define where and how each element is displayed. These HTML tags can also include attributes related to placing the elements on specific parts of the web page. The head mounted system 160 will detect these HTML tags and their attributes and coordinate with other components of the system to use as input for the location where a specific element can be displayed.

[0086] Extracting hints or tags from each element is performed at 430. Hints or tags are typically formatting hints or formatting tags provided by a content designer of the 2D content / web page and / or a web page developer. As discussed above, a content designer may provide instructions or hints, for example in the form of HTML tags shown in the "Sample HTML web page code provided by a web page developer", to instruct the web browser 110 to display elements of the 2D content in a specific portion of the page or screen. In some embodiments, the web page designer may use additional HTML tag attributes to define additional formatting rules. For example, if a user has a reduced sensitivity to a particular color (e.g., red), red is not displayed, but another color is used, or as discussed above, if a video with a preference to be displayed on a vertical surface cannot be displayed on a vertical surface, the video may be displayed on another (physical) surface instead, or a virtual surface may be created, and the video may then be displayed on the virtual surface. The following is an example HTML page parser implemented in a browser for parsing an HTML page to extract hints / tags from each element within the HTML page.

[0087] Example HTML page parser implemented in a browser

[0088]

[0089]

[0090]

[0091] The example HTML page parser shows how to parse, identify and / or extract and copy an HTML page containing HTML tags for providing display preferences for specific elements / objects within 2D content (e.g., a web page). As disclosed in the example HTML page parser, the disclosed sample code can be used to parse elements within 2D content (e.g., a web page). The HTML page parser (e.g., ML.layout, ML.container, etc.) can identify / extract certain HTML tags using various element names and values ​​to determine how to display specific elements to a user in a 3D environment (e.g., by mapping the element to a specific surface).

[0092] A lookup / search is performed at 440 for alternative display forms for one or more elements. Certain formatting rules may be specified for images on a web page. For example, if the web browser 110 is capable of displaying a 3D version of an image, the web page designer may place additional tags or define certain attributes of a particular tag to allow the web browser 110 to recognize that the image may have an alternative version of the image (e.g., a 3D version of the image). The web browser 110 may then access the alternative version of the image (e.g., a 3D version of the image) to be displayed in the 3D-enabled browser.

[0093] The identified elements are stored within the 2D content at 450. The method may store the identified elements in a non-transitory storage medium for use by a mapping routine to map the elements to a particular surface (e.g., in Figure 3 The non-transitory storage medium may include a data storage device such as storage device 130 or local storage device 140. The elements may be stored in a storage device such as described below. Figure 5 In some embodiments, the identified elements within the 2D content may be stored in a transient storage medium.

[0094] Figure 5 An example of a table for storing elements deconstructed from 2D content according to some embodiments is shown. Figure 4 The results of identifying elements within 2D content at 410 are stored in an exemplary table in a database. The element table 500 includes, for example, information about one or more elements within the 2D content, including an element identification (ID) 510, a preference indicator 520 for a location on a 3D surface for placing the element, a parent element ID 530 (if the particular element is included within the parent element), a child element ID 540 (if the element can contain child elements), and a multiple entity indicator 550 (to indicate whether the element contains multiple embodiments that can ensure that the surface or virtual object used to display the element is compatible with multiple versions of the display element). The parent element is an element / object within the 2D content that can contain child elements (e.g., child elements). For example, an element ID with a value of 220 (e.g., main video 220) has a parent element ID with a value of 260 (e.g., active tag 260), which indicates that the main video 220 is a child element of the active tag 260. Or stated differently, the main video 220 is included within the active tag 260. Continuing with the same example, the main video 220 has a sub-element ID 230 (e.g., user comment 230) indicating that the user comment 230 is associated with the main video 220. A person of ordinary skill in the art will appreciate that the element table 500 may be a table in a relational database or any type of database. In addition, the element table 500 may be a table containing Figure 4 The matrix of results of identifying elements within the 2D content at 410 is stored in a computer memory (eg, a cache).

[0095] Each of rows 560 of element table 500 corresponds to an element within a web page. Element ID 510 is a column that contains a unique identifier (e.g., element ID) for each element. In some embodiments, the uniqueness of an element may be defined as a combination of the element ID 510 column within the table and another column (e.g., preference 520 column if the content designer has identified more than one preference). Preference 520 is a column whose value may be based at least in part on a preference defined by a content designer / developer (e.g., a web page designer) and determined by the content designer / developer. Figure 4 The preference 520 column may be determined based at least in part on predefined browser rules to specify where certain types of elements within a web page should be displayed within the 3D environment. These predefined rules may provide suggestions to the system and method to determine where to optimally place elements within the 3D environment.

[0096] Parent Element ID 530 is a column containing the element ID of the parent element that the particular element in the current row is displayed in or is related to. A particular element within a web page may be embedded in, placed within, or related to another element on the page. For example, in the current embodiment, the first entry in the Element ID 510 column stores the ID of the parent element that the particular element in the current row is displayed in or is related to. Figure 2 The preference values ​​in the preference 520 column corresponding to the primary video 220 are determined based on HTML tags and / or attributes, and in the current embodiment, the element should be placed in a "primary" location in the user's physical environment 105. Depending on the current location of the user 108, the primary location may be a wall in a living room, or a stove hood in a kitchen that the user 108 is currently looking at, or if in a large space, a virtual object projected in front of the user's 108 site line on which the primary video 220 may be projected. More information on how elements of 2D content are displayed to the user 108 will be disclosed in a later section. Continuing with the current example, the parent element ID 530 column stores the value of the element ID 220 corresponding to the primary video 220. Figure 2 The value of the element ID 260 corresponding to the activity tag 260. Therefore, the main video 220 is a child element of the activity tag 260.

[0097] Subelement ID 540 is a column containing the element IDs of the subelement in which the particular element in the current row is displayed or is related to. A particular element within a web page may be embedded, placed within, or related to another element on the page. Continuing with the current example, the Subelement ID 540 column stores the IDs of the subelement in which the particular element is displayed or is related to. Figure 2 The value of the element ID 230 corresponding to the user comment 230 .

[0098] The multiple entity indicator 550 is a column that indicates whether an element contains multiple entities, which may warrant the need to make the surface or virtual object (which is used to display the element) compatible with displaying multiple versions of the element (e.g., the element may be a user comment 230, where there may be more than one comment available for the primary video 220). Continuing with the current example, the multiple entity indicator 550 column stores a value of "N" to indicate that the primary video 220 does not have or does not correspond to multiple primary videos in the activity label 260 (e.g., there are "no" multiple versions of the primary video 220).

[0099] Continuing with the current example, the second entry in the element ID 510 column stores the Figure 2 The value of the element ID 230 corresponding to the user comment 230 in the preference 520 column shows a preference of "horizontal" to indicate that the user comment 230 should be placed on a "horizontal" surface somewhere in the user's physical environment 105. As discussed above, the horizontal surface will be determined based on the available horizontal surfaces of the user's physical environment 105. In some embodiments, the user's physical environment 105 may not have a horizontal surface, in which case the systems and methods of the present disclosure can identify / create a virtual object with a horizontal surface to display the user comment 230. Continuing with the current example, the parent element ID 530 column stores the value of the element ID 230 corresponding to the user comment 230. The preference value corresponding to the user comment 230 in the preference 520 column shows a preference of "horizontal" to indicate that the user comment 230 should be placed on a "horizontal" surface somewhere in the user's physical environment 105. As discussed above, the horizontal surface will be determined based on the available horizontal surfaces of the user's physical environment 105. In some embodiments, the user's physical environment 105 may not have a horizontal surface, in which case the systems and methods of the present disclosure can identify / create a virtual object with a horizontal surface to display the user comment 230. Continuing with the current example, the parent element ID 530 column stores the value of the element ID 230 corresponding to the user comment 230. Figure 2 The value element ID 220 corresponding to the main video 220 is stored in the multiple entity indicator 550 column, and the value "Y" is stored to indicate that the user comment 230 can contain more than one value (eg, more than one user comment).

[0100] The remaining rows in the element table 500 contain information about the remaining elements of interest to the user 108. One of ordinary skill in the art will appreciate that storing the results of identifying elements within the 2D content at 410 can improve the functionality of the computer itself, because after the analysis has been performed on the 2D content, if another user is interested in the same 2D content, the analysis can be retained by the system and method for future analysis of the 2D content. Because the deconstruction of the particular 2D content has been completed before, the system and method for deconstructing the particular 2D content can be avoided.

[0101] In some embodiments, the element table 500 may be stored in the storage device 130. In other embodiments, the element table 500 may be stored in the local storage device 140 for quick access to recently viewed 2D content or for possible re-access to recently viewed 2D content. In other embodiments, the element table 500 may be stored in both the storage device 130 remote from the user 108 and the local storage device 140 local to the user 108.

[0102] return Figure 3, the method continues at 330 to identify surrounding surfaces. The user 108 may view the user's physical environment 105 through the head mounted system 160 to allow the head mounted system 160 to collect and identify surrounding surfaces, such as walls, tables, paintings, window frames, stoves, refrigerators, televisions, etc. The head mounted system 160 is aware of real objects within the user's physical environment 105 due to sensors and cameras on the head mounted system 160 or any other type of similar device. In some embodiments, the head mounted system 160 may match real objects observed within the user's physical environment 105 with virtual objects stored in the storage device 130 or the local storage device 140 to identify surfaces that can be used for the virtual objects. Real objects are objects identified within the user's physical environment 105. Virtual objects are objects that do not physically exist within the user's physical environment, but can be displayed to the user to appear as if the virtual objects exist in the user's physical environment. For example, the head mounted system 160 may detect an image of a table within the user's physical environment 105. The image of the table can be simplified into a 3D point cloud object for fast and efficient comparison and matching at the storage device 130 or the local storage device 140. If a match is detected between a real object and a 3D point cloud object (e.g., of a table), the system and method will recognize the table as having a horizontal surface because the 3D point cloud object representing the table is defined as having a horizontal surface. Figure 6 A more detailed description of identifying surrounding surfaces is disclosed in .

[0103] In some embodiments, the virtual object may be an extracted object, where the extracted object may be a physical object that is recognized within the user's physical environment 105, but is displayed to the user as a virtual object in the place of the physical object, so that additional processing and associations can be performed on the extracted object that would not be possible on the physical object itself (e.g., changing the color of the physical object to highlight specific features of the physical object, etc.). In addition, the extracted object may be a virtual object extracted from 2D content (e.g., a web page from a browser) and displayed to the user 108. For example, the user 108 may select an object such as a sofa from a web page displayed on the 2D content / web page to be displayed within the user's physical environment 105. The system may recognize the selected object (e.g., sofa) and display the extracted object (e.g., sofa) to the user 108 as if the extracted object (e.g., sofa) physically existed in the user's physical environment 105. In addition, the virtual object may also include an object with a surface for displaying content (e.g., a transparent display screen near the user for viewing certain content), which may not even physically exist in the user's physical environment 105, but from the perspective of the 2D content, it may be an ideal display surface for presenting certain content to the user from the displayed content.

[0104] Figure 6is a flow chart illustrating a method for identifying a surface from a user's local environment according to some embodiments. Figure 6 It is publicly available Figure 3 Detailed process of identifying surrounding surfaces at 330 . Figure 6 Beginning at 610, identifying the user's current surroundings, similar to Figure 3 The surrounding surface is identified at 330. The method continues with the next step of determining the user's posture at 620.

[0105] Determining the user's posture at 620 is an important step in identifying the user's current surroundings because the user's posture will provide the user 108 with a perspective on objects within the user's physical environment 105. For example, referring back to Figure 1 , user 108 is viewing the user's physical environment 105 using head mounted system 160. Determining the user's pose (i.e., vector and / or origin position information relative to the world) at 620 helps head mounted system 160 understand, for example: (1) how high user 108 is relative to the ground; (2) the angle at which user 108 rotates his head to move and capture images of the room; and (3) the distance between user 108 and table 188, main wall 180, and side wall 184. In addition, the pose of user 108 helps determine the angle at which head mounted system 160 is viewing vertical surfaces 182 and 186, as well as other surfaces within the user's physical environment 105.

[0106] At 630, the method identifies the size of the surrounding surfaces. Each candidate surface within the user's physical environment 105 is labeled and categorized with a corresponding size. In some embodiments, each candidate surface within the user's physical environment 105 is also labeled and categorized with a corresponding orientation. This information will help identify which elements should be mapped to which surfaces based at least in part on the size of the surface, the orientation of the surface, the distance of the user 108 from the particular surface, and the type of information that needs to be displayed for the element. For example, a video may be displayed farther away than a blog or article, which may contain a lot of information, where if the article is displayed at a small size on a distant wall, the text size of the article may be too small for the user to see.

[0107] At 640, the method stores the list of surrounding surfaces in a non-transitory storage medium for use by a mapping routine (e.g., Figure 3 340 maps the element to the identified surface) is used to map the element to a specific surface. The non-transitory storage medium may include a data storage device such as storage device 130 or local storage device 140. The identified surface may be stored in a storage device such as described below. Figure 7 In some embodiments, the identified surfaces may be stored in a transient storage medium.

[0108] Figure 7 An example of a table for storing a list of surfaces identified from a user's local environment is shown in accordance with some embodiments. Surface table 700 is an exemplary table in which the results of a process of identifying surrounding surfaces may be stored in a database. Surface table 700 includes, for example, information about surfaces within the user's physical environment 105, the user's physical environment 105 having data columns including surface ID 710, width 720, height 730, orientation 740, real or virtual indicator 750, multiple 760, and position 770. It will be appreciated by those of ordinary skill in the art that surface table 700 may be a table in a relational database or any type of database. Additionally, surface table 700 may be a table stored in a database. Figure 3 The results of identifying surrounding surfaces at 330 are stored in a matrix in a computer memory (eg, cache).

[0109] Each of the rows 780 of the surface table 700 may correspond to a surface from the user's physical environment 105 or a virtual surface that may be displayed to the user 108 within the user's physical environment 105. The surface ID 710 is a column containing a unique identifier (e.g., surface ID) for uniquely identifying a particular surface. The dimensions of the particular surface are stored in the width 720 column and the height 730 column.

[0110] Orientation 740 is a column that indicates the orientation of the surface relative to the user 108 (e.g., vertical, horizontal, etc.). Real / virtual 750 is a column that indicates whether a particular surface is located on a real object within the user's physical environment 105 as perceived by the user 108 using the head mounted system 160 or whether the particular surface is located on a virtual object that will be generated by the head mounted system 160 and displayed within the user's physical environment 105. The head mounted system 160 may have to generate a virtual object for situations where the user's physical environment 105 may not contain enough surfaces to display the amount of content that the user 108 wishes to display. In these embodiments, the head mounted system 160 may search a database for an existing virtual object that may have an appropriate surface size to display certain types of elements identified for display. The database may be from the storage device 130 or the local storage device 140.

[0111] Multiple 760 is a column that indicates whether the surface / object is compatible with multiple versions of the display element (e.g., the element can be Figure 2 Auxiliary tabs 250, where there may be more than one auxiliary (ie, inactive) tab for a particular web browser 110 (eg, one web page per tab). If the multiple 760 column has a "multiple" value, such as for storage and Figure 2 The fourth entry of the Surface ID column of the Virtual Rolodex 190 corresponds to the value 190 and stores the Figure 2In the case of the fifth entry in the surface ID column having value 194 corresponding to multiple stacked virtual objects 194, the systems and methods will know that if an element can have multiple versions of the element (such as is the case with an inactive label), then there is a type of surface that can accommodate multiple versions.

[0112] Position 770 is a column that indicates the location of the physical surface relative to a reference frame or reference point. Figure 7 As shown in the column heading of position 770 in the surface table 700, the position of the physical surface can be predetermined as the center of the surface. In other embodiments, the position can be predetermined as another reference point of the surface (e.g., the front, back, top, or bottom of the surface). The position information can be expressed as a vector and / or position information from the center of the physical surface relative to some reference frame or reference point. There can be several ways to represent the positions in the surface table 700. For example, the position value of surface ID 194 in the surface table 700 is represented in the summary to illustrate the vector information and the reference information frame (e.g., "frame" subscript). x, y, z are 3D coordinates in each spatial dimension, and frame indicates which reference frame the 3D coordinates are relative to.

[0113] For example, surface ID 186 shows that the center of surface 186 is located at (1.3, 2.3, 1.3) relative to the real world origin. As another example, surface ID 192 shows that the center of surface 192 is located at (x, y, z) relative to the user reference system, and surface ID 190 shows that the center of surface 190 is located at (x, y, z) relative to another surface 182. The reference system is important for eliminating which reference system is currently being used. In the case where the origin of the real world is used as the reference system, it is usually a static reference system. However, in other embodiments, when the reference system is a user reference system, the user may be a mobile reference system, in which case, if the user is moving and the user reference system is used as the reference system, the plane (or vector information) may move and change with the user. In some embodiments, the reference system of each surface may be the same (e.g., the user reference system). In other embodiments, the reference system of the surface stored in the surface table 700 may be different depending on the surface (e.g., the user reference system, the world reference system, another surface or object in the room, etc.).

[0114] In the current example, the values ​​stored in the surface table 700 are contained in Figure 2 Physical surfaces (e.g., vertical surfaces 182 and 186, and horizontal surface 192) and virtual surfaces (e.g., virtual Rolodex 190 and multi-stack virtual object 194) identified within the user's physical environment 105. For example, in the current embodiment, the first entry of the surface ID 710 column stores the Figure 2The value of surface ID 182 corresponding to the vertical surface 182 of FIG. 7 is a value of width 720 in the width column and a value of height 730 in the height column, which correspond to the width and height of the vertical surface 182, respectively, indicating that the vertical surface 182 has a size of 48 inches (width) by 36 inches (height). Similarly, the orientation value in the orientation 740 column indicates that the vertical surface 182 has a "vertical" orientation. In addition, the real / virtual value in the real / virtual 750 column indicates that the vertical surface 182 is an "R" (e.g., real) surface. The multiplier value in the multiplier 760 column indicates that the vertical surface 182 is "single" (e.g., can only store a single content). Finally, the position 770 column uses (2.5, 2.3, 1.2) 用户 The vector information indicates the position of the vertical surface 182 relative to the user 108.

[0115] Other rows in the surface table 700 contain information about other surfaces in the user's physical environment 105. It will be appreciated by those skilled in the art that the information stored in the surface table 700 may be stored in the user's physical environment 105. Figure 3 The results of identifying the surrounding surfaces at 330 can improve the function of the computer itself because after the analysis of the surrounding surfaces has been performed, the analysis can be retained by the head mounted system 160 for future analysis of the user's surrounding surfaces if another user or the same user 108 is interested in different 2D content in the same physical environment 105. The processing steps of identifying the surrounding surfaces at 330 can be avoided because these processing steps have been completed previously. The only difference may include identifying additional or different available virtual objects based at least in part on the element table 500 identifying elements with different 2D content.

[0116] In some embodiments, the surface table 700 is stored in the storage device 130. In other embodiments, the surface table 700 is stored in the local storage device 140 of the user 108 for quick access to recently viewed 2D content or for possible re-access to recently viewed 2D content. In other embodiments, the surface table 700 may be stored in both the storage device 130 remote from the user 108 and the local storage device 140 local to the user 108.

[0117] return Figure 3, the method uses a combination of the identified elements in identifying elements in 2D content 320 and the identified surrounding surfaces in identifying surrounding surfaces at 330, and in some embodiments, uses virtual objects as additional surfaces to continue mapping the elements to the identified surfaces at 340. Mapping the identified elements to the identified surfaces may involve multiple factors, some of which may include using an HTML page parser (such as the example HTML page parser discussed above) to analyze hints provided by the 2D content designer / author through HTML tag elements defined by the 2D content designer / author. Other factors may include selecting how and where to map certain 2D content from a predefined set of rules provided by the AR browser, AR interface, and / or cloud storage. Figure 8 A detailed flow of a mapping process for mapping one or more elements from 2D content to an identified surface is provided.

[0118] Figure 8 Depicted is a flow chart showing a method for mapping elements from 2D content to a surface, in accordance with some embodiments. Figure 8 It is publicly available Figure 3 Detailed process of mapping elements to the identified surface at 340 .

[0119] At 810, the method determines whether the identified element contains a hint provided by a 2D content designer. When the 2D content designer initially designs the 2D content, the 2D content designer may provide hints about where to best display a particular element. For example, Figure 2The main video 220 can be a YOUTUBE video displayed on a web page within the active tab 260. A 2D content designer (e.g., a web page designer) can provide a hint to indicate that the main video 220 is best displayed on a vertical plane in the direct perspective of the user 108. In some embodiments, this can be accomplished by using existing HTML tag elements originally designed for 2D web page content to further define how specific content elements in the 2D content are displayed if a 3D display environment is available. As another example, a 2D content designer can provide a hint to indicate that 3D images are available for a particular web page, rather than 2D images. For example, in the case of a 2D image, in addition to providing basic HTML tags to identify the source of the 2D content, the 2D content designer can also provide other less commonly used HTML tags to identify the source of the 3D version of the 2D image, and in addition, a hint is provided that if a 3D version of the image is used, it is highlighted in front of the user's perspective (e.g., in the main system of the 3D layout). In some embodiments, where the web browser 110 that renders the 2D content may have 3D display capabilities to take advantage of the enhanced 3D image, the 2D content designer may provide this additional "hint" for the 3D image position of the 2D image. Those skilled in the art will appreciate that there are many other ways in which a 2D content designer may provide hints for where a particular content element should be placed in a 2D layout in addition to what is disclosed herein, and those disclosed herein are only some examples of different ways in which a 2D content designer may provide hints for optimally displaying some or all elements within the 2D content.

[0120] In another embodiment, the HTML tag standard may include the creation of new HTML tags or similar markup languages ​​for providing AR / VR specific types of browsers (such as the example HTML web pages provided by the web developer mentioned above) with hints on the placement of 3D objects around the user. At the time of this writing, these new HTML tags have not yet been created and / or adopted as standard tags within the HTML language. However, after the HTML standard includes these types of additional tags, certain embodiments of the current method and system will utilize these new tags to further provide a mapping from identified elements to identified surfaces. Those skilled in the art will appreciate that in addition to HTML tags, there are many other languages ​​that can be modified or adopted to further provide hints on how content elements should be optimally displayed in a 3D environment, and the new HTML tag standard is just one way to achieve this goal.

[0121] At 820, the method determines whether to use hints provided by the 2D content designer or to use a predefined set of rules to map one or more content elements from the 2D content to certain types of 3D surfaces. In some embodiments, in the absence of hints for specific content elements provided by the 2D content designer, the system and method may use a predefined set of rules to determine the optimal way to map the content elements to the surface. In other embodiments, even when there may be hints for the placement of content elements provided by the 2D content designer, the system and method may determine that it may be best to use a predefined set of rules to map the content elements to the surface. However, in other embodiments, the system and method may determine that the hints provided by the 2D content designer are sufficient, and therefore the hints may be used to map the content elements to the surface. Finally, the final decision of the AR browser is to determine whether to use the hints provided by the 2D content designer or to use predefined rules to map the content elements to the surface.

[0122] At 830, assuming the determination is made using a hint provided by the 2D content designer, the method analyzes the hint and searches a list of identified surrounding surfaces that can be used to display the particular content element based at least in part on the hint (e.g., querying the surface table 700). At 840, the method runs a best fit algorithm to select a best fit surface for the particular content element based on the hint provided. For example, the best fit algorithm may hint at the "main content" of a particular content element within a particular web page and attempt to identify a 3D surface that is front and center relative to the user 108 in the 3D environment from the available identified surrounding surfaces. For example, because the main video 220 is within the active tab 260, the 3D surface of the active tab 260 may be selected. Figure 5 The element table 500 has a preference value of “primary” in the preference 520 column, and the vertical surface 182 is a surface in the direct line of sight of the user 108 and has an optimal size for displaying the primary video 220, so Figure 2 The main video 220 is mapped to the vertical surface 182 .

[0123] At 850, the method stores the mapping results of the content elements in the mapping of the elements to the surface table in a non-transitory storage medium for use by the display algorithm to display the content elements on their corresponding mapped surfaces, whether these surfaces are identified surrounding surfaces or virtual objects displayed in the user's surrounding environment. The non-transitory storage medium may include a data storage device such as the storage device 130 or the local storage device 140. The mapping results may be stored in a specific table, such as the following described Fig. 9 Table disclosed in .

[0124] Fig. 9An example of a table for storing mappings of content elements from 2D content to surfaces is shown according to some embodiments. Mapping table 900 is an exemplary table that stores the results of a process of mapping content elements to surfaces in a database. Mapping table 900 includes, for example, information about content elements (e.g., element IDs) and information about surfaces to which content elements are mapped (e.g., surface IDs). It will be appreciated by those of ordinary skill in the art that mapping table 900 may be a table stored in a relational database or any type of database or storage medium. In addition, mapping table 900 may be a matrix in a computer memory (e.g., a cache) that contains Figure 3 The result of mapping the element to the identified surrounding surface is 340 .

[0125] Each row in mapping table 900 corresponds to a content element from the 2D content that is mapped to a surface in the user's physical environment 105 or a surface in a virtual object displayed to user 108 (where the virtual object looks like an object in the user's physical environment 105). For example, in the current embodiment, the first entry in the element ID column stores the value of element ID 220 corresponding to primary video 220. The surface ID value in the surface ID column corresponding to primary video 220 is 182, which corresponds to vertical surface 182. In this manner, primary video 220 is mapped to vertical surface 182. Similarly, user comments 230 are mapped to horizontal surface 192, suggested videos 240 are mapped to vertical surface 186, and auxiliary tags 250 are mapped to virtual Rolodex 190. The element IDs in mapping table 900 may be mapped to the elements stored in the element ID column. Figure 5 The surface ID in the mapping table 900 can be associated with the element ID in the element table 500. Figure 7 The surface ID in the surface table 700 is associated with the surface.

[0126] return Figure 8 At 860, assuming that the determination is to be made using a predefined rule, the method queries a database containing content element to surface mapping rules and determines which type of surface should be considered for mapping the content element for a specific content element in the web page. Figure 2The rules returned by the primary video 220 may indicate that the primary video 220 should be mapped to a vertical surface, and thus after searching the surface table 700, multiple candidate surfaces are revealed (e.g., vertical surfaces 182 and 186 and the virtual Rolodex 190). At 870, the predefined rule set may run a best fit algorithm to select from the available candidate surfaces which surface is the best fit for the primary video 220. Based at least in part on the best fit algorithm, it is determined that the primary video 220 should be mapped to the vertical surface 182 because, of all the candidate surfaces, the vertical surface 182 is the surface within the direct line of sight of the user 108, and the vertical surface 182 has the optimal size for displaying the video. After determining the mapping of one or more elements, at 850, the method stores the mapping results of the content elements in a mapping of the element to the surface table in a non-transitory storage medium as described above.

[0127] return Figure 3 , the method continues at 350 to display one or more elements as virtual content on the mapped surface. The head mounted system 160 may include one or more display devices within the head mounted system 160 to display information, such as a micro projector (not shown). As at 340, the one or more elements are displayed on the mapped corresponding mapped surface. Using the head mounted system 160, the user 108 will see the content on the corresponding mapped surface. It will be appreciated by those of ordinary skill in the art that the content elements are displayed as if they are physically attached to various surfaces (physical or virtual), but in fact, the content elements are actually projected onto the physical surface perceived by the user 108, and in the case of virtual objects, the virtual objects are displayed as if they are attached to the corresponding surface of the virtual object. It will be appreciated by those of ordinary skill in the art that when the user 108 turns his head or looks up or down, the display device within the head mounted system 160 may continue to fix the content elements on their corresponding surfaces to further provide the user 108 with the feeling that the content is fixed to the mapped surface. In other embodiments, the user 108 may change the content of the user's physical environment 105 through movements made by the user's 108 head, hands, eyes, or voice.

[0128] Physical manipulation of objects

[0129] Some embodiments of the present invention are directed to methods that provide for the creation, physical navigation, and manipulation of objects. Unlike known computing systems that limit the scope of interaction with a user to only objects displayed in 2D space by using traditional mouse / keyboard type interactions to manipulate objects, the present embodiments are directed to user interaction in 3D space, which allows physical interaction with the computing system. This allows the user to control the movement and placement of objects in an interactive environment using body movements and gestures.

[0130] Fig. 10A A flow chart of a method for implementing physical placement and manipulation of objects according to some embodiments is shown. At 1001, one or more interactive objects are identified and presented to a user via a display device of a mixed reality system for a host application. Interactive objects may include real objects and virtual objects. Virtual objects are objects that exist only as constructs within a virtual world and are generated / displayed to a user as a computer-generated rendering via a head-mounted display device. Real objects may be displayed as digital computer-generated display object renderings corresponding to real objects (e.g., for VR applications), or may also be viewed as real objects themselves through a transparent portion of a display device (e.g., for AR applications). In some embodiments, an interactive object may be a real or virtual object that causes a change in the HMD system when an interaction is detected between the interactive object and the user or the HMD system.

[0131] At 1003, a user control input is received to select one or more interactive objects. The selection can be facilitated by hovering the pointer object using a controller device (such as a mouse or haptic controller), wherein the triggering of a button or trigger identifies that the user desires to select the object on which the pointer is currently hovering. The object on which the pointer is hovering can be an optional link / URL corresponding to the source data of the interactive object. Alternatively, the selection can occur only by the user's body movement / gesture without the user manipulating the control device. For example, the mixed reality system may employ one or more camera devices that track the movement of the user's hand / arm, and thus can be recognized when the user's hand hovers over the interactive object / link. Certain gestures can be pre-configured in the system to be recognized as "select" gestures (e.g., when the hand is configured in a selection position, such as when the user's hand is held to point to / point to the object with a pointing finger, a two-way handshake). Therefore, when a selection gesture is recognized while the user's hand is hovering over the interactive object, the object will be recognized by the system as selected by the user. Other embodiments of the selection gesture are described below in conjunction with the "release" gesture.

[0132] At 1005, the mixed reality system identifies and tracks the user's body movement relative to the selected object. For example, when the user has selected an object at a first position, the user may wish to move the object to a second position. With this embodiment, the movement of the object to the second position can be achieved by the user's body movement from the first position to the second position. For example, this may occur due to the movement of the user's selection arm from the first position to the second position and / or the user walking to the second physical position. The user's body movement can be identified and tracked by identifying any changes in the position of the controller device held by the user (e.g., by tracking the physical movement of the tactile controller held by the user via changes in the position of any sensor / transmitter that can be embedded in the controller device and / or visually tracking the movement of any marker on the controller device). Alternatively, the user's body movement can be identified and tracked by identifying the body movement of the relevant part of the user's body, for example, the camera device visually tracks the movement and position of the user's selection arm / hand.

[0133] As the user performs physical movement, at 1007, the data associated with the (multiple) interactive objects are displayed accordingly to the new position / orientation associated with the user's movement. For example, if the user has selected an object with a selection arm and is currently moving the selection arm from a first position to a new position, the data associated with the interactive object (i.e., the selected object) will be visually displayed in a second position corresponding to the movement of the user's arm. To improve efficiency, some embodiments may display only a thumbnail display and / or a low-resolution preview of the data associated with the interactive object during any ongoing movement, and only implement a high-resolution display version at the final placement of the object.

[0134] Objects may be rendered into bounded volumes, hereinafter referred to as "prisms". Each prism may have properties and attributes that allow a universe application to manage and display the prisms in a VR / AR environment, such that the universe application may manage the placement and display of virtual content in the VR / AR environment by managing the prisms themselves. In some embodiments, the prisms are implemented as cubic and / or rectangular spaces into which virtual content from an application is displayed, wherein the application is an application instance of the application when the application is rendered into more than one prism. The prisms may represent a subtree of a multi-application scene graph at a current location. Obtaining one or more prisms previously deployed at the current location includes obtaining instance data of one or more prisms from an external database, and rebuilding a local prism database using the instance data of the one or more prisms, wherein the instance data of each prism includes a data structure defining prism attributes of the prism, the prism attributes including at least one of position, orientation, range width, range height, range depth, anchor type, or anchor position, wherein the instance data of each prism includes a key-value pair of application-specific attributes, the key-value pair including state information of virtual content previously rendered into the prism by the application. Further details regarding the manner of implementing prisms are described in co-pending U.S. patent application Ser. No. 62 / 610,101, filed on Dec. 22, 2017, and entitled “METHODS AND SYSTEM FOR MANAGING AND DISPLAYING VIRTUAL CONTENTIN AN AUGMENTED REALITY SYSTEM,” the entire contents of which are incorporated herein by reference.

[0135] When the user selects an interactive object while performing a recognizable body movement, a new prism is created that stores data corresponding to the interactive object and moves to a new location to effectuate the movement of the data corresponding to the interactive object. The coordinate position of the prism of the selected object is modified within the world application to represent the new location of the selected object, wherein appropriate properties of the prism (such as a position property) are modified to represent the new position coordinates.

[0136] Other properties of the object may also need to be modified to represent the new location of the object. For example, the orientation of the object may change in the new location where the object is to be displayed. A deformation may be applied to modify the display orientation of the object in the new display location. To the extent that unacceptable video artifacts may appear as a result of the deformation (e.g., the deformation may excessively distort the displayed image of the object and / or the deformation may cause an excessive amount of the newly oriented object data to be undisplayable), a re-rendering may occur to render the object for display in the new location.

[0137] Any suitable method may be used to determine the orientation of the data associated with the interactable object. For example, in some embodiments, the orientation of the data associated with the interactable object may be a surface normal of substantially planar content, or the orientation may be predefined and associated with the data associated with the interactable object to indicate a preferred direction in which the host application is intended to view the content. As an example, an avatar will have an orientation vector with a surface normal starting from a point between the avatar's eyes. The orientation may also be determined in any other suitable manner that may be applied.

[0138] In some cases, the user may seek to achieve final placement of the selected object at a new location. This may be identified, for example, by identifying when the user releases a selection button / trigger on the controller device. Additionally, even when the controller device is not in use, a "release gesture" may be preconfigured and recognized by the mixed reality system to identify a location for placing an object by recognizing that the user has made an appropriate gesture. At 1009, placement of the selected object is identified, and final rendering of the data associated with the selectable object is then performed at the new location.

[0139] In some embodiments, the start of a release gesture may be the same as the end of a selection gesture. For example, a user's hand may start in an open position and close into a pinch (e.g., a selection gesture), and the user input may continue to start with the same pinch and open into an open position (e.g., a release gesture). In this way, a single user input / interaction may be used to select (i.e., a selection gesture) and release (i.e., a continuous release gesture) an interactable object. In some embodiments, a user may pinch (e.g., start with an open hand and move a finger so that the fingertip of the finger touches) a selectable object, such as a URL link, drag the URL link to the outside of the browser window, and then release the URL (e.g., by reversing the pinch movement), which may be considered a single user interaction. In other embodiments, a first pinch may be used to select a virtual object, and a second pinch may be used to release the interactable object.

[0140] By way of illustrative embodiment, the present disclosure will now describe window creation with physical navigation for a browser. To illustrate, consider that in a mixed reality system, the user's workspace is not limited by the size of the display screen. Therefore, unlike traditional browsers, browser windows in a mixed reality system can be placed and retained anywhere within the user's environment. The problem is that traditional browser technology is configured under the assumption that the displayable browser position must be limited to the display screen.

[0141] To address this limitation of traditional browsers, some embodiments of the present invention are directed to methods for providing physical navigation and manipulation of browser windows. The general idea is to provide a user of a mixed reality system with the ability to place a browser window at any eligible location in the user's environment, where the placement of the window is achieved through physical manipulation activities of the user.

[0142] Fig. 10B A flow chart of a method for implementing physical placement and manipulation of a browser window is shown. At 1002, a user manipulates a pointing object (such as a cursor / pointer) to hover over a link on a browser window. Within an underlying computing system / OS implementing the browser, the position of the pointer object is tracked and identified (1052). For example, this can occur by tracking and watching one or more event objects within the system, where the event objects are associated with movement events of a mouse pointer. The mouse / cursor position can be reported on the event object and received by a handler attached to the window.

[0143] At 1004, the user presses and holds a trigger control that uses a manipulation control mechanism. For example, a haptic controller implemented as a totem can be used to allow the user to press and hold a trigger on a link. A totem is a handheld controller that tracks its position and orientation relative to the AR / VR headset. In some embodiments, the totem can be a six-degree-of-freedom (6 DOF) controller where the user can move a pointer in altitude and azimuth (on a spherical shell) by moving the totem up or down. To move an object closer or farther away, the user can use a joystick on the totem to "push" or "pull". At 1054, the base OS receives an event object indicating that a trigger event has occurred.

[0144] At 1006, a preview of the linked content may be generated. In some embodiments, this is performed by accessing a URL associated with the link on which the trigger was detected (at 1056). The URL is parsed to identify a location (e.g., a website), from which a location identifier (e.g., an IP address) for the location is identified. A connection (e.g., a TCP connection) is then established, wherein communication with an HTTP server is performed to identify content that is then rendered. In the current embodiment, content rendering is performed based on the preview, such that a "thumbnail" based rendering of the linked content is generated.

[0145] While continuing to hold the totem trigger, the user performs physical navigation (e.g., hand, arm, and body movement) to direct the totem to the desired new window position. This results in physical navigation of the window position (1008). The body movement can be converted into an event object generated by and / or related to the operation of the totem device, which is associated with data for the movement and / or positioning of the totem (1058). The movement / positioning detected for the user's body movement is applied to the movement / positioning of the preview window, wherein the dynamic positioning of the preview window is displayed to the user corresponding to the user's body movement.

[0146] At 1010, the user then releases the totem trigger to place the window at the desired location. At this point, at 1060, a browser window is created at the coordinates corresponding to the location where the totem trigger was released. According to some embodiments, instead of the browser window being managed independently by each application on a one-to-one basis in the VR / AR environment, the window is rendered as a prism.

[0147] Fig.11A -F provides an illustrative example of this method for physically placing a window in a mixed reality environment. Fig.11A A representation of a mixed reality environment is shown that includes a physical environment viewed by a user 108 through a head mounted system 160. Content (e.g., a web page) may be viewed through a web browser 1104 accessed by a network 120. The content shown in the browser 1104 may include one or more selectable links 1106.

[0148] The network 120 may be the Internet, an internal network, a private cloud network, a public cloud network, etc. The web browser 1104 is also operably coupled to the processor 170 via the network 120. Although the processor 170 is shown as a separate component from the head mounted system 160, in alternative embodiments, the processor 170 may be integrated with one or more components of the head mounted system 160, and / or may be integrated into other system components within the environment 100, such as, for example, a network 120 accessing a computing network 125 and a storage device 130. The processor 170 may be configured with software for receiving and processing information (such as video, audio, and content) received from the head mounted system 160, the local storage device 140, the web browser 1104, the computing network 125, and the storage device 130.

[0149] The head mounted system 160 may be a virtual reality (VR) or augmented reality (AR) head mounted system that includes a user interface, a user sensing system, an environment sensing system, and a processor (all not shown). The head mounted system 160 presents an interface for interacting with and experiencing the digital world to the user 108. The interaction may involve the user and the digital world, one or more other users interacting with the environment 100, and objects within the digital and physical worlds.

[0150] The user interface may include receiving 2D content and selecting elements within the 2D content by user input through the user interface. The user interface may be at least one or a combination of a tactile interface device, a keyboard, a mouse, a joystick, a motion acquisition controller, an optical tracking device, and an audio input device. A tactile interface device (such as totem 1102) is a device that allows a person to interact with a computer through physical sensation and movement. Haptics refers to a type of human-computer interaction technology that includes tactile feedback or other physical sensations to perform actions or processes on a computing device.

[0151] The user sensing system may include one or more sensors operable to detect certain features, characteristics, or information about the user 108 wearing the head mounted system 160. For example, in some embodiments, the sensor may include a camera or optical detection / scanning circuitry capable of detecting real-time optical characteristics / measurements of the user 108, such as, for example, one or more of: pupil constriction / dilation, angle measurement / positioning of each pupil, sphericity, eye shape (changes in eye shape over time), and other anatomical data. This data may provide or be used to calculate information that may be used by the head mounted system 160 to enhance the user's viewing experience (e.g., the user's visual focus). The environment sensing system may also include one or more sensors for obtaining data from the user's physical environment. Objects or information detected by the sensors may be provided as input to the head mounted system 160. In some embodiments, the input may represent the user's interaction with the virtual world.

[0152] like Fig. 11B As shown in , the user can press and hold a selectable link 1106 within the browser 1104. A pointer 1108 controllable by the user 108 can be used to hover over the selectable link 1108. When the user hovers over the link, the user can press and hold the trigger on the totem 1102.

[0153] like Fig. 11C As shown in , when the user presses and holds the totem trigger, a preview window 1110a is displayed to the user. The preview window 1110a corresponds to the content related to the link 1106.

[0154] The preview window 1110a can be physically manipulated by the user 108 to change the display position of the preview window 1110a. For example, while continuing to hold the totem trigger, the user can physically move a part of his / her body (e.g., by moving the arm used to hold the totem) to change the display position of the preview window 1110a. Fig.11D The new position / location is shown in . Fig.11E As shown in , the body movement of the user's body changes the position of the preview window 1110a. For example, this can be achieved by detecting the change in the position / orientation of the totem 1102 held in the moving body part (for example, detecting the movement of the totem held in the user's hand), wherein the detection of the body movement is converted into the corresponding movement of the preview window. Like this, the display of the new position of the preview window (and the window display in motion) can be controlled by, for example, the body movement speed of the user, the distance of the body movement, the direction of the body movement, and / or any other suitable window / display parameters that can correspond to the body movement. For example, a relatively small displacement in the arm position will be converted into a relatively small movement of the preview window displayed, while moving the arm a relatively large distance will be converted into a relatively large movement of the display window position. As another example, the slower movement of the arm will be converted into a slower display movement of the preview window when in motion, while moving the arm relatively quickly will be converted into a relatively fast movement of the preview window when in motion.

[0155] The user can then select where to place the browser window. Fig.11F As shown in , when the user releases the totem trigger, the position of the preview window changes to the position of the new browser window to be created when the totem trigger is released.

[0156] Thus, the current embodiments provide a method of selecting a link, moving it to a new location, and releasing the object to create a new window that allows the user to place, organize, and save content within their physical environment. Unlike traditional desktop and mobile environments, this placement is done via physical navigation, allowing the user to use their hands, arms, and / or body to move content around the user's space. This form of navigation is more efficient than the virtual navigation used by desktop and mobile browsers, and gives the user greater control over creating and placing browser windows within their space.

[0157] As described above, this method is also advantageous because it allows the user to open and place a new window with one user interaction (e.g., pressing a button only once). This is a significant improvement over other interaction methods that require the user to first press a button to open the new window and then press a button to move / place the new window to a second location.

[0158] As another illustrative embodiment, the present disclosure will now describe the use of physical manipulation to place any interactable object. A specific illustrative embodiment relates to the placement of wall hanging objects (such as paintings, virtual televisions, virtual monitors, or any other object that can be placed on a wall location). This example scenario is related to the idea that a user may have a set of wall hangings / pictures that the user would like to hang on walls throughout the house, but the user may not know exactly where to place them. Therefore, the system will create a digital copy of the picture / painting in the wall hanging application that can be physically manipulated by the user to a placement location on the wall.

[0159] Fig. 10C A flow chart of a method for implementing physical placement and manipulation of a wall object in accordance with some embodiments is shown. At 1022, a user manipulates a user selection object (such as a cursor / pointer) on a host application such as a browser window to hover over interactive content (such as a wall object representation (icon) or a link to a wall object). In an underlying computing system / OS implementing the host application, the location of the user selection object is tracked and identified (1072). For example, this may occur by tracking and watching one or more event objects within the system, where the event objects are associated with movement events of the user selection object. The user selection object location may be reported on the event object and received by a handler attached to the host application.

[0160] At 1024, the user presses and holds a manipulation control mechanism, such as a trigger on a totem. For example, a haptic controller implemented as a totem may be employed to allow a user to press and hold a trigger to select a link or other interactive content, such as a file, application content, etc., so that when the interactive content is manipulated, additional data is accessed compared to before the manipulation occurred. At 1074, the base OS receives an event object indicating that a manipulation control mechanism event (such as pressing or holding a totem trigger) has occurred. The manipulation control mechanism event may indicate a selection of an interactive object or a manipulation of a selected interactive object.

[0161] At 1026, a preview may be generated for the data associated with the interactable object. In some embodiments, this is performed (at 1076) by accessing data associated with the interactable object, such as a URL associated with a link whose manipulation control mechanism is detected. In some embodiments, the interactable object data is parsed to identify a location (e.g., a website), from which a location identifier (e.g., an IP address) for the location is identified. A connection (e.g., a TCP connection) is then established, wherein communication with an HTTP server is performed to identify content, which is then rendered. In some embodiments, content rendering is performed based on the preview, such that a digital outline of the wall hanging is generated as a preview.

[0162] While continuing to activate the manipulation control mechanism, the user performs physical navigation (e.g., hand, arm, and / or body movement) to direct the user-selected object to the desired wall location of the data associated with the interactive object. In some embodiments, this may result in physical navigation of the object location (1028). The body movement is converted into an event object generated by and / or related to the operation of the totem device, which is associated with data for the movement and / or positioning of the totem (1078). The movement / positioning detected for the user's body movement is applied to the movement / positioning of the preview wall hanging, wherein the dynamic positioning of the preview wall hanging is displayed to the user corresponding to the user's body movement.

[0163] The user then releases the totem trigger at the desired location to place the data associated with the interactable object at the desired wall location at 1030. At this point, a final version of the interactable object is created at the coordinates corresponding to the location where the totem trigger was released at 1080. The complete digital copy of the wall hanging (e.g., at the source URL) is used to generate a final rendering of the interactable object at the placement location.

[0164] Fig. 12A -F provides an example of a method of physically placing a wall hanging in a mixed reality environment. The figure shows a representation of a mixed reality environment that includes a physical environment viewed by a user 108 through a head mounted system 160. The user can be located in various locations including a wall space 105 where a wall hanging object can be placed. Content (e.g., an interactable object / wall hanging) can be accessed, for example, through one or more icons / selectable links / icons 1106.

[0165] like Fig. 12B As shown in , a user may press and hold an interactable object 1106, for example, where a controller operated by user 108 may be used to hover over the interactable object 1106. When the user is hovering over an object, the user may press and hold a trigger on the controller to select the object.

[0166] like Fig. 12C As shown in , when the user presses and holds the controller trigger, a preview 1202 of the wall hanging object is displayed to the user. The preview 1202 corresponds to a digital outline and / or thumbnail of the wall hanging located at the link 1106, and is a visual representation of the data associated with the interactive object.

[0167] The preview 1202 can be physically manipulated by the user 108 to change the display position of the preview 1202. For example, while continuing to hold the totem trigger, the user can physically move a part of his / her body (e.g., by moving the arm holding the totem) to change the display position of the preview 1202. Fig.12D The new orientation / position is shown in . Fig.12EAs shown in , the physical movement of the user's body changes the position of the preview 1202 to the corresponding wall position. For example, this can be achieved by detecting a change in the position / orientation of the totem 1102 held in the moving body part (e.g., detecting the movement of the totem held in the user's hand), wherein the detection of the physical movement is converted into a corresponding movement of the preview. In this way, the display of the new position of the preview (and the display of the preview when moving) can be controlled by, for example, the speed of the user's physical movement, the distance of the physical movement, the direction of the physical movement, and / or any other suitable display parameters that may correspond to the physical movement. For example, a relatively small displacement in the position of the arm will be converted into a relatively small movement of the displayed preview, while moving the arm a relatively large distance will be converted into a relatively large movement of the preview display. As another example, a slower movement of the arm will be converted into a slower displayed movement of the preview when moving, while moving the arm relatively faster will be converted into a relatively faster movement of the preview when moving.

[0168] The user can then select where the wall mount will be placed. Fig.12F As shown in , when the user releases the totem trigger, the location previewed at the time the totem trigger was released changes to the location of the newly created wall hanging. A complete digital copy of the wall hanging from the interactable object is then taken and rendered to display at the selected placement location.

[0169] This method has many benefits.For example, this physical method realizing object placement allows the user to organize its task in space, and uses its physical space to group content, establishes relationship and creates the meaning of object as in the real world.In addition, physical navigation is not to place only using virtual menu and default object / window, and can also make the user manually place content by moving its hand, arm and body.This is a more efficient interface, because it utilizes the familiar intrinsic physical skills of most people's instinct, such as spatial memory, and provides more control for the user.This is an improvement to current technology, for traditional desktop and mobile browser, new object / window can only be created via the file menu or link context menu of browser, particularly in the case of creating new object / window in default position (it usually covers current position).In addition, computing usually requires the user to create a new digital object in a user interaction and then place the newly created digital content with a second user interaction, and improvement only needs a user interaction to complete the two.

[0170] Additional Embodiments

[0171] Other embodiments of the present disclosure are described below. These additional embodiments may incorporate elements from the embodiments disclosed above, and elements of these additional embodiments may be incorporated into the embodiments disclosed above.

[0172] 1. A method comprising

[0173] Identify interactable objects;

[0174] identifying a user's body movements corresponding to the interactable objects;

[0175] determining a coordinate location of data associated with the interactable object based on the user's body movement; and

[0176] Data associated with the interactable objects are placed at coordinate locations corresponding to the user's body movements.

[0177] 2. The method according to embodiment 1, wherein a user input device is used to detect the user's body movement.

[0178] 3. The method of embodiment 2, wherein the user input device is a tactile controller.

[0179] 4. The method of embodiment 3, wherein the haptic controller corresponds to a totem device having at least six degrees of freedom.

[0180] 5. The method of embodiment 3, wherein, at a processing system associated with the haptic controller, the user's body movement is converted into an event object, wherein the event object is associated with data indicating the movement or positioning of the haptic controller.

[0181] 6. A method according to embodiment 3, wherein the coordinate position of the interactive object is determined by identifying the position where the trigger for the tactile controller is released after the user's body movement.

[0182] 7. The method of embodiment 2, wherein the user input device is a camera for detecting user gestures.

[0183] 8. The method of embodiment 1, wherein the open interactable object is identified by identifying a link that interacts with a part of the user's body.

[0184] 9. The method of embodiment 8, wherein the preview of the link is generated by accessing the content of a URL associated with the link.

[0185] 10. The method of embodiment 9, wherein the user's body movement corresponds to physical movement of the preview.

[0186] 11. The method of embodiment 1, wherein the interactable object is opened and placed via a single user interaction of a computing device.

[0187] 12. The method of embodiment 1, wherein the orientation of the interactable object is determined by:

[0188] Determine the surface normal at the coordinate location;

[0189] determining an orientation of data associated with the interactable object; and

[0190] Aligns the surface normal of the coordinate location with the orientation of the data associated with the interactable object.

[0191] 13. The method of embodiment 1, wherein the interactable object comprises a wall hanging.

[0192] 14. A system comprising:

[0193] a mixed reality display device configured to display three-dimensional content;

[0194] a user input device configured to detect body movement of a user;

[0195] a processor that receives and processes data from a user input device;

[0196] A memory for holding programmable code executable by a processor, wherein the programmable code includes instructions for identifying an interactive object, identifying a user's body movement corresponding to the interactive object, determining a coordinate position of data associated with the interactive object based on the user's body movement, and placing the data associated with the interactive object at the coordinate position corresponding to the user's body movement.

[0197] 15. The system of embodiment 14, wherein the user input device is a haptic controller.

[0198] 16. The system of embodiment 15, wherein the haptic controller corresponds to a totem device having at least six degrees of freedom.

[0199] 17. The system of embodiment 15, wherein the user's body movements are converted into event objects at the processor, wherein the event objects are associated with data indicating the movement or positioning of the haptic controller.

[0200] 18. A system according to embodiment 15, wherein the coordinate position of the interactable object is determined by identifying the location where the trigger for the haptic controller is released after the user's body movement.

[0201] 19. The system of embodiment 14, wherein the user input device is a camera for detecting user gestures.

[0202] 20. The system of embodiment 14, wherein an open interactable object is identified by identifying a link that interacts with a part of the user's body.

[0203] 21. The system of embodiment 20, wherein the preview of the link is generated by accessing the content of a URL associated with the link.

[0204] 22. The system of embodiment 20, wherein the user's body movement corresponds to physical movement of the preview.

[0205] 23. The system of embodiment 14, wherein the interactable object comprises a wall hanging.

[0206] 24. The system of embodiment 14, wherein the programmable code further comprises instructions for determining the orientation of the interactable object by:

[0207] Determine the surface normal at the coordinate location;

[0208] determining an orientation of data associated with the interactable object; and

[0209] Aligns the surface normal of the coordinate location with the orientation of the data associated with the interactable object.

[0210] 25. A computer program product embodied on a computer readable medium having stored thereon sequences of instructions which, when executed by a processor, cause the processor to perform a method comprising:

[0211] Identify interactable objects;

[0212] identifying a user's body movements corresponding to the interactable objects;

[0213] determining a coordinate location of data associated with the interactable object based on the user's body movement; and

[0214] Data associated with the interactable objects are placed at coordinate locations corresponding to the user's body movements.

[0215] 26. The computer program product of embodiment 25, wherein a user input device is employed to detect body movement of the user corresponding to the window.

[0216] 27. The computer program product of embodiment 26, wherein the user input device is a haptic controller.

[0217] 28. The computer program product of embodiment 27, wherein the haptic controller corresponds to a totem device having at least six degrees of freedom.

[0218] 29. The computer program product of embodiment 27, wherein, at a processing system associated with the haptic controller, the user's body movement is converted into an event object, wherein the event object is associated with data indicating the movement or positioning of the haptic controller.

[0219] 30. The computer program product of embodiment 27, wherein the coordinate position of the interactable object is determined by identifying a location where a trigger for a haptic controller is released following a body movement of the user.

[0220] 31. The computer program product of embodiment 26, wherein the user input device is a camera for detecting user gestures.

[0221] 32. The computer program product of embodiment 25, wherein the window to open is identified by identifying a link that interacts with a part of the user's body.

[0222] 33. The computer program product of embodiment 32, wherein the preview of the link is generated by accessing content of a URL associated with the link.

[0223] 34. The computer program product of embodiment 33, wherein the user's body movement corresponds to physical movement of the preview.

[0224] 35. The computer program product of embodiment 25, wherein the interactable object is opened and placed with a single user interaction of the computing device.

[0225] 36. The computer program product of embodiment 35, wherein the single user interaction comprises a single press of a button on the computing device.

[0226] 37. The computer program product of embodiment 25, wherein the sequence of instructions further determines the orientation of the interactable object by:

[0227] Determine the surface normal at the coordinate location;

[0228] determining an orientation of data associated with the interactable object; and

[0229] Aligns the surface normal of the coordinate location with the orientation of the data associated with the interactable object.

[0230] 38. A method comprising:

[0231] receiving data indicating selection of an interactable object at the beginning of a user interaction,

[0232] receiving data indicating an end of a user interaction with the interactable object,

[0233] Determine the position of the interactive object at the end of the user interaction, and

[0234] The data associated with the interactable object is placed at the location of the interactable object at the end of the user interaction.

[0235] 39. The method of embodiment 38 further comprises: a user input device providing data indicating a selection of an interactable object and data indicating an end of a user interaction with the interactable object.

[0236] 40. The method of embodiment 39, wherein the user input device is a tactile controller.

[0237] 41. The method of embodiment 40, wherein the haptic controller corresponds to a totem device having at least six degrees of freedom.

[0238] 42. The method of embodiment 40, wherein the user's body movements are converted into event objects at a processing system associated with the haptic controller, wherein the event objects are associated with data for the movement or positioning of the haptic controller.

[0239] 43. A method according to embodiment 40, wherein the position of the interactable object at the end of the user interaction is determined by identifying the location where a trigger for the haptic controller is released after selection of the interactable object.

[0240] 44. The method of embodiment 39, wherein the user input device is a camera for detecting user gestures.

[0241] 45. The method of embodiment 44, wherein the data indicative of a selection of the interactable object corresponds to a selection gesture.

[0242] 46. ​​The method of embodiment 44, wherein the data indicating the end of the user interaction with the interactable object corresponds to a release gesture.

[0243] 47. A method according to embodiment 38, wherein the data indicating the selection of the interactable object corresponds to at least a portion of the user's body that interacts with the link.

[0244] 48. The method of embodiment 47, wherein the preview of the link is generated by accessing the content of a URL associated with the link.

[0245] 49. The method according to embodiment 48 further includes: receiving data indicating the user's body movement, wherein the user's body movement corresponds to the physical movement of the preview.

[0246] 50. The method of embodiment 38, wherein the interactive objects are opened and placed with the aid of user interaction.

[0247] 51. The method of embodiment 38, wherein the orientation of the interactable object is determined by:

[0248] Determine the surface normal of the position of the interactable object at the end of the user interaction;

[0249] determining an orientation of data associated with the interactable object; and

[0250] Aligns the surface normal of the coordinate location with the orientation of the data associated with the interactable object.

[0251] 52. The method of embodiment 39, wherein the interactable object comprises a wall hanging.

[0252] 53. A method comprising:

[0253] receiving data indicating selection of an interactable object at the beginning of a user interaction,

[0254] receiving data indicating an end of a user interaction with the interactable object,

[0255] Determine the position of the interactive object at the end of the user interaction, and

[0256] When the position of the interactable object at the end of the user interaction is outside the initial prism of the interactable object, a new prism is created at the end of the user interaction for data associated with the interactable object.

[0257] 54. The method of embodiment 53, further comprising: when the interactable object is outside the boundaries of the initial prism, displaying a visual preview of data associated with the interactable object at the location of the interactable object between the start and end of the user interaction.

[0258] 55. The method of embodiment 53 further comprises: a user input device providing data indicating selection of an interactable object and data indicating an end of user interaction with the interactable object.

[0259] 56. The method of embodiment 55, wherein the user input device is a tactile controller.

[0260] 57. The method of embodiment 56, wherein the haptic controller corresponds to a totem device having at least six degrees of freedom.

[0261] 58. The method of embodiment 56, wherein, at a processing system associated with the haptic controller, the user's body movement is converted into an event object, wherein the event object is associated with data for the movement or positioning of the haptic controller.

[0262] 59. A method according to embodiment 56, wherein the position of the interactable object at the end of the user interaction is determined by identifying the location where a trigger for the haptic controller is released after selection of the interactable object.

[0263] 60. The method of embodiment 55, wherein the user input device is a camera for detecting user gestures.

[0264] 61. The method of embodiment 60, wherein the data indicating selection of the interactable object corresponds to a selection gesture.

[0265] 62. The method of embodiment 60, wherein the data indicating the end of the user interaction with the interactable object corresponds to a release gesture.

[0266] 63. A method according to embodiment 53, wherein the data indicating the selection of the interactive object corresponds to at least a portion of the user's body that interacts with the link.

[0267] 64. The method of embodiment 63, wherein the preview of the link is generated by accessing the content of a URL associated with the link.

[0268] 65. The method according to embodiment 64 further includes: receiving data indicating the user's body movement, wherein the user's body movement corresponds to the physical movement of the preview.

[0269] 66. A method according to embodiment 53, wherein the interactive objects are opened and placed with the help of user interaction.

[0270] 67. The method of embodiment 53, wherein the orientation of the interactable object is determined by:

[0271] Determine the surface normal of the position of the interactable object at the end of the user interaction;

[0272] determining an orientation of data associated with the interactable object; and

[0273] Aligns the surface normal of the coordinate location with the orientation of the data associated with the interactable object.

[0274] 68. The method of embodiment 53, wherein the interactable object comprises a wall hanging.

[0275] 69. A method comprising:

[0276] At the start of a user interaction, receiving data indicating a selection of an interactable object contained within the first prism, and

[0277] When data representing body movement of the user corresponding to a user interaction between the beginning and the end of the user interaction indicates that the interactable object is outside the first prism at the end of the user interaction, a second prism is created at the location of the interactable object at the end of the user interaction to contain data associated with the interactable object.

[0278] 70. The method of embodiment 69 further comprises a user input device providing data indicating selection of an interactable object and data indicating body movement of the user corresponding to a user interaction between a start and an end of the user interaction.

[0279] 71. The method of embodiment 70, wherein the user input device is a tactile controller.

[0280] 72. The method of embodiment 71, wherein the haptic controller corresponds to a totem device having at least six degrees of freedom.

[0281] 73. A method according to embodiment 71, wherein the user's body movement is converted into an event object at a processing system associated with the haptic controller, wherein the event object is related to data for the movement or positioning of the haptic controller.

[0282] 74. A method according to embodiment 71, wherein the position of the interactable object at the end of the user interaction is determined by identifying the location where a trigger for the haptic controller is released after selection of the interactable object.

[0283] 75. The method of embodiment 70, wherein the user input device is a camera for detecting user gestures.

[0284] 76. The method of embodiment 75, wherein the data indicating selection of the interactable object corresponds to a selection gesture.

[0285] 77. The method of embodiment 75, wherein the data indicating the end of the user interaction with the interactable object corresponds to a release gesture.

[0286] 78. A method according to embodiment 69, wherein the data indicating selection of the interactive object corresponds to at least a portion of the user's body that interacts with the link.

[0287] 79. The method of embodiment 78, wherein the preview of the link is generated by accessing the content of a URL associated with the link.

[0288] 80. The method of embodiment 79, wherein the user's body movement corresponds to physical movement of the preview.

[0289] 81. A method according to embodiment 69, wherein the interactive objects are opened and placed with the help of user interaction.

[0290] 82. The method of embodiment 69, wherein the orientation of the interactable object is determined by:

[0291] Determine the surface normal of the position of the interactable object at the end of the user interaction;

[0292] determining an orientation of data associated with the interactable object; and

[0293] Aligns the surface normal of the coordinate location with the orientation of the data associated with the interactable object.

[0294] 83. The method of embodiment 69, wherein the interactable object comprises a wall hanging.

[0295] 84. A system comprising:

[0296] a mixed reality display device configured to display three-dimensional content;

[0297] a user input device configured to detect body movement of a user;

[0298] a processor that receives and processes data from a user input device;

[0299] A memory for storing programmable code executable by a processor, wherein the programmable code includes instructions for: receiving data for a selection of an interactable object contained within a first prism at the beginning of a user interaction, receiving data indicating an end of a user interaction with the interactable object, receiving data indicating a body movement of the user corresponding to removal of the interactable object from the first prism between the beginning and the end of the user interaction, and, at the end of the user interaction with the interactable object, creating a second prism to contain data associated with the interactable object.

[0300] 85. A system comprising:

[0301] a mixed reality display device configured to display three-dimensional content;

[0302] a user input device configured to detect body movement of a user;

[0303] a processor that receives and processes data from a user input device;

[0304] A memory for storing programmable code executable by a processor, wherein the programmable code includes instructions for: receiving data indicating a selection of an interactable object at the beginning of a user interaction, receiving data indicating an end of a user interaction with the interactable object, determining a position of the interactable object at the end of the user interaction, and when the position of the interactable object at the end of the user interaction is outside of an initial prism of the interactable object, creating a new prism at the end of the user interaction for data associated with the interactable object.

[0305] 86. A system comprising:

[0306] a mixed reality display device configured to display three-dimensional content;

[0307] a user input device configured to detect body movement of a user;

[0308] a processor that receives and processes data from a user input device;

[0309] A memory for storing programmable code executable by a processor, wherein the programmable code includes instructions for: receiving data indicating a selection of an interactable object contained within a first prism at the beginning of a user interaction, and when data representing a body movement of the user corresponding to the user interaction between the beginning and the end of the user interaction indicates that the interactable object is outside the first prism at the end of the user interaction, creating a second prism at the location of the interactable object at the end of the user interaction to contain data associated with the interactable object.

[0310] System Architecture Overview

[0311] Fig.13 14 is a block diagram of an illustrative computing system 1400 suitable for implementing embodiments of the present disclosure. The computing system 1400 includes a bus 1406 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1407, a system memory 1408 (e.g., RAM), static storage devices 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., a modem or Ethernet card), a display 1411 (e.g., a CRT or LCD), and input devices 1412 (e.g., a keyboard and mouse).

[0312] According to one embodiment of the present invention, computing system 1400 performs specific operations by processor 1407 executing one or more sequences of one or more instructions contained in system memory 1408. The instructions can be read into system memory 1408 from another computer-readable / usable medium such as static storage device 1409 or disk drive 1410. In alternative embodiments, hard-wired circuits can be used instead of software instructions or in combination with software instructions to implement the present disclosure. Therefore, embodiments of the present disclosure are not limited to any specific combination of hardware circuits and / or software. In one embodiment, the term "logic" represents any combination of software or hardware used to implement all or part of the present disclosure.

[0313] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1407 for execution. The medium can take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 1410. Volatile media include dynamic memory, such as system memory 1408.

[0314] Common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette or any other medium from which a computer can read.

[0315] In an embodiment of the present disclosure, the execution of the instruction sequence for practicing the present disclosure is performed by a single computing system 1400. According to other embodiments of the present disclosure, two or more computing systems 1400 coupled by a communication link 1415 (e.g., a LAN, PTSN, or wireless network) can cooperate with each other to execute the instruction sequence required to practice the present disclosure.

[0316] The computing system 1400 can send and receive messages, data, and instructions, including programs (i.e., application code), via the communication link 1415 and the communication interface 1414. The received program code can be executed by the processor 1407 upon receipt, and / or stored in the disk drive 1410 or other non-volatile memory for later execution. The computing system 1400 can communicate with a database 1432 on an external storage device 1431 via a data interface 1433.

[0317] In the foregoing description, the present disclosure has been described with reference to specific embodiments of the present disclosure. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above-described process flow is described with reference to a specific order of processing actions. However, the order of many of the described processing actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for creating an object using physical operations, comprising: receiving data indicating a selection of an interactable object contained within a first prism at the start of a user interaction, wherein the data indicating the selection of the interactable object corresponds to at least a portion of a body of the user interacting with a link; generating a preview of the link by accessing content of a URL associated with the link; displaying a preview of at least a portion of content including the URL associated with the link; receiving data indicating an end of the user interaction with the interactable object; receiving data indicative of a body movement of the user corresponding to removal of the interactable object from the first prism between the beginning and the end of the user interaction, wherein the body movement of the user corresponds to movement of the preview, such that a visual display of the preview moves according to the body movement during the user interaction; and At the end of the user interaction with the interactable object, a second prism is created to contain the data associated with the interactable object.

2. The method according to claim 1, further comprising: When the interactable object is outside the boundaries of the first prism, the preview is displayed at the location of the interactable object between the start and the end of the user interaction.

3. The method according to claim 1, further comprising: The data indicative of the selection of the interactable object and the data indicative of the end of the user interaction with the interactable object are provided via user input means.

4. The method according to claim 3, wherein: The user input device is a haptic controller.

5. The method according to claim 4, wherein: The haptic controller corresponds to a totem device having at least six degrees of freedom.

6. The method according to claim 4, wherein: At a processing system associated with the haptic controller, the user's body movement is converted into an event object, wherein the event object is associated with data for the movement or positioning of the haptic controller.

7. The method according to claim 4, wherein: A position of the interactable object at the end of the user interaction is determined by identifying a position at which a trigger for the haptic controller is released after the selection of the interactable object.

8. The method according to claim 3, wherein: The user input device is a camera for detecting user gestures.

9. The method according to claim 8, wherein: The data indicative of the selection of the interactable object corresponds to a selection gesture.

10. The method according to claim 8, wherein: The data indicative of the end of the user interaction with the interactable object corresponds to a release gesture.

11. The method according to claim 1, wherein: The interactable object is opened and placed by means of the user interaction.

12. The method according to claim 1, wherein: The orientation of the interactable object is determined by: determining a surface normal for a position of said interactable object at said end of said user interaction; determining the orientation of the data associated with the interactable object; as well as The surface normal of the location is aligned with the orientation of the data associated with the interactable object.

13. The method according to claim 1, wherein: The interactable object includes a wall hanging.

14. A method for creating an object using physical operations, comprising: receiving data indicating selection of an interactable object at the start of a user interaction, wherein the data indicating the selection of the interactable object corresponds to at least a portion of a body of the user interacting with a link; generating a preview of the link by accessing content of a URL associated with the link; displaying a preview of at least a portion of content including the URL associated with the link; receiving data indicating an end of the user interaction with the interactable object; determining a position of said interactable object at said end of said user interaction; receiving data indicative of a body movement of the user, wherein the body movement of the user corresponds to movement of the preview, such that during the user interaction a visual display of the preview moves according to the body movement; and When the position of the interactable object at the end of the user interaction is outside the initial prism of the interactable object, a new prism is created for data associated with the interactable object at the end of the user interaction.

15. A method for creating an object using physical operations, comprising: at the beginning of a user interaction, receiving data indicative of a selection of an interactable object contained within a first prism, wherein the data indicative of the selection of the interactable object corresponds to at least a portion of a body of the user interacting with a link; generating a preview of the link by accessing content of a URL associated with the link; displaying a preview of at least a portion of content including the URL associated with the link; receiving data indicative of a body movement of the user, wherein the body movement of the user corresponds to movement of the preview, such that during the user interaction a visual display of the preview moves according to the body movement; and When data indicative of the user's body movement indicates that the interactable object is outside the first prism at the end of the user interaction, a second prism is created at the location of the interactable object at the end of the user interaction to contain the data associated with the interactable object.

Images