Virtual personal interface for control and travel between virtual worlds

Abstract

Methods and systems described herein relate to a virtual personal interface (herein referred to as a "personal interface") for controlling an artificial reality (super-reality (XR)) environment, for example by providing a detailed view of a selected item by providing a user interface for interacting with a current XR application, navigating between multiple virtual worlds without having to transition into or out of a homepage hall for these worlds, executing aspects of a second XR application within a world controlled by a first XR application, and providing 3D content separate from the current world. In at least one of the worlds, the personal interface itself may present content corresponding to an item, action, or application of the world within runtime separate from the current virtual world. The XR application may be defined for use with a personal interface to create a 2D interface portion and a 3D world portion displayed via the personal interface.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for controlling and navigating between multiple virtual worlds in artificial reality without having to transition in and out of a home lobby. Background Art

[0002] Artificial reality systems provide users with many opportunities to experience what it might be like to visit desired places, participate in certain events, interact with specific individuals, etc. Typically, these activities take place in the context of a virtual world implemented by an artificial reality application designed to simulate real-life situations. For example, such a virtual world may depict scenes of locations that can be controlled to immerse the user in events within the world as if the user were actually there.

[0003] In some scenarios, the user may want to travel to a subsequent virtual world because of an interest in an activity or thing experienced in the current world. Otherwise, this desire to travel may be a result of the user having finished a virtual world event or simply becoming more interested in various artificial reality products.

[0004] In an artificial reality environment, some objects that users see and interact with are "virtual objects", i.e., computer-generated representations of objects. Virtual objects can be presented, for example, by head-mounted displays, mobile devices, projection systems, etc. Typically, users can interact with virtual objects using controllers and / or gestures. In a system including hand tracking, images of user hands can be interpreted to create a 3D model of the user's hands or to estimate hand gestures in other ways. In some systems, user gestures can perform "interactions" with virtual objects, which can include selecting, moving, rotating, resizing, activating controls, changing colors or skins, defining interactions between real objects or virtual objects, setting virtual forces acting on virtual objects, or any other actions or changes to objects that users can imagine. Summary of the invention

[0005] According to a first aspect of the present disclosure, a method for navigating multiple virtual worlds in an artificial reality is provided, the method comprising: providing a personal interface having multiple 2D interfaces to multiple applications, wherein each application has a 3D world construction portion and a 2D interface portion, the 3D world construction portion controlling one of the multiple virtual worlds executed in a first runtime, and the 2D interface portion controlling one of the multiple 2D interfaces executed in a second runtime of the personal interface; and switching to a subsequent virtual world by the following steps: receiving a selection of an application corresponding to a subsequent virtual world among the multiple applications; displaying the 2D interface portion of the selected application via the second runtime of the personal interface, wherein the 2D interface portion of the selected application includes one or more travel cards, each travel card defining a link to a corresponding travel destination of the subsequent virtual world; receiving a selection of a travel destination via the displayed 2D interface portion of the selected application; and in response to the selection, causing the selected application to generate and display a 3D world corresponding to the selected travel destination in the first runtime.

[0006] In some embodiments, receiving a selection of an application corresponding to a subsequent virtual world occurs simultaneously with displaying the current virtual world.

[0007] In some embodiments, the personal interface appears with consistent controls in both the current virtual world and subsequent virtual worlds.

[0008] In some embodiments, the travel destination corresponds to one or more of the following: (a) a place in the subsequent virtual world; (b) an event in the subsequent virtual world; (c) a person in the subsequent virtual world; or (d) any combination of the above.

[0009] In some embodiments, the method further includes: receiving a selection of an item in a subsequent virtual world that corresponds to the controller; accessing content from the controller via a deep link associated with the selected item; and presenting the content from the controller on the personal interface in a second runtime of the personal interface.

[0010] In some embodiments, the method also includes: generating, by the personal interface, 3D content in a second runtime of the personal interface other than the first runtime in response to receiving a 3D content trigger action for an item in a subsequent virtual world or on the personal interface; and displaying, by the personal interface, the generated 3D content according to display options, wherein the display options specify that one of the 3D contents is displayed in the following manner: A) at a specified position relative to the personal interface; B) using the personal interface as a window into another 3D world, which is the 3D content; or C) using the personal interface as a portal to a volume containing the 3D content.

[0011] According to a second aspect of the present disclosure, there is provided a computing system for navigating multiple virtual worlds in an artificial reality, the computing system comprising: one or more processors; and one or more memories storing multiple instructions, which, when executed by the one or more processors, cause the computing system to execute a process, the process comprising: providing a personal interface having multiple 2D interfaces to multiple applications, wherein each application has a 3D world construction portion and a 2D interface portion, the 3D world construction portion controlling execution of one of the multiple virtual worlds in a first runtime, and the 2D interface portion controlling execution of one of the multiple 2D interfaces in a second runtime of the personal interface; and switching to a subsequent virtual world by the following steps: receiving a selection of one of the multiple applications corresponding to the subsequent virtual world; displaying the 2D interface portion of the selected application via the second runtime of the personal interface; receiving a selection of a travel destination via the displayed 2D interface portion of the selected application; and in response to the selection, causing the selected application to generate and display a 3D world corresponding to the selected travel destination in the first runtime.

[0012] In some embodiments, the selected 2D interface portion of the application includes one or more travel cards, each travel card defining a link to a corresponding travel destination of a subsequent virtual world.

[0013] In some embodiments, the travel destination corresponds to one or more of: (a) a place within a subsequent virtual world; (b) an event within a subsequent virtual world; (c) a person within a subsequent virtual world; or (d) any combination of the above.

[0014] In some embodiments, receiving a selection of an application corresponding to a subsequent virtual world occurs simultaneously with displaying the current virtual world.

[0015] In some embodiments, the personal interface appears with consistent controls in both the current virtual world and subsequent virtual worlds.

[0016] In some embodiments, the process also includes: receiving a selection of an item corresponding to the controller in a subsequent virtual world; accessing content from the controller via a deep link associated with the selected item; and presenting the content from the controller on the personal interface in a second runtime of the personal interface.

[0017] In some embodiments, the method further includes: generating, by the personal interface, 3D content in a second runtime of the personal interface other than the first runtime in response to receiving a 3D content trigger action for an item in a subsequent virtual world or on the personal interface; and displaying, by the personal interface, the generated 3D content according to display options, wherein the display options specify that the 3D content be displayed in one of the following ways: A) at a specified position relative to the personal interface; B) using the personal interface as a window into another 3D world, which is the 3D content; or C) using the personal interface as a portal to a volume containing the 3D content.

[0018] According to a third aspect of the present disclosure, a machine-readable storage medium is provided, on which a plurality of machine-executable instructions are stored, which, when executed by one or more processors, cause the one or more processors to execute a method for navigating a plurality of virtual worlds in an artificial reality, the method comprising: providing a personal interface having a plurality of 2D interfaces to a plurality of applications, wherein each application has a 3D world construction portion and a 2D interface portion, the 3D world construction portion controlling execution of one of the plurality of virtual worlds in a first runtime, and the 2D interface portion controlling execution of one of the plurality of 2D interfaces in a second runtime of the personal interface; and switching to a subsequent virtual world by the following steps: receiving a selection of an application among the plurality of applications corresponding to the subsequent virtual world; displaying the 2D interface portion of the selected application via the second runtime of the personal interface; receiving a selection of a travel destination via the displayed 2D interface portion of the selected application; and in response to the selection, causing the selected application to generate and display a 3D world corresponding to the selected travel destination in the first runtime.

[0019] In some embodiments, the selected 2D interface portion of the application includes one or more travel cards, each travel card defining a link to a corresponding travel destination of a subsequent virtual world.

[0020] In some embodiments, the travel destination corresponds to one or more of: (a) a place within a subsequent virtual world; (b) an event within a subsequent virtual world; (c) a person within a subsequent virtual world; or (d) any combination of the above.

[0021] In some embodiments, receiving a selection of an application corresponding to a subsequent virtual world occurs simultaneously with displaying the current virtual world.

[0022] In some embodiments, the personal interface appears with consistent controls in both the current virtual world and subsequent virtual worlds.

[0023] In some embodiments, the method further includes: receiving a selection of an item in a subsequent virtual world that corresponds to the controller; accessing content from the controller via a deep link associated with the selected item; and presenting the content from the controller on the personal interface in a second runtime of the personal interface.

[0024] In some embodiments, the method further includes: generating, by the personal interface, 3D content in a second runtime of the personal interface other than the first runtime in response to receiving a 3D content trigger action for an item in a subsequent virtual world or on the personal interface; and displaying, by the personal interface, the generated 3D content according to a display option, wherein the display option specifies that the 3D content is displayed in one of the following ways: A) displayed at a specified position relative to the personal interface; B) using the personal interface as a window into another 3D world, which is the 3D content; or C) using the personal interface as a portal to a volume containing the 3D content.

[0025] It will be appreciated that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be universal in any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art from the specification, claims and drawings of the present disclosure. The above summary description and the following detailed description are exemplary and illustrative only and are not limitations of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram showing an overview of a number of devices on which some embodiments of the present technology may operate.

[0027] Figure 2A is a line diagram illustrating a virtual reality headset that may be used in some embodiments of the present technology.

[0028] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0029] Figure 2C is a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold with one or both hands to interact with an artificial reality environment.

[0030] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0031] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0032] Figure 5 is a conceptual block diagram illustrating an exemplary artificial reality (hyper-reality (XR)) application that may be used in some implementations to generate and control personal interfaces and 3D world content, respectively.

[0033] Figure 6 is a flow chart illustrating a process for directly navigating between multiple virtual worlds using a personal interface of an artificial reality (XR) navigation system in some embodiments of the present technology.

[0034] Figure 7 is a flow chart illustrating a process for presenting content from a controller corresponding to a selected item of a virtual world using a personal interface in some implementations of the present technology.

[0035] Figure 8 is a flow chart illustrating a process for generating and displaying 3D content via a personal interface in some implementations of the present technology.

[0036] Fig. 9 is a flow chart illustrating a process for populating a dedicated space within a virtual world with content originating from an application external to the virtual world in some implementations of the present technology.

[0037] Fig.10 is a schematic diagram illustrating an exemplary personal interface.

[0038] Fig.11 is a schematic diagram showing an exemplary personal interface that depicts a virtual world destination that a user can execute when selecting an XR application on the personal interface.

[0039] Fig. 12A is an exemplary schematic diagram showing a personal interface that provides 3D content separate from the world in which the user is currently traveling.

[0040] Fig. 12B is an exemplary schematic diagram showing a personal interface that provides a window into a virtual world that is different from the world in which the user is currently traveling.

[0041] Fig. 12C is an exemplary schematic diagram showing a personal interface that provides a portal for viewing and interacting with 3D content that is separate from the world in which the user is currently traveling.

[0042] Fig.13 is an exemplary diagram illustrating the populating of a dedicated space within a virtual world with content originating from an application external to the virtual world.

[0043] The technology described herein may be better understood by referring to the following detailed description and accompanying drawings, in which like reference numbers refer to identical or functionally similar elements. DETAILED DESCRIPTION

[0044] Aspects of the present disclosure relate to a virtual personal interface (herein a "personal interface") that controls an artificial reality (XR) environment, for example, by providing a user interface for interacting with a current XR application, providing a detailed view of a selected item, navigating between multiple virtual worlds without having to transition into or out of a home lobby for these worlds, executing aspects of a second XR application within a world controlled by a first XR application, and providing 3D content separate from the current world. The personal interface can be separate from the current virtual world, allowing the personal interface to appear consistently across multiple virtual worlds, and allowing the personal interface to either display controls from the XR application controlling the current virtual world, or display elements and controls for other VR worlds (e.g., controls that enable virtual world selection and transport).

[0045] For example, an XR system may define a platform for XR applications, wherein multiple XR applications may each include a controller, a system for providing output in a personal interface, and a system for providing a 3D world. For example, when a user is in a first virtual world, she may access a personal interface, and may navigate within the personal interface to the personal interface output of the XR application for controlling the current world or a personal interface output from another XR application. In some cases, the personal interface output from the other application is referred to herein as a 2D interface, but may include 3D content in some cases. A 2D interface may allow a user to teleport directly to one or more locations in a 3D world controlled by the other application. When such a teleportation control is activated, the 3D world building portion of the corresponding XR application may be activated to create a world into which the user is brought. In other words, a personal interface may host a 2D interface from a personal interface builder of an application, wherein the 2D interface may enable teleportation to a 3D world and / or control of the 3D content of the corresponding virtual world. In some cases, the personal interface builder and / or 3D world builder may cause the display of a 2D interface and / or a 3D world by orchestrating content hosted by a server for the corresponding application and / or content stored locally by the personal interface.

[0046] When the user desires to travel from the current virtual world to a subsequent virtual world, she may simply select the desired application on the personal interface (i.e., simply select the desired application via the 2D interface of the personal interface builder from the application) and select the teleportation control to be transported to the corresponding virtual world. In this case, the current virtual world may be displayed simultaneously with the personal interface (which may provide a 2D interface from the selected other application) until the 2D interface is used to change the world through the corresponding 3D world builder of the other application. In some embodiments, as part of the 2D interface of the application, the personal interface may present various specific destinations in the corresponding virtual world to which the user may be directly transported. For example, such destinations may be specified by one or more travel cards, which are implemented as deep links to places, events, or characters in the corresponding virtual worlds. In this way, the personal interface may facilitate direct travel between multiple virtual worlds in artificial reality. In response to the selection of one of such multiple links, the 3D world building portion corresponding to the selected application may then responsively (i.e., be caused to) build a 3D world corresponding to the selected destination.

[0047] In some embodiments, the personal interface may define various controls applicable to each of the multiple virtual worlds corresponding to the multiple applications. For example, these controls may orchestrate the following: avatar movement and / or appearance; navigation to specific areas in the virtual world; personal content that may be included in the virtual world; access to credit required to pay for transactions, etc. In these ways, the personal interface may provide a full set of controls that can be similarly applied regardless of which virtual world the user is operating in. Alternatively, the personal interface may bridge these spaces by providing these controls universally across multiple virtual worlds to allow these spaces to be experienced in an unconstrained manner from the user's perspective.

[0048] As can be understood from the above, the personal interface can be used as an output medium for content originating from an application corresponding to the current virtual world. However, in some embodiments, the personal interface can also provide content originating from an external application, that is, content that is not explicitly specified by the application controlling the current virtual world or content that is not known to the application controlling the current virtual world. In some cases, the personal interface can even provide 3D content that is not controlled by the following application: the application controls the current virtual world. In some cases, the personal interface can provide controls for a dedicated area of ​​the virtual world that is enhanced with content from another application. For example, a virtual world can include a dedicated broadcast (e.g., casting) space (e.g., a 2D area or a 3D area), and the personal interface can coordinate a deep link to the space provided to a third-party application, and then the third-party application can project its content into the dedicated space. Therefore, the user can select a content provider (i.e., a third-party application) on the personal interface, which can then cause the content to be relayed to the dedicated broadcast space.

[0049] In some implementations, the personal interface can display various corresponding content for other items selected by the user in the virtual world. For example, when a selectable item is selected by the user and is deeply linked to a controller for the item (e.g., the same application as the virtual world application being traveled, another application, a system controller (e.g., a profile manager or a social graph module), etc.), the personal interface can access this deep link to obtain and display corresponding content from the controller. In this way, the user can browse the corresponding content to, for example, learn additional information about the selected item.

[0050] In some embodiments, the personal interface can generate 3D content that does not include the application of the current virtual world in which the user is traveling (i.e., is not controlled by the application). To this end, the 2D interface that controls the personal interface can react to the user's 3D content triggering action so that the personal interface accesses and displays 3D content for the action. Exemplary content triggering actions may include the user approaching or selecting a specific item in the virtual world, gazing or staring at a specific item in the virtual world or on the personal interface, selecting an item on the personal interface, navigating a website in the virtual world, being present in the virtual world (the presence is detected by the control application of the virtual world (to provide content specified by the application)), selecting one or more personal content items that the user expects to include in the virtual world, etc. The personal interface can display the corresponding 3D item content near the personal interface in response to receiving the 3D content triggering action, the personal interface can become a window into another virtual world, or the personal interface can provide a portal to the following volume: the volume extends through the personal interface and enables the user to view and interact with 3D content provided in the volume but may not be controlled by the application corresponding to the current virtual world. In these ways, the personal interface can provide the user with an opportunity to preview items that trigger actions for 3D content so that the user can be initially informed of the item's knowledge or existing knowledge can be enhanced without having to provide the 3D content through or access the 3D content by an application controlling the current virtual world. In some cases, the personal interface can provide the type of preview (nearby area, window, portal) based on the item and its controller (e.g., the corresponding application). In this way, the personal interface can optimize the scope and degree of relevant information that can be conveyed to the user about the item.

[0051] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality, and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0052] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture multiple real-world images and then display them on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system (e.g., by adding virtual objects). "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially constitutes light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a transparent display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR; AR; MR; or any combination or hybrid thereof.

[0053] The existing way of presenting, navigating and controlling virtual worlds in artificial reality provides discrete ways to determine and reach these worlds. For example, these worlds can only become the target of access and manipulation through the transmission of the home lobby. That is, the home lobby can be similar to a mental switch, and only after the conversion through this lobby can the user be redirected from the world currently traveling to the subsequent virtual world. In these ways, the home lobby is an obstacle to being able to seamlessly navigate in multiple worlds for items of interest in the current world and other worlds. That is, traveling between multiple virtual worlds in artificial reality requires users to switch between these worlds through the home lobby, in which the current world is closed and the new world is opened. In other words, seamless movement between multiple worlds without involving this conversion step has avoided the currently available architecture. Therefore, the corresponding artificial reality experience may be unnecessarily disappointing, distracting and time-consuming. In contrast, the implementation of the present technology provides users with the ability to switch between virtual worlds uninterruptedly, regardless of whether it is associated with a specific item of interest. That is to say, this uninterrupted conversion can, for example, provide users with the ability to maintain a psychological association with the previous virtual world and the future virtual world. For example, applicable transitions may be enabled through deep links to items of interest that may allow a user to travel directly to a virtual world. Thus, instances of such direct travel may be programmatically formulated to enhance travel between multiple virtual worlds, thereby causing users of the artificial reality navigation system (unlike existing systems) to specifically remember the relative associations of virtual worlds and items contained within / defined for those worlds. Furthermore, embodiments of the present technology may conserve system resources (e.g., battery power) and reduce processing time for artificial reality systems by enabling such direct travel because there is no need to load, store, and access home lobbies.

[0054] Another way in which embodiments of the present technology differ from existing systems includes the ability to provide enhanced security for user content when run through a personal interface that operates independently of any application controlling the virtual world. Thus, the risk of such applications accessing user content and potentially manipulating such content (e.g., credit cards, inventory items, etc.) is substantially eliminated.

[0055] Another way that embodiments of the present technology improve upon existing artificial reality systems includes the ability to provide multiple avenues for the introduction and display of 3D content, thereby demonstrating greater usability and flexibility than these current systems. For example, whereas existing systems are limited to providing 3D content from only a single application at any given time, the present technology can provide 3D content via a personal interface as well as from applications outside the current virtual world (e.g., projection applications).

[0056] Further, embodiments of the present technology may enable direct access to content that may be displayed in a personal interface. For example, current embodiments may provide a system of deep links between selectable entities and corresponding controllers, thereby allowing third-party systems to provide contextual and control information without requiring the user to filter through another external system to access and obtain that information.

[0057] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 1 is a block diagram showing an overview of multiple devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of a computing system 100 that enable control of an artificial reality (XR) environment, for example, by providing a user interface for interacting with a current XR application, providing a detailed view of a selected item, navigating between these worlds without having to transition into or out of a home lobby for multiple virtual worlds, executing aspects of a second XR application within a world controlled by a first XR application, and providing 3D content separate from the current world. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head mounted view device and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head mounted view device and other processing operations are offloaded to the core processing component. Figure 2A and Figure 2B An example head mounted view device is described. In some implementations, the location data and environmental data may be collected solely by sensors incorporated into the head mounted view device, while in other implementations, one or more of the plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.

[0058] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0059] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets signals received from the input devices and transmits the information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or another light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0060] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus, or a wireless connection. The processor 110 can communicate with the hardware controller of each device (e.g., a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices include: a liquid crystal display (LCD) display screen; a light emitting diode (LED) display screen; a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or card, a video chip or card, an audio chip or card, a universal serial bus (USB), FireWire or another external device, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0061] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, GPS units, LiDAR or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment while tracking the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that maps the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other changes in position, etc.

[0062] The computing system 100 may include such a communication device: the communication device can communicate with other local computing devices or network nodes wirelessly or based on wired communication. The communication device can communicate with another device or server through a network, for example, using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol. The computing system 100 can use the communication device to distribute operations on multiple network devices.

[0063] Processor 110 may access memory 150, which may be included on one of the computing devices of computing system 100, or may be distributed across the computing devices of computing system 100, or multiple other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM); various cache memories; CPU registers; read-only memory (ROM); and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, tape drive, etc. Memory is not a propagating signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160 that stores programs and software, such as operating system 162, artificial reality (XR) control and navigation system 164, and other applications 166. Memory 150 may also include data storage 170 which may include, for example, virtual world location data, virtual world item location data, configuration data, settings, user options or user preferences, etc. which may be provided to program storage 160 or any element of computing system 100 .

[0064] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the present technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments including any of the foregoing systems or devices, and the like.

[0065] Figure 2A 2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a strap 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a plurality of locators 225, and one or more computing units 230. Position sensors 220, IMU 215, and computing units 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensors 220, and locators 225 may track the movement and position of HMD 200 in the real world and artificial reality environments with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locators 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position sensors, force sensors, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shape and position of real objects around the HMD 200.

[0066] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD); an organic light-emitting diode (OLED) display; an active-matrix organic light-emitting diode display (AMOLED); a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels; a projector unit (e.g., micro-LED; laser (LASER)), etc.); some other display, or some combination thereof.

[0067] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC can use the HMD in conjunction with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0068] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a pass-through display 258 and a frame 260. The frame 260 can accommodate various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye tracking sensors, micro-electromechanical systems (MEMS) components, network components, etc.

[0069] The projector can be coupled to a transparent display 258, for example, via optical elements, to display media to the user. These optical elements can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for directing light from the projector to the user's eyes. Image data from the core processing component 254 can be transmitted to the HMD 252 via a link 256. A controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted to the user's eyes as output light via optical elements. The output light can be mixed with the light passing through the display 258, so that the output light can present virtual objects as follows: these virtual objects appear to exist in the real world.

[0070] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0071] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, and in some embodiments, a user can hold a controller 270 with one or both hands to interact with the artificial reality environment presented by the HMD 200 and / or HMD 250. The controller 270 can communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controller can have its own IMU unit, position sensor, and / or can emit a light point farther away. The HMD 200 or HMD 250, external sensors, or sensors in the controller can track these controller light points to determine the position and / or orientation of the controller (e.g., tracking the controller with 3DoF or 6DoF). The computing unit 230 or the core processing component 254 in the HMD 200 can use this tracking to monitor the user's hand position and movement in combination with the IMU output and the position output. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A and 274B) that a user may actuate to provide input and interact with objects.

[0072] In various embodiments, HMD 200 or HMD 250 may also include a number of additional subsystems for monitoring user interactions and indications of intent, such as an eye tracking unit, an audio system, various network components, etc. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or HMD 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate one or both eyes of the user, and HMD 200 or HMD 250 can use an eye-facing camera to capture reflected light of that light to determine eye position (e.g., based on a set of reflected light around the user's cornea) to model the user's eyes and determine gaze direction.

[0073] Figure 3 305B may be a HMD 200 or a HMD system 250. The client computing device 305 may be a logical connection to one or more remote computers (e.g., a server computing device) through a network 330 to operate in a network environment.

[0074] In some implementations, server 310 may be an edge server that receives client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing device 310 and server computing device 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, these server computing devices may each be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0075] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can each be a distributed computing environment comprising multiple computing devices, can be located in its corresponding server, or can be located at the same physical location or at geographically different physical locations.

[0076] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.

[0077] Figure 44 is a block diagram showing a plurality of components 400 that may be used in a system employing the disclosed technology in some embodiments. The plurality of components 400 may be included in one device of the computing system 100, or may be distributed across multiple devices of the computing system 100. The plurality of components 400 include hardware 410, middleware 420, and a plurality of specialized components 430. As discussed above, a system implementing the disclosed technology may use various hardware including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.), and a storage memory 418. In various embodiments, the storage memory 418 may be one or more of the following: a local device; an interface to a remote storage device; or a combination thereof. For example, the storage memory 418 may be one or more hard drives or flash drives accessible via a system bus, or may be a cloud storage provider (e.g., in memory 315 or 325) or other network storage device accessible via one or more communication networks, a cloud storage provider, or other network storage device. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0078] Middleware 420 may include components that coordinate resources between hardware 410 and multiple dedicated components 430. For example, middleware 420 may include an operating system, a service system (service), a driver, a basic input output system (BIOS), a controller circuit, or other hardware or software systems.

[0079] The dedicated components 430 may include components configured to perform operations for controlling an artificial reality (XR) environment, for example, by providing a user interface for interacting with a current XR application, providing a detailed view of a selected item, navigating between these worlds without having to transition into or out of a home lobby for multiple virtual worlds, executing aspects of a second XR application within a world controlled by a first XR application, and providing 3D content separate from the current world. The dedicated components 430 may include an information retrieval module 434, an information evaluation module 436, a travel execution module 438, a content pattern generation module 440, a content promotion module 442, a content selection module 444, a content transmission module 446, a content presentation module 448, and components and APIs (e.g., interface 432) that can be used to provide a user interface, transmit data, and control the dedicated components. In some embodiments, the plurality of components 400 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the plurality of dedicated components 430. Although multiple specialized components 430 are depicted as separate components, multiple specialized components 430 may have logical or other non-physical partitions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0080] In some embodiments, the information retrieval module 434 can retrieve information (i.e., "data") that can be used to activate the personal interface. For example, such data can include gestures, utterances, and other activities of the user while traveling within the virtual world. In some embodiments, the information retrieval module 434 can retrieve the user's selection of an XR application corresponding to the virtual world in which the user desires to travel (including a specific destination in that world). In some embodiments, for a virtual world, the information retrieval module 434 can retrieve data corresponding to the user's actions or the space with which the user interacts. Such data can be about the selection of a specific item (e.g., an object, a person, a dedicated broadcast space), the gaze of a specific item in the virtual world or on the personal interface, the selection of a control on the personal interface, a conversation with a person in the virtual world, and the like. In addition, the information retrieval module 434 can retrieve data for a 3D content triggering action that can cause the personal interface to display 3D content in addition to the virtual world in which the user is traveling. In this case, a non-exhaustive list of triggering actions may include the user approaching or selecting a particular item in the virtual world, looking at or gazing at a particular item in the virtual world or on the personal interface, selecting an item on the personal interface, navigating a website in the virtual world, being present in the virtual world (which presence is detected by the control application of the virtual world (to provide content specified by the application)), and selecting one or more personal content items that the user desires to include in the virtual world. Figure 6 Boxes 602, 604 and 608 in FIG. Figure 7 In box 702, Figure 8 Box 802 and Fig. 9 Block 902 in provides additional details regarding retrieving the above-described types of data.

[0081] In some embodiments, the information evaluation module 436 can perform specific evaluations regarding travel to and actions within a virtual world. For example, the information evaluation module 436 can evaluate which XR application corresponds to a virtual world that a user has selected for travel, and whether the user has selected a specific travel destination designated for the virtual world. In some embodiments, the information evaluation module 436 can evaluate the types of 3D content triggering actions that a user has taken within a virtual world and which 3D content triggering actions can result in the generation of 3D content on a personal interface. In addition, for a virtual world, the information evaluation module 436 can estimate the type of broadcast space (e.g., 2D or 3D) within the virtual world that can be dedicated to displaying or otherwise presenting content for the virtual world. In conjunction with Figure 6 Boxes 604 and 608 in Figure 8 Box 802 and Fig. 9 Box 902 in provides additional details of the evaluation performed by the information evaluation module 436.

[0082] In some implementations, the travel execution module 438 can execute travel to a specific destination (e.g., a place, an event, or one or more individuals) within the virtual world. For example, in response to a user selecting an XR application corresponding to the virtual world on the personal interface, travel can be executed according to the user's selection of a travel card presented by the application. Figure 6 Block 608 in provides additional details regarding the types of travel that may be performed by this module.

[0083] In some embodiments, the content mode generation module 440 can determine a specific generation mode for the content that should be generated. For example, the content mode generation module 440 can select how the 3D content should be presented to the user via the personal interface when the personal interface is used to generate 3D content that does not include a world in which the user is traveling. In these aspects, exemplary modes can include presenting the content according to display options, such as near the personal interface, through a window provided in the personal interface, or through a volumetric space generated by the personal interface. In some cases, the content mode generation module 440 can determine a specific mode based on the type of item that is the subject of the 3D content triggering action, for example based on a mapping of item types to display modes. In other cases, the content mode generation module 440 can always use the same display mode. In other cases, the application that controls the display of the 3D content can further specify which display mode to use. About Figure 8 Block 804 in provides additional details regarding the content pattern generation performed by the content pattern generation module 440 .

[0084] In some embodiments, the content promotion module 442 can facilitate the processing of specific types of content that can enhance the virtual world in which the user is traveling. For example, for a selected virtual space within the virtual world in which the user is traveling, such a module can enable the transmission of 2D content and / or 3D content to the space. In this regard, the content promotion module 442 can detect a deep link associated with the space. Once a link is detected, the content promotion module 442 can transmit the link to the personal interface, such as so that the user can then select an application corresponding to a content provider, which can transmit the content to the selected virtual space. In conjunction with Fig. 9 Boxes 904 and 906 in provide additional details regarding content promotion that may be performed by content promotion module 442.

[0085] In some implementations, the content selection module 444 can perform the selection of content that is processed according to one or more operations of the content promotion module 442. That is, the content selection module 444 can receive a 2D interface selection of the personal interface for an application on the personal interface that can provide content that can enhance the virtual world. Figure 6 Block 906 in provides additional details regarding the content selection performed by the content selection module 444 .

[0086] In some embodiments, the content transmission module 446 can transmit 3D content to change or enhance the virtual world in which the user is traveling. An example transmission may involve content of an XR application corresponding to the virtual world in which the user desires to travel. Another example transmission may involve content that fills the virtual space, the content coming from a content provider whose content is facilitated by the content facilitation module 442. Figure 6 Box 610 and Fig. 9 Block 906 in provides more details regarding the 3D content transfer performed by the content transfer module 446 .

[0087] In some implementations, the content presentation module 448 can present 3D content based on a user's selection of an XR application corresponding to the virtual world or a 2D application that can enhance the content of the world. Figure 6 Frame 610, Figure 7 Frame 706, Figure 8 Box 806 and Fig. 9 Additional details regarding content presentation performed by content presentation module 448 are provided in block 906 of FIG.

[0088] Those skilled in the art will appreciate that the above Figures 1 to 4 The components shown in each of the flowcharts described above and in the flowcharts discussed below may be changed in a variety of ways. For example, the order of the logic may be rearranged, multiple sub-steps may be performed in parallel, the logic shown may be omitted, other logic may be included, etc. In some embodiments, one or more of the above-described components may perform one or more of the multiple processes described below.

[0089] Figure 5is a conceptual block diagram of an exemplary artificial reality (XR) application 500 that can be used to generate a 2D interface displayed via a personal interface and control the 3D content of the corresponding virtual world in the artificial reality in some embodiments. Among them, the application 500 includes a controller 502, a personal interface builder 504, and a 3D world builder 506. In operation, the XR system may include multiple runtimes (runtime systems or environments), one runtime is controlled by the currently active application that defines the current virtual world, and one runtime is the personal interface. Although the personal interface can be a virtual object in the 3D world with properties such as size, reaction to physical phenomena, display properties, etc., the content displayed by the personal interface can be a runtime separate from the virtual world. This separation between the virtual world and the personal interface provides the active application with the ability to output the 3D world when the user accesses the features of other XR applications (e.g., a travel card that directly teleports to another virtual world). In various embodiments, the personal interface execution environment can be hidden from the active application, or can have the following access ports: the application executing on the personal interface can control these access ports to present only the desired information to the active application. Thus, the XR application can be conceptually divided into general control and access functions in the controller 502, a personal interface builder 504 module that executes when the personal interface is running to provide a 2D interface on the personal interface, and a 3D world builder module 506 that executes when the world is running to generate the current 3D world. In this way, the 3D world builder 506 of the XR application is only active when the XR application is generating the current 3D world (or in some cases, when a window in the world is needed), while the personal interface builder 504 can be executed in the following situations: when the XR application is the currently active application; or when the XR application is selected in the personal interface when another application is the currently active application.

[0090] The controller 502 may include all necessary coding and programming for coordinating the operation of the personal interface builder 504 and the 3D world builder 506. For example, the controller 502 may include an API for, for example, triggering the personal interface builder 504 to generate a 2D interface in response to a deep link activation, or triggering the 3D world builder 506 to generate a 3D world. The personal interface builder 504 may respond to activation of an application on the personal interface received via the controller 502. The activation may cause the personal interface builder 504 to generate a 2D interface portion of the XR application: the 2D interface portion may, for example, control travel to a location in the virtual world of the application, access features in the current virtual world, such as user preferences, social activities, use of accessories, etc. When a user of the XR control and navigation system 164 uses a 2D interface from the personal interface builder 504 to travel to a location in the corresponding world, the 3D world builder 506 may cause the virtual world to be loaded (i.e., through communication performed by the controller 502). In various cases, the personal interface builder 504 and / or 3D world builder 506 can use local content and / or content retrieved from a remote source to build 2D interfaces and 3D worlds. For example, a 3D world can include multiple 3D models (e.g., trees, buildings, etc.) that can be manipulated by multiple users in the 3D world. These 3D models can be stored locally or can be retrieved from a server, and the representations of other users and their states can be synchronized with such a server. As another example, the personal interface builder 504 can have templates that are populated with content stored locally or live content retrieved from a server.

[0091] In some embodiments, the personal interface builder 504 itself can generate 3D content that can be displayed via the personal interface according to display options corresponding to the content of one or more items. In this way, the following personal interface can host and display content independently of the virtual world: the personal interface can be a 2D application with a 2D interface corresponding to the XR application and other applications. Therefore, the personal interface can be used as an output medium for the XR application in two different ways. First, the personal interface can be a tool that simply presents the 2D interface of the application, for example, enabling the user to travel in multiple virtual worlds corresponding to these applications and interact with the multiple virtual worlds. In this case, the 3D content is under the control of the XR application corresponding to the virtual world that the user is traveling in and generated outside the personal interface. Secondly, the personal interface can be a tool that generates 3D content of the XR application and displays the 3D content by itself. Such 3D content may not include the virtual world in which the user is traveling. In this way, the user can "preview" one or more aspects of the virtual world through the personal interface, for example, without actually entering a given virtual world.

[0092] Figure 6 6 is a flow chart illustrating a process 600 for navigating multiple virtual worlds in an artificial reality using a personal interface of an XR control and navigation system 164 in some embodiments of the present technology. For example, process 600 may be used to allow a user to instantly and interchangeably travel between multiple virtual worlds in which the user's acquaintances reside. In some embodiments, process 600 may be initiated by a user executing a program that creates an artificial reality environment. In some embodiments, process 600 may be performed on a server system that controls the artificial reality environment; and in some cases, all or part of process 600 may be performed on an artificial reality client device.

[0093] At block 602, a user of the XR control and navigation system 164 may activate a personal interface for accessing one or more controls (e.g., avatar controls, content items that may be accessed, payment methods and account access, navigation tools, etc.) while navigating in a current virtual world. For example, the personal interface may be activated and displayed to the user in response to the user making one or more gestures, a series of motions, speech, activating a user interface (UI) element, etc. In some implementations, the user may activate the personal interface using one or more of these activation cues to display the personal interface in order to navigate directly from the current virtual world to a subsequent virtual world.

[0094] In some cases, when the personal interface is activated, it can display a 2D interface of the currently active application (i.e., the application that controls the current virtual world in which the user is located). At box 604, the process 600 can receive a selection of another XR application from a plurality of available XR applications through the personal interface. At this point, the selected XR application can represent a subsequent virtual world that the user wishes to interact with and / or the user expects to seamlessly travel from the current virtual world to. In order to make a selection of an XR application, the process 600 can present the user with an application list (in which the application can be stored locally on the personal interface) and / or the following application shells: the process 600 has selected these application shells as being of interest to the user and components of these application shells can be downloaded upon selection. To facilitate selection, the process 600 can configure the application to be searchable and / or organized according to a predetermined ranking format (e.g., most commonly used).

[0095] At box 606, process 600 can display the 2D interface portion of the selected application via the personal interface. For example, the displayed 2D interface portion can control the 2D interface of the personal interface so as to output content corresponding to the selected application. During such control, such 2D interface portion can be operated according to the runtime of the personal interface so as not to be subject to any control of the controller of the current virtual world, that is, when displaying the content of the selected application, the 2D interface portion can operate under its own authority. In some embodiments, the displayed content corresponding to the selected application may include, for example, information about the source of the selected application (e.g., its origin), the type of virtual world corresponding to the application, a list of travel cards that enable the user to travel to a specific destination in the corresponding virtual world, a list of occupants of the corresponding virtual world, an avatar that will be available to the user when the user arrives in the virtual world, and the like.

[0096] At block 608, process 600 may receive a selection of a travel destination through the displayed 2D interface portion of the selected application. For example, the 2D interface portion may present to the user a plurality of travel cards for a subsequent virtual world corresponding to the application, wherein the travel cards may indicate destinations, including places (museums, schools, resorts, etc.), events (sports games, music performances, parties, etc.), and people (friends, family members, managers, etc.) within the world. In this regard, the travel cards may have corresponding deep links so that the user can travel directly to the corresponding associated destination. In these aspects, the travel cards may be initiated, for example, by one or more of the following (i.e., defined by one or more of the following): initiated natively by the selected application; initiated due to scheduling certain events for the subsequent virtual world (e.g., the timing of a sports or music event); initiated based on membership or presence in the world (e.g., a friend of the user, a celebrity, a user with a special status in the world); initiated when another user designates a location in the world as a location of interest, and the like. In some embodiments, specific aspects that may be included in a progress card may include one or more of the following: avatar selections that will be provided to the user when the user arrives in a subsequent virtual world; descriptions of specific events and their start times, such as activities in a subsequent virtual world; people that the user can join in such a world, etc.

[0097] At block 610, process 600 may cause the selected application to generate and display 3D content for a subsequent virtual world corresponding to the selected travel destination. As an example, if the subsequent virtual world includes a building with multiple rooms, and the selected travel destination corresponds to a particular room in the rooms, process 600 may generate a representative 3D description of the room. In this regard, it is understood that process 600 may generate the room in 3D format based on the operation of controller 502 and 3D world construction portion 506 and the operation between them. In some embodiments, process 600 may generate 3D content for a virtual world corresponding to the selected travel destination (e.g., the above-mentioned room) from elements stored locally on a personal interface or otherwise (i.e., elements from a remote server). In some embodiments, process 600 may generate 3D content for such a virtual world, wherein, for an exemplary multiplayer environment, one or more states of the displayed objects may be synchronized between users in the world (e.g., by one or more host servers of the world).

[0098] At block 612, in the event that the user has not completed traveling among multiple virtual worlds in the artificial reality, process 600 may return to block 602 to initiate travel to another subsequent virtual world. In the event that the personal interface is still activated (i.e., not cancelled by the user in accordance with one or more prompts of similar type to the prompts that enabled activation but, for example, in a different number or order), it is contemplated that the user may simply tap the personal interface to wake it up. That is, the XR control and navigation system 164 may configure the personal interface to execute a sleep mode after a predetermined period of inactivity in order to reduce distraction to the user while traveling in a virtual world.

[0099] Figure 7 is a flowchart showing a process 700 for presenting content corresponding to a selected item from a controller using a personal interface in some embodiments of the present technology. Process 700 can be initiated when a user is traveling in a virtual world corresponding to a selected XR application, and process 700 can be executed either on a server system that controls the XR application or on an artificial reality client device that operates the selected XR application. For example, for an item in a virtual world in which a user is traveling, process 700 can present the content of the item in real time on a personal interface. For example, such an item can be a painting hanging in a virtual world of a museum, where the content presented can include the name of the artist, the artist's history, other paintings painted by the artist, and the like.

[0100] At block 702, process 700 may receive a selection of an item, such as an item in the current virtual world in which the user is traveling or an item selected through a personal interface. For example, the selection may be the result of the user interacting with the item (e.g., touching, looking at the item, etc.); being within a predetermined proximity of the item; passing the item a predetermined number of times within a certain time interval; tapping the item when it is displayed on the personal interface; speaking a command indicating the item, etc.

[0101] At box 704, process 700 may access a deep link to a controller corresponding to the selected item. The deep link may be included in the data structure of the selected item and may specify a destination controller and one or more parameters to be passed to the controller (e.g., the identity of the selected item, the identity of the user, selected context information, etc., such as which virtual world it was previously in, who else is present). For example, the deep link may specify the destination as a module or function call in the current application, another local XR application, a uniform resource identifier (URI) of a remote service, etc. In some cases, the controller may correspond to an application of the current virtual world in which the user is traveling. In other cases, the controller may correspond to an application of an external local or remote service, that is, an application separate from the application that controls the current virtual world. In some embodiments, the controller may correspond to the following items: a system component mapped to the selected item, such as a profile manager, a social graph module, or a contact module mapped to a person type item; a file picker mapped to a media player type item; a digital wallet mapped to a payment portal type item; a scheduling manager that defines the availability of certain areas within the current virtual world, etc. In some cases, the selected item may be an area of ​​the virtual world, and at block 704, process 700 may access a controller deep link as a result of the user entering the area. When process 700 accesses the deep link corresponding to the selected item, process 700 may provide, via the personal interface runtime, a request to the controller to display corresponding information in the personal interface, wherein the request may specify, for example, specific parameters of the selected item.

[0102] At block 706, process 700 may receive the requested content from the controller of the selected item, and may display or present the content on the personal interface. There is no limitation on the type of content that may be provided, but examples include a detail page for the selected item, user annotations to the selected item, a 2D interface of an XR application that is a controller for the selected item, a default page provided by a system component that provides metadata from the selected item, another 3D model, a view of another virtual world, etc.

[0103] Figure 88 is a flow chart illustrating a process 800 for generating and displaying 3D content via a personal interface in some embodiments of the present technology. Process 800 may be initiated when a user is traveling within a virtual world. Process 800 may be performed on an XR device, or in some cases, on a server system supporting such an XR device, where operations such as “displaying 3D content” are understood to cause such content to be displayed on an XR device. For example, process 800 may be performed via a personal interface for a user in conjunction with a virtual world. Figure 7 The paintings discussed generate and display 3D content.

[0104] As the user moves within the current virtual world, process 800 may receive one or more 3D content triggering actions at box 802. For the user's activities within the current virtual world, such triggering actions may be actions and / or contexts that the XR control and navigation system 164 has mapped to display corresponding 3D content during the virtual personal interface runtime. For example, content triggering actions may include the user approaching or selecting a specific item in the virtual world, looking at or gazing at a specific item in the virtual world or on the personal interface, selecting an item on the personal interface, navigating to a website in a virtual web browser, being present in the virtual world (which presence is detected by the control application of the virtual world (to provide content specified by the application)), selecting one or more personal content items that the user desires to include in the virtual world, selection of an item (wherein the corresponding deep-linked controller provides 3D content in response (as described above in combination with Figure 7 Discussed) etc.

[0105] At box 804, process 800 can generate 3D content based on the display options. Specifically, process 800 can generate such 3D content during the runtime of the personal interface, wherein the generated content may be inaccessible or uncontrolled by the corresponding application controlling the current virtual world. The generated content can be related to the item that is the target or subject of the given 3D content trigger action. For example, if the user approaches an advertisement in the current virtual world, the corresponding generated content can be provided by an application under the control of the associated advertiser. At this point, it can be understood that the personal interface builder of the source application (see Figure 5 The personal interface builder 504 of FIG. 5 and the accompanying discussion) can provide 3D content to the personal interface as part of the content generated according to block 804 - this can be triggered by deep link activation to the source application, such as in conjunction with Figure 7as discussed. Depending on the nature of the content to be generated, process 800 may generate the content according to a particular display option (i.e., the controlling application specifies how the item is to be displayed relative to the personal interface). In various embodiments, the system may allow only one display option, which may be set by the content provider or as metadata about the content, or there may be a mapping of content types to display options. Display options executable by the personal interface may include displaying the generated content near the personal interface in the following manner: through the personal interface serving as a window to another virtual world; or through a portal (e.g., volume) that extends through the personal interface and through which the user can access the item. For example, in the case where the content is a 3D model that is smaller than a certain size, a portal display option may be used, in the case where the content is a 3D model that is larger than a certain size, a nearby display option may be used, and in the case where the content is a link to another virtual world, a window display option may be used.

[0106] At block 806, process 800 may display the generated content according to the corresponding display options. In this way, process 800 may provide a personal interface as an output medium for 3D content in a manner that is not controlled by the current virtual world in which the user is traveling or does not interfere in any way with the display of the current virtual world.

[0107] Fig. 9 is a flow chart illustrating a process 900 for populating a dedicated space within a virtual world with content originating from an application external to the virtual world in some embodiments of the present technology. Process 900 may be initiated when a user is traveling within a virtual world corresponding to a selected XR application, for example, process 900 may be available as part of an interface between a virtual world and a personal interface, in which access to designated spaces in the virtual world results in selection of those spaces in the personal interface, or conversely, in which content selected through coordination in the personal interface may then be provided for access to designated spaces in the virtual world. Process 900 may be executed on an XR device, such as in a runtime of a personal interface. In some cases, process 900 may be executed on a server system controlling such an XR device. In these aspects, for example, process 900 may populate a dedicated movie screen in a virtual world with content of a movie selected by the user from a list of movies provided to the user according to an application stored on the personal interface.

[0108] At block 902, process 900 may receive a selection of a virtual space designated for external content within a current virtual world in which a user is traveling. In this regard, the virtual space may be, for example, a 2D space and / or a 3D space. For example, a space designed for displaying a movie selected by a user may be configured to emit video and / or audio content. Such a virtual space may be associated with a deep link representing an addressing location of the virtual space.

[0109] At block 904, process 900 may receive a selection of a provider of content that will be output for the specified virtual space. For example, the selection may be the result of a user selecting a specific external application of a content provider on a personal interface that provides content that the user desires to output to the specified virtual space. Alternatively, the specified virtual space may be assigned to a specific content provider so that when the user (as at block 902) selects the virtual space, the content provider is pre-selected. In some embodiments, the selection may be a selection of a specific 2D or 3D content item that the user wishes to display in the specified virtual space.

[0110] In some implementations, process 900 can be performed so that the user first selects a content item in box 904 and then selects a virtual space designated for external content. For example, the user can select a video and then select a control that brings up a list of dedicated virtual spaces in the area that support playing external videos. As another example, the user can select a 3D model and then select a control that brings up a list of dedicated volumes in the area that are large enough to accommodate the selected 3D model or are designated to accommodate the type of the selected 3D model.

[0111] At box 906, process 900 can provide access to the selected virtual space to the selected content provider, wherein providing access causes the content provider to fill the content into the selected virtual space. The selection of the content provider can cause the deep link of the dedicated space to be published to the content provider, which allows the content provider to access the dedicated space and display content in the dedicated space. For example, the deep link can be the address or ID of the virtual projection screen, and the streaming video content provider can then use the virtual projection screen to output streaming video. In these aspects, the content provider can fill the selected virtual space with 2D content and / or 3D content, that is, if the space includes 3D capacity, the provider can further fill the space with corresponding 3D content.

[0112] Fig.101000 is a schematic diagram showing an exemplary personal interface 1002. It can be understood from the discussion so far that the personal interface can provide interfaces such as various 2D widget applications or 3D widget applications, which can present 2D interfaces of applications, system controls, user preferences, etc. That is, some of the multiple interfaces in the personal interface can be 2D interface parts from XR applications (e.g., Figure 5 The personal interface may include a 2D interface for controlling applications of the current virtual world (e.g., elements 1006, 1008, 1010, and 1012 - the 2D interface allows the user to interact with the current "Dreamland" virtual world 1004 through a uniform array of exemplary controls. As shown, such controls may include avatar controls 1006 that enable the user to control, for example, the movement of an avatar representing the user or other entity in the virtual world. In addition, such controls may include selections of content items 1008 that the user may desire to include within the virtual world (e.g., avatar clothing, avatar accessories, collectibles, etc.); payment methods / account access 1010 as a means of purchasing goods and / or services; and navigation tools 1012 that enable users to manipulate and record experiences within virtual worlds (e.g., a list of places in the current world that the user may want to travel to, events that have occurred, bookmarked locations, etc.). To enable users to seamlessly travel between multiple virtual worlds in artificial reality, the personal interface can also present access to 2D interfaces of multiple user-selectable XR applications, where the XR applications installed on the XR device and with corresponding virtual worlds are displayed in list 1014.

[0113] Fig.111 is a schematic diagram 1100 showing an exemplary personal interface 1102 that depicts virtual world destinations that can be executed by a user when an artificial reality (XR) application on the personal interface 1102 is selected. For example, in the case where the user selects an XR application corresponding to a virtual world 1104 of "Clubhouse", the personal interface 1102 can present a 2D interface of the XR application, which in this example includes a user list of destinations 1106 that the user can travel to within the world. That is, the destinations 1106 can be associated with deep links that enable the user to travel directly to the corresponding locations. The exemplary destinations can correspond to locations within the world (e.g., places, events, such as Location 1 and Location 2) and people or groups within the world (e.g., Group 1), so that the user can travel to join these people and share the experience of a particular event (e.g., a concert). Additionally, in this example, the 2D interface of the Clubhouse XR application includes a user 1108 of the Clubhouse virtual world, which the user can select to, for example, view the user's current location, corresponding social media presence, and other types of public postings.

[0114] FIG. 12A to FIG. 12C An example of how a personal interface of the XR control and navigation system 164 can present 3D content in artificial reality according to various display options is shown. For example, Fig. 12A 1 is an exemplary schematic diagram 1200 showing a personal interface 1202 providing 3D content 1204 separate from the world in which the user is currently traveling. The 3D content 1204 is located near the personal interface (i.e., in a volume under the control of the personal interface). In particular, the generated 3D content 1204 can be combined with (as described above) Figure 8 The ) corresponds to the world-related of the item to which the 3D content triggering action is directed.

[0115] As another example 1250, Fig. 12B 12 is an exemplary diagram showing a personal interface 1252 that provides a window 1254 for entering a virtual world "Clubhouse" that is different from the world of "Car Show" 1256 that is currently traveling. Fig. 12A Similarly, the generated 3D content may correspond to a virtual world of an item corresponding to the 3D content triggering action, and the personal interface 1252 may present the content in a window 1254, wherein the other virtual world is depicted as viewed through the personal interface 1252. Here, the other virtual world corresponds to Fig.11 A portion of the clubhouse is pictured.

[0116] As another example 1275, Fig. 12Cis an exemplary diagram showing a personal interface 1277 providing a portal into a 3D volume 1280 that is separate from the world (“Clubhouse” 1279) that the user is currently traveling in. In particular, the personal interface 1277 can provide the volume 1280 as a space within the personal interface 1277 through which the user can view, and in some embodiments interact with, 3D content presented in the 3D volume (as shown here with the user's hand 1282 reaching into the volume 1280 to interact with content items 1281) at runtime that is separate from the current virtual world.

[0117] It will be appreciated that embodiments of the present technology may implement one or more of the multiple display options discussed above depending on the specific item to which the 3D content trigger action has been directed. For example, if the user is traveling in the current world and is interacting with a given item in "My Content Items", the controller controlling that item may determine which display option is applicable. In other embodiments, 3D item types may be mapped to specific display options. In still other embodiments, the system may be configured to use only one or two of the multiple display options.

[0118] Fig.13 is an exemplary schematic 1300 showing populating a dedicated space 1304 within a virtual world with content originating from an application external to the virtual world. For example, in response to the user traveling through the virtual world of "Clubhouse" 1302 and selecting a projection space 1304 (projection screen, blank banner, open volume, etc.), the user may use a personal interface 1306 displayed to her to select content 1308 to be projected into the space. That is, the user may select, for example, Video 2, which may correspond to a video from a content provider capable of projecting one or more of 2D content and 3D content according to a deep link for projection space 1304 that describes the corresponding address and content capacity of the projection space. In this way, the user experience of a virtual world such as a dream may be enhanced by including material that may be personally selected by the user and provided by third-party sources.

[0119] References in this specification to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0120] As used herein, being higher than a threshold value means that the value of a comparison item is higher than other values ​​specified, the comparison item is located in a certain specified number of items with a maximum value, or the value of a comparison item is within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of a comparison item is lower than other values ​​specified, the comparison item is located in a certain specified number of items with a minimum value, or the value of a comparison item is within a specified minimum percentage value. As used herein, being located within a threshold value means that the value of a comparison item is between two other values ​​specified, the comparison item is located in a middle specified number of items, or the value of a comparison item is within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" may be understood to mean selecting a connection with a value corresponding to a connection speed assigned thereto that is higher than a threshold value.

[0121] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items, such as: A and A; B, B and C; A, A, B, C and C; etc.

[0122] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except as limited by the appended claims.

Claims

1. A method for navigating multiple virtual worlds in an artificial reality, the method comprising: Provides a personal interface with multiple 2D interfaces to multiple applications, Each application has a 3D world construction part and a 2D interface part, the 3D world construction part controls execution of one of the multiple virtual worlds in a first runtime, and the 2D interface part controls execution of one of the multiple 2D interfaces in a second runtime of the personal interface; and Transition to the subsequent virtual world by following these steps: receiving a selection of an application among the plurality of applications that corresponds to the subsequent virtual world; displaying the selected 2D interface portion of the application via a second runtime of the personal interface, wherein the selected 2D interface portion of the application includes one or more travel cards, each travel card defining a link to a corresponding travel destination of the subsequent virtual world; receiving a selection of a travel destination via the displayed 2D interface portion of the selected application; and In response to the selection, the selected application is caused to generate and display, in the first runtime, a 3D world corresponding to the selected travel destination.

2. The method according to claim 1, in, Receiving a selection of an application corresponding to the subsequent virtual world is performed simultaneously with displaying the current virtual world.

3. The method according to claim 1 or 2, in, The personal interface appears with consistent controls in both the current virtual world and the subsequent virtual world.

4. A method according to any preceding claim, in, The travel destination corresponds to one or more of the following: (a) a place in the subsequent virtual world; (b) an event in the subsequent virtual world; (c) a person in the subsequent virtual world; or (d) any combination of the above items.

5. The method according to any preceding claim, further comprising: receiving a selection of an item in the subsequent virtual world that corresponds to the controller; accessing content from the controller via a deep link associated with the selected item; as well as The content from the controller is presented on the personal interface in the second runtime of the personal interface.

6. The method according to any preceding claim, further comprising: generating, by the personal interface, 3D content in a second runtime other than the first runtime of the personal interface in response to receiving a 3D content triggering action for an item in the subsequent virtual world or on the personal interface; as well as The generated 3D content is displayed by the personal interface according to display options, wherein the display options specify that the 3D content is displayed in one of the following ways: A) displayed at a specified position relative to the personal interface; B) using the personal interface as a window into another 3D world, the other 3D world being the 3D content; or C) using the personal interface as a portal to a volume containing the 3D content.

7. A computing system for navigating multiple virtual worlds in an artificial reality, the computing system comprising: one or more processors; as well as One or more memories storing instructions, which, when executed by the one or more processors, cause the computing system to perform the method according to any one of claims 1 to 6. 8 . A computer-readable storage medium, comprising instructions, which, when executed by a computer, cause the computer to perform the steps according to claim 1 .

Images