Augmentation of world control and enhancement of application control in artificial reality environment

Abstract

The present disclosure provides enhancement of world control and enhancement of application control. The enhancement of world control may be directly controlled through a housing in an artificial reality environment. In order to enable even unexperienced users to develop enhancement of world control, a builder system of world control is provided. The enhancement of application control may be resource-intensive (e.g., using eye movement tracking, social media binding, etc.), may support complex interactions between themselves, may require or enumerate use of input and licensed resources, and may be controlled by its hosting application. When a running application is suspended, the application closes its application control enhancement, but the application can select an enhancement that causes the XR system to run "placeholder" world control as long as the application is not running. Augmentation of placeholder world control retains the appearance of the application in an artificial reality environment, but uses few system resources. Upon user selection of an enhancement of placeholder world control, the XR system may run an associated application and close the enhancement of placeholder world control.

Description

Technical Field

[0001] The present disclosure relates to supporting multiple core types of augmented virtual objects in an artificial reality environment. Background Art

[0002] An "augmentation" in an artificial reality (XR) environment is a virtual object that can include presentation data, context, and logic. XR systems can use augmentations as basic building blocks for displaying two-dimensional (2D) and three-dimensional (3D) models in an artificial reality environment. For example, augmentations can represent people, places, and things in an artificial reality environment, and can respond to context, such as the current display mode, time of day, type of surface the augmentation is on, relationship to other augmentations, etc.

[0003] An XR system provides the richest user experience when it supports both (i) enhancements with a wide range of capabilities and (ii) developers with a wide range of expertise. However, these requirements may conflict because enhancements with a wide range of capabilities require a lot of expertise to develop. Summary of the invention

[0004] According to the present invention, a method for supporting world-controlled enhancements and application-controlled enhancements in an artificial reality environment is provided, the method comprising: running a first active application in the artificial reality environment, wherein the first active application is associated with one or more application-controlled enhancements, wherein each of the one or more application-controlled enhancements is under the control of its associated application at runtime and inherits functions and / or permissions from the first active application, wherein at least one world-controlled enhancement is associated with at least one of the one or more application-controlled enhancements, wherein the at least one world-controlled enhancement is under the control of a shell running in the artificial reality environment at runtime; identifying a trigger for switching from the first active application to a second active application; in response to an indication from the first active application, running, by the shell, the at least one world-controlled enhancement associated with the at least one application-controlled enhancement of the first active application in the artificial reality environment; closing, by the first active application, one or more application-controlled enhancements of its associated currently running application-controlled enhancements and the first active application; and running, by the second application, one or more application-controlled enhancements associated with the second active application in the artificial reality environment.

[0005] Optionally, the trigger for switching is one or more of: selection of a second active application by a user in the artificial reality environment; or a resource shortage in the shell.

[0006] Optionally, running the enhancements to the one or more application controls associated with the second active application includes applying a state to the enhancements to the one or more application controls associated with the second active application, the state saved from a previous execution of the enhancements to the one or more application controls associated with the second active application.

[0007] Optionally, the at least one world-controlled enhancement is created based on an enhancement template customized using selected parameters.

[0008] Optionally, the method further comprises identifying a trigger for activating the first active application.

[0009] Optionally, the trigger for activating the first active application comprises: interaction of a user with the at least one world control enhancement associated with the at least one application control enhancement of the first active application.

[0010] Optionally, the method further comprises running the first active application and the second active application in the artificial reality environment.

[0011] Optionally, the method further comprises closing the second active application, and optionally running the first active application in the artificial reality environment, wherein running the first active application in the artificial reality environment comprises re-running the enhancements of the one or more application controls.

[0012] According to the present invention, a computer-readable storage medium is also provided, which stores instructions, which, when executed by a computing system, cause the computing system to perform a process for supporting world control enhancements and application control enhancements in an artificial reality environment, the process comprising: running a first active application, the first active application being associated with one or more application control enhancements, wherein at least one world control enhancement is associated with at least one of the one or more application control enhancements; identifying a trigger for executing a second active application; in response to the trigger, running one or more world control enhancements of the at least one world control enhancement associated with the at least one application control enhancement of the first active application by a shell of the artificial reality environment, wherein the one or more world control enhancements of the at least one world control enhancement are displayed using the same display characteristics as the associated one or more application control enhancements; closing the first active application and the currently running application control enhancements associated with the first active application; and running the second active application and the one or more application control enhancements associated with the second active application in the artificial reality environment.

[0013] Optionally, running the enhancements to the one or more application controls associated with the second active application includes applying a state to the enhancements to the one or more application controls associated with the second active application, the state saved from a previous execution of the enhancements to the one or more application controls associated with the second active application.

[0014] Optionally, the one or more application-controlled enhancements associated with the first active application inherit functionality and / or permissions from the first active application.

[0015] Optionally, the process further comprises identifying a trigger for activating the first active application.

[0016] Optionally, the trigger for activating the first active application comprises an interaction of a user with an enhancement of one of the at least one world control enhancements associated with the enhancement of at least one application control of the first active application.

[0017] Optionally, each of the one or more application-controlled enhancements is under the control of its associated application at runtime.

[0018] Optionally, the at least one world-controlled augmentation is under control of a shell running in the artificial reality environment at runtime.

[0019] According to the present invention, a computing system for supporting world control enhancement and application control enhancement in an artificial reality environment is also provided, the computing system comprising: one or more processors; and one or more memories storing instructions, which when executed by the one or more processors cause the computing system to execute a process, the process comprising: running a first active application, the first active application being associated with one or more application control enhancements, wherein at least one world control enhancement is associated with at least one of the one or more application control enhancements; identifying a trigger for executing a second active application; in response to the trigger, running one or more world control enhancements of the at least one world control enhancement associated with at least one application control enhancement of the first active application by a shell of the artificial reality environment, wherein the one or more world control enhancements of the at least one world control enhancement are displayed to look the same as the associated one or more application control enhancements; closing the first active application and the currently running application control enhancements associated with the first active application; and running a second active application and one or more application control enhancements associated with the second active application in the artificial reality environment.

[0020] Optionally, running the enhancements to the one or more application controls associated with the second active application includes applying a state to the enhancements to the one or more application controls associated with the second active application, the state saved from a previous execution of the enhancements to the one or more application controls associated with the second active application.

[0021] Optionally, the one or more application-controlled enhancements associated with the first active application inherit functionality and / or permissions from the first active application.

[0022] Optionally, the at least one world control enhancement associated with at least one application control enhancement of the one or more application control enhancements associated with the first application is automatically created based on the at least one application control enhancement of the one or more application control enhancements associated with the first application.

[0023] Optionally, the process further includes identifying a trigger for activating the first active application.

[0024] Optionally, the trigger for activating the first active application comprises an interaction of a user with an enhancement of one of the at least one world control enhancements associated with the enhancement of at least one application control of the first active application.

[0025] Optionally, the process further includes running the first active application and the second active application in the artificial reality environment.

[0026] Optionally, each of the one or more application-controlled enhancements is under the control of its associated application at runtime.

[0027] Optionally, the at least one world-controlled augmentation is under control of a shell operating in the artificial reality environment at runtime.

[0028] Optionally, the at least one world-controlled enhancement is created based on an enhancement template customized using user-selected parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram showing an overview of a number of devices upon which some embodiments of the present technology may operate.

[0030] Figure 2A is a line diagram illustrating a virtual reality head mounted viewer that may be used in some embodiments of the present technology.

[0031] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0032] Figure 2Cis a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold in one or both hands to interact with an artificial reality environment.

[0033] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0034] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0035] Figure 5 is a flow chart illustrating a process for creating world controlled enhancements in some implementations of the present technology.

[0036] Figure 6 is a flow chart illustrating an enhanced process for creating applications and application controls thereof in some embodiments of the present technology.

[0037] Figure 7 is a flow chart illustrating an enhanced process for running one or more applications and world control in an artificial reality environment in some embodiments of the present technology.

[0038] Figure 8 is a flow chart illustrating a process for switching from running one application with enhancements to its controls to running another application in some embodiments of the present technology.

[0039] Fig. 9 is a conceptual diagram illustrating an example of enhancement of application control.

[0040] Fig.10 is a conceptual diagram illustrating an example of an enhancement of world control that preserves the location of applications that are not currently running in an artificial reality environment.

[0041] The various techniques described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to the same elements or functionally similar elements. DETAILED DESCRIPTION

[0042] To avoid limiting enhancement functionality or a set of potential developers, the present disclosure provides a system that supports two core types of enhancements: world-controlled enhancements and application-controlled enhancements. In an artificial reality environment, world-controlled enhancements can be controlled directly by the XR shell, while application-controlled enhancements can be controlled by their hosting application. When a running application is suspended, the application turns off its application-controlled enhancements, but can choose to have the XR system run a "placeholder" world-controlled enhancement. The placeholder world-controlled enhancement retains the appearance of the application in the artificial reality environment, but uses very few system resources. Once the user selects a placeholder world-controlled enhancement, the XR system can run the associated application and turn off the placeholder world-controlled enhancement.

[0043] An XR shell (or simply "shell") can be an artificial reality device operating system or another application that controls an artificial reality environment. The shell can create content in the artificial reality environment. The shell can define rules for how applications and virtual objects (i.e., augmentations) access hardware and artificial reality environment resources, what applications and virtual objects must do to create new virtual objects, how applications and virtual objects share information; and the shell can coordinate the operation of the artificial reality environment. World-controlled augmentations do not require the installation of applications, but are instead run directly by the XR shell. The XR shell directly controls resource usage through multiple world-controlled augmentations, so the XR shell can choose to run many of the multiple world-controlled augmentations at the same time. World-controlled augmentations will persist as long as the XR shell keeps them running, or until the user turns them off.

[0044] Augmentations can be developed from within the XR system or on a traditional system (e.g., a laptop or other tablet system). To allow even inexperienced users to develop world-controlled augmentations, a world-controlled builder system is provided. Users invoke the world-controlled augmentation builder system to quickly develop world-controlled augmentations from provided templates. Users fill in values ​​into the fields of the templates under the guidance of the world-controlled augmentation builder system. These templates limit the scope of functionality of the world-controlled augmentations (e.g., do not include augmentations that require hand tracking, cannot access licensed resources, etc.), while also ensuring consistency and allowing on-the-fly creation in artificial reality environments.

[0045] In contrast to world-controlled enhancements, app-controlled enhancements are limited only by the expertise level of their developers and the resources available on their host systems. App-controlled enhancements can be resource-intensive (e.g., using eye tracking, social media tie-ins, hand tracking, artificial intelligence (AI) support, etc.), can support complex interactions between themselves, and can make heavy use of input, licensed resources. When the app controlling the app-controlled enhancements is suspended, these app-controlled enhancements disappear. To prevent overuse of limited resources, the XR shell can only allow one or a few active apps and their corresponding app-controlled enhancements at a time.

[0046] As an example of place-holding, a checkers game application may present a fully interactive application-controlled augmentation that may present the current state of the game, allow the user to make their next move using hand tracking, and respond using moves received through a communication channel with a remote player, etc. If the user chooses to activate another application or leaves the vicinity of the checkers game application-controlled augmentation for at least a threshold period of time, the checkers game application will be closed along with its application-controlled checkers augmentation. The augmentation is replaced by a placeholder world-controlled augmentation that is run directly by the XR shell, e.g., the placeholder world-controlled augmentation may only display a checkers board. If the user interacts with the placeholder augmentation, the XR shell may replace the placeholder augmentation by running the checkers game application again, which in turn runs its interactive checkers game application-controlled augmentation.

[0047] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0048] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system (e.g., by adding virtual objects). "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially comes from light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a transparent display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.

[0049] Existing XR systems do not support the use of many complex virtual objects at the same time. In addition, only the most experienced developers can create fully functional virtual objects with features such as communication with third parties, use of hand tracking and eye tracking, AI assistant functions, etc. The system and method of the world-controlled enhancement builder disclosed herein are expected to overcome these shortcomings in existing systems. The present disclosure provides a template-based enhancement builder through which even the most inexperienced users can easily create world-controlled enhancements. Application-controlled enhancements with a wider range of comprehensive functions are also supported and can be created by more knowledgeable developers. These two types of enhancements can interact in a beneficial way. As long as the application is not running, the placeholder enhancement of the world control can retain the appearance of the application in the artificial reality environment, even while consuming very little system resources. Once the user selects the placeholder world-controlled enhancement, the XR system can run the associated application and turn off the placeholder world-controlled enhancement. By allowing these two types of enhancements to work together, the present technology avoids restrictions on enhancement functionality or a group of potential developers. By allowing enhanced control of placeholder worlds to preserve the appearance of an application, the present technique allows for rich artificial reality environments even in the face of limited resources, and thereby increases the value of the system running the artificial reality environment.

[0050] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1is a block diagram showing an overview of multiple devices on which some embodiments of the disclosed technology may run. These devices may include the following hardware components of the computing system 100: These hardware components can control virtual objects. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103), which communicate through a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a stand-alone head-mounted view device that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted view device and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted view device and other processing operations are transferred to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted view device is described. In some implementations, location data and environmental data may be collected solely by sensors incorporated into the head mounted view device, while in other implementations, one or more of a plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.

[0051] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0052] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets signals received from the input devices and transmits information to the processors 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0053] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, or a wireless connection, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with a hardware controller for each device (e.g., a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices are: liquid crystal display (LCD) display screen, light emitting diode (LED) display screen, projection display, holographic display, or augmented reality display (e.g., head-up display device or head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0054] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, GPS units, LiDAR or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment and track the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0055] The computing system 100 may include such a communication device: the communication device can communicate with other local computing devices or network nodes wirelessly or based on wired communication. The communication device can communicate with another device or server through a network, for example, using the Transmission Control Protocol / Internet Protocol (TCP / IP protocol). The computing system 100 can use the communication device to distribute operations on multiple network devices.

[0056] Processor 110 can access memory 150, which can be included in one of the multiple computing devices of computing system 100, or can be distributed on multiple computing devices of computing system 100, or multiple other external devices. Memory includes one or more hardware devices for volatile storage or non-volatile storage, and can include both read-only memory and writable memory. For example, memory can include one or more of the following: random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, and tape drive. Memory is not a propagation signal separated from the underlying hardware; therefore, memory is non-transient. Memory 150 can include program memory 160, which stores programs and software, such as operating system 162, enhanced builder 164 for world control, and other applications 166. Memory 150 may also include data storage 170 which may include, for example, a library of world control enhancements, templates for creating world control enhancements, configuration data, settings, user options or preferences, etc. that may be provided to program storage 160 or any element of computing system 100 .

[0057] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices.

[0058] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a band 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. Position sensor 220, IMU 215, and computing unit 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensor 220, and locator 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locator 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shape and position of real objects around the HMD 200.

[0059] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), some other display, or some combination thereof.

[0060] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), which the PC can use in conjunction with outputs from the IMU 215 and position sensor 220 to determine the position and movement of the HMD 200.

[0061] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality HMD system 250 includes only a head-mounted view device without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0062] The projector can be coupled to a transparent display 258, for example, via optical elements to display media to the user. These optical elements can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc., for guiding the light from the projector to the user's eyes. Image data can be transmitted to the HMD 252 from the core processing component 254 via a link 256. The controller in the HMD 252 can convert the image data into a plurality of light pulses from the projector, which can be transmitted to the user's eyes as output light via optical elements. The output light can be mixed with the light passing through the display 258, thereby allowing the output light to present the following virtual objects: these virtual objects look as if they exist in the real world.

[0063] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0064] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held by a user in one or both hands to interact with an artificial reality environment presented by HMD 200 and / or HMD 250. Controllers 270 can communicate with the HMD directly or via an external device (e.g., core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit light points farther away. The HMD 200 or HMD 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., tracking the controllers with 3DoF or 6DoF). The computing unit 230 or core processing component 254 in the HMD 200 can combine the IMU output and the position output and use this tracking to monitor the position and movement of the user's hands. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A-274B) that a user may actuate to provide input and interact with objects.

[0065] In various embodiments, HMD 200 or HMD 250 may also include additional subsystems (e.g., an eye tracking unit, an audio system, various network components, etc.) that monitor indications of user interaction and intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or HMD 250 or from a plurality of external cameras may monitor the position and posture of a user's hands to determine posture and other hand and body movements. As another example, one or more light sources may illuminate one or both eyes of a user, and HMD 200 or HMD 250 may use an eye-facing camera to capture reflections of that light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes, and determine gaze direction.

[0066] Figure 3 3 is a block diagram showing an overview of an environment 300 in which some embodiments of the disclosed technology may operate. The environment 300 may include one or more client computing devices 305A to 305D, examples of which may include the computing system 100. In some embodiments, some of the plurality of client computing devices (e.g., client computing device 305B) may be HMD 200 or HMD system 250. The plurality of client computing devices 305 may operate in a network environment using logical connections to one or more remote computers (e.g., server computing devices) through a network 330.

[0067] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, each of these server computing devices may be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0068] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, which can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0069] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.

[0070] Figure 44 is a block diagram showing a plurality of components 400 that can be used in a system that employs the disclosed technology in some embodiments. Each component 400 can be included in one device of the computing system 100, or can be distributed on multiple devices of the computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, a system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., in a storage device 315 or 325) or other network storage device accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0071] Middleware 420 may include components that communicate resources between hardware 410 and a plurality of specialized components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

[0072] The specialized components 430 may include software or hardware configured to perform operations to support world-controlled enhancements and application-controlled enhancements. The multiple specialized components 430 may include: an XR shell 434; a world-controlled enhancement builder 436; a library of world-controlled enhancements 438; one or more applications 440 and 442 with their associated application-controlled enhancements; and components and APIs (e.g., interface 432) that can be used to provide user interfaces, transfer data, and control each specialized component. In some embodiments, each component 400 may be in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the specialized components 430. Although each specialized component 430 is depicted as a separate component, these specialized components may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0073] The XR shell 434 may be an application that presents an artificial reality environment to a user and may coordinate the user's interaction with the artificial reality environment. The XR shell 434 may coordinate the work of XR subcomponents such as applications 440 and 442 and may control the interactions between the various enhancements. The XR shell 434 may be responsible for running the world control enhancements.

[0074] A user may invoke a world-controlled enhancement builder 436 to build their enhancements. The world-controlled enhancement builder 436 may present templates to the user. The user, under the guidance of the world-controlled enhancement builder 436, may select a template and fill in values ​​into the fields of the selected template. The world-controlled enhancement builder may, through the use of templates, limit the functional scope of the world-controlled enhancements and may ensure consistency between world-controlled enhancements built by different users. In some variations, the world-controlled enhancement builder 436 may allow a user to create enhancements "on the fly" while in an artificial reality environment. In other cases, the world-controlled enhancement builder 436 may be used by an alternative system such as a conventional computer, thereby allowing a user to create enhancements for future use in an artificial reality environment.

[0075] Because the XR shell 434 can directly run all world-controlled enhancements, the XR shell can access these world-controlled enhancements. The library of world-controlled enhancements 438 can be a repository of these world-controlled enhancements that the XR shell 434 can run. In some cases, world enhancements can exist in the cloud and be served to endpoint devices based on their context (e.g., geographic location, user profile / preferences, system or user state, recent actions, etc.).

[0076] Applications 440 and 442 represent any number of applications that may be running in the user's artificial reality environment at some time or another. Each application may be associated with one or more application-controlled enhancements.

[0077] It will be understood by those skilled in the art that the above Figures 1 to 4 The multiple components shown in each of the multiple flow charts discussed above and below can be changed in various ways. For example, the order of the logic can be rearranged, multiple sub-steps can be performed in parallel, the logic shown can be omitted, other logic can be included, etc. In some embodiments, one or more of the multiple components described above can perform one or more of the multiple processes described below.

[0078] The difference between world-controlled enhanced functionality and application-controlled enhanced functionality is reflected in their different creation processes. Figure 5 and Figure 6 These creation processes are depicted, and Figure 7 and Figure 8 Depicts the different modes of operation of these enhancements and how these enhancements can interact.

[0079] Figure 5 is a flow chart illustrating a process 500 for creating a world-controlled augmentation in some embodiments. In some variations, process 500 is invoked by a user while the user is outside of an artificial reality environment (e.g., process 500 may be run on a user's personal computer). In some variations, process 500 begins when a user in an artificial reality environment interacts with a world-controlled augmentation builder associated with creating a world-controlled augmentation. The user's interaction may include looking at, pointing at, or touching a representation of the world-controlled augmentation builder (e.g., from an application launcher). In some variations, process 500 may begin when a user verbally expresses a command to create a world-controlled augmentation, and the expression is mapped to a semantic space (e.g., by applying a natural language processing model) to determine the user's intent from the words of the command.

[0080] At block 502, process 500 receives a user's selection of a template suitable for the enhancement of a world control that the user wishes to create. In some variations, the user's selection is from a menu of templates presented to the user by the world control enhancement builder 436. These templates may present the user with some fields already populated with default values ​​(e.g., values, permissions, actions, etc.). In some variations, the "template" is an already created enhancement of a world control, and the user's new world control enhancement may copy or inherit features of the already created enhancement of the world control.

[0081] At block 504 , process 500 presents the selected template to the user, wherein in some variations, prompts are presented regarding how to populate the fields of the template.

[0082] At block 506, process 500 interacts with the user to populate a template for an enhancement of a new world control. This may include populating fields for values, permissions, actions, etc. Some value fields may be metadata, such as the name of the enhancement of the world control, the author or owner of the enhancement, and a unique identifier for the enhancement of the world control. Other value fields may include visual elements of the enhancement of the world control, including images and 3D models, parameters, and links to external data sources (e.g., time of day, news feed).

[0083] With the help of templates, users can define actions that allow this type of world-controlled enhancements, and can define permissions associated with these actions. Actions typically define how the world-controlled enhancements interact with the user and the artificial reality environment. Action types can include object behaviors, spatial behaviors, data management interactions, notification actions, privacy actions, system actions, etc. Object behaviors can include how the world-controlled enhancements react to other objects, inputs, and events in the artificial reality environment, and can specify that the world-controlled enhancements respond to the triggering of display, configuration, or other output actions. Spatial behaviors can include how the world-controlled enhancements move, react to surface layouts, set view states in different contexts, etc. Other actions may define how the world-controlled enhancement responds to changing circumstances, including: data management interactions (e.g., what data the world-controlled enhancement retrieves or updates in response to a given trigger), notifications (e.g., how the world-controlled enhancement presents detected events in the artificial reality environment), privacy and security actions (e.g., what data and interfaces the world-controlled enhancement provides to users, other virtual objects, and other entities, and how the world-controlled enhancement protects its internal resources), and / or system actions (e.g., changing the state of power management, managing and requesting resources, hardware access, interacting with the operating system (OS) or other control application interface, thread execution management, etc.).

[0084] In some variations, a user may define how world control enhancements respond to user input, in some cases defining touch, voice, and other interactive user input via a user interface.

[0085] A user may specify permissions for a new world-controlled enhancement, including, for example, the volume in which the world-controlled enhancement is authorized to write to within the user's artificial reality environment. In some variations, world-controlled enhancements may be defined using multiple view states and contextual breakpoint rules for transitioning between the multiple view states. Each view state may control what and how the world-controlled enhancement presents its data, as well as what virtual objects may use as anchors when in that state. Other permissions may be associated with the other action types listed above and their associated data.

[0086] Although the range of data, data types, and data sources that may be associated with a world-controlled augmentation is virtually unlimited, specific templates may require specific fields. For example, a world-controlled augmentation representing a character in an artificial reality environment may require a name, current state, and avatar.

[0087] As described above, a template can include default values ​​for some or all of a plurality of parameters and values ​​for an object. In some cases, a "template" is an enhancement of an existing world control from which the enhancement being created copies or inherits values, actions, parameters, etc. In general, a user can add to, remove values, actions, parameters from, or replace them with user-specified substitutes for the inherited component. In some cases, a "parent" enhancement can specify which of these inheritances can be overwritten by the user's selection. In the same manner, a user can be granted permission to change certain defaults included in a template.

[0088] As described above, the template may limit the functionality of the enhancement added to the world control at block 506. For example, in some variations, the world control enhancement is prohibited from accessing any licensed resources (e.g., camera, messaging service) and is prohibited from using very much computing power.

[0089] Process 500 throughout block 506 may guide the user in filling out a template, thereby simplifying the creation of a new world-control enhancement, while the template also constrains what actions the enhancement may perform.

[0090] At block 508, process 500 may add the newly created world-controlled enhancement to library 438, from which it may be later extracted as needed. In some variations, the newly created world-controlled enhancement may be associated with an application to serve as a placeholder enhancement when the application is not running. Figure 8 Step 806 and Fig. 9 and Fig.10 This scenario is shown. In some variations, the newly created world-controlled enhancement is triggered to automatically display in the user's artificial reality environment in certain contexts. In some cases, the library is supplemented on a remote system to provide the world-controlled enhancement to users other than the user who created the world-controlled enhancement.

[0091] Combine the above description of enhanced template creation for world control with Figure 6 Similar to process 500 described above, in some variations, when a user outside of an artificial reality environment begins to create a new application that will control its own augmentation, Figure 6 In one embodiment, the process 600 of the user's computer can be started (e.g., the process 600 is run on a user's personal computer or a remote server). In this case, the developer uses the process 600 to create an application and its enhancements, and then imports the application into the artificial reality environment. In other cases, the developer can use the application creation system from within the artificial reality environment to create an application and its corresponding application-controlled enhancements.

[0092] At block 602, process 600 begins by creating a new application, for example in an artificial reality environment or a traditional computer application development system. Figure 5 As with the appendix, the creation process 600 of an application with its enhancements may use templates, but the process may also include defining and controlling enhancements with a greater range of properties and capabilities. Application developers have greater flexibility in how to create applications and what functions and permissions to assign to new applications (see in particular the description of blocks 606 and 608 below).

[0093] At block 604, in some variations, the process 600 defines at least one application-controlled enhancement to be controlled by the application being developed. As in developing the application itself, the developer has a great deal of freedom in how to develop the application-controlled enhancement. In some variations, the application developer may choose to begin the creation process using a template from the world-controlled enhancement builder 436, or may choose to begin using an existing enhancement as a starting point.

[0094] Combined with the above Figure 5 The functionality and features of any world control enhancements discussed in block 506 of may also be applied to application control enhancements. For example, while an application control enhancement may have any number and type of data elements, an example application control enhancement may include: metadata (e.g., the name of the application control enhancement, the author or owner of the enhancement, and a unique identifier for the application control enhancement), visual elements including images and 3D models, parameters, and links to external data sources (e.g., messaging applications, gesture trackers). Developers may define the actions allowed for application control enhancements and may define permissions associated with those actions. Specific actions may include object behaviors, spatial behaviors, data management interactions, notification actions, privacy actions, system actions, and the like. In some variations, developers may define how application control enhancements respond to user input, such as user input through a user interface defining gestures, touch, voice, and other interactions.

[0095] When developing application-controlled enhancements, developers are free to override the restrictions imposed by the XR system on the functionality of world-controlled enhancements. The details of these restrictions may vary from system to system, but may include a refusal to permit world-controlled enhancements to interact richly with other enhancements or with the user, and a refusal to use permitted resources. Therefore, in box 606, developers can provide advanced features with more functionality to application-controlled enhancements. These features may include, for example, user gaze tracking, hand tracking, complex interaction patterns with the user and between multiple enhancements controlled by the application, access to messaging or other network resources, access to AI assistants or other machine learning driven systems, and generally any functionality provided by the XR system that will host the application. This is one aspect of distinguishing an application and its application-controlled enhancements from world-controlled enhancements. In order to make world-controlled enhancements easy to create and reduce the requirements for computing resources, these world-controlled enhancements are not allowed to access these advanced features and / or are restricted in what the world-controlled enhancements can do using various resources and inputs.

[0096] In box 608, process 600 sets permissions for the application and sets permissions for the application-controlled enhancements by inheriting from the application. In order to implement many of the multiple advanced features added in box 606, the application must ensure permission to access certain system resources such as cameras, microphones, and profile information about the interacting user. Once these permissions are granted to the application, the application can in turn extend these same permissions or a strict subset of these same permissions to its application-controlled enhancements. Using permission inheritance, just like using field definition inheritance, allows application-controlled enhancements to present a unified experience to the user. This is one aspect that distinguishes an application and its application-controlled enhancements from world-controlled enhancements. The permissions of an application may have broader permissions than world-controlled enhancements.

[0097] At block 610, process 600 associates the newly created application control enhancement with the application being developed. At block 612, process 600 associates at least one world control enhancement with at least one application control enhancement of the plurality of application control enhancements of the new application. The creation of such a world control enhancement may be as described above in conjunction with Figure 5 In some embodiments, a world-controlled enhancement can be automatically created based on a newly created application-controlled enhancement, taking features such as the 3D model and display properties from the newly created application-controlled enhancement, but excluding advanced features that are only available to application-controlled enhancements. When the application being developed has called an application-controlled enhancement, but has subsequently been closed or replaced by another active application, the world-controlled enhancement can be used to represent the application-controlled enhancement (so it is no longer able to present its application-controlled enhancement). Figure 8The process and Fig. 9 and Fig.10 The conceptual scenario is used to describe additional aspects of the work on the association between enhancements to application control and enhancements to world control.

[0098] At block 614, process 600 specifies the conditions under which the application can run. For examples of this, see Figure 7 and Figure 8 The process and Fig. 9 and Fig.10 Concept scene.

[0099] Figure 7 is a flow chart illustrating an enhanced process 700 for running one or more applications and world control in an artificial reality environment in some embodiments. Figure 5 The process 500 is used to create an enhancement of world control and can be achieved by Figure 6 The process 700 may be executed as part of an XR system and may begin when a user first enters an artificial reality environment (e.g., by wearing HMD 200 or executing a particular environment with HMD 200).

[0100] At block 702, process 700 runs one or more world control enhancements associated with the user's current location. For example, process 700 may identify anchor points in the current environment and select world control enhancements that match those anchor points, and / or may identify various contextual factors (e.g., who the user is, what other objects are in the area, the time, date, other people around, etc.) and perform an analysis of the available world control enhancements to rank the world control enhancements according to which best match the contextual factors. Process 700 may use a local and / or cloud-based version of library 438 to perform these checks on world control enhancements, and may display the selected or highest ranked world control enhancements. The set of displayed world control enhancements may change as the user changes context, location, and / or orientation.

[0101] Process 700 may perform enhancements of one or more applications (i.e., one or more currently active applications) and their associated application controls. The application selected to run may be a system default, may depend on the user's circumstances when entering the artificial reality environment, or may be based on user selection. For example, a user may open a launcher listing available applications, and may select an application as the currently active application. As another example, a user may select an enhancement of world controls that may be associated with an application in the environment, and cause the application to execute as the currently active application (as discussed in the next box).

[0102] At block 704, process 700 receives a user selection of an enhancement for a world control. In some variations, the user interacts with an enhancement for a world control associated with an application, and the enhancement for a world control acts as a placeholder when the application is not running. (In some cases, not all enhancements for world controls are placeholders for enhancements for application controls of a stopped application.) The user's interaction may include, for example, looking at, pointing at, or touching the enhancement for a world control, or indicating the enhancement for a world control with a voice command. In some variations, the user verbally expresses a command selecting an application or an enhancement for its associated world control, and the expression is mapped to a semantic space (e.g., by applying an NLP model) to determine the user's intent from the words of the command.

[0103] At block 706, process 700 may close the active application that is executing (e.g., executing at block 702 or previously executing) and may begin running the application selected by the user at block 704. The newly selected application may begin running one or more of its application-controlled enhancements. For more information on what may happen when one application replaces another application, see Figure 8 and attachment.

[0104] Figure 8 800 is a flow chart illustrating a process 800 for switching from running one application with its controls augmented to running another application in some embodiments. Process 800 runs in the XR system and can begin immediately when a user first enters an artificial reality environment (e.g., by wearing HMD 200). Process 800 can be a more detailed implementation of process 700, or can be Figure 7 A sub-process of process 700.

[0105] At block 802, process 800 runs a first application with its application-controlled enhancements. Unlike the case of world-controlled enhancements that are run directly by the XR shell, application-controlled enhancements are controlled by their hosting application. The application can decide under what circumstances each of the multiple enhancements it controls will run, and under what circumstances each enhancement stops running. When the application itself stops running, all of its application-controlled enhancements also stop running, but can be replaced by world-controlled enhancements as described below.

[0106] In block 804, the user selects a second application to run. There are many ways to select. For example, Figure 7Boxes 704 and 706 of FIG. 10 show when a user selects an augmentation of a world control that is a placeholder associated with an application that is not running. The user may also select an application to run through a launcher selection, using text or gestures, or the XR system may receive another trigger from the user or another source (e.g., a network signal indicating an incoming message or notification) to launch a different active application.

[0107] At block 806, process 800 triggers the closing of the already running active application. This occurs because each application may consume a large amount of resources, and the XR system can only run one or a limited number of applications at a time. In this scenario, the XR system cannot start running a second application selected by the user while running the first application. Closing the active application may include saving the state of the active application (including one or more of the plurality of application-controlled enhancements). This state allows the application to rerun itself and its application-controlled applications from the same point the next time the application is executed.

[0108] Note that while in the scenario of block 806, it is the user's selection of the second application that causes the first application to close, in other scenarios, the first application is closed for other reasons. For example, the system may be running out of resources and may close the first application as an "emergency" measure. In another example, the user selects to close the first application, but does not yet want to run another application. In all of these cases, the process of block 806 can be followed.

[0109] When the first active application stops running, all of its application-controlled enhancements stop running. In the scenario of block 806, the first active application prepares for this by causing one or more of its multiple application-controlled enhancements to be replaced with placeholder world-controlled enhancements. The placeholder world-controlled enhancements are run by the XR shell in response to a request from the first active application, or are passed to the XR shell. The placeholder world-controlled enhancements from the first application are then run directly by the XR shell, allowing the placeholder world-controlled enhancements to persist after the active application is closed. The first active application is then paused, and all of its application-controlled enhancements are turned off.

[0110] The selected second application is run at block 808. As with the first application while it is running, during the lifecycle of the second application, the second application may run application-controlled enhancements, which may include providing application-controlled enhancements to replace any world-controlled enhancements (e.g., enhancements selected to be executed by the new active application) that were placeholders for application-controlled enhancements for the application.

[0111] If the user chooses to switch back to the first application, that choice may be implemented by selecting enhancement of the placeholder world control initiated in block 806 (effectively re-executing process 800 ).

[0112] In block 810, if the user selects a second application through an enhancement of a placeholder world control that was specifically designed for that purpose, or there are other placeholder world control enhancements for the active application that have been replaced by the enhancement of the application control, then the placeholder world control enhancements have served their purpose and process 800 may stop running those placeholder world control enhancements.

[0113] Fig. 9 and Fig.10 is to illustrate some of the concepts described above (especially the Figure 8 806) of the conceptual diagrams 900 and 1000. Fig. 9 , a user 902 plays checkers by interacting with an application-controlled enhancement 904. The board application-controlled enhancement 904 has a main application that updates the board as the game progresses and handles communications with the user's 902 opponent, or the board application-controlled enhancement 904 is the opponent itself. The application also uses system resources such as a tracking camera to determine how to create a control ray 906 from the user's 902 hand to the application-controlled enhancement 904.

[0114] Finally, the user 902 leaves the game and selects another active application (causing the corresponding application to be executed) by selecting the corresponding world control augmentation 1002. The main application of the checkers game saves its state (e.g., the game state) and closes, rather than continuing to consume valuable system resources unnecessarily. As part of the main application closing process, the main application presents Fig.10 The XR shell runs the placeholder world-controlled enhancement 1004 in place of the application-controlled enhancement 904. The main application of the checkers game then stops running, and the checkers application-controlled enhancement 904 also stops running. However, this is not obvious to the user because the placeholder world-controlled enhancement 1004 is still there. However, the placeholder world-controlled enhancement 1004 cannot play checkers, but its visual presence indicates that playing checkers is possible. If the user interacts with the placeholder world-controlled enhancement 1004 by, say, touching it, the XR shell stops running the placeholder world-controlled enhancement 1004 and restarts the main application, which then starts the application-controlled enhancement 904, using the main application's previously saved state to continue the game from the same point.

[0115] References in this specification to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that the particular features, structures, or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present disclosure. These expressions appearing in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be rendered by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0116] As used herein, being higher than a threshold value means that the value of a comparison item is higher than other values ​​specified, the comparison item is located in a certain specified number of items with a maximum value, or the comparison item has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of a comparison item is lower than other values ​​specified, the comparison item is located in a certain specified number of items with a minimum value, or the comparison item has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of a comparison item is between two other values ​​specified, the comparison item is located in a middle specified number of items, or the comparison item has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" may be understood to mean selecting a connection with a value corresponding to a connection speed assigned thereto that is higher than a threshold value.

[0117] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of the following: A, B, C or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items (e.g., A and A; B, B and C; A, A, B, C and C, etc.).

[0118] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The above-mentioned specific features and actions are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except in the appended claims.

[0119] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for supporting augmented world control and augmented application control in an artificial reality environment, the method comprising: running a first active application in the artificial reality environment, wherein the first active application is associated with one or more application-controlled enhancements, wherein each of the one or more application-controlled enhancements is under control of its associated application at runtime and inherits functionality and / or permissions from the first active application, and wherein at least one world-controlled enhancement is associated with at least one of the one or more application-controlled enhancements, wherein the at least one world-controlled enhancement is under control of a shell running in the artificial reality environment at runtime; identifying a trigger for switching from the first active application to a second active application; in response to an instruction from the first active application, executing, by the shell, in the artificial reality environment, the at least one world-controlled augmentation associated with at least one application-controlled augmentation of the first active application; Closing, by the first active application, one or more of its associated currently running application-controlled enhancements and the first active application; and One or more application-controlled augmentations associated with the second active application are executed by the second active application in the artificial reality environment.

2. The method according to claim 1, wherein: The trigger for switching is one or more of the following: Selection of the second active application by a user in the artificial reality environment; or Resources are scarce in the shell.

3. The method according to claim 1, wherein: Running the enhancements to the one or more application controls associated with the second active application includes: applying a state to the enhancements to the one or more application controls associated with the second active application, the state saved from a previous execution of the enhancements to the one or more application controls associated with the second active application; and / or Wherein the at least one world-controlled enhancement is created based on an enhancement template customized using selected parameters.

4. The method according to claim 1, further comprising: identifying a trigger for activating the first active application; Wherein, the trigger for activating the first active application comprises: an interaction of a user with the at least one world control enhancement associated with the at least one application control enhancement of the first active application.

5. The method according to claim 4, further comprising: running both the first active application and the second active application in the artificial reality environment; and / or The method further comprises: closing the second active application; as well as Running the first active application in the artificial reality environment, running the first active application in the artificial reality environment includes re-running enhancements of the one or more application controls.

6. A computer-readable storage medium storing instructions which, when executed by a computing system, cause the computing system to perform a process for supporting augmentation of world control and augmentation of application control in an artificial reality environment, the process comprising: running a first active application, the first active application being associated with one or more application-controlled enhancements, wherein at least one world-controlled enhancement is associated with at least one application-controlled enhancement of the one or more application-controlled enhancements; identifying a trigger for executing a second active application; In response to the trigger, executing, by a shell of the artificial reality environment, one or more of the at least one world-controlled augmentations associated with at least one application-controlled augmentation of the first active application, wherein the one or more of the at least one world-controlled augmentations are displayed using the same display characteristics as the associated one or more application-controlled augmentations; Closing the first active application and enhancements to currently running application controls associated with the first active application; and The second active application and one or more application control enhancements associated with the second active application are executed in the artificial reality environment.

7. The computer-readable storage medium according to claim 6, wherein: executing the enhancement of the one or more application controls associated with the second active application includes applying a state to the enhancement of the one or more application controls associated with the second active application, the state being saved from a previous execution of the enhancement of the one or more application controls associated with the second active application; and / or Wherein, the one or more application-controlled enhancements associated with the first active application inherit functionality and / or permissions from the first active application.

8. The computer-readable storage medium according to claim 6, wherein: The process also includes: identifying a trigger for activating the first active application; Wherein, the trigger for activating the first active application comprises: an interaction of a user with an enhancement of one of the at least one world control enhancements associated with the enhancement of at least one application control of the first active application.

9. The computer-readable storage medium according to claim 6, in, Each of the one or more application-controlled enhancements is under control of its associated application at runtime; and Wherein the at least one world-controlled enhancement is under control of the shell operating in the artificial reality environment at runtime.

10. A computing system for supporting augmented world control and augmented application control in an artificial reality environment, the computing system comprising: one or more processors; as well as one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: running a first active application, the first active application being associated with one or more application-controlled enhancements, wherein at least one world-controlled enhancement is associated with at least one application-controlled enhancement of the one or more application-controlled enhancements; identifying a trigger for executing a second active application; In response to the trigger, running, by a shell of the artificial reality environment, one or more of the at least one world-control augmentations associated with the at least one application-controlled augmentation of the first active application, wherein the one or more of the at least one world-control augmentations are displayed to appear identical to the associated one or more application-controlled augmentations; Closing the first active application and an enhancement of currently running application controls associated with the first active application; and The second active application and one or more application control enhancements associated with the second active application are executed in the artificial reality environment.

11. The computing system of claim 10, wherein: Running the enhancements to the one or more application controls associated with the second active application includes: applying a state to the enhancements to the one or more application controls associated with the second active application, the state saved from a previous execution of the enhancements to the one or more application controls associated with the second active application; and / or Wherein, the one or more application-controlled enhancements associated with the first active application inherit functionality and / or permissions from the first active application.

12. The computing system of claim 10, wherein: The at least one world control enhancement associated with at least one of the one or more application control enhancements associated with the first application is automatically created based on the at least one application control enhancement of the one or more application control enhancements associated with the first application.

13. The computing system of claim 10, wherein: The process also includes: identifying a trigger for activating the first active application; Wherein, the trigger for activating the first active application comprises: an interaction of a user with an enhancement of one of the at least one world control enhancements associated with the enhancement of at least one application control of the first active application.

14. The computing system of claim 13, wherein: The process also includes: The first active application and the second active application are run in the artificial reality environment.

15. The computing system according to claim 10, in, Each of the one or more application-controlled enhancements is under control of its associated application at runtime; and wherein the at least one world-controlled augmentation is, when operating, under control of the shell operating in the artificial reality environment; and / or Wherein, the at least one world-controlled enhancement is created based on an enhancement template customized using user-selected parameters.

Images