Dynamic boundaries for artificial reality systems

Abstract

Dynamic boundaries for artificial reality systems are disclosed. Aspects of the present disclosure relate to triggering dynamic boundaries of operational changes at an artificial reality system. Embodiments of a dynamic boundary manager may execute dynamic boundary criteria. For example, when an artificial reality system is operated under specific operating conditions, a dynamic boundary corresponding to a user can be automatically generated. User movement may trigger the dynamic boundary criteria and cause one or more of the following: extension of the dynamic boundary; popping up a dynamic boundary; and / or reformation of dynamic boundaries. The dynamic boundary manager may also cause an operation change at the artificial reality system in response to these triggers, such as suspending an application being executed and / or changing an artificial reality environment displayed to the user (e.g., to and from virtual reality environment, perspective vision, etc.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 18 / 631,179, filed on April 10, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to dynamic boundaries that trigger operational changes at artificial reality systems. Background Art

[0004] Artificial reality devices are becoming increasingly common. As they become more popular, the applications implemented on such devices are becoming increasingly complex. Augmented reality applications or mixed reality applications can provide interactive three-dimensional experiences that combine real-world environments with virtual objects. Virtual reality applications can provide independent three-dimensional computer environments. Users can observe augmented reality experiences, mixed reality experiences, and / or virtual reality experiences through a head-mounted display (e.g., glasses or a headset) of an artificial reality system. Some artificial reality experiences may hinder the user's perception of the real world, so some technologies can be implemented to protect and / or improve the user experience. Summary of the Invention

[0005] Aspects of the present disclosure relate to dynamic boundaries that trigger operational changes in artificial reality systems. Artificial reality systems provide immersive visual displays to users, such as through head-mounted displays. Sometimes, immersive displays can obstruct the user's perception of the real-world environment. Some artificial reality systems implement boundaries that enforce safety conditions when operating these systems. For example, the operation of the artificial reality system (e.g., the artificial reality environment displayed to the user) can be changed in response to user movement approaching and / or crossing a boundary. Implementations of a dynamic boundary manager enforce dynamic boundaries for the user. The dynamic boundary manager can, for example, automatically generate dynamic boundaries corresponding to the user when the artificial reality system operates under specific operating conditions. The dynamic boundary manager can enforce dynamic boundary criteria. For example, user movement can trigger the expansion, pop-up, and / or reformatting of a dynamic boundary. The dynamic boundary manager can also cause changes to the operation of the artificial reality system in response to these triggers, such as pausing an executing application and / or changing the artificial reality environment displayed to the user (e.g., switching to or from a virtual reality environment, see-through viewing, etc.).

[0006] The dynamic boundary manager can automatically generate a dynamic boundary for the user, for example, while the artificial reality system is operating in one or more particular modes of operation. The one or more example modes of operation include a stationary mode, a virtual reality mode, a seated mode, a standing mode, or any combination thereof. In some implementations, the dynamic boundary is formed relative to a predefined portion of the user’s body, for example, the user’s head, torso, or any other suitable reference point on the user’s body. The dynamic boundary can include a predefined size and shape, for example, a cylinder with a predefined radius and height, a sphere with a predefined radius, a cuboid with a predefined dimension, or any other suitable shape. In some implementations, the size of the automatically generated dynamic boundary can be based on the user’s body measurements (e.g., arm span, height, stride length, etc.).

[0007] While the user is within the dynamic boundary, the artificial reality system can display an immersive environment, for example, a virtual reality environment, to the user. The dynamic boundary manager can detect triggers of the dynamic boundary criteria in response to user movements, for example, when the user moves within a threshold distance of the dynamic boundary and / or when the user’s speed meets or exceeds a speed criterion. The dynamic boundary manager can cause changes to the operation at the artificial reality system in response to these triggers, for example, transitioning from the virtual reality environment to a see-through view of the real-world environment that includes the user. For example, while the user is in the virtual reality environment, certain user movements can compromise the user’s safety and / or can indicate user activity that is more conducive to a see-through view display. The dynamic boundary manager can trigger changes to the dynamic boundary and / or the operation at the artificial reality system in response to these user movements to improve the user’s experience. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a block diagram showing an overview of a number of devices on which some implementations of the technology can operate.

[0009] Figure 2A is a line drawing showing a virtual reality headset that can be used in some implementations of the technology.

[0010] Figure 2B is a line drawing showing a mixed reality headset that can be used in some implementations of the technology.

[0011] Figure 2C is a line drawing showing a controller that can be held in a single hand or both hands by a user in some implementations to interact with an artificial reality environment.

[0012] Figure 3 is a block diagram showing an overview of an environment in which some implementations of the technology can operate.

[0013] Figure 4 is a block diagram illustrating a number of components that can be used in systems employing the disclosed technology in some implementations.

[0014] Figure 5A is a schematic diagram illustrating a computer-generated environment displayed to a user.

[0015] Figure 5B is a schematic diagram illustrating a perspective view displayed to a user.

[0016] Figure 6 is a state diagram illustrating example active states, example broken states, example expanded states, example popped states, and example reforming states of a dynamic boundary.

[0017] Figure 7 is a conceptual diagram illustrating different states of a dynamic boundary.

[0018] Figure 8 is a conceptual diagram illustrating a number of example dynamic boundary shapes.

[0019] Figure 9 is a conceptual diagram illustrating an example variation for reforming a dynamic boundary.

[0020] Figure 10A is a first portion of a flow diagram illustrating a process for triggering a change in operation at an artificial reality (meta reality (XR)) system through a dynamic boundary in some implementations of the present technology.

[0021] Figure 10B is a second portion of the flow diagram illustrating the above process for a popped state of the triggering of the dynamic boundary in some implementations of the present technology.

[0022] The technology presented herein can be better understood with reference to the following detailed description together with the accompanying drawings, in which like reference numerals refer to similar elements or functionally similar elements throughout the drawings. DETAILED DESCRIPTION

[0023] Aspects of the disclosure relate to dynamic boundaries that trigger changes in operation at an artificial reality system. Artificial reality systems provide immersive visual displays to users, for example, through head-mounted displays. At times, the immersive displays can hinder the user’s perception of the real-world environment. Some artificial reality systems implement boundaries that enforce safety conditions when operating these systems. For example, operation at the artificial reality system (e.g., the artificial reality environment displayed to the user) can change in response to user movements that approach and / or cross the boundary. Implementations of a dynamic boundary manager enforce dynamic boundaries for users. The dynamic boundary manager can automatically generate a dynamic boundary corresponding to a user, for example, when the artificial reality system is operating under certain operating conditions. The dynamic boundary manager can enforce dynamic boundary criteria. For example, user movements can trigger an expansion of the dynamic boundary, a pop-out of the dynamic boundary, and / or a reforming of the dynamic boundary. The dynamic boundary manager can also cause changes to operation at the artificial reality system in response to these triggers, such as pausing an application being executed and / or changing the artificial reality environment displayed to the user (e.g., to a virtual reality environment, a transition to or from a see-through vision, etc.).

[0024] The dynamic boundary manager can automatically generate a dynamic boundary for a user, for example, when the artificial reality system is operating under one or more certain operating modes. The one or more example operating modes include a stationary mode, a virtual reality mode, a sitting mode, a standing mode, or any combination thereof. In some implementations, the dynamic boundary is formed relative to a predefined portion of the user’s body, such as the user’s head, torso, or any other suitable reference point on the user’s body. The dynamic boundary can include a predefined size and shape, such as a cylinder with a predefined radius and height, a sphere with a predefined radius, a cuboid with a predefined dimension, or any other suitable shape. In some implementations, the size of the automatically generated dynamic boundary can be based on body measurements of the user (e.g., arm span, height, step length, etc.).

[0025] While the user is within the dynamic boundary, the artificial reality system can display an immersive environment, such as a virtual reality environment, to the user. The dynamic boundary manager can detect triggers of the dynamic boundary criteria in response to user movements, such as when the user moves within a threshold distance of the dynamic boundary and / or when the user’s speed meets or exceeds a speed criterion. The dynamic boundary manager can cause changes to operation at the artificial reality system in response to these triggers, such as a transition from the virtual reality environment to a see-through vision of the real-world environment that includes the user. For example, while the user is in the virtual reality environment, certain user movements can endanger the user’s safety and / or can indicate user activities that are more conducive to a see-through vision display. The dynamic boundary manager can trigger changes to the dynamic boundary and / or operation at the artificial reality system in response to these user movements to improve the user’s experience.

[0026] In some implementations, the dynamic boundary manager can enforce expansion criteria and pop criteria. For example, the expansion criteria can include one or more distance thresholds triggered by user movement that is very close to the dynamic boundary and / or that exceeds the dynamic boundary. The dynamic boundary manager can expand the dynamic boundary in response to the triggered expansion criteria, e.g., expand the size of the boundary, and / or cause a see-through vision from the displayed virtual reality environment to the user’s real-world environment. The dynamic boundary can be expanded based on predefined metrics, physical measurements of the user (e.g., arm span, height, stride length, etc.), and / or speed of movement of the user. In some implementations, the see-through vision of the user’s real-world environment to which the user is transitioned in response to the expansion criteria includes a see-through vision that is displayed with a first opacity (e.g., an opacity of 60%, 70%, 80%, etc.). In this example, triggering the expansion criteria can indicate to the user via the display transition that the user’s movement has triggered the dynamic boundary, however the dynamic boundary has not yet been fully triggered (e.g., “popped”), so the user can maintain the virtual reality environment by staying within the dynamic boundary (e.g., by stopping movement and / or moving back toward the center of the dynamic boundary).

[0027] The pop criteria can be triggered after the dynamic boundary is expanded (e.g., after the expansion criteria are triggered). For example, the pop criteria can include one or more distance thresholds triggered by user movement that is very close to the expanded dynamic boundary or that exceeds the expanded dynamic boundary, and / or a speed criterion triggered by user movement that meets or exceeds a speed threshold. The dynamic boundary manager can pop the dynamic boundary, pause an application that is executing at the artificial reality system (e.g., a virtual reality application that is executing), and / or cause a transition from the see-through vision of the user’s real-world environment that is displayed with the first opacity to a see-through vision of the user’s real-world environment that is displayed with a second opacity (e.g., 90%, 100%, etc.), where the second opacity is greater than the first opacity, in response to the triggered pop criteria. In this example, triggering the pop criteria can indicate to the user via the display transition that the user’s movement has popped the dynamic boundary, causing the virtual reality environment to be paused (e.g., to ensure user safety, to improve user experience when moving, etc.).

[0028] The dynamic boundary manager can also enforce a reform criteria. For example, after expanding and / or popping the dynamic boundary, the dynamic boundary manager can reform the dynamic boundary when user movement satisfies a reform criteria. The reform criteria can include a reform speed criteria triggered by user movement that satisfies or falls below a reform speed threshold for a threshold duration of time. In this example, when the user's movement indicates that the user is stationary (e.g., below the reform speed threshold for the threshold duration of time), the boundary manager can reform the dynamic boundary so that the user can again be immersed in the virtual reality environment. For example, the dynamic boundary manager can reform the dynamic boundary in response to the triggered reform criteria (e.g., with respect to the user, at the time of reform), resume execution of one or more applications at the artificial reality system (e.g., the paused virtual reality application), and / or cause a transition from the displayed perspective view of the user's real-world environment to the resumed virtual reality environment. In this example, triggering the reform criteria allows for resuming user interaction with the virtual reality environment within the safe range of the dynamic boundary space. In some implementations, the reformed dynamic boundary can be generated with an initial size and enlarged to a predefined size associated with the automatically generated dynamic boundary.

[0029] Embodiments of the disclosed technology can include or be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some manner before presentation to a user, which can include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and / or derivatives thereof. Artificial reality content can include completely generated content or generated content combined with captured content (e.g., recordings of a real-world scene). Artificial reality content can include video, audio, haptic feedback, or some combination of these, any of which can be presented in a single channel or in multiple channels (such as stereo video that can produce a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality can also be associated with applications, products, accessories, services, or some combination of these, that are used to create content, or to visualize content, in an artificial reality setting. The artificial reality system that provides the artificial reality content can be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a “cave” environment, or other projection systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0030] “Virtual reality” or “VR” as used herein refers to an immersive experience in which a user’s visual input is controlled by a computing system. “Augmented reality” or “AR” refers to a system in which a user views real-world images after they pass through a computing system. For example, a tablet with a camera on the back can capture real-world images and then display those images on a screen on the opposite side of the tablet from the camera. The tablet can process and adjust or “augment” the images as they pass through the system, for example by adding virtual objects. “Mixed reality” or “MR” refers to a system in which light entering a user’s eye is partially generated by a computing system and partially composed of light reflected by objects in the real world. For example, an MR headset can be shaped like a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, allowing the MR headset to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “ultra reality,” or “XR” refers to any of VR; AR; MR; or any combination or mixture thereof.

[0031] Conventional XR systems can include boundaries that maintain user safety, for example, when a user is immersed in a virtual reality environment, however these boundaries lack flexibility and include rigid forms that trigger system functionality. For example, often such boundaries require a setup procedure and then use of such boundaries is limited to the real environment in which the boundary was set. Further, conventional systems lack dynamic boundaries that are capable of changing size and shape in response to user movement. Further, XR systems often lack different levels of functionality in response to triggered boundaries, instead resulting in complete disruption of the user experience.

[0032] The disclosed implementations of dynamic boundaries increase flexibility and selective XR system responses to triggered dynamic boundary criteria. For example, triggered dynamic boundary criteria can cause one or more of the following: dynamic boundary expansion; transition from a VR environment to see-through vision at a limited opacity (e.g., less than 100%); transition to full see-through vision; and pausing of an executing VR application, among others. The triggered variety of XR system operations (each configured for dynamic boundary criteria satisfied by user movement) adapts system functionality to specific user movement scenarios (e.g., standing, sitting, fast movement, large distance movement, slow movement, small distance movement, etc.). Moreover, the reforming criteria of dynamic boundaries support resumption of a user’s artificial reality experience and minimizes disruption to the user. Dynamic boundaries can be automatically generated with respect to a user (e.g., the user’s body) and thus improve the cumbersome setup procedures required by systems that execute traditional boundaries.

[0033] Several implementations are discussed in greater detail below with reference to the drawings. Figure 1 is a block diagram that illustrates a generalized architecture for a variety of devices on which the disclosed technology can run. The devices can include hardware components of computing system 100 that trigger operational changes at an artificial reality system through dynamic boundaries. In various implementations, computing system 100 can include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate through wired or wireless channels to distribute processing and share input data. In some implementations, computing system 100 can include a standalone head-mounted viewer that is capable of providing a computer-created or augmented experience for a user without the need for external processing or external sensors. In other implementations, computing system 100 can include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), where some processing operations are performed on the head-mounted viewer and other processing operations are offloaded to the core processing component. Example head-mounted viewers are described below in connection with Figure 2A and Figure 2B In some implementations, position data and environment data can be collected by sensors contained in a head-mounted viewer device only, while in other implementations one or more of the multiple non-head-mounted viewer computing devices can include sensor components that can track environment data or position data.

[0034] Computing system 100 can include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). Processors 110 can be a single processing unit or multiple processing units that are located in one device or distributed across multiple devices (e.g., distributed across two or more of computing devices 101-103).

[0035] Computing system 100 can include one or more input devices 120 that provide input to processors 110, thereby informing these processors of actions. These actions can be communicated by a hardware controller that interprets received signals from the input device and communicates information to processors 110 using a communication protocol. For example, individual input devices 120 can include a mouse, a keyboard, a touchscreen, a touchpad, a wearable input device (e.g., a haptic glove, bracelet, ring, earring, necklace, watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0036] Processors 110 can be coupled to other hardware devices, for example, by using internal buses, external buses (e.g., a peripheral component interconnect (PCI) bus, a small computer system interface (SCSI) bus), or wireless connections. Processors 110 can communicate with hardware controllers of devices such as display 130. Display 130 can be used to display text and graphics. In some implementations, display 130 includes an input device, for example, when the input device is a touchscreen or is equipped with an eye direction monitoring system as part of the display. In some implementations, the display is separate from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen; a light-emitting diode (LED) display screen; a projection, holographic, or augmented reality display (e.g., a heads-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 can also be coupled to the processors, such as network chips or network cards, video chips or video cards, audio chips or audio cards, universal serial bus (USB), Firewire, or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0037] In some implementations, the computing system 100 can use input from the I / O devices 140 (e.g., a camera, a depth sensor, an IMU sensor, a GPS unit, a lidar (LiDAR) or other time-of-flight sensor, etc.) to identify and map a user’s physical environment while tracking the user’s location in that environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, a building, an outdoor space, etc.) and / or acquire a map previously generated by the computing system 100 or another computing system that has mapped the area. The SLAM system can track the user within the area based on factors such as GPS data, match identified objects and structures to mapped objects and structures, monitor acceleration and other location changes, etc.

[0038] The computing system 100 can include a communication device that enables wireless or wired communication between the computing system 100 and other local computing devices or network nodes. This communication device can communicate with another device or server over a network, for example, using the transmission control protocol / Internet protocol (TCP / IP). The computing system 100 can utilize the communication device to distribute operations among multiple network devices.

[0039] The processor 110 can access memory 150, which can be included on one of the multiple computing devices of the computing system 100 or can be distributed across multiple computing devices of the computing system 100 or multiple other external devices. The memory includes one or more hardware devices for volatile or non-volatile storage and can include both read-only memory and writable memory. For example, the memory can include one or more of random access memory (RAM); various types of cache memory; CPU register; read-only memory (ROM); and writable non-volatile memory such as flash memory, hard drives, floppy disks, optical disks (CDs), DVDs, magnetic storage devices, tape drives, etc. The memory is not a propagating signal away from the underlying hardware; thus the memory is non-transitory. The memory 150 can include program memory 160 storing programs and software such as an operating system 162, dynamic boundary manager 164, and other application programs 166. The memory 150 can also include data memory 170, which can include, for example, XR application data, dynamic boundary thresholds and criteria, user’s location information and / or historical boundaries, configuration data, settings, user options or preferences, etc., which can be provided to the program memory 160 or any element of the computing system 100.

[0040] Some implementations can be run with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that can be suitable for use with the technology include, but are not limited to, XR head-mounted viewers, personal computers, server computers, hand-held or laptop devices, cellular telephones, wearable electronic devices, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, or the like.

[0041] Figure 2Ais a line drawing of a virtual reality head-mounted display (HMD) 200 according to some embodiments. In this example, the virtual reality HMD 200 also includes augmented reality features, using see-through cameras 225 to render portions of the real world that can have computer-generated overlays. The virtual reality HMD 200 includes a front rigid body 205 and a band 210. The front rigid body 205 includes one or more electronic display elements of one or more electronic displays 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a plurality of cameras and a plurality of locators 225, and one or more computing units 230. The position sensors 220, the IMU 215, and the computing units 230 can be located inside the virtual reality HMD 200 and can not be visible to a user. In various implementations, the IMU 215, the position sensors 220, and the cameras and locators 225 can track movement and position of the virtual reality HMD 200 in the real world and in an artificial reality environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locators 225 can emit infrared light beams that create light points on real objects around the virtual reality HMD 200 and / or the cameras 225 capture images of the real world and localize the virtual reality HMD 200 within that real world environment. As another example, the IMU 215 can include one or more of the following components that can be used in the localization process: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras 225 integrated with the virtual reality HMD 200 can detect the light points described above. The computing units 230 in the virtual reality HMD 200 can use the detected light points and / or position points to extrapolate the position and movement of the virtual reality HMD 200 and to identify shapes and positions of real objects around the virtual reality HMD 200.

[0042] One or more electronic displays 245 can be integrated with the front rigid body 205 and can provide image light to a user as instructed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of electronic displays 245 include liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, active-matrix organic light-emitting diode displays (AMOLEDs), displays including one or more quantum dot light-emitting diode (QOLED) sub-pixels, projector units (e.g., micro-LEDs, lasers (LASERs), etc.), some other display, or some combination thereof.

[0043] In some implementations, the virtual reality HMD 200 can be coupled to a core processing component, such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the virtual reality HMD 200 (e.g., by emitting light from the virtual reality HMD 200), and the PC can use the HMD in conjunction with output from the IMU 215 and the position sensor 220 to determine the position and movement of the virtual reality HMD 200.

[0044] Figure 2B is a line diagram of a mixed reality HMD system 250 that includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 gigahertz (GHz) link) as indicated by link 256. In other implementations, the mixed reality HMD system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye tracking sensors, microelectromechanical systems (MEMS) components, networking components, etc.

[0045] The projector can be coupled to the see-through display 258, e.g., via optical elements, to display media to the user. These optical elements can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. to direct light from the projector to the user's eye. Image data from the core processing component 254 can be transmitted to the mixed reality HMD 252 via the link 256. A controller in the mixed reality HMD 252 can convert the image data to light pulses from the projector, which can be transmitted as output light to the user's eye via the optical elements. This output light can mix with light passing through the see-through display 258, allowing the output light to appear to present virtual objects that appear to exist in the real world.

[0046] Like the virtual reality HMD 200, the mixed reality HMD system 250 can also include motion and position tracking units, cameras, light sources, etc. that allow the mixed reality HMD system 250 to track itself, e.g., in 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), render virtual objects to appear to be stationary as the mixed reality HMD 252 moves, and make virtual objects react to hand gestures and other real world objects.

[0047] Figure 2C Controllers 270 (including controller 276A and controller 276B) are shown that a user can hold in one or both hands to interact with the artificial reality environment presented by the virtual reality HMD 200 and / or the mixed reality HMD system 250 in some implementations. The controllers 270 can communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit additional light points. The virtual reality HMD 200 or the mixed reality HMD system 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., to track the controllers in 3DoF or 6DoF). The computing unit 230 in the virtual reality HMD 200 or the core processing component 254 can use this tracking in conjunction with IMU output and position output to monitor the position and motion of the user's hands. The controllers can also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A and 274B) that the user can actuate to provide input and interact with objects.

[0048] In various implementations, virtual reality HMD 200 or mixed reality HMD system 250 can also include additional subsystems (e.g., eye tracking units, audio systems, various networking components, etc.) to monitor indications of user interaction and intent. For example, in some implementations, one or more cameras included in virtual reality HMD 200 or mixed reality HMD system 250 or from a plurality of external cameras can monitor the position and pose of a user's hands, in lieu of or in addition to controllers, to determine hand gestures and other hand and body motions. As another example, one or more light sources can illuminate a user's eye or eyes, and virtual reality HMD 200 or mixed reality HMD system 250 can use an eye-facing camera to capture reflections of the light to determine eye position (e.g., based on a set of reflections around a user's cornea), model the user's eyes, and determine a gaze direction.

[0049] Figure 3 FIG. 3 is a block diagram illustrating an overview of an environment 300 in which some implementations of the disclosed technology can operate. Environment 300 can include one or more client computing devices 305A-305D, examples of which can include computing system 100. In some implementations, some of the plurality of client computing devices (e.g., client computing device 305B) can be virtual reality HMD 200 or mixed reality HMD system 250. Client computing devices 305 can operate in a networked environment using logical connections to one or more remote computers, such as server computing devices, through a network 330.

[0050] In some implementations, server computing device 310 can be an edge server that receives a plurality of client requests and coordinates fulfillment of the requests through other servers (e.g., server computing devices 320A-320C). Server computing devices 310 and 320 can include computing systems, such as computing system 100. Although each server computing device 310 and 320 is shown logically as a single server, each can be a plurality of server computing devices in a distributed computing environment that includes a plurality of computing devices located at the same physical location or at geographically different physical locations.

[0051] The client computing device 305, and the server computing devices 310 and 320, can each act as a server or a client to other servers / one or more client devices. The server computing device 310 can be connected to a database 315. The server computing devices 320A through 320C can each be connected to a corresponding database 325A through 325C. As described above, each server computing device 310 or 320 can correspond to a group of servers, and individual ones of these server computing devices can share a database, or can have its own database. Although the databases 315 and 325 are shown logically as single units, each of the databases 315 and 325 can be a distributed computing environment that includes multiple computing devices, can be located within its corresponding server, or can be located at the same physical location or at geographically different physical locations.

[0052] The network 330 can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 can be the Internet or some other public or private network. The client computing device 305 can connect to the network 330 via a network interface, such as through wired or wireless communication. Although the connections between the server computing device 310 and the multiple server computing devices 320 are shown as separate connections, these connections can be any type of local area, wide area, wired, or wireless network, including the network 330 or a separate public or private network.

[0053] Figure 4is a block diagram illustrating a plurality of components 400 that can be used in systems employing the disclosed technology, in some implementations. The plurality of components 400 can be included in one device of the computing system 100, or can be distributed across multiple devices of the computing system 100. The plurality of components 400 includes a plurality of hardware 410, middleware 420, and a plurality of specialized components 430. As described above, systems implementing the disclosed technology can use a variety of hardware, including processing units 412, working memory 414, input and output devices 416 (e.g., cameras, displays, IMU units, network connections, etc.), and storage memory 418. In various implementations, the storage memory 418 can be one or more of: a local device; an interface to a remote storage device; or a combination thereof. For example, the storage memory 418 can be one or more hard disk drives or flash drives accessible through a system bus, or can be a cloud storage provider (e.g., located in the databases 315 or 325) or other network storage device accessible through one or more communication networks. In various implementations, the plurality of components 400 can be implemented in a client computing device (e.g., the client computing device 305), or on a server computing device (e.g., the server computing device 310 or 320).

[0054] The middleware 420 can include components that transfer resources between the hardware 410 and the plurality of specialized components 430. For example, the middleware 420 can include an operating system, services, drivers, a basic input output system (BIOS), controller circuitry, or other hardware or software systems.

[0055] The plurality of specialized components 430 can include software or hardware configured to perform operations that trigger changes in operations at an XR system through dynamic boundaries. The plurality of specialized components 430 can include an XR mode manager 434, a user monitor 436, a boundary controller 438, one or more XR applications 440, and components and application program interfaces (APIs) that can be used to provide user interfaces, transfer data, and control the plurality of specialized components, e.g., the interface 432. In some implementations, the plurality of components 400 can be located in a computing system distributed across multiple computing devices, or can be an interface to a server-based application that performs one or more of the plurality of specialized components 430. Although the plurality of specialized components 430 are depicted as separate components, the plurality of specialized components 430 can be logically functional or other non-physically distinct functional, and / or can be sub-modules or blocks of code of one or more applications.

[0056] The XR mode manager 434 is a manager for operating the XR system in different modes (e.g., VR mode or passthrough mode). In VR mode, the XR system can display a computer-generated environment to the user, such as a VR experience. In passthrough mode, the XR system can display to the user a real-world environment of the user that is captured by one or more cameras of the XR system. As used herein, passthrough mode can refer to a display that includes a real-world environment of the user, and / or a mix of a real-world environment of the user and one or more computer-generated components (e.g., virtual objects).

[0057] The XR mode manager 434 can transition between VR mode and passthrough mode, for example, based on triggers identified by the boundary controller 438. The XR mode manager 434 can implement different transition variants, such as a gradual transition from one environment to the next (e.g., passthrough to VR, VR to passthrough, etc.), a gradual transition with different timing parameters (e.g., at different time intervals (e.g., 1 second, 1.5 seconds, 2 seconds, 3 seconds, 5 seconds, etc.), and / or at different rates of gradual transition (e.g., from slow to fast, from fast to slow, etc.), and a quick transition from one environment to the next (e.g., to perform a safety protocol), etc. The XR mode manager 434 can also pause and / or resume applications executing at the XR system, such as one or more XR applications 440. Reference is made to Figure 10A and Figure 10B Blocks 1004, 1010, 1018, 1022, 1024, 1030, and 1032 of FIGS. 10, 11, 18, 22, 24, 30, and 32, respectively, describe more details regarding the XR mode manager 434.

[0058] The user monitor 436 can monitor movements of a user of the XR system. For example, one or more sensors (e.g., cameras, handheld controllers including IMUs, worn sensors, etc.) can track movements of the user within a real environment of the user. The user monitor 436 can detect, based on the tracked movements of the user, a user movement that satisfies a dynamic boundary criterion. Reference is made to Figure 10A and Figure 10B Blocks 1006, 1012, 1014, and 1026 of FIGS. 10, 11, 18, and 22, respectively, describe more details regarding the user monitor 436.

[0059] The boundary controller 438 can dynamically control the dynamic boundary and trigger XR operation changes in response to user movements. For example, the boundary controller 438 can generate a dynamic boundary around the user (e.g., relative to the user’s body), e.g., a boundary that includes a predefined shape and size. The boundary controller 438 can compare user movements to criteria for the generated dynamic boundary, e.g., one or more distance thresholds and / or one or more velocity thresholds. Example dynamic boundary criteria include an expansion criterion, a pop-out criterion, and a reform criterion. The boundary controller 438 can communicate with the XR mode manager 434 to cause operation changes at the XR system, e.g., transitions between displayed environments, pausing and / or resuming one or more XR applications 440, etc., in response to triggered boundary criteria. Reference is made to Figure 10A and Figure 10B Blocks 1002, 1006, 1008, 1012, 1014, 1016, 1026, and 1028 of FIG. 10 describe more details regarding the boundary controller 438.

[0060] The one or more XR applications 440 can be applications that execute on the XR system, e.g., applications that generate XR environments (e.g., three-dimensional immersive environments). One or more example XR applications 440 can generate fully computer-generated environments, e.g., VR environments, see-through environments with virtual components, or any other suitable XR environment. The one or more XR applications 440 can execute at least partially at the XR system to generate the XR environment. Reference is made to Figure 10B Blocks 1022 and 1030 of FIG. 10 describe more details regarding the one or more XR applications.

[0061] The XR system can display content to the user through a see-through visual display, e.g., a VR environment and / or the user’s real-world environment. Figure 5A is a diagram 500A that illustrates a computer-generated environment displayed to a user. The diagram 500A includes a user 502 and a VR environment 504. The XR system can display the VR environment 504 to the user 502 through a system shell that provides an immersive environment, e.g., through an immersive gaming application, or any other suitable application that provides a VR environment displayed to the user.

[0062] When displaying the VR environment 504 to the user 502, the user’s perception of the real world can be obstructed. As a result, certain user movements can pose safety risks and / or certain user activities can be obstructed. To address these risks / obstructions, in certain scenarios, the XR system can display a see-through visual that includes the user’s real-world environment.

[0063] Figure 5Bis a diagram 500B illustrating a see-through vision displayed to a user. The diagram 500B includes a user 502 and a see-through environment 506. When displaying the see-through environment 506 to the user 502, the user’s perception of the real-world environment is no longer completely obstructed, and the user 502 can more easily be able to perform certain activities or move safely.

[0064] Implementations of the dynamic boundary can trigger a transition from the VR environment 504 to the see-through environment 506 and / or from the see-through environment 506 to the VR environment 504. For example, the XR system can monitor movements of the user 502 and can compare the monitored movements to dynamic boundary criteria (e.g., expansion criteria, pop-out criteria, reforming criteria) to trigger a change in operation at the XR system. Example changes in operation include expanding the dynamic boundary, popping out the dynamic boundary and pausing execution of an application being executed, transitioning from the VR environment to see-through vision, transitioning from see-through vision to the VR environment, reforming the dynamic boundary and resuming execution of the application, and any other suitable XR system function. In some implementations, the dynamic boundary can include different states, and transitions between these states can be triggered by user movements that satisfy the dynamic boundary criteria.

[0065] Figure 6 is a state diagram 600 illustrating example active states, example broken states, example expanded states, example popped-out states, and example reformed states of a dynamic boundary. The diagram 600 includes a state diagram that includes an active state 602, a broken state 604, an expanded state 606, a popped-out state 608, and a reformed state 610. Any suitable states of a dynamic boundary can be added or omitted. Figure 7 is a conceptual diagram 700 illustrating different states of a dynamic boundary. The diagram 700 includes a formed dynamic boundary 702, a broken dynamic boundary 704, an expanded dynamic boundary 706, and a reformed dynamic boundary 708.

[0066] Returning to Figure 6 In the active state 602, a dynamic boundary can be formed around the user, and a computergenerated environment, such as a VR environment, can be displayed to the user. In the active state 602, movements of the user can be favorable and / or safe for the VR environment. Figure 7 The formed dynamic boundary 702 illustrates a dynamic boundary in the active state 602. For example, the formed dynamic boundary 702 can be positioned relative to the user’s body (e.g., around the user) and include a defined shape and size when formed. Figure 8 Variations of the dynamic boundary formed relative to the user are further described.

[0067] User movements can cause state transitions of the dynamic boundary and resulting XR system functionality. The transition from the active state 602 to the broken state 604 can be triggered by a user movement that satisfies an extension criterion of the dynamic boundary. For example, the extension criterion can include one or more threshold distance metrics. Example user movements include any suitable body movement (e.g., arm movement, head movement, walking, running, jumping, standing, sitting, etc.). Example threshold proximities include 0 meters (e.g., a user movement that breaks the dynamic boundary), 0.01 meters, 0.05 meters, 0.1 meters, and 0.2 meters, etc.

[0068] In some implementations, the extension criterion can include a first threshold distance and a second threshold distance. For example, a user movement that satisfies the first threshold distance can cause a transition from the active state 602 to the broken state 604. A user movement that satisfies the second threshold distance can cause a transition from the broken state 604 to the extended state 606. Figure 7 The broken dynamic boundary 704 illustrates an inner circle that can correspond to the first threshold distance and an outer circle that can correspond to the second threshold distance. A user movement that breaks the inner circle (e.g., satisfies the first threshold distance) can trigger a transition from the active state 602 to the broken state 604.

[0069] In the broken state 604, the XR system display can be transitioned. For example, the XR system display can be transitioned from the VR environment to a see-through vision (e.g., a see-through of a real-world environment of the user captured by an XR system camera) at a first opacity that is less than 100% (e.g., 60%, 70%, 80%, etc.). The semi-transparent nature of the see-through vision at the first opacity can indicate to the user that the user can return to the VR environment by repositioning within the dynamic boundary. In some implementations, a fade from the VR environment to the see-through vision at the first opacity can occur within a defined duration (e.g., 0.3 seconds, 0.4 seconds, 0.5 seconds, etc.) or based on a movement of the user. For example, the fade from the VR environment to the see-through vision at the first opacity can correspond to a user movement that progresses from an inner boundary to an outer boundary of the broken dynamic boundary 704.

[0070] For example, the break state 604 can transition to an extended state 606 after user movement satisfies a second distance threshold of the extension criteria. In the extended state 606, the dynamic boundary can be extended. The extension can include an extension of a predefined size (e.g., 0.2 meters, 0.3 meters, 0.4 meters, etc.), an extension based on user’s body measurements (e.g., arm length, height, step length, etc.), or any other suitable extension. For example, the extension can include a first predefined size when the user is detected to be sitting and a second predefined size when the user is detected to be standing. In some embodiments, a speed of the user’s movement over a period of time can be calculated and the size of the dynamic boundary can be extended based on the speed of the user (e.g., extended larger for higher speeds, extended smaller for lower speeds, etc.).

[0071] The transition from the extended state 606 to the pop-out state 608 can be triggered by user movement satisfying pop-out criteria of the dynamic boundary. In some embodiments, the pop-out criteria can include a first threshold distance and a second threshold distance. For example, user movement satisfying the first threshold distance can trigger a fade from the see-through vision at a first opacity to the see-through vision at a second opacity (e.g., fade from 80% to 100%, etc.). User movement satisfying the second threshold distance can trigger a pause of one or more applications being executed.

[0072] Figure 7 The extended dynamic boundary 706 of FIG. 7A illustrates an inner circle that can correspond to the first threshold distance of the pop-out criteria and an outer circle that can correspond to the second threshold distance of the pop-out criteria. User movement breaking the inner circle (e.g., user movement satisfying the first threshold distance) can trigger a fade from the see-through vision displayed at a first opacity (e.g., less than 100%) to the see-through vision displayed at a second opacity (e.g., 100%). The fade can occur over a defined duration or the fade progress can correspond to user movement from the first threshold distance (e.g., the inner circle of the extended dynamic boundary 706) to the second threshold distance (e.g., the outer circle of the extended dynamic boundary 706).

[0073] User movement breaking the outer ring of the extended dynamic boundary 706 (e.g., user movement satisfying a second threshold distance) can trigger a pause of one or more applications being executed. For example, one or more applications being executed at the XR system can provide a VR environment displayed to the user. In response to the second threshold distance of the pop-out criteria being broken by the user movement, the one or more applications being executed can be paused. In some implementations, audio from the one or more applications being executed can be output to the user (e.g., at 100% volume, 80% volume, 50% volume, 20% volume, etc.) when the XR system performs one or more gradual transitions from the VR environment to the see-through vision (e.g., at a different level of opacity) in response to satisfying the expansion criteria and / or the pop-out criteria. The pause of the one or more applications being executed can also pause the audio from the one or more applications being executed, signaling to the user that the one or more applications have been paused.

[0074] Once the second threshold distance of the pop-out criteria is satisfied, the dynamic boundary can transition from the expanded state 606 to the popped-out state 608. In some implementations, the pop-out criteria can include a velocity criterion, and user movement satisfying the velocity criterion can cause: a) a transition from the see-through vision displayed at the first opacity to the see-through vision displayed at the second opacity; and b) a pause of the one or more applications being executed. For example, once the velocity criterion is satisfied, the dynamic boundary can transition from the expanded state 606 to the popped-out state 608. The velocity criterion can provide an effective mechanism for quickly transitioning to the popped-out state 608 if the user is moving quickly, thus making it highly likely that the dynamic boundary will pop out.

[0075] In the popped-out state 608, the one or more applications being executed can remain paused, and the XR system can display the see-through vision of the user’s real-world environment until a reforming criterion of the dynamic boundary is satisfied. For example, the user can move to perform certain actions in the user’s real-world environment that are disadvantageous to performing while displaying an immersive VR environment to the user. The transition from the popped-out state 608 to the reforming state 610 can be triggered by user movement satisfying the reforming criterion of the dynamic boundary. For example, the reforming criterion can be a velocity threshold and a threshold duration, so user movement below the velocity threshold (e.g., 0.3 meters per second (m / s), 0.2 m / s, 0.1 m / s, etc.) for the threshold duration (e.g., 0.4 seconds, 0.5 seconds, 0.8 seconds, 1 second, 2 seconds, etc.) satisfies the reforming criterion.

[0076] In the reforming state 610, the XR system can resume execution of the suspended one or more applications (e.g., the one or more applications suspended in the pop-out state 608) and transition from the see-through vision to a VR environment provided by the resumed one or more applications. In some implementations, the transition can include a fade from the see-through vision (e.g., at a current displayed opacity) to the VR environment over a period of time (e.g., 1 second, 1.2 seconds, 1.5 seconds, etc.). In some implementations, the dynamic boundary reforming in the reforming state 610 can include a similar size and shape as the initially formed dynamic boundary (e.g., a cylinder with a defined radius and height). In another example, the dynamic boundary reforming in the reforming state 610 can initially reform at a smaller size and grow over a defined period of time (e.g., 1 second, 1.2 seconds, 1.5 seconds, etc.).

[0077] Figure 7 The reforming dynamic boundary 708 illustrates an inner circle, which can correspond to an initial size of the reforming dynamic boundary, and an outer circle, which can correspond to a size of the dynamic boundary after it has grown over a defined period of time. While the dynamic boundary is reforming (e.g., growing), a user movement that satisfies the distance criteria of the reforming dynamic boundary (e.g., crossing the dynamic boundary) can cancel the reforming and cause a transition from the reforming state 610 to the expanded state 606. For example, the cancelled reforming can maintain display of the see-through vision to the user until the reforming dynamic boundary is complete.

[0078] Once the dynamic boundary has completed its reforming, the reforming state 610 can transition to the active state 602. In the active state 602, the XR system displays the VR environment to the user, for example, until the user movement satisfies the expansion criteria of the dynamic boundary.

[0079] In some scenarios, the dynamic boundary can transition from the expanded state 606 to the reforming state 610 without proceeding to the pop-out state 608. For example, the transition from the expanded state 606 to the reforming state 610 can be triggered by a user movement that satisfies the reforming criteria of the dynamic boundary, such as when the user movement is below a speed threshold of the reforming criteria for a threshold duration of the reforming criteria. In this example, the suspended one or more applications are not resumed because no one or more applications were suspended as a result of not proceeding to the pop-out state 608. However, a transition (e.g., a fade over a duration of time) from the see-through vision at a displayed opacity (e.g., 80%) to the VR environment can be performed. Further, the dynamic boundary can be reforming (e.g., generated at an initial size and grown), and once the dynamic boundary has completed the reforming of the dynamic boundary, the reforming state 610 can transition to the active state 602.

[0080] In some implementations, a map of the user’s real-world environment can be drawn and one or more real-world objects drawn can be used as criteria for popping the dynamic boundary. For example, the XR system can draw a map of the user’s real-world environment through one or more cameras. In some implementations, the user can be directed to move around the real-world environment to capture different portions of the environment so that the map can be generated.

[0081] One or more locations of real-world objects can be used to form and / or expand the dynamic boundary. For example, the defined shape and / or size of the dynamic boundary can be changed to account for the drawn real-world objects. In some implementations, the size and / or shape of the dynamic boundary can be adjusted so that the boundary does not overlap with the drawn real-world objects. In another example, the size and / or shape of the dynamic boundary can be adjusted so that the boundary is at least a threshold distance from the drawn real-world objects.

[0082] In some implementations, the location of the drawn real-world objects can be used to perform additional criteria that triggers the popping of the dynamic boundary. For example, the additional criteria can include a threshold distance and the dynamic boundary can be popped when user movement is detected within the threshold distance from the real-world objects. For example, the XR system can transition (e.g., fade or snap) from the displayed VR environment to a see-through view of the user’s real-world environment and / or can pause one or more applications that are executing. When the user satisfies the reforming criteria and the user is no longer proximate to the real-world objects (e.g., outside a distance threshold), the XR system can similarly reform the dynamic boundary so that the paused one or more applications can be resumed and the user can return to the VR environment.

[0083] Figure 8 is a conceptual diagram 800 illustrating example dynamic boundary shapes. The diagram 800 includes a user 802, a dynamic boundary 804, a user 806, and a dynamic boundary 808. The dynamic boundary 804 is an example of a dynamic boundary generated with respect to a portion of the user’s 802 body, in particular the user’s head. The dynamic boundary 804 includes a cylinder with a defined radius and height that is positioned according to the user’s head (e.g., the location of the user’s head when the dynamic boundary is formed). In this example, the user’s head movement can be monitored and used to trigger dynamic boundary criteria (e.g., expansion criteria, pop criteria, reforming criteria). A dynamic boundary with respect to the user’s head can detect user movements such as standing and sitting and trigger the resulting XR system functionality (e.g., fade into or out of a VR environment, see-through view, etc.).

[0084] Dynamic boundary 808 is an example of a dynamic boundary generated with respect to the entire body of user 806 (or the torso of user 806). Dynamic boundary 806 includes a defined radius and height that is positioned according to the user’s body (e.g., the position of the user’s body at the time the dynamic boundary is formed). In this example, any suitable movement of user 806 (e.g., arm movement, leg movement, head movement, etc.) can be monitored and used to trigger a dynamic boundary criterion (e.g., an expansion criterion, a pop criterion, a reform criterion).

[0085] A formed dynamic boundary can include any suitable shape and / or size. For example, a dynamic boundary can include a cylinder, a sphere, a cuboid, or any other suitable shape. Further, the size (e.g., the dimensions of the shape) can be predefined and / or based on the user’s body (e.g., arm length, height, step length, etc.). In some embodiments, the dimensions of the dynamic boundary can be defined using historical tracking of the user. For example, some users can naturally move more than other users while experiencing a VR environment. Based on historical tracking of user movement, some users can be given more freedom (e.g., a large dynamic boundary) than other users in order to tailor the dynamic boundary to different user behavior, effectively triggering XR system functionality.

[0086] Figure 9 is a conceptual diagram 900 illustrating an example variation for reforming a dynamic boundary. Diagram 900 includes a user 902, a formed dynamic boundary 904, a reformed dynamic boundary 906, and scenes 910, 912, and 914. Scene 910 illustrates user 902 popping formed dynamic boundary 904 with a user movement. Scenes 912 and 914 illustrate different variations of reforming a dynamic boundary. For example, a movement of user 902 can satisfy a reform criterion of the dynamic boundary, and in response, a reformed dynamic boundary 906 can be generated around user 902.

[0087] In scenario 912, the reformatted dynamic boundary 906 is formed relative to the user’s 902 body at the time of reformattmg (e.g., relative to the position of the user’s head and / or any other suitable body part(s) of the user’s body). In scenario 914, the reformatted dynamic boundary 906 is formed relative to a combination of the user’s 902 body at the time of reformattmg and the previous position of the formed dynamic boundary 904 before it was popped. For example, scenario 910 shows that the user 902 has moved into the formed dynamic boundary 904 to pop the boundary. Thus, in scenario 914, the reformatted dynamic boundary 906 is positioned between the user’s 902 body and the previous position of the formed dynamic boundary 904. In this variation, using the history of the dynamic boundary’s position can position the reformatted dynamic boundary 906 in a more optimal position for the user’s 902 movements. For example, the user 902 can sway and / or lean, and the reformatted dynamic boundary 906 that takes advantage of the history of the formed dynamic boundary 904 is less likely to be triggered by these movements of the user 902. In other examples, the reformatted dynamic boundary 906 shown in scenario 912 can be a more optimal position. The reformatted dynamic boundary 906 can be reformatted (e.g., resized and / or repositioned) in any other suitable manner.

[0088] Those skilled in the art will realize that the components shown in the above-described Figures 1 to 9 modifications can be made to the components illustrated in the figures and described above, as well as those in each of the flow diagrams discussed below. For example, the order of the logic can be rearranged, multiple sub-steps can be performed in parallel, logic can be omitted, other logic can be included, etc. In some embodiments, one or more of the components described above can perform one or more of the processes described below.

[0089] Figure 10A is a process 1000 that illustrates a process for triggering operational changes at an artificial reality (XR) system with dynamic boundaries in some embodiments of the technology. In some embodiments, the process 1000 can be performed by an XR system. In some embodiments, the process 1000 is triggered by operational modes at the XR system, such as VR mode, still mode, standing / sitting mode, any combination thereof, or any other suitable mode.

[0090] At block 1002, the process 1000 can generate a dynamic boundary. For example, the dynamic boundary can be generated for the user when the artificial reality system is operating in a particular mode of operation (e.g., a stationary mode, a VR mode, a sitting mode, a standing mode, or any combination thereof). In some implementations, the dynamic boundary is formed relative to a predefined portion of the user’s body, such as the user’s head, torso, or any other suitable reference point on the user’s body. The dynamic boundary can include a predefined size and shape, such as a cylinder with a predefined radius and height, a sphere with a predefined radius, a cuboid with a predefined dimension, or any other suitable shape. In some implementations, the size of the automatically generated dynamic boundary can be based on body measurements of the user (e.g., arm span, height, stride length, etc.).

[0091] At block 1004, the process 1000 can display a VR environment to the user. For example, one or more applications executing at the XR system can provide a VR environment for display to the user. The VR environment can be a three-dimensional immersive environment displayed to the user, and the dynamic boundary can enforce safety and / or advantageous conditions when the VR environment is displayed to the user.

[0092] At block 1006, the process 1000 can detect whether the user movement satisfies an expansion criterion. For example, the expansion criterion can include one or more distance thresholds triggered by a user movement that comes very close to the dynamic boundary and / or exceeds the dynamic boundary. The user movement can be monitored to detect when the one or more distance thresholds are satisfied. When the user movement satisfies the expansion criterion, the process 1000 can proceed to block 1008. When the user movement does not satisfy the expansion criterion, the process 1000 can loop back to block 1004, where the VR environment can continue to be displayed until a user movement that satisfies the expansion criterion is detected.

[0093] At block 1008, the process 1000 can expand the dynamic boundary. For example, the dynamic boundary manager can expand the size of the dynamic boundary in response to the triggered expansion criterion. The dynamic boundary can be expanded based on a predefined metric, body measurements of the user (e.g., arm span, height, stride length, etc.), and / or a speed of the user’s movement. In some implementations, the dynamic boundary is formed around the user in a predefined size and a predefined shape, and expanding the size of the dynamic boundary includes scaling up the predefined size while maintaining the predefined shape.

[0094] In some implementations, the dynamic boundary is in the shape of a cylinder with a predefined height and radius. In some implementations, the predefined height and radius of the cylinder and the distance thresholds of the expansion criterion are such that the dynamic boundary is expanded in response to the user moving from sitting to standing or from standing to sitting.

[0095] At block 1010, process 1000 can display the perspective view with a first opacity. For example, the perspective view of the user’s real-world environment that is transitioned to in response to the expansion criteria being met includes the perspective view displayed with a first opacity (e.g., an opacity of 60%, 70%, 80%, etc.). In this example, triggering the expansion criteria can indicate to the user by the display transition that the user’s movement has triggered the dynamic boundary, however the dynamic boundary has not yet been fully triggered (e.g., popped up), so the user can maintain the virtual reality environment by staying within the dynamic boundary.

[0096] At block 1012, process 1000 can detect whether the user movement meets pop-up criteria. For example, the pop-up criteria can include one or more distance thresholds triggered by the user movement being very close to the expanded dynamic boundary or exceeding the expanded dynamic boundary; and / or a speed criterion triggered by the user movement meeting or exceeding a speed threshold. The user movement can be monitored to detect whether one or more pop-up distance thresholds and / or a pop-up speed threshold are met. When the user movement meets the pop-up criteria, process 1000 can proceed to process 1020 of FIG. 10. Figure 10B When the user movement does not meet the pop-up criteria, process 1000 proceeds to block 1014.

[0097] At block 1014, process 1000 can detect whether the user movement meets reform criteria. For example, the reform criteria can include a reform speed criterion triggered by the user movement meeting or falling below a reform speed threshold for a threshold duration of time. The dynamic boundary can be reformed in response to the user movement indicating that the user is stationary (e.g., below the reform speed threshold for the threshold duration of time). When the user movement meets the reform criteria, process 1000 can proceed to block 1016. When the user movement does not meet the reform criteria, process 1000 loops back to block 1010, where the perspective view can be displayed with the first opacity until the user movement meets the pop-up criteria and / or the reform criteria.

[0098] At block 1016, process 1000 can reform the dynamic boundary. For example, the dynamic boundary can be reformed at the time of reform relative to the user in response to the reform criteria being triggered. In some implementations, the reformed dynamic boundary can be generated with an initial size and grown to a predefined size associated with the automatically generated dynamic boundary. In some implementations, the dynamic boundary is reformed at the time of reform relative to a predefined portion of the user’s body.

[0099] At block 1018, process 1000 can display the VR environment to the user. For example, the XR system can transition from the perspective view of the user’s real-world environment displayed with the first opacity to the VR environment (e.g., provided by one or more applications being executed) in response to the reform criteria being triggered.

[0100] Figure 10B FIG. 10 is a flowchart illustrating a process 1020 for a pop-out state of a dynamic boundary in some embodiments of the technology. In some embodiments, the process 1020 can be performed by an XR system. In some embodiments, the process 1020 is triggered in response to user movement satisfying pop-out criteria for a dynamic boundary, e.g., at block 1012 of FIG. 10.

[0101] At block 1022, the process 1020 can pause one or more XR applications that are executing. For example, one or more applications that are executing at an XR system can provide a VR environment for display to a user. In response to user movement satisfying pop-out criteria, the one or more applications that are executing can be paused.

[0102] At block 1024, the process 1020 can display see-through visuals at a second opacity. For example, an XR system can transition from displaying see-through visuals at a first opacity to displaying see-through visuals at a second opacity (e.g., fade from 80% opacity to 100% opacity, etc.) in response to satisfying pop-out criteria.

[0103] In some embodiments, when an XR system performs one or more transitions from a VR environment to see-through visuals (e.g., at different levels of opacity) in response to satisfying expansion criteria and / or pop-out criteria, audio from one or more applications that are executing can be output to a user. Pausing one or more applications that are executing (as at block 1022) can also pause audio from the one or more applications that are executing, signaling to the user that the one or more applications have been paused.

[0104] At block 1026, the process 1020 can detect whether user movement satisfies reform criteria. For example, reform criteria can include a reform speed criterion triggered by user movement that satisfies or falls below a reform speed threshold for a threshold duration of time. A dynamic boundary can be reformed in response to user movement indicating that the user is stationary (e.g., below the reform speed threshold for the threshold duration of time). When user movement satisfies reform criteria, the process 1020 can proceed to block 1028. When user movement does not satisfy reform criteria, the process 1020 loops back to block 1024, where see-through visuals can be displayed at a second opacity until user movement satisfies reform criteria.

[0105] At block 1028, process 1020 can reform the dynamic boundary. For example, responsive to the triggered reform criteria, the dynamic boundary can be reformed relative to the user at the time of the reform. In some implementations, the reformed dynamic boundary can be generated at an initial size and grown to a predefined size associated with the automatically generated dynamic boundary. In some implementations, the dynamic boundary is reformed relative to: a) a predefined portion of the user's body at the time of the reform, or b) a combination of the predefined portion of the user's body at the time of the reform and a location of the dynamic boundary prior to being popped (e.g., prior to suspending the one or more applications being executed at block 1022).

[0106] At block 1030, process 1020 can resume the suspended one or more XR applications. For example, the one or more applications providing the VR environment can be suspended responsive to the user movement satisfying the pop criteria (e.g., the dynamic boundary can be popped, thereby causing the one or more applications being executed to be suspended). Responsive to the user movement satisfying the reform criteria, the suspended one or more applications can be resumed.

[0107] At block 1032, process 1020 can display the VR environment. For example, the XR system can transition from the perspective view of the user's real-world environment displayed at the second opacity to the VR environment provided by the resumed one or more applications responsive to the triggered reform criteria.

[0108] Several implementations of the disclosed technology are described above with reference to the accompanying drawings. Computing devices on which the technology can be implemented can include one or more central processing units, memories, input devices (e.g., keyboards and pointing devices), output devices (e.g., display devices), storage devices (e.g., disk drives), and network devices (e.g., network interfaces). The memory and storage devices are computer-readable storage media that can store instructions that implement at least portions of the technology. In addition, the data structures and message structures can be stored or transmitted via a data transmission medium, e.g., a signal on a communication link. Various communications links can be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. Thus, computer readable media can include computer-readable storage media (e.g., "non-transitory" media) and computer-readable transmission media.

[0109] In this specification, references to “an implementation” (e.g., “some implementations,” “various implementations,” “one implementation,” “an implementation,” etc.) mean that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation of the disclosure. The appearances of the phrases in various places in the specification are not necessarily all referring to the same implementation, or to mutually exclusive implementations or alternative implementations. Furthermore, various features that are described can be present in some implementations and not in others. Similarly, various requirements that are described can be requirements for some implementations and not for others.

[0110] As used herein, above threshold means that the value of the comparison item is higher than a specified other value, the comparison item is among a specified number of items with the highest values, or the value of the comparison item is within a specified top percentage of values. As used herein, below threshold means that the value of the comparison item is lower than a specified other value, the comparison item is among a specified number of items with the lowest values, or the value of the comparison item is within a specified bottom percentage of values. As used herein, within threshold means that the value of the comparison item is between two specified other values, the comparison item is among an intermediate specified number of items, or the value of the comparison item is within an intermediate specified percentage range. When not otherwise defined, relative terms such as high or insignificant can be understood to assign a value and determine how that value compares to an established threshold. For example, the phrase “select a fast connection” can be understood to mean selecting a connection with a value assigned corresponding to a connection speed that is above a threshold.

[0111] As used herein, the word “or” means any possible arrangement of items in a set. For example, the phrase “A, B, or C” means at least one of A, B, C, or any combination thereof, for example, any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiple any items, for example, A and A; B, B, and C; A, A, B, C, and C; etc.

[0112] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Specific examples and implementations have been described herein for illustrative purposes, and various modifications are possible without departing from the scope of the examples and implementations. The specific features and acts are disclosed as example forms of implementing the appended claims. Therefore, the examples and implementations are not limited to the specific features and acts described, except as limited by the claims.

[0113] Any patents, patent applications, and other references noted above are incorporated herein by reference. Modifications can be made to various aspects of the disclosure to adopt systems, functions, and concepts described in the various references above to provide yet further embodiments. To the extent statements or subject matter in the documents incorporated by reference conflict with the statements or subject matter of the present disclosure, the present disclosure controls.

Claims

1. A method for triggering an operational change at an artificial reality (XR) system via a dynamic boundary, the method comprising: displaying a VR environment to a user via the XR system operating in a stationary mode, wherein the dynamic boundary is automatically formed relative to the user of the XR system; In response to a user movement that satisfies the expansion criteria of the dynamic boundary: expanding the size of the dynamic boundary, and fading the display of the XR system from the VR environment to a see-through view of the user's real-world environment at a first opacity, wherein the extended criteria include one or more distance thresholds; and After the expansion, in response to a user movement that satisfies the popup criteria for the dynamic boundary: Pause the running VR application, and fading the display of the XR system from the see-through view at the first opacity to a see-through view of the user's real-world environment at a second opacity, The pop-up criteria include a speed threshold and / or one or more pop-up distance thresholds.

2. The method according to claim 1, wherein The dynamic boundary is formed around the user in a predefined size and a predefined shape, and extending the size of the dynamic boundary includes enlarging the predefined size while maintaining the predefined shape.

3. The method according to claim 2, wherein: The dynamic boundary is formed relative to the position of a predefined portion of the user's body.

4. The method according to claim 3, wherein: The predefined portion of the user's body includes the user's head.

5. The method according to claim 2, wherein: The dynamic boundary comprises a cylinder having a predefined height and radius.

6. The method according to claim 5, wherein: The predefined height and radius of the cylinder and the distance threshold of the expansion criterion cause the dynamic boundary to expand in response to the user moving from sitting to standing or from standing to sitting.

7. The method according to claim 1, wherein: The XR system monitors the user's movement through one or more sensors. When user movement is detected within one or more distance thresholds relative to the dynamic boundary, the user movement satisfies the expansion criteria, and The user movement satisfies the pop-up criteria when: the user movement is within the one or more pop-up distance thresholds relative to the dynamic boundary; and / or the user movement meets or exceeds the speed threshold.

8. The method according to claim 1, further comprising: After performing the expansion and pausing the VR application, in response to a user movement that satisfies the reshaping criteria of the dynamic boundary: Resuming the suspended VR application; fading the display of the XR system from the see-through view at the second opacity to the VR environment; and reshaping the dynamic boundary relative to the user, The reforming criterion includes a reforming speed threshold.

9. The method according to claim 8, wherein The dynamic boundary is automatically formed around the user in a predefined size and a predefined shape, and the reformed dynamic boundary includes the predefined size and the predefined shape.

10. The method according to claim 9, wherein: The dynamic boundary is reformed relative to: a predefined portion of the user's body at the time of reformation; or a combination of the predefined portion of the user's body at the time of reformation and the position of the dynamic boundary before pausing the executing VR application.

11. The method according to claim 8, wherein: The XR system monitors the user's movement through one or more sensors. When user movement is detected within the one or more distance thresholds relative to the dynamic boundary, the user movement satisfies the expansion criterion, The user movement satisfies the pop-up criteria if: the user movement is detected within the one or more pop-up distance thresholds relative to the dynamic boundary; and / or the user movement is detected meeting or exceeding the speed threshold; and The user movement satisfies the reforming criterion when, after the dynamic boundary is extended and the VR application is paused, user movement that meets or falls below the reforming speed threshold is detected within a threshold duration.

12. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for triggering an operational change at an artificial reality (XR) system via a dynamic boundary, the process comprising: displaying a VR environment to a user via the XR system operating in a stationary mode, wherein the dynamic boundary is automatically formed relative to the user of the XR system; In response to a user movement that satisfies the expansion criteria of the dynamic boundary: expanding the size of the dynamic boundary, and fading the display of the XR system from the VR environment to a see-through view of the user's real-world environment at a first opacity, wherein the extended criteria include one or more distance thresholds; and After the expansion, in response to a user movement that satisfies the popup criteria for the dynamic boundary: Pause the running VR application, and fading the display of the XR system from the see-through view at the first opacity to a see-through view of the user's real-world environment at a second opacity, The pop-up criteria include a speed threshold and / or one or more pop-up distance thresholds.

13. The computer-readable storage medium of claim 12, wherein: The dynamic boundary is formed around the user in a predefined size and a predefined shape, and extending the size of the dynamic boundary includes enlarging the predefined size while maintaining the predefined shape.

14. The computer-readable storage medium of claim 13, wherein: The dynamic boundary is formed relative to the position of a predefined portion of the user's body.

15. The computer-readable storage medium of claim 12, wherein: The dynamic boundary comprises a cylinder having a predefined height and radius.

16. The computer-readable storage medium of claim 15, wherein: The predefined height and radius of the cylinder and the distance threshold of the expansion criterion cause the dynamic boundary to expand in response to the user moving from sitting to standing or from standing to sitting.

17. The computer-readable storage medium of claim 12, wherein: The XR system monitors the user's movement through one or more sensors. When user movement is detected within one or more distance thresholds relative to the dynamic boundary, the user movement satisfies the expansion criteria, and The user movement satisfies the pop-up criteria when: the user movement is within one or more pop-up distance thresholds relative to the dynamic boundary; and / or the user movement meets or exceeds the speed threshold.

18. The computer-readable storage medium of claim 12, wherein: The process also includes: After performing the expansion and pausing the VR application, in response to a user movement that satisfies the reshaping criteria of the dynamic boundary: Resuming the suspended VR application; fading the display of the XR system from the see-through view at the second opacity to the VR environment; and reshaping the dynamic boundary relative to the user, The reforming criterion includes a reforming speed threshold.

19. An artificial reality (XR) system that triggers operational changes via dynamic boundaries, the XR system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: Displaying the VR environment to the user; Software is executed at the XR system using the dynamic boundary, wherein: The dynamic boundary includes an expansion standard, a pop-up standard and a reformation standard; the expansion criteria comprising one or more distance thresholds relative to the dynamic boundary, and the dynamic boundary transitioning to an expanded state in response to user movement satisfying the expansion criteria; The pop-up criteria include one or more distance thresholds and / or speed criteria, and when the dynamic boundary is in the extended state, in response to a user movement that satisfies the pop-up criteria, the dynamic boundary transitions to the pop-up state, and the executing VR application is paused; and The reformatting criteria include speed criteria, and: a) while the dynamic boundary is in the expanded state, in response to a user movement that satisfies the reshaping criterion, The dynamic boundary transitions to a reforming state, and b) while the dynamic boundary is in the pop-up state, in response to a user movement that satisfies the reshaping criteria, The dynamic boundary transitions to a reforming state, and the suspended VR application is resumed.

20. The system of claim 19, wherein: In response to user movement that satisfies the extended criteria, the XR system transitions from displaying the VR environment to displaying see-through vision at a first opacity, When the dynamic boundary is in the expanded state, in response to a user movement that satisfies the pop-up criteria, the XR system transitions from displaying the see-through visual at the first opacity to displaying the see-through visual at a second opacity, While the dynamic boundary is in the expanded state, in response to user movement that satisfies the reshaping criteria, the XR system transitions from displaying the see-through visuals at the first opacity to displaying the VR environment, and While the dynamic boundary is in the pop-up state, in response to user movement that meets the reformatting criteria, the XR system transitions from displaying the see-through visuals at the second opacity to displaying the VR environment.

Images