Automatic boundary creation and relocation

Abstract

The invention discloses automatic boundary creation and relocation. Aspects of the present disclosure relate to instant boundary creation for virtual reality (VR) experiences. An artificial reality (XR) system may scan and collect visual feature data of a user's real-world environment in the background while rendering an augmented reality (AR) environment or a mixed reality (MR) environment. When the XR system detects an intent to enter a VR mode (e.g., by launching a VR application), the XR system may generate a recommendation to a boundary of a real-world space, which may include a type of interaction mode (e.g., a movable mode or a stationary mode). Based on the user's response to the recommendation, the XR system may prompt the user to further scan the real-world space and / or manually adjust the boundaries while continuing to scan and collect visual feature data in the background. Alternatively or additionally, some embodiments may automatically relocate the real-world space based on the generated boundaries.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Non-Provisional Patent Application No. 18 / 629,769, filed April 8, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to automatic boundary creation and localization for rendering of an XR environment by artificial reality (super reality (XR)) systems. BACKGROUND

[0004] Artificial reality (XR) devices are becoming increasingly common. As they become more popular, the applications implemented on such devices are becoming more complex. Mixed reality (MR) and augmented reality (AR) applications can provide interactive three-dimensional (3D) experiences that combine images of the real world with virtual objects, while virtual reality (VR) applications can provide entirely independent 3D computer environments. For example, an MR or AR application can be used to overlay virtual objects on a real scene viewed through a camera. Real-world users in the scene can then make gestures that are captured by the camera, which can provide for interaction between the real-world users and the virtual objects. Users can view AR, MR, and VR (collectively, XR) experiences through a head-mounted display (HMD), such as glasses or a headset. The HMD can have a see-through display that allows light from the real world to pass through the lens in combination with light from a waveguide that is simultaneously emitting light from a projector in the HMD, allowing the HMD to present virtual objects that are mixed together with real objects that the user can actually see. SUMMARY

[0005] Aspects of the disclosure relate to instant boundary creation for virtual reality (VR) experiences. Artificial reality (XR) systems, when rendering an augmented reality (AR) environment or a mixed reality (MR) environment, which do not require a boundary, can scan and collect visual feature data of a user’s real-world environment in the background. When the XR system detects an intent to enter VR mode, e.g., by launching a VR application, the XR system can generate a recommendation for a boundary of the real-world space, which can include a type of interaction mode (e.g., a movable mode or a stationary mode). Based on the user’s response to the recommendation, the XR system can prompt the user to further scan the real-world space and / or manually adjust the boundary, while continuing to scan and collect visual feature data in the background. However, in some implementations, the initial background scan can be sufficient to establish a boundary, and the XR system can render the VR mode without further scanning, thereby not requiring a rendered manual scene capture.

[0006] Aspects of the disclosure also relate to XR system repositioning in a real-world space. When entering an XR experience, automatic repositioning of the XR system can fail if the user is in a new room or if the XR system cannot identify the room. Accordingly, the XR system can prompt the user to look around the room, thereby generating a mesh that can be compared to an existing room mesh. If the mesh matches the existing room mesh, the XR system can align the meshes, extract existing spatial anchors and scene data, and resume from where the user stopped in a previous XR experience. If the mesh does not match the existing room mesh, the XR system can use the newly scanned mesh to establish a new room to execute the XR experience. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a block diagram showing an overview of a number of devices on which some implementations of the technology can operate.

[0008] Figure 2A is a line drawing showing a virtual reality headset that can be used in some implementations of the technology.

[0009] Figure 2B is a line drawing showing a mixed reality headset that can be used in some implementations of the technology.

[0010] Figure 2C is a line drawing showing a controller that can be held in one or both hands by a user to interact with an artificial reality environment in some implementations.

[0011] Figure 3 is a block diagram showing an overview of an environment in which some implementations of the technology can operate.

[0012] Figure 4is a block diagram illustrating a number of components that can be used in systems employing the disclosed technology in some implementations.

[0013] Figure 5 is a flowchart illustrating a process for automatically generating boundaries for a virtual reality (VR) experience in some implementations of the technology.

[0014] Figure 6A is a conceptual diagram illustrating an example view of an XR experience rendered on an XR system while scanning a real-world space around the XR system.

[0015] Figure 6B is a conceptual diagram illustrating an example view of options to launch a VR experience on an XR system.

[0016] Figure 6C is a conceptual diagram illustrating an example view of a recommendation of an interaction mode for a VR experience on an XR system.

[0017] Figure 6D is a conceptual diagram illustrating an example view of boundaries for a VR experience generated on an XR system while rendering the XR experience.

[0018] Figure 6E is a conceptual diagram illustrating an example view of a VR experience rendered on an XR system in relation to boundaries generated by the XR system.

[0019] Figure 7 is a flowchart illustrating a process for positioning an XR system in a real-world space in some implementations of the technology.

[0020] Figure 8A is a conceptual diagram illustrating an example view of a mesh of a real-world space generated on an XR system while a user looks around the real-world space.

[0021] Figure 8B is a conceptual diagram illustrating an example view of a complete mesh of a real-world space on an XR system in which the XR system has been positioned or repositioned.

[0022] Figure 8C is a conceptual diagram illustrating an example view of an XR experience rendered on an XR system relative to a positioned or repositioned real-world space.

[0023] The technology presented herein can be better understood with reference to the following detailed description together with the accompanying drawings, in which like reference numerals refer to similar elements or functionally similar elements throughout the drawings. DETAILED DESCRIPTION

[0024] Aspects of the disclosure relate to instant boundary creation for virtual reality (VR) experiences. Artificial reality (XR) systems, when rendering an augmented reality (AR) environment or a mixed reality (MR) environment, which do not require a boundary, can scan and collect visual feature data of a user’s real-world environment in the background. When the XR system detects an intent to enter VR mode, for example by launching a VR application, the XR system can generate a recommendation for a boundary of the real-world space, which can include a type of interaction mode (e.g., a movable mode or a stationary mode). Based on the user’s response to the recommendation, the XR system can prompt the user to further scan the real-world space and / or manually adjust the boundary, while continuing to scan and collect visual feature data in the background. In some implementations, however, the initial background scan can be sufficient to establish a boundary, and the XR system can render the VR mode without further scanning, thereby eliminating the need to render a manual scene capture.

[0025] Aspects of the disclosure also relate to XR system repositioning in a real-world space. When entering an XR experience, automatic repositioning of the XR system can fail if the user is in a new room or if the XR system is unable to recognize the room. Accordingly, the XR system can prompt the user to look around the room, thereby generating a mesh that can be compared to existing room meshes. If the mesh matches an existing room mesh, the XR system can align the meshes, extract existing spatial anchors and scene data, and resume from where the user stopped in a previous XR experience. If the mesh does not match an existing room mesh, the XR system can use the newly scanned mesh to establish a new room in which to execute the XR experience.

[0026] Embodiments of the disclosed technology can include or can be implemented in an artificial reality system. Artificial reality or a virtual reality (VR) is a form of reality that has been adjusted in some manner before presentation to a user, which might be for purposes of either simulation or simulation as in simulations of real world environments. Artificial reality content might also be generated by a user's own content, such as using a head-mounted display (HMD) with video capture functionality. Artificial reality content can include completely generated content or generated content combined with captured content (such as a real-world video), for example. The artificial reality content can include video, audio, haptic feedback, or some combination of these, any of which can be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality can also be associated with applications, products, accessories, services, or some combination of these. The artificial reality content can be generated by one or more artificial reality systems. The artificial reality system that provides the artificial reality content can be on a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a "cave" environment, or other projection systems, or any other hardware platform that can provide artificial reality content to one or more viewers.

[0027] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet computer with a camera on the back can capture real-world images and then display those images on a screen on the side of the tablet opposite the camera. The tablet computer can process and adjust or "augment" the images as they pass through the system, for example by adding virtual objects. "Mixed reality" or "MR" refers to a system in which light into a user's eye is generated in part by a computing system and in part by light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped like a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide, which simultaneously emits light from a projector in the MR head-mounted viewer, allowing the MR head-mounted viewer to present virtual objects that are mixed with real objects that the user can see. As used herein, "artificial reality," "extra reality," or "XR" refers to any of VR; AR; MR; or any combination or mixture of these.

[0028] The implementations described herein provide technical improvements in the field of artificial reality. According to some implementations, an XR system, while in an AR or MR mode, can scan a real-world space “in the background” to generate a boundary that can be used to seamlessly transition to a VR mode. In some implementations, the XR system can automatically attempt to reposition the XR system in a real-world space that the XR system initially fails to recognize by generating a mesh of the real-world space and comparing the generated mesh to existing, stored meshes of known real-world spaces. If the generated mesh matches a stored mesh, the XR system can retrieve the stored mesh (and any associated data such as spatial anchor data and / or scene data) and render an XR experience relative to the stored mesh. If the generated mesh does not match a stored mesh, the XR system can establish the real-world space as a “new room” using the generated mesh and render the XR experience relative to the generated mesh. Accordingly, the implementations described herein can reduce latency in rendering an XR experience, thereby improving latency and overall user experience by minimizing or eliminating manual scene gathering and setup for artificial reality.

[0029] Several implementations are discussed in greater detail below with reference to the drawings. Figure 1 is a block diagram that illustrates a generalized architecture for a plurality of devices on which a number of implementations of the disclosed technology can run. The plurality of devices can include a plurality of hardware components of a computing system 100 that can automatically generate a boundary for a virtual reality (VR) experience and / or position an artificial reality (XR) system in a real-world space. In various implementations, the computing system 100 can include a single computing device 103 or a plurality of computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate over wired or wireless channels to distribute processing and share input data. In some implementations, the computing system 100 can include a standalone head-mounted viewer that is capable of providing a computer-created or augmented experience for a user without the need for external processing or external sensors. In other implementations, the computing system 100 can include a plurality of computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), where some processing operations are performed on the head-mounted viewer and other processing operations are offloaded to the core processing component. The following discussion is presented to enable those skilled in the art to make and use embodiments of the technology, and is provided in the context of a computer program Figure 2A and Figure 2BAn example head-mounted viewer is described. In some implementations, the location data and environmental data can be collected only by sensors contained in the head-mounted viewer device, while in other implementations one or more of the plurality of non-head-mounted viewer computing devices can include sensor components that can track environmental data or location data.

[0030] Computing system 100 can include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). Processors 110 can be a single processing unit or multiple processing units that are located in one device or distributed across multiple devices (e.g., distributed across two or more of computing devices 101-103).

[0031] Computing system 100 can include one or more input devices 120 that provide input to processors 110, thereby informing these processors of actions. These actions can be communicated by a hardware controller that interprets received signals from the input device and communicates information to processors 110 using a communication protocol. For example, individual input devices 120 can include a mouse, a keyboard, a touchscreen, a touchpad, a wearable input device (e.g., a haptic glove, bracelet, ring, earring, necklace, watch, etc.), a camera (or other light-based input device such as an infrared sensor), a microphone, or other user input device.

[0032] The processor 110 can be coupled to other hardware devices, for example, through the use of an internal bus, an external bus (e.g., a peripheral component interconnect (PCI) bus, a small computer system interface (SCSI) bus), or a wireless connection. The processor 110 can communicate with hardware controllers of devices such as the display 130. The display 130 can be used to display text and graphics. In some implementations, the display 130 includes an input device, for example, when the input device is a touchscreen or is equipped with an eye direction monitoring system as part of the display. In some implementations, the display is separate from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen; a light-emitting diode (LED) display screen; a projection, holographic, or augmented reality display (e.g., a heads-up display device or a head-mounted device); and the like. Other input / output (I / O) devices 140 can also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), Firewire, or other external device, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, and the like.

[0033] In some implementations, the computing system 100 can use input from the I / O devices 140 (e.g., a camera, a depth sensor, an IMU sensor, a GPS unit, a lidar (LiDAR) or other time-of-flight sensor, and the like) to identify and map a user’s physical environment while tracking the user’s location in that environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., a topology, a grid, and the like) of an area (which can be a room, a building, an outdoor space, and the like) and / or obtain a map previously generated by the computing system 100 or another computing system that has mapped the area. The SLAM system can track the user within the area based on factors such as GPS data, match identified objects and structures to mapped objects and structures, monitor acceleration and other location changes, and the like.

[0034] The computing system 100 can include a communication device that enables wireless or wired communication between the computing system 100 and other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using the transmission control protocol / Internet protocol (TCP / IP). The computing system 100 can utilize the communication device to distribute operations among multiple network devices.

[0035] The processor 110 can access memory 150, which can be included on one of the multiple computing devices of the computing system 100 or can be distributed across multiple computing devices of the computing system 100 or multiple other external devices. The memory includes one or more hardware devices for volatile or non-volatile storage and can include both read-only memory and writable memory. For example, the memory can include one or more of random access memory (RAM); various types of cache memory; CPU register; read-only memory (ROM); and writable non-volatile memory such as flash memory, hard drives, floppy disks, compact discs (CDs), DVDs, magnetic storage devices, tape drives, and the like. The memory is not a propagating signal away from the underlying hardware; thus the memory is non-transitory. The memory 150 can include program memory 160 storing programs and software such as an operating system 162, a boundary creation and repositioning system 164, and other application programs 166. The memory 150 can also include data memory 170, which can include, for example, rendering data, virtual object data, XR environment / experience data, mesh data, visual feature data, real-world space data, scan data, trigger data, boundary data, threshold data, spatial anchor data, scene data, configuration data, settings, user options or preferences, and the like, which can be provided to the program memory 160 or any element of the computing system 100.

[0036] Some implementations can be run with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that can be suitable for use with the technology include, but are not limited to, XR head-mounted viewers, personal computers, server computers, hand-held or laptop devices, cellular telephones, wearable electronic devices, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, or the like.

[0037] Figure 2Ais a line drawing of a virtual reality head mounted display (HMD) 200 according to some embodiments. In this example, the virtual reality HMD 200 also includes augmented reality features, using see-through cameras 225 to render portions of the real world that can have computer-generated overlays. The virtual reality HMD 200 includes a front rigid body 205 and a band 210. The front rigid body 205 includes one or more electronic display elements of one or more electronic displays 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a plurality of cameras and locators 225, and one or more computing units 230. The position sensors 220, the IMU 215, and the computing units 230 can be located inside the virtual reality HMD 200 and can not be visible to the user. In various implementations, the IMU 215, the position sensors 220, the cameras and locators 225 can track movement and position of the virtual reality HMD 200 in the real world and in an artificial reality environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locators 225 can emit infrared light beams that create light points on real objects around the virtual reality HMD 200, and / or the cameras 225 capture images of the real world and locate the virtual reality HMD 200 within the real world environment. As another example, the IMU 215 can include one or more of the following components that can be used in the localization process: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras 225 integrated with the virtual reality HMD 200 can detect the light points described above. The computing units 230 in the virtual reality HMD 200 can use the detected light points and / or position points to extrapolate the position and movement of the virtual reality HMD 200, and to identify shapes and positions of real objects around the virtual reality HMD 200.

[0038] One or more electronic displays 245 can be integrated with the front rigid body 205 and can provide image light to a user as instructed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of electronic displays 245 include liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, active-matrix organic light-emitting diode displays (AMOLEDs), displays including one or more quantum dot light-emitting diode (QOLED) sub-pixels, projector units (e.g., micro-LEDs, lasers (LASERs), etc.), some other display, or some combination thereof.

[0039] In some implementations, the virtual reality HMD 200 can be coupled to a core processing component, such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the virtual reality HMD 200 (e.g., by emitting light from the virtual reality HMD 200), and the PC can use the HMD in conjunction with output from the IMU 215 and the position sensor 220 to determine the position and movement of the virtual reality HMD 200.

[0040] Figure 2B is a line diagram of a mixed reality HMD system 250 that includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 gigahertz (GHz) link) as indicated by link 256. In other implementations, the mixed reality HMD system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye tracking sensors, microelectromechanical systems (MEMS) components, networking components, etc.

[0041] The projector can be coupled to the see-through display 258, e.g., via optical elements, to display media to the user. These optical elements can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. to direct light from the projector to the user's eyes. Image data from the core processing component 254 can be transmitted to the mixed reality HMD 252 via the link 256. A controller in the mixed reality HMD 252 can convert the image data to light pulses from the projector, which can be transmitted as output light to the user's eyes via the optical elements. This output light can mix with light passing through the see-through display 258, allowing the output light to appear to present virtual objects that appear to exist in the real world.

[0042] Like the virtual reality HMD 200, the mixed reality HMD system 250 can also include motion and position tracking units, cameras, light sources, etc. that allow the mixed reality HMD system 250 to track itself, e.g., in 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), render virtual objects to appear to be stationary as the mixed reality HMD 252 moves, and make virtual objects react to hand gestures and other real world objects.

[0043] Figure 2C Controllers 270 (including controller 276A and controller 276B) are shown that a user can hold in one or both hands to interact with the artificial reality environment presented by the virtual reality HMD 200 and / or the mixed reality HMD system 250 in some implementations. The controllers 270 can communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit additional light points. The virtual reality HMD 200 or the mixed reality HMD system 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., to track the controllers in 3DoF or 6DoF). The computing unit 230 in the virtual reality HMD 200 or the core processing component 254 can use this tracking in conjunction with IMU output and position output to monitor the user's hand positions and hand motions. The controllers can also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A and 274B) that the user can actuate to provide input and interact with objects.

[0044] In various implementations, virtual reality HMD 200 or mixed reality HMD system 250 can also include additional subsystems (e.g., eye tracking units, audio systems, various networking components, etc.) to monitor indications of user interaction and intent. For example, in some implementations, one or more cameras included in virtual reality HMD 200 or mixed reality HMD system 250 or from a plurality of external cameras can monitor the position and pose of a user's hands, in lieu of or in addition to controllers, to determine hand gestures and other hand and body motions. As another example, one or more light sources can illuminate a user's eye or eyes, and virtual reality HMD 200 or mixed reality HMD system 250 can use an eye-facing camera to capture reflections of the light to determine eye position (e.g., based on a set of reflections around a user's cornea), model the user's eyes, and determine a gaze direction.

[0045] Figure 3 FIG. 1 is a block diagram illustrating a computing system 100 in accordance with some embodiments of the disclosed technology. Computing system 100 can be used to implement any of the client computing devices 305A-305D, server computing devices 310 and 320, and other computing devices described herein. As shown, computing system 100 includes a communication mechanism such as a bus 101 or other communication circuitry for communicating information and

[0046] In some implementations, server computing device 310 can be an edge server that receives a plurality of client requests and coordinates fulfillment of the requests through other servers (e.g., server computing devices 320A-320C). Server computing devices 310 and 320 can include computing systems such as computing system 100. Although each server computing device 310 and 320 is shown logically as a single server, each can be a distributed computing environment comprising multiple computing devices located at the same physical location or at geographically different locations.

[0047] Client computing device 305, as well as server computing devices 310 and 320, can each act as a server or a client to other servers / one or more client devices. Server computing device 310 can be connected to database 315. Server computing devices 320A through 320C can each be connected to a corresponding database 325A through 325C. As described above, each server computing device 310 or 320 can correspond to a group of servers, and individual ones of these server computing devices can share a database, or can have its own database. Although databases 315 and 325 are shown logically as single units, databases 315 and 325 can each be a distributed computing environment that includes multiple computing devices, can be located within its corresponding server, or can be located at the same physical location or at geographically different physical locations.

[0048] Network 330 can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 can be the Internet or some other public or private network. Client computing device 305 can connect to network 330 via a network interface, such as through wired or wireless communication. Although the connections between server computing device 310 and multiple server computing devices 320 are shown as separate connections, these connections can be any type of local area, wide area, wired, or wireless network, including network 330 or a separate public or private network.

[0049] Figure 4is a block diagram illustrating a plurality of components 400 that can be used in systems employing the disclosed technology, in some implementations. The plurality of components 400 can be included in one device of the computing system 100, or can be distributed across multiple devices of the computing system 100. The plurality of components 400 includes a plurality of hardware 410, middleware 420, and a plurality of specialized components 430. As described above, systems implementing the disclosed technology can use a variety of hardware, including processing units 412, working memory 414, input and output devices 416 (e.g., cameras, displays, IMU units, network connections, etc.), and storage memory 418. In various implementations, the storage memory 418 can be one or more of: a local device; an interface to a remote storage device; or a combination thereof. For example, the storage memory 418 can be one or more hard disk drives or flash drives accessible over a system bus, or can be a cloud storage provider (e.g., located in the databases 315 or 325) or other network storage device accessible over one or more communication networks. In various implementations, the plurality of components 400 can be implemented in a client computing device (e.g., the client computing device 305), or on a server computing device (e.g., the server computing device 310 or 320).

[0050] The middleware 420 can include components that transfer resources between the hardware 410 and the plurality of specialized components 430. For example, the middleware 420 can include an operating system, services, drivers, a basic input output system (BIOS), controller circuitry, or other hardware or software systems.

[0051] The plurality of specialized components 430 may include software or hardware configured to perform operations for generating a boundary for a virtual reality (VR) experience and / or positioning an artificial reality (XR) system in real-world space. The plurality of specialized components 430 may include an XR environment rendering module 434; a real-world space scanning module 436; a trigger detection module 438; a boundary generation module 440; a VR experience execution module 442; a mesh generation module 444; a mesh comparison module 446; a spatial data retrieval module 448; a mesh completion module 450; and components and application programming interfaces (APIs) that may be used to provide a user interface, transfer data, and control the plurality of specialized components, such as interface 432. In some embodiments, the plurality of components 430 may reside in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the plurality of specialized components 430. Although the plurality of specialized components 430 are depicted as separate components, the plurality of specialized components 430 may be logical functions or other non-physically distinct functions, and / or may be submodules or code blocks of one or more applications.

[0052] The XR environment rendering module 434 can render the XR environment on the XR system. In some embodiments, such as in augmented reality (AR) or mixed reality (MR), the XR environment rendering module 434 can render one or more virtual objects overlaid on a view of the real world space. In some embodiments, the XR environment rendering module 434 can render the XR environment relative to boundaries established for the real world space (e.g., a grid (further described herein), one or more spatial anchors, scene data, etc.). Figure 5 502, and Figure 7 Blocks 710 and 714 describe further details regarding rendering the XR environment.

[0053] The real-world space scanning module 436 can scan the real-world space around the XR system to automatically detect features of the real-world space. The real-world space scanning module 436 can scan the real-world space while the XR system is moving (e.g., while the user of the XR system is moving around in the real-world space and / or looking around the real-world space). The real-world space scanning module 436 can scan the real-world space using, for example, one or more cameras, one or more depth sensors, or any combination thereof that may be included in the input / output device 416. Figure 5 Box 504 and Figure 7 Block 702 describes further details regarding scanning the real-world space around the XR system.

[0054] In some implementations, the trigger detection module 438 can detect a trigger that indicates an intent to execute a VR experience on the XR system. In some implementations, the trigger can be a user-initiated, XR application-initiated, or system-initiated request to launch a VR experience. For example, the user can select a virtual button from a virtual menu that corresponds to launching a VR experience. In another example, the XR application can request to switch from an AR mode or MR mode to a VR mode. Further details regarding detecting a trigger that indicates an intent to execute a VR experience are described herein with reference to block 506 of FIG. 5. Figure 5 Further details regarding generating a boundary for a real-world space are described herein with reference to block 508 of FIG. 5.

[0055] In some implementations, the boundary generation module 440 can generate a boundary for the scanned real-world space based on the detected features of the real-world space scanned by the real-world space scanning module 436. In some implementations, the boundary can be a “guardian.” As used herein, a “guardian” can be an XR use space defined in the real-world environment. If a user wearing the XR system crosses the boundary while accessing an XR experience, one or more system actions or restrictions can be triggered on the XR system. For example, the XR system can display a warning message on the XR system, can activate at least partial see-through on the XR system, can display a boundary on the XR system, can pause rendering or updating of the XR environment, etc., as further described herein. In some implementations, the boundary can be a “mesh” for the real-world space. In such implementations, it is contemplated that the boundary generation module 440 and the mesh generation module 444 can perform similar functions, and then one or the other of these two modules can be omitted from the specialized components 430. In some implementations, the boundary generation module 440 can also update the generated boundary based on further scans performed by the real-world space scanning module 436, e.g., upon determining that the generated boundary covers less than a threshold amount of the real-world space, as further described herein. Further details regarding generating a boundary for a real-world space are described herein with reference to block 508 of FIG. 5. Figure 5 Further details regarding generating a boundary for a real-world space are described herein with reference to block 508 of FIG. 5.

[0056] In some implementations, the VR experience execution module 442 can execute a VR experience for the real-world space relative to the boundary generated by the boundary generation module 440. For example, the VR experience execution module 442 can render the VR experience as a fully immersive, computer-generated artificial environment that occupies the entire field of view of the XR system. However, if a user wearing the XR system approaches the generated boundary while the VR experience is being rendered, the VR experience rendering module can, for example, stop rendering the VR experience, display the boundary overlaid on the VR experience, display a warning, etc., as described above and further herein.

[0057] In some implementations, the mesh generation module 444 can generate a mesh corresponding to a real-world space scanned by the real-world space scanning module 436. The mesh, for example, can be a three-dimensional (3D) model of the boundaries of the real-world space including one or more walls, a ceiling, a floor, one or more physical objects, etc. In some implementations, the mesh generation module 444 can generate the mesh using one or more cameras, one or more depth sensors, or any combination thereof, which can be included in the input / output device 416. However, in some implementations, it is contemplated that depth data need not be captured, but rather can be predicted from one or more images, e.g., through a machine learning model. In some implementations, the mesh generation module 444 can also perform post-processing on the mesh to refine and / or simplify the mesh, as further described herein. Further details regarding generating a mesh corresponding to a scanned real-world space are described herein with reference to block 508 of FIG. 5 and block 704 of FIG. 7. Figure 5 Further details regarding comparing a generated mesh to one or more stored meshes are described herein with reference to block 706 of FIG. 7. Figure 7 Further details regarding generating a mesh corresponding to a scanned real-world space are described herein with reference to block 508 of FIG. 5 and block 704 of FIG. 7.

[0058] In some implementations, the mesh comparison module 446 can compare the mesh generated by the mesh generation module 444 to one or more meshes stored corresponding to one or more previously scanned real-world spaces to determine whether the generated mesh matches the stored mesh above a threshold. For example, the mesh comparison module 446 can identify visual features (e.g., corners, edges, curves, etc.) in the generated mesh and attempt to align these visual features in the generated mesh to the stored one or more meshes. In some implementations, the mesh comparison module 446 can determine a percentage of the generated mesh that matches the stored mesh, e.g., 75% similarity, and determine that the meshes match if the threshold percentage is met. In some implementations, the mesh comparison module 446 can select the threshold by taking into account dynamic objects and / or movable objects that can be present in the real-world space. For example, in some implementations, the mesh comparison module 446 can identify stationary components of the real-world space, e.g., walls, floors, ceilings, large furniture, etc., and determine whether the mesh meets the threshold based only on these components. Further details regarding comparing a generated mesh to one or more stored meshes are described herein with reference to block 706 of FIG. 7. Figure 7 Further details regarding generating a mesh corresponding to a scanned real-world space are described herein with reference to block 508 of FIG. 5 and block 704 of FIG. 7.

[0059] In some implementations, when the mesh comparison module 446 determines that the generated mesh matches the stored mesh above a threshold, the spatial data retrieval module 448 can retrieve stored spatial anchor data, stored scene data, or both corresponding to the stored mesh. In some implementations, the stored mesh can have one or more spatial anchors associated with it. As used herein, a "spatial anchor" can be a designated, persistent location in a real-world space to which a virtual object can be persistently positioned relative to, as further defined herein. As used herein, "scene data" can be marker data for physical objects (e.g., a table, a chair, a sofa, etc.) in the identified real-world space that can be captured in a mesh and associated with a spatial anchor, as further defined herein. Further details regarding retrieving stored spatial anchor data, stored scene data, or both corresponding to the stored mesh for the real-world space are described herein with reference to block 708 of FIG. 7. Figure 7 Further details regarding determining whether the generated mesh is complete are described herein with reference to block 712 of FIG. 7.

[0060] In some implementations, the mesh completeness module 450 can determine whether the generated mesh is complete, e.g., whether the mesh generated by the mesh generation module 444 meets or exceeds a threshold. In some implementations, the threshold can be a percentage of the real-world space, e.g., 90% of the real-world space. In some implementations, the threshold can be defined by the XR application based on its requirements for rendering a particular XR experience (e.g., rendering all four walls, a ceiling, a floor, etc.). If the generated mesh is sufficiently complete, the mesh completeness module 450 can instruct the XR environment rendering module 434 to render the XR environment relative to the generated mesh. If the generated mesh is not sufficiently complete, the mesh completeness module 450 can instruct the real-world space scanning module 436 to further scan the real-world space, the mesh generation module 444 can update the generated mesh, and the mesh completeness module 450 can again determine whether the mesh is sufficiently complete. Further details regarding determining whether the generated mesh is complete are described herein with reference to block 712 of FIG. 7. Figure 7 Further details regarding determining whether the generated mesh is complete are described herein with reference to block 712 of FIG. 7.

[0061] Although the specialized components 430 are described herein as including all of the XR environment rendering module 434, the real-world space scanning module 436, the trigger detection module 438, the boundary generation module 440, the VR experience execution module 442, the mesh generation module 444, the mesh comparison module 446, the spatial data retrieval module 448, and the mesh completeness module 450, in some implementations, it is contemplated that one or more of these modules can be omitted. For example, as described above, when the boundary generation module 440 generates a mesh, it is contemplated that the boundary generation module 440 and the mesh generation module 444 can be combined. Further, to perform only XR experiences, it is contemplated that the VR experience execution module 442 can be omitted. Figure 5The dedicated component 430 can be configured to perform the process 500 of FIG. 5. It is contemplated that the dedicated component 430 need only include the XR environment rendering module 434, the real-world space scanning module 436, the trigger detection module 438, the boundary generation module 440, and the VR experience execution module 442, and omit the other modules, when performing the process 500 of FIG. 5. To perform only the process 500 of FIG. 5, the dedicated component 430 can be configured to receive a request to render an XR environment, to render the XR environment, to receive a request to generate a boundary for the XR environment, to generate the boundary for the XR environment, and to execute the VR experience within the boundary for the XR environment. Figure 7 The dedicated component 430 can be configured to perform the process 700 of FIG. 7. It is contemplated that the dedicated component 430 need only include the real-world space scanning module 436, the mesh generation module 444, the mesh comparison module 446, the mesh completion module 450, and the XR environment rendering module 434, and omit the remaining modules, when performing the process 700 of FIG. 7. Further, it is noted that the dedicated component 430 can optionally include the space data retrieval module 448 when performing the process 700 of FIG. 7. Figure 7 The dedicated component 430 can be configured to perform the process 700 of FIG. 7. It is contemplated that the dedicated component 430 need only include the real-world space scanning module 436, the mesh generation module 444, the mesh comparison module 446, the mesh completion module 450, and the XR environment rendering module 434, and omit the remaining modules, when performing the process 700 of FIG. 7. Further, it is noted that the dedicated component 430 can optionally include the space data retrieval module 448 when performing the process 700 of FIG. 7.

[0062] Those skilled in the art will realize that the components illustrated in the above-described Figures 1 to 4 and illustrated in each of the plurality of flowcharts discussed below can be changed in a variety of ways. For example, the order of the logic can be rearranged, multiple sub-steps can be performed in parallel, logic shown can be omitted, other logic can be included, and the like. In some embodiments, one or more of the components described above can perform one or more of the processes described below.

[0063] Figure 5 is a flowchart illustrating a process 500 for generating a boundary for a virtual reality (VR) experience in some embodiments. In some embodiments, the process 500 can be performed when rendering an artificial reality (XR) environment, such as a mixed reality (MR) or an augmented reality (AR) environment. The process 500 can render the XR environment based on, for example, a user request, an application-level request, and / or a system-level request. In some embodiments, the process 500 can automatically render the XR environment, for example, upon activation or donning of an XR system, which in some embodiments can resume from where the user left the XR environment in a previous session.

[0064] In some embodiments, the process 500 can be performed by an XR system including one or more XR devices, such as an XR head-mounted display (HMD) (e.g., the virtual reality HMD 200 of FIG. 2 and / or the mixed reality HMD 252 of FIG. 2), one or more external processing components, one or more controllers (e.g., the controller 254 of FIG. 2), and / or the like. Figure 2A Figure 2B In some embodiments, the process 500 can be performed by an XR system including one or more XR devices, such as an XR head-mounted display (HMD) (e.g., the virtual reality HMD 200 of FIG. 2 and / or the mixed reality HMD 252 of FIG. 2), one or more external processing components, one or more controllers (e.g., the controller 254 of FIG. 2), and / or the like. Figure 2C ​controller 276A and / or 276B of the XR system, etc. In some implementations, in addition to VR experiences, the XR system is also capable of rendering AR experiences and / or MR experiences. In some implementations, one or more blocks of process 500 can be performed by a server or computing system remote from the XR system, such as a cloud computing system or an edge computing system associated with a platform of the XR system.

[0065] At block 502, process 500 can render an XR environment on the XR system. In some implementations, the XR environment can be an AR environment or an MR environment. In some implementations, the rendered XR environment can include one or more virtual objects overlaid on a view of a real-world space surrounding the XR system. However, it is contemplated that in some implementations, a view on the XR system while in an AR mode or an MR mode need not always include virtual objects overlaid on a real-world space, but can simply display a perspective view of the real-world space.

[0066] At block 504, while rendering the XR environment, process 500 can automatically detect features of the real-world space by scanning the real-world space as the XR system moves. In some implementations, process 500 can scan the real-world space “in the background,” e.g., without notifying a user of the XR system that it is scanning the real-world space and / or without an instruction or explicit input from the user. In other words, in some implementations, process 500 can scan the real-world space based on a system-level command that is automatically generated. In some implementations, e.g., where the XR application is capable of rendering AR experiences or MR experiences in addition to VR experiences, the XR application can generate the command to scan the real-world space, e.g., through a call to an application programming interface (API) of the system.

[0067] Process 500 can scan the real-world space using one or more image capture devices (e.g., cameras that detect light in the visible wavelength range and / or the non-visible wavelength range), one or more depth sensors, or any combination thereof. In some implementations, process 500 can scan the real-world space using one or more cameras without using depth sensors, capturing one or more two-dimensional (2D) images of the real-world space without corresponding depth data that can later be predicted from features of the 2D images by applying one or more machine-learned models. Further details regarding locally applying and updating trained models for generating depth predictions for 2D images are further described in U.S. Patent Application No. 18 / 454,349, titled “Assisted Scene Capture for an Artificial Reality Environment,” filed August 23, 2023, the entirety of which is incorporated by reference herein.

[0068] In some implementations, the features of the real-world space can include visual features of the real-world space, such as walls, ceilings, floors, physical objects within the real-world space, etc. In some implementations, the features of the real-world space can be captured as an XR space model (also referred to as a “room box”) corresponding to the real-world space, which can include at least one of: an XR wall corresponding to a physical wall; an XR ceiling corresponding to a physical ceiling; an XR floor corresponding to a physical floor; or any combination thereof. To obtain the XR space model, a user of the XR system can scan the real-world space by moving an XR device and / or looking around the real-world space with the XR device using one or more cameras and / or one or more depth sensors, where process 500 uses such image and / or depth data to automatically identify one or more planar surfaces (e.g., walls, floors, ceilings) in the real-world space. For example, process 500 can identify planar surfaces by analyzing image and / or depth data for large areas having the same color, having consistently increasing and / or decreasing depth relative to the XR system, and / or having a particular orientation (e.g., above, below, or around the XR system).

[0069] In some implementations, the features of the real-world space can be captured as a three-dimensional (3D) mesh corresponding to the scanned real-world space and stored on the XR system until when the VR experience needs to be rendered. In some implementations, the mesh can be stored as a mesh having one or more interconnected shapes (e.g., squares, triangles, etc.). In some implementations, process 500 can capture both the XR space model and the mesh in order to further refine the mesh, as further described herein. In some implementations, when running in the background and capturing the XR space model and / or mesh, process 500 does not need to display the XR space model and / or mesh on the XR system, whether at the time of capturing the XR space model and / or mesh or upon completion of the capturing.

[0070] At block 506, process 500 can detect a trigger indicating an intent to execute a VR experience on the XR system. In some implementations, the trigger can be a request by a user of the XR system to initiate the VR experience, such as by selecting a virtual or physical button corresponding to the VR experience for the request by gesture, controller selection, audible notification, and / or eye gaze dwell. In some implementations, such as where the XR application rendering the XR environment has a VR mode in addition to a MR mode or AR mode, the trigger can be a request by the XR application to initiate the VR experience.

[0071] In some implementations, based on detecting the trigger at block 506, process 500 can determine whether the XR system can be repositioned in the real-world space. In some implementations, process 500 can attempt to reposition the XR system in the real-world space by comparing one or more features of the detected real-world space to one or more features of a stored previously captured known real-world space. If process 500 determines that the features of the scanned real-world space match the features of the stored real-world space above a threshold, process 500 can bypass block 508 and continue to block 510, executing the VR experience relative to the stored boundaries corresponding to the known real-world space. If process 500 determines that the features of the scanned real-world space do not match the features of the stored real-world space above a threshold (e.g., indicating a repositioning failure due to the real-world space being new to the XR system and / or otherwise not being able to be identified), process 500 can continue to block 508. Further details are described herein with reference to Figure 7 Further details are described herein regarding positioning and / or repositioning of the XR system in the real-world space.

[0072] At block 508, the process 500 can generate a boundary of the real-world space based on the detected features of the real-world space in response to the trigger detected at block 506. In some implementations, the boundary can correspond to the generated XR space model and / or mesh. In some implementations where a mesh corresponding to the real-world space is captured, the process 500 can also process the generated mesh. For example, in some implementations, processing the generated mesh can include collapsing one or more variations in the mesh corresponding to identified flat surfaces (e.g., walls, ceilings, floors, or any combination thereof) onto the plane created by the identified flat surfaces. For example, when scanning the real-world space, the process 500 can have incorrectly detected a slight variation in depth of a wall, ceiling, and / or floor, resulting in a mesh that cannot be tiled onto the surface. In such examples, the process 500 can collapse the mesh onto the corresponding flat surface such that the mesh tiles without the slight variation in depth. In some implementations, the process 500 can identify the corresponding flat surfaces from the generated XR space model.

[0073] In some implementations, processing the generated mesh can include clipping the mesh that extends through, beyond, and / or around physical objects that do not correspond to the real-world space intended to be captured. For example, when scanning the real-world space, the process 500 can detect a depth of a window, sliding glass door, open doorway, and / or other see-through element that extends beyond the real-world space. In such examples, the process 500 can modify the mesh such that the mesh does not extend to such areas. In some implementations, when capturing as described above, the process 500 can clip the mesh from such areas by clipping the mesh to the generated XR space model.

[0074] In some implementations, processing the generated mesh can include simplifying the mesh corresponding to identified flat surfaces (e.g., walls, ceilings, floors, or any combination thereof). For example, because the surfaces are flat, a denser, higher definition mesh can not be needed because such surfaces contain less detail. Thus, in such examples, the process 500 can simplify the mesh by merging the mesh corresponding to the flat surfaces (e.g., merging 2 triangles into a square, merging all shapes corresponding to a flat surface into a rectangle, etc.). In some implementations, when capturing as described above, the flat surfaces can be identified from the generated XR space model.

[0075] In some implementations, the processing can further include determining that at least one of the one or more physical objects captured by the mesh is dynamic. Process 500 can determine that a physical object is dynamic by, for example, analyzing image data and / or depth data captured by the XR system to determine that the physical object is moving. In another example, process 500 can determine that a physical object is dynamic by performing object recognition (e.g., recognizing a person, an animal, etc.) on image data captured by the XR system. In implementations in which process 500 determines that a physical object is dynamic, process 500 can filter at least one of the one or more physical objects from the mesh. For example, process 500 can remove the recognized physical object from the mesh.

[0076] In some implementations, processing the generated mesh can include determining that the mesh is complete by identifying that the mesh covers a threshold percentage of the real-world space and / or that a threshold percentage of the real-world space required to render the VR experience has been scanned. For example, process 500 can compare the portion of the real-world space that has been scanned to the amount of real-world space required to render the VR experience to determine how much of the real-world space has been scanned and whether it exceeds a threshold, such as 90%. In some implementations, the threshold can be a threshold amount or a threshold number of required surfaces (e.g., floor, ceiling, one or more walls). In some implementations, process 500 can determine the percentage of the real-world space that has been scanned based on movement of the XR system relative to the real-world space (e.g., whether the user has looked 360 degrees around, whether the user has looked up and down from multiple angles, etc.). When above the threshold, process 500 can determine that the mesh is complete enough and stop scanning the real-world space when rendering the XR environment.

[0077] In some implementations, process 500 can display a prompt to a user of the XR system indicating that the mesh is complete and available to render the VR experience. However, in some implementations, such a prompt is not displayed so that the user is not aware that the XR system is scanning the real-world space in the background while the XR environment is being rendered. Further details regarding generating and processing a mesh corresponding to a real-world space are described further in U.S. Patent Application No. 18 / 454,349, filed August 23, 2023, and entitled “Assisted Scene Capture for an Artificial Reality Environment,” the entirety of which is incorporated by reference herein.

[0078] In some implementations, process 500 can determine that the generated boundary covers less than a threshold percentage of the real-world space. In such implementations, process 500 can further scan the real-world space as the XR system moves. In some implementations, process 500 can prompt a user of the XR system to continue scanning the real-world space, for example, through a visual message or an audible message. In some implementations, process 500 can indicate locations of the real-world space that the user is to further scan (e.g., locations where the performed scans were insufficient to generate the boundary), thereby obtaining additional features of the real-world space. In some implementations, process 500 can display the generated boundary (which can indicate locations of the real-world space that require further scanning) and instruct the user to continue scanning, for example, with reference to the illustrated and described user interface 700. Figure 8A Based on the further scans, process 500 can update the generated boundary.

[0079] In some implementations, prior to performing a VR experience relative to the generated boundary, process 500 can determine an interaction mode for the VR experience based on the generated boundary. In some implementations, for example, the interaction mode can be “movable” when process 500 detects that the XR system moves more than a threshold amount. Process 500 can detect the movement of the XR system, for example, from one or more images acquired by the XR system over time (e.g., from the XR HMD and / or an external image acquisition device), one or more sensors of an inertial measurement unit (IMU) integrated with the XR system, etc. For example, process 500 can detect that the user is not fixed to a particular pivot point in the XR environment, for example, that the user moves more than a threshold amount in the x-direction, the y-direction, and / or the z-direction. Alternatively or additionally, process 500 can determine a “movable” interaction mode based on the completeness of the generated boundary being above a threshold, as further described above. In the “movable” interaction mode, the user can move freely in the real-world space while the VR experience is rendered.

[0080] In some implementations, the interaction mode can be“stationary,” for example, when the process 500 detects that the XR system is moving less than a threshold amount. For example, the process 500 can detect that the user is fixed to a particular pivot point in the XR environment, indicating that the user is in a seated or standing position, e.g., as detected from one or more images, one or more sensors of the IMU, etc., the user is moving less than a threshold amount in the x-direction, y-direction, and / or z-direction. In some implementations, the process 500 can alternatively or additionally detect a seated or standing position by analyzing one or more images, e.g., by capturing and detecting a user’s legs in a seated or standing position. Alternatively or additionally, as further described above, the process 500 can determine a“stationary” interaction mode based on the completeness of the generated boundaries being below a threshold, as the boundaries that are not sufficient for a“movable” interaction mode are likely to be sufficient for a“stationary” interaction mode. In the“stationary” interaction mode, the user must remain fixed to the pivot point, and / or move only a threshold amount from the pivot point.

[0081] In some implementations, the process 500 can recommend and / or notify the user of the determined interaction mode, e.g., through a display prompt and / or through an audible notification by the XR system, although in some implementations such a recommendation and / or notification is not necessary. In some implementations, the process 500 can modify the determined interaction mode based on input received by the XR system from the user. For example, the user can request to perform a VR experience in a“movable” mode rather than a“stationary” mode. In such implementations, if the generated boundaries are not sufficiently complete to support a“movable” mode in the VR experience, the process 500 can continue to scan the real-world space by moving the XR system and / or looking around the real-world space, and / or prompt the user to continue scanning the real-world space. In some implementations, the process 500 can continue to scan the real-world space in the background while processing and / or generating boundaries, determining an interaction mode, and / or recommending an interaction mode, and continually update and refine the generated boundaries without prompting the user and / or without the user explicitly instructing to do so.

[0082] In some implementations, at any point in the process 500, the process 500 can display the detected features of the real-world space and / or the generated boundaries on the XR system. In some implementations, as described above, displaying the generated boundaries can indicate to the user locations where additional scanning can be needed to perform a VR experience. In some implementations, after displaying the generated boundaries, the process 500 can modify the generated boundaries. For example, the process 500 can receive input from a user of the XR system to at least partially modify the generated boundaries, e.g., by detecting one or more controllers (e.g., Figure 2Cone or more manual adjustments to the generated boundaries based on the position of the controller and / or the position of the tracked hand or other body part. For example, the user of the XR system can move the controller or body part around the real-world space to, e.g., trace the correct position of a wall, ceiling, and / or floor with a light ray projected from the controller. In another example, the user of the XR system can position the controller or body part at the correct position of a wall, ceiling, and / or floor to identify the wall, ceiling, and / or floor based on the position of the controller or body part (e.g., detected by one or more cameras on the XR device, detected by one or more sensors of an IMU). In some implementations, the user of the XR system can pinch (with a hand) or select (with a controller) the predicted wall, ceiling, and / or floor and drag the wall, ceiling, and / or floor to the correct position. Further details regarding capturing and realigning the generated boundaries are described in greater detail in U.S. Patent Application No. 18 / 346,379, filed July 3, 2023, and titled “Artificial Reality Room Capture Realignment,” the entirety of which is incorporated by reference herein.

[0083] At block 510, the process 500 can execute the VR experience relative to the generated boundaries of the real-world space. For example, the process 500 can execute the VR experience in a “movable” mode or a “stationary” mode as recommended and / or modified by the user of the XR system. In some implementations, when the user approaches a boundary in the generated real-world space (e.g., enters within a threshold distance and / or moves toward the generated boundary at a threshold amount of speed) while executing the VR experience, the process 500 can display a warning, audibly announce a warning, turn on see-through, display at least a portion of the generated boundaries overlaid on the VR experience, or any combination thereof, thereby preventing potential harm to the user (or other users or movable objects in the real-world space), damage to physical objects in the real-world space, and the like.

[0084] Figure 6A6 is a conceptual diagram illustrating an example view 600A of an XR experience 604 rendered on an XR system while scanning a real-world space 602 surrounding the XR system. The XR experience 604 can be an augmented reality (AR) or mixed reality (MR) checkers game in which a virtual checkerboard 606 is overlaid on a physical table 610 in the real-world space 602. While rendering the XR experience 604, the XR system can scan the real-world space 602 “in the background” (i.e., without alerting a user of the XR system and without affecting the XR experience 604) to generate boundary data that can be used, for example, to render a virtual reality (VR) experience.

[0085] Figure 6B is a conceptual diagram illustrating an example view 600B of an XR overlay 612 on an XR system that includes an option to launch a VR experience. For example, based on a user selection from a menu, an audible notification to the user, and / or based on a system-level suggestion to the user, the XR system can render the XR overlay 612 over the XR experience 604. By selecting the XR overlay 612, the user can indicate a desire to stop executing the XR experience 604 and instead proceed to the VR experience indicated by the XR overlay 612.

[0086] Figure 6C is a conceptual diagram illustrating an example view 600C of a recommendation 614 for an interaction mode for a VR experience on an XR system. Figure 6B In view 600B of FIGURE 6, a user can select XR overlay 612 to initiate a VR experience. XR overlay 612 can be a trigger indicating an intent to enter VR mode. Upon detecting an intent to enter VR mode, the XR system can evaluate the XR system's scanning of real-world space 602 and / or the XR system's movement within real-world space 602 to recommend a "movable" mode or a "stationary" mode for the VR experience, as further described herein. In the example shown in views 600A through 600C, the XR system can determine that the user is in a seated position in front of physical table 610, for example, based on detecting that the XR system's movement within real-world space 602 is less than a threshold amount and / or by detecting that the user's legs are in a seated position from one or more images (not shown). Therefore, the XR system can recommend "stationary" mode for the VR experience via XR overlay 612. However, in some implementations, the user can modify the suggested interaction mode to "movable" mode, as further described herein (not shown), which may require the user to further scan real-world space 602 with the XR system.

[0087] Figure 6Dis a conceptual diagram illustrating an example view 600D of a boundary 616 for a VR experience generated on a XR system while rendering an XR experience. In this example, the boundary 616 demarcates a wall, floor, ceiling (not shown), and physical table 610 (e.g., physical object) in the real-world space 602 that are identified by scanning the real-world space 602 with the XR system. In some implementations, the XR system can display the boundary 616 so that a user of the XR system can modify it, e.g., by clicking and dragging (e.g., with a controller) or pinching and dragging (e.g., with a gesture) the boundary 616 to the correct or desired location, as further described herein.

[0088] Figure 6E is a conceptual diagram illustrating an example view 600E of a VR experience 618 on a XR system that is related to the boundary 616 (shown in view 600D) generated by the XR system. Once the boundary 616 is established for the real-world space 602, the XR system can render the VR experience 618, which can be a fully immersive, computer-generated view of an artificial environment. Since the VR experience 618 can occupy the entire field of view of the user on the XR system, generating the boundary 616 in the real-world space 602 can ensure that the user does not hit or otherwise collide with physical objects in the real-world space 602, thereby preventing injury and / or damage. In some implementations, if the user approaches the boundary 616 in the real-world space 602 while the VR experience 618 is rendered, the XR system can render a warning, open a see-through, and / or display the boundary 616, etc., as further described herein. Figure 6D

[0089] Figure 7 is a flowchart illustrating a process 700 for positioning an artificial reality (XR) system in a real-world space in some implementations of the present technology. In some implementations, the process 700 can be performed when the XR system is activated or donned in a real-world space. In some implementations, the process 700 can be performed based on a user request, an application-level request, or a system-level request to launch an XR experience. In some implementations, the process 700 can be performed when entering a real-world space that the XR system did not identify and / or discover in an initial scan, thereby causing an initial failure of the XR system to be repositioned in the real-world space, and facilitating creation, restoration, and / or identification of the scene by the process 700.

[0090] In some implementations, the process 700 can be performed by an XR system that includes one or more XR devices, such as an XR head-mounted display (HMD) (e.g.,​Figure 2A virtual reality HMD 200 and / or Figure 2B mixed reality HMD 252), one or more external processing components, one or more controllers (e.g., Figure 2C controllers 276A and / or 276B), etc. In some implementations, one or more blocks of process 700 can be performed by a server or computing system that is remote from the XR system, such as a cloud computing system or an edge computing system associated with a platform of the XR system.

[0091] At block 702, process 700 can scan a real-world space around an artificial reality (XR) system as the XR system is moved (e.g., as a user of the XR system moves around and / or looks around a real-world space). Process 700 can scan the real-world space using one or more cameras, one or more depth sensors, or any combination thereof. In some implementations, process 700 can scan the real-world space as described herein with reference to block 504 of process 500. Figure 5

[0092] At block 704, process 700 can generate a mesh corresponding to the scanned real-world space. The mesh can include, for example, one or more walls, a floor, a ceiling, and / or one or more physical objects in the real-world space. In some implementations, process 700 can generate the mesh as described herein with reference to block 504 of process 500. An example view of a mesh of a real-world space generated on an XR system as it is scanned is shown and described herein with reference to FIG. 6. In some implementations, process 700 can perform further processing of the generated mesh (e.g., as described herein with reference to block 508 of process 500). In some implementations, process 700 can generate the mesh in order to attempt to re-localize the XR system to a real-world space if the XR system previously knew the real space. Figure 5 Figure 8A In some implementations, process 700 can perform further processing of the generated mesh (e.g., as described herein with reference to block 508 of process 500). In some implementations, process 700 can generate the mesh in order to attempt to re-localize the XR system to a real-world space if the XR system previously knew the real space. Figure 5

[0093] ​​​At block 706, the process 700 can determine whether the generated mesh matches the stored mesh above a threshold. In some implementations, the process 700 can access locally and / or remotely stored meshes corresponding to real-world spaces previously scanned by the XR system or other XR systems. The process 700 can determine whether the generated mesh matches the stored mesh above a threshold by, for example, attempting to align the generated mesh with the stored mesh and comparing one or more visual features of the generated mesh with one or more visual features of the stored mesh. For example, the process 700 can determine whether a threshold number of visual features in the generated mesh match a threshold number of visual features in the stored mesh, whether the generated mesh matches the stored mesh within a threshold amount of variance, and the like.

[0094] If the process 700 determines at block 706 that the mesh generated at block 704 matches the stored mesh above a threshold, the process 700 can proceed to blocks 708-710. In some implementations, at block 708, the process 700 can retrieve stored spatial anchor data, stored scene data, or any combination thereof associated with the stored mesh. The stored spatial anchor data can include one or more spatial anchors of the real-world space. Individual spatial anchors of the one or more spatial anchors can define respective locations that can be automatically recognized by the XR system. As the XR system moves between real-world locations, the XR system can scan these locations and define certain anchors (e.g., at surfaces, edges, corners, doorways, etc.). These spatial anchors can define a map of the world around the XR system and can store the map locally or to a centralized map service. By aligning the detected one or more spatial anchors with the stored corresponding spatial anchors, the XR system can recognize itself within the map defined by the larger set of anchors from the map service. The XR system can then use the map to track itself in the real-world space and ensure that virtual objects appear to stay at the same locations and orientations in the scene (i.e., are world-locked).

[0095] In some implementations, the process 700 can retrieve the one or more spatial anchors from a local storage device on the XR system. In some implementations, the process 700 can retrieve the one or more spatial anchors from a platform computing system or other server located in the cloud. In some implementations, the one or more spatial anchors can be pre-established for the scene by another XR system that previously accessed the scene and can be uploaded to the cloud for storage and later accessed by other XR systems (e.g., the XR system executing the process 700) that later access the scene.

[0096] In some implementations, at least one of the one or more spatial anchors can be associated with scene data previously collected by the XR system or another XR system. The XR system can define the scene data when initially accessing the real-world space. In some cases, the XR system can define the scene data by, for example, capturing one or more images of the scene with a camera integrated with or in operable communication with the XR system, and identifying objects having types that match types in a scene dictionary. In other implementations, a user can manually specify locations and corresponding object types from the scene dictionary. Accordingly, the XR system can identify one or more object types (e.g., wall, door, window, table, chair, etc.) in a set of object types defined as scene components, each object type corresponding to a physical object in the one or more images of the scene. The XR system can generate object data associated with the one or more physical objects having the identified one or more object types. The XR system can also generate scene data by storing the object data with reference to one or more locations in the scene, and associate the scene data with at least one of the one or more spatial anchors.

[0097] At block 710, the process 700 can render the XR experience relative to the stored mesh, and in some implementations, the process 700 can render the XR experience relative to the stored spatial anchor data, the stored scene data, or any combination thereof. For example, the process 700 can render virtual objects of a mixed reality (MR) experience or an augmented reality (AR) experience at particular locations relative to any of the stored spatial anchors, and in some implementations, the process 700 can render virtual objects of a mixed reality (MR) experience or an augmented reality (AR) experience at persistent locations relative to previous locations of virtual objects in a previously executed XR experience session. In some implementations, the process 700 can render virtual objects relative to the scene data (e.g., the identified real-world objects in the real-world space). For example, the process 700 can render a virtual vase on the identified physical kitchen island. In some implementations, the process 700 can render virtual objects relative to the stored mesh. For example, the process 700 can render virtual objects relative to, and / or interacting with, the walls, floor, ceiling, and / or physical objects identified in the mesh (e.g., a virtual ball bouncing off a physical table identified in the mesh). In some implementations, for example when the XR experience is a VR experience, the process 700 can use the stored mesh as a boundary of the real-world space, and provide a warning or other safety measure if the user approaches or enters within a threshold distance of the boundary, as described herein with respect to block 510 of FIG. 5. Figure 5

[0098] ​If the process 700 determines at block 706 that the generated mesh does not match the stored mesh above a threshold, the process 700 can proceed to block 712. At this point, the XR system has scanned a sufficient amount of the real-world space for the process 700 to determine that the XR system is not in a pre-existing room. However, the XR system has scanned enough of the real-world space to establish (or begin establishing) a new room. Accordingly, at block 712, the process 700 can determine whether the generated mesh has at least a threshold level of completeness with respect to the real-world space. For example, the process 700 can determine whether the generated mesh covers at least a threshold percentage of the real-world space and / or whether a threshold number of particular physical objects of the real-world space are covered by the mesh (e.g., all walls, floors, ceilings, etc.). In some implementations, the threshold level of completeness can be defined by the requirements of a particular XR experience.

[0099] If at block 712, the process 700 determines that the generated mesh covers more than a threshold amount of the real-world space, the process 700 can proceed to block 714. At block 714, the process 700 can render the XR experience with respect to the generated mesh, e.g., by rendering virtual objects with respect to the stored mesh. For example, the process 700 can render virtual objects with respect to and / or interacting with the walls, floors, ceilings, and / or physical objects identified in the mesh (e.g., a virtual ball bouncing off a physical table detected in the mesh). In some implementations, e.g., when the XR experience is a VR experience, the process 700 can use the stored mesh as a boundary of the real-world space and provide a warning or other safety measure if the user approaches or enters within a threshold distance of the boundary, as described further herein with reference to block 510 of FIG. 5. Figure 5 In some implementations, it is contemplated that the process 700 can continue to scan the real-world space and collect mesh data and / or spatial anchor data for the real-world space that can be used to render the XR experience at block 714 while performing blocks 706 and / or 712.

[0100] If at block 712, the process 700 determines that the generated mesh does not cover more than a threshold amount of the real-world space, the process 700 can proceed to blocks 716-718. At block 716, the process 700 can further scan the real-world space as the XR system moves in the real-world space (e.g., by the user looking around the real-world space and / or moving around in the real-world space). In some implementations, the process 700 can prompt the user to continue looking around the real-world space and / or moving around in the real-world space through the XR system, e.g., in a pop-up message or other user interface element. Figure 8AIn some implementations, the process 700 can display the generated grid overlaid on a view of the real-world space to the user in order to indicate to the user one or more areas of the real-world space that need to be scanned, as further described herein.

[0101] At block 718, the process 700 can update the generated grid based on the further scanning. The process 700 can then return to block 712 and again determine whether the updated grid covers more than a threshold amount of the real-world space. If the updated grid does not cover more than the threshold amount of the real-world space, the process 700 can return to block 716 and continue scanning the real-world space, and so on. If the updated grid covers more than the threshold amount of the real-world space, the process 700 can proceed to block 714 and render the XR experience relative to the generated grid, as further described above.

[0102] Although the process 700 is primarily described herein as generating and comparing grids at blocks 704-706 to determine whether a matching stored grid already exists, in some implementations, it is contemplated that the process 700 instead identifies visual features of the real-world space (e.g., using computer vision techniques) and compares these visual features to features of stored grids in order to determine whether a pre-existing grid exists. In such implementations, if a match exists based on the visual features, at block 710, the process 700 can render the XR experience relative to the stored grid. If a match does not exist based on the visual features, it is contemplated that the process 700 can then generate a grid (as described with reference to block 704), bypass block 706, and continue to block 712, and so on.

[0103] Figure 8A FIG. 8A is a conceptual diagram illustrating an example view 800A of a grid 802 generated on an XR system as the XR system scans a real-world space 804. As the XR device is scanning the real-world space 804, the XR device can display an XR overlay 806 that prompts the user to continue moving around in the real-world space 804 and / or to continue looking around the real-world space 804 in order to generate the grid 802. In some implementations, the XR device can post-process the grid 802 after the grid 802 is complete (e.g., when a threshold amount of the real-world space 804 has been covered by the grid 802 and / or when the grid 802 is sufficient to render an XR experience defined by a particular XR application). In some implementations, the XR system can render the view 800A while initially attempting to match the grid 802 to a stored grid and / or in order to obtain additional grid data for rendering an XR environment.

[0104] Figure 8Bis a conceptual diagram illustrating an example view 800B of a complete mesh 802 of a real-world space 804 on an XR system in which the XR system has been located or re-located. In some implementations, when the XR system matches a threshold number of visual features of the mesh 802 to a stored mesh, the XR system can obtain the stored mesh and re-locate into the real-world space 804. In some implementations, when the XR system is unable to re-locate by matching the mesh 802 to a stored mesh, the XR system can establish the real-world space 804 as a new room with the mesh 802. In either case, the XR system can display a view 800B indicating that the XR system has re-located or located into the room (e.g., office) through an XR overlay 808 and can allow the user to select whether to continue with the XR experience in the real-world space 804 or cancel. However, in some implementations, it is contemplated that the XR system need not display the XR overlay 808 and can automatically continue to the XR experience 810 as Figure 8C shown.

[0105] Figure 8C is a conceptual diagram illustrating an example view 800C of an XR experience 810 rendered on the XR system relative to the re-located or located real-world space 804. When the user of the XR system selects to continue with the XR experience in the real-world space 804 (or automatically performs the XR experience upon re-location or location of the XR system), the XR system can render the XR experience 810 as shown in view 800C. For example, the XR system can render virtual objects 812A and 812B relative to physical objects in the real-world space 804 identified through the mesh 802. For example, the XR system can render a virtual art piece 812A hanging on a physical wall identified through the mesh and / or any available scene data, and a virtual clock 812B on a physical desk identified through the mesh and / or any available scene data.

[0106] Several implementations of the disclosed technology are described above with reference to the accompanying drawings. Computing devices on which the technology can be implemented can include one or more central processing units, memories, input devices (e.g., keyboards and pointing devices), output devices (e.g., display devices), storage devices (e.g., disk drives), and network devices (e.g., network interfaces). The memory and the storage devices are computer-readable storage media that can store instructions that implement at least portions of the technology. In addition, the data structures and message structures can be stored or transmitted via a data transmission medium, e.g., a signal on a communication link. Various communications links can be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. Thus, computer readable media can include computer-readable storage media (e.g., "non-transitory" media) and computer-readable transmission media.

[0107] In this specification, references to "an implementation" (e.g., "some implementations," "various implementations," "one implementation," "an implementation," etc.) mean that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation of the disclosure. The appearances of the phrases in various places in the specification are not necessarily all referring to the same implementation, or to a single implementation or alternative implementation, that is mutually exclusive from other implementations. Furthermore, various features that are described can be present in some implementations and not in others. Similarly, various requirements that are described can be requirements for some implementations and not for others.

[0108] As used herein, above threshold means that the value of the comparison item is higher than a specified other value, the comparison item is among a specified number of items with the highest values, or the value of the comparison item is within a specified top percentage of values. As used herein, below threshold means that the value of the comparison item is lower than a specified other value, the comparison item is among a specified number of items with the lowest values, or the value of the comparison item is within a specified bottom percentage of values. As used herein, within threshold means that the value of the comparison item is between two specified other values, the comparison item is among an intermediate specified number of items, or the value of the comparison item is within an intermediate specified percentage range. When not otherwise defined, relative terms such as high or insignificant can be understood to assign a value and determine how that value compares to an established threshold. For example, the phrase "select a fast connection" can be understood to mean selecting a connection with a value assigned corresponding to a connection speed that is above a threshold.

[0109] As used herein, the word "or" means any possible arrangement of items on the set. For example, the phrase "A, B, or C" means at least one of A, B, C, or any combination thereof, for example, any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiple any (e.g., A and A; B, B, and C; A, A, B, B, C, and C; etc.).

[0110] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Specific examples and implementations have been described herein for illustrative purposes, and various modifications are possible without departing from the scope of the examples and implementations. The specific features and acts are disclosed as example forms of implementing the appended claims. Therefore, the examples and implementations are not limited to the specific features and acts described, except as limited by the claims.

[0111] Any patents, patent applications, and other references noted above are incorporated herein by reference. Modifications can be made to various aspects of the disclosure to adopt systems, functions, and concepts described in the various references above to provide yet further embodiments. To the extent statements or subject matter in the documents incorporated by reference conflict with the statements or subject matter of the present disclosure, the present disclosure controls.

Claims

1. A method for automatically generating a boundary for a virtual reality experience, the method comprising: rendering an artificial reality environment on the user's artificial reality system, the artificial reality environment comprising one or more virtual objects overlaid on a view of the real-world space; while rendering the artificial reality environment, automatically detecting features of the real-world space by scanning the real-world space as the artificial reality system moves; detecting a trigger indicating an intent to perform the virtual reality experience on the artificial reality system; In response to detecting the trigger, generating a boundary of the real-world space based on detected features of the scanned real-world space; determining that the generated boundary covers less than a threshold percentage of the real-world space; further scanning the real-world space via the artificial reality system based on determining that the generated boundary covers less than a threshold percentage of the real-world space; updating the generated boundary based on further scans of the real-world space; as well as The virtual reality experience is performed relative to the generated boundaries of the real-world space.

2. The method according to claim 1, in, Further scanning the real world space is responsive to a cue rendered by the artificial reality system to further move the artificial reality system, and The method further comprises: Additional features of the real-world space are obtained in response to further movement of the artificial reality system.

3. The method according to claim 1, further comprising: Prior to executing the virtual reality experience, determining an interaction mode for the virtual reality experience based on the generated boundary, the interaction mode based on either: detecting movement of the artificial reality system greater than a threshold amount while scanning the real-world space; or detecting movement of the user in a seated or standing position less than the threshold amount.

4. The method according to claim 3, wherein: The determined interaction mode can be modified by the user through the artificial reality system.

5. The method according to claim 1, wherein The method further comprises: The generated boundary is modified based on one or more manual adjustments made by the user.

6. The method according to claim 1, wherein The trigger detected is to start the virtual reality experience on the artificial reality system.

7. The method according to claim 1, wherein Scanning the real-world space includes generating a mesh corresponding to the real-world space, and wherein generating the boundary includes: Processing the grid, the processing comprising at least one of: folding one or more variations in the mesh corresponding to at least one of a wall, a ceiling, a floor, or any combination thereof onto the detected flat surface, clipping the mesh into an artificial real-space model in which the mesh extends beyond the detected flat surface, simplifying the mesh corresponding to at least one flat surface, determining that the grid is complete by identifying a threshold percentage of the real-world space covered by the grid, or Any combination of the above.

8. The method according to claim 1, wherein The artificial reality system scans the real-world space using at least one of: one or more cameras; one or more depth sensors; or any combination of the above.

9. The method according to claim 1, in, The boundary is generated by estimating depth data of the scanned real-world space, and Wherein the depth data is estimated by applying a machine learning model to one or more scanned images of the real-world space.

10. The method according to claim 1, further comprising: Based on detecting the trigger, and before generating the boundary, it is determined that relocalization of the artificial reality system in the real-world space has failed.

11. The method according to claim 1, wherein The real world space is scanned by the artificial reality system without notifying the user.

12. The method according to claim 1, further comprising: Before executing the virtual reality experience, the generated boundary is displayed on the artificial reality system.

13. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for automatically generating a boundary for a virtual reality experience, the process comprising: rendering an artificial reality environment on the user's artificial reality system, the artificial reality environment comprising one or more virtual objects overlaid on a view of the real-world space; while rendering the artificial reality environment, automatically detecting features of the real-world space by scanning the real-world space as the artificial reality system moves, without notifying a user of the artificial reality system; detecting a trigger indicating an intent to perform the virtual reality experience on the artificial reality system; In response to detecting the trigger, generating a boundary of the real-world space based on detected features of the scanned real-world space; as well as The virtual reality experience is performed relative to the generated boundaries of the real-world space.

14. The computer-readable storage medium of claim 13, wherein: The process also includes: determining that the generated boundary covers less than a threshold percentage of the real-world space; Based on determining that the generated boundary covers less than a threshold percentage of the real-world space, further scanning the real-world space with the artificial reality system; and The generated boundary is updated based on further scans of the real-world space.

15. The computer-readable storage medium according to claim 13, in, Further scanning the real world space is responsive to a cue rendered by the artificial reality system to further move the artificial reality system, and The process further includes: Additional features of the real-world space are obtained in response to further movement of the artificial reality system.

16. The computer-readable storage medium of claim 13, wherein: The process also includes: Prior to executing the virtual reality experience, determining an interaction mode for the virtual reality experience based on the generated boundary, the interaction mode based on either: detecting movement of the artificial reality system greater than a threshold amount while scanning the real-world space; or detecting movement of the user in a seated or standing position less than the threshold amount.

17. The computer-readable storage medium of claim 16, wherein: The determined interaction mode can be modified by the user through the artificial reality system.

18. A computing system for automatically generating a boundary for a virtual reality experience, the computing system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: rendering an artificial reality environment on the user's artificial reality system, the artificial reality environment comprising one or more virtual objects overlaid on a view of the real-world space; automatically detecting features of the real-world space by scanning the real-world space as the artificial reality system moves while rendering the artificial reality environment, without requiring user input causing scanning of the real-world space; detecting a trigger indicating an intent to perform the virtual reality experience on the artificial reality system; In response to detecting the trigger, generating a boundary of the real-world space based on detected features of the scanned real-world space; and The virtual reality experience is performed relative to the generated boundaries of the real-world space.

19. The computing system of claim 18, wherein: The process also includes: determining that the generated boundary covers less than a threshold percentage of the real-world space; Based on determining that the generated boundary covers less than a threshold percentage of the real-world space, further scanning the real-world space with the artificial reality system; and The generated boundary is updated based on further scans of the real-world space.

20. The computing system of claim 18, wherein: The process also includes: Based on detecting the trigger, and before generating the boundary, it is determined that relocalization of the artificial reality system in the real-world space has failed.

Images