Automatic boundaries of artificial reality environments

Abstract

An automatic boundary of an artificial reality environment is provided. For an artificial reality (XR) application requiring a protected zone, aspects of the present disclosure may automatically determine which location in a real-world environment should set a boundary. The automatic boundary system may detect a floor plane using a machine learning model and then generate a height map of the real-world environment, the height map being floor zero. The height map may include a positive height (e.g., an object on the floor) and a negative height (e.g., a stair leading to below). The automatic boundary system may then generate a boundary of the region based on the detected heights by applying a threshold to ignore objects at certain heights; applying a threshold to ignore open regions at certain widths; and the strange-shaped area and the like which can continuously trigger the boundary if the user is nearby are excluded. The user may further manually adjust the generated boundary in both directions (e.g., closer to or farther from the user).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 585,360, filed on September 26, 2023, entitled “Automatic Boundary for an Artificial Reality Environment,” and priority to U.S. Non-Provisional Application No. 18 / 771,009, filed on July 12, 2024, both of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to generating automatic boundaries of an artificial reality (XR) environment provided by an XR system. Background Art

[0004] Artificial reality (XR) devices are becoming more and more common. As they become more popular, the applications implemented on such devices are becoming more and more complex. Mixed reality (MR) applications and augmented reality (AR) applications can provide interactive three-dimensional (3D) experiences that combine images of the real world with virtual objects, while virtual reality (VR) applications can provide a completely independent 3D computer environment. For example, MR applications or AR applications can be used to superimpose virtual objects on a real scene observed by a camera. Then, real-world users in the scene can make gestures captured by the camera, which can provide interactivity between real-world users and virtual objects. Users can observe AR experiences, MR experiences, and VR experiences (collectively referred to as XR experiences) through a head-mounted display (HMD) (such as glasses or a head-mounted viewer (headset)). The HMD can have a see-through display that allows light from the real world to pass through a lens and combine with light from a waveguide that simultaneously emits light from a projector in the HMD, thereby allowing the HMD to present virtual objects mixed with real objects that the user can actually see. Summary of the invention

[0005] For example, for XR applications that require a protected area, the automatic boundary system described herein can automatically determine where in the user's real-world environment the boundary should be set. For example, the automatic boundary system can use one or more machine learning models to detect a floor plane and generate a height map corresponding to the real-world environment, with the floor plane as zero. The height map can include positive heights (e.g., objects on the floor) and negative heights (e.g., objects below the floor, such as stairs leading below). The automatic boundary system can determine the user's position relative to the detected height and generate a boundary of the area relative to the user based on the detected height. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram showing an overview of a device on which some embodiments of the present technology may operate.

[0007] Figure 2A is a line diagram illustrating a virtual reality head mounted viewer that may be used in some embodiments of the present technology.

[0008] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0009] Figure 2C is a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold in one or both hands to interact with an artificial reality environment.

[0010] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0011] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0012] Figure 5 is a flow chart illustrating a process for providing automatic boundaries of an artificial reality (XR) environment provided by an XR system in some embodiments of the present technology.

[0013] Figure 6 is a conceptual diagram illustrating an example perspective view of a determined floor plane in a real-world environment.

[0014] Fig. 7A is a conceptual diagram showing an example overhead view of a height map corresponding to a real-world environment.

[0015] Figure 7B is a conceptual diagram illustrating an example top view of a determined continuous floor area corresponding to a real-world environment.

[0016] Figure 7C is a conceptual diagram illustrating an example top view of a modified continuous floor area corresponding to a real-world environment.

[0017] Fig.7D is a conceptual diagram illustrating an example top view of a zoomed-out continuous floor area corresponding to a real-world environment.

[0018] Fig. 7E is a conceptual diagram showing an example top view of a pared-down continuous floor area corresponding to a real-world environment.

[0019] Figure 7F is a conceptual diagram illustrating an example top view of a continuous floor plane corresponding to a boundary of an artificial reality (XR) environment.

[0020] The various techniques described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to the same elements or functionally similar elements. DETAILED DESCRIPTION

[0021] Various aspects of the present disclosure relate to providing automatic boundaries (which may be used interchangeably with the term "guardian" herein) of an artificial reality (XR) environment. As used herein, a "guardian" may be a space for XR use defined in a real-world environment. If a user wearing an XR system crosses the boundary while accessing an XR experience, one or more system actions or restrictions may be triggered on the XR system. For example, as further described herein, the XR system may display an alert message on the XR system, may activate at least partial perspective on the XR system, may display a boundary on the XR system, may pause rendering or updating of the XR environment, and the like.

[0022] For example, for XR applications that require a protected area, the automatic boundary system described herein can automatically determine where in the user's real-world environment the boundary should be set. For example, the automatic boundary system can use one or more machine learning models to detect a floor plane and generate a height map corresponding to the real-world environment, with the floor plane as zero. The height map can include positive heights (e.g., objects on the floor) and negative heights (e.g., objects below the floor, such as stairs leading below). The automatic boundary system can determine the user's position relative to the detected height and generate a boundary of the area relative to the user based on the detected height.

[0023] When generating the boundary, the automatic boundary system can apply thresholds to ignore objects at certain heights, such as objects above 2 meters (which can be, for example, a low-hanging chandelier) or objects below 15 centimeters (e.g., a carpet or mat). The automatic boundary system can "floodfill" closed areas on the floor plane to determine the continuous floor area. The automatic boundary system can also apply thresholds to ignore open areas at certain widths (e.g., below 1 meter); thus, strangely shaped areas that will constantly trigger the boundary if the user is nearby can be removed. In some embodiments, the automatic boundary system can also cut out areas above the threshold in the height map (e.g., a coffee table) that are surrounded (i.e., surrounded) by the continuous floor area, so that the "island" whose height is excluded is no longer surrounded. In some embodiments, the user can further manually adjust the generated boundary in both directions (e.g., to make it closer or farther away from the user). Therefore, the automatic boundary system can reduce the friction of spatial setup in XR by automatically creating a safe boundary for the user.

[0024] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0025] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system (e.g., by adding virtual objects). "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially constitutes light reflected by objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.

[0026] The various embodiments described herein provide specific technical improvements in the field of artificial reality. According to some embodiments, the XR system can scan the real-world space to automatically determine the floor plane and the height map of each object in the space. The XR system can use this height map to automatically determine the continuous floor area, which can include the following objects: objects with a height lower than the threshold height difference relative to the floor plane (e.g., a floor mat), or objects that do not touch the floor (e.g., a low-hanging ceiling chandelier). The XR system can also automatically exclude areas that do not meet certain standards (e.g., small areas that will continuously trigger the display of the boundary if crossed, areas with negative heights relative to the floor plane, etc.), and / or reduce the determined continuous floor area so that the display of the boundary is triggered before the user may make physical contact. Therefore, relative to conventional technologies, the various embodiments described herein can generate improved boundaries for the XR experience, which do not require user input or require less user input. Therefore, each embodiment can reduce the latency of rendering the XR experience by minimizing or eliminating manual scene acquisition and setup of artificial reality, thereby improving latency and overall user experience. Further, the various embodiments described herein reduce the likelihood, risk, and / or incidence of collisions and / or other accidents (e.g., falling down stairs) between users of the XR system and physical objects in real-world space.

[0027] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1is a block diagram showing an overview of devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of a computing system 100 that may provide automatic boundaries for an XR environment of an artificial reality (XR) device. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted view device that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted view device and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted view device and other processing operations are offloaded to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted view device is described. In some implementations, location data and environmental data may be collected solely by sensors incorporated into the head mounted view device, while in other implementations, one or more of the plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.

[0028] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0029] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits the information to the processors 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0030] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, or a wireless connection, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with a hardware controller for each device (e.g., for a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen, a light emitting diode (LED) display screen, a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0031] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, GPS units, LiDAR or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment and track the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0032] The computing system 100 may include such a communication device: the communication device can communicate with other local computing devices or network nodes wirelessly or based on wired communication. The communication device can communicate with another device or server through a network, for example, using the Transmission Control Protocol / Internet Protocol (TCP / IP protocol). The computing system 100 can use the communication device to distribute operations on multiple network devices.

[0033] Processor 110 may access memory 150, which may be included in one of the computing devices of computing system 100, or may be distributed across the computing devices of computing system 100, or across multiple other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, and tape drive. Memory is not a propagation signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160, which stores programs and software, such as operating system 162, automatic boundary system 164, and other application programs 166. Memory 150 may also include data storage 170, which may include, for example, floor plane data, height map data, boundary data, continuous floor area data, closed area data, threshold data, reduction data, reduction data, rendering data, configuration data, settings, user options or preferences, etc., which may be provided to program memory 160 or any element in computing system 100.

[0034] In various embodiments, the techniques described herein may include a non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform the steps as shown and described herein. In various embodiments, the techniques described herein may include a computing system that includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform the steps as shown and described herein.

[0035] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0036] Figure 2A 2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. In this example, the HMD 200 also includes augmented reality features that use a see-through camera 225 to render portions of the real world that may have computer-generated overlays. The HMD 200 includes a front rigid body 205 and a strap 210. The front rigid body 205 includes one or more electronic display elements of one or more electronic displays 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a camera and positioner 225, and one or more computing units 230. The position sensors 220, the IMU 215, and the computing unit 230 may be located inside the HMD 200 and may not be visible to the user. In various embodiments, the IMU 215, the position sensor 220, and the camera and positioner 225 can track the movement and position of the HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the positioner 225 can emit infrared beams that produce light spots on real objects around the HMD 200; and / or the camera 225 captures images of the real world and positions the HMD 200 within the real world environment. As another example, the IMU 215 can include, for example, one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof that can be used in the positioning process. One or more cameras 225 integrated with the HMD 200 can detect the light spots. The calculation unit 230 in the HMD 200 may use the detected light points and / or position points to infer the position and movement of the HMD 200 and recognize the shapes and positions of real objects around the HMD 200 .

[0037] One or more electronic displays 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of electronic displays 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), etc.), some other display, or some combination thereof.

[0038] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC can use the HMD in conjunction with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0039] Figure 2B 2 is a line diagram of a mixed reality HMD system 250 that includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 may communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality HMD system 250 includes only a head mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 may house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0040] The projector can be coupled to the see-through display 258, for example, via an optical element to display media to the user. The optical element can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for directing light from the projector to the user's eyes. The image data can be transmitted from the core processing component 254 to the HMD 252 via a link 256. The controller in the HMD 252 can convert the image data into a plurality of light pulses from the projector, which can be transmitted to the user's eyes via the optical element as output light. The output light can be mixed with the light passing through the display 258, thereby allowing the output light to present the following virtual objects: these virtual objects appear to exist in the real world.

[0041] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0042] Figure 2C Controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held by a user in one or both hands to interact with the artificial reality environment presented by the HMD 200 and / or the HMD system 250. The controllers 270 can communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit light points farther away. The HMD 200 or HMD system 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., tracking the controllers with 3DoF or 6DoF). The computing unit 230 or the core processing component 254 in the HMD 200 can combine the IMU output and the position output and use this tracking to monitor the position and movement of the user's hands. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A-274B) that a user may actuate to provide input and interact with objects.

[0043] In various embodiments, the HMD 200 or HMD system 250 may also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor indications of user interaction and intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in the HMD 200 or HMD system 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate one or both eyes of the user, and the HMD 200 or HMD system 250 can use eye-facing cameras to collect reflections of light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes and determine gaze direction.

[0044] Figure 3 305B may be a HMD 200 or a HMD system 250. The plurality of client computing devices 305 may be operated in a network environment using logical connections to one or more remote computers (e.g., server computing devices) through a network 330.

[0045] In some implementations, server computing device 310 may be an edge server that receives multiple client requests and coordinates fulfillment of these requests through other servers (e.g., server computing devices 320A-320C). Server computing devices 310 and 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, each of these server computing devices may be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0046] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other server / client devices. Server computing device 310 can be connected to database 315. Server computing devices 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server computing device 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0047] Network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 may be the Internet or some other public or private network. Client computing device 305 may be connected to network 330 via a network interface, such as by wired or wireless communications. Although the connections between server computing device 310 and multiple server computing devices 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including network 330 or a separate public or private network.

[0048] Figure 44 is a block diagram showing a plurality of components 400 that can be used in a system that employs the disclosed technology in some embodiments. Each component 400 can be included in a device of the computing system 100, or can be distributed on multiple devices of the computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, a system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., in a database 315 or 325) or other network storage accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0049] Middleware 420 may include components that communicate resources between hardware 410 and a plurality of specialized components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

[0050] The specialized components 430 may include software or hardware configured to perform operations for providing automatic boundaries of an artificial reality (XR) environment provided by an artificial reality (XR) system. The specialized components 430 may include: a floor plane determination module 434; a height map generation module 436; a continuous floor area determination module 438; an exclusion identification module 440; a boundary determination module 442; and components and application programming interfaces (APIs), such as interface 432, that may be used to provide a user interface, transfer data, and control the specialized components. In some embodiments, the components 400 may be in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more specialized components 430. Although the specialized components 430 are depicted as separate components, the specialized components may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0051] The floor plane determination module 434 can determine the floor plane in the real world environment. In some embodiments, the floor plane determination module 434 can determine the floor plane via one or more images captured by one or more cameras, alone or in combination with input from one or more depth sensors, which can indicate a continuous area below the XR system (e.g., an area of ​​continuous color, an area of ​​continuously increasing depth from the XR system, etc.). In some embodiments, the floor plane determination module 434 can determine the floor plane based on, for example, the following items, by tracking information from one or more controllers (e.g., Figure 2C The floor plane can be determined by input from controllers 276A and / or 276B in the user's control panel: the user places the controller on the floor plane (thereby indicating the location of the floor plane); the user outlines the floor plane using light projected from the controller; the user touches the floor with a finger or hand (e.g., via hand tracking), etc. In some embodiments, as further defined herein, the floor plane determination module 434 can determine the floor plane based on a generated XR space model or mesh corresponding to the real world environment. Figure 5 Block 502 in describes further details regarding determining the floor plane in the real-world environment.

[0052] The height map generation module 436 can generate a height map of the real-world environment by detecting multiple heights corresponding to multiple physical objects in the real-world environment. These physical objects may include fixed physical objects and / or movable physical objects (including walls, ceilings, floors, and other objects such as furniture) in the real-world space. The multiple heights can be detected relative to the floor plane, which can be set to zero height in the height map. For example, the height map generation module 436 can detect the height relative to the floor plane by collecting image data and / or depth data indicating physical objects in the space, and / or the height of these physical objects relative to the XR system (which can be compared with the height of the floor plane relative to the XR system to determine the height of these physical objects and the size of these physical objects, thereby generating a height map). In some embodiments, the height map generation module 436 can detect the height based on the generated grid mentioned above and further described herein, which grid may include features indicating physical objects in the real-world environment.

[0053] In some embodiments, the height map generation module 436 can identify each physical object according to its detected height based on scene data collected or established for the real-world environment. The scene data can be generated by storing object data associated with one or more physical objects in the real-world environment. With reference to one or more locations in the real-world environment, the scene data can provide identified object types from a set of object types defined as scene components in the real-world environment. For example, object types may include semantic labels, such as walls, ceilings, doors, tables, windows, counters, etc. In some embodiments, the identified object types and / or semantic labels can be automatically identified by applying object detection and / or recognition techniques. In some embodiments, the scene data can be collected in advance or in real time by the XR system performing process 500; while in other embodiments, the scene data may have been collected in advance by another XR system and shared with the XR system performing process 500. Reference is made to the description herein. Figure 5 Block 504 in describes further details regarding generating a height map of the real-world environment by detecting a plurality of heights corresponding to a plurality of physical objects in the real-world environment.

[0054] In some embodiments, the continuous floor area determination module 438, the exclusion identification module 440, and the boundary determination module 442 can perform functions sequentially to automatically generate the boundary of the XR environment. The continuous floor area determination module 438 can determine the continuous floor area by determining a closed area of ​​the floor plane determined by the floor plane determination module 434, in which a threshold number of heights in the closed area detected by the height map generation module 436 are at the height of the floor plane. The area can be surrounded by surrounding physical objects, such as detected walls of the real-world environment, or other objects within a threshold depth of the wall, such as windows, doors, etc. For example, the continuous floor area determination module 438 can determine a continuous floor area including all detected heights at the floor plane (i.e., all areas at zero height). Therefore, in some embodiments, the continuous floor area determination module 438 can exclude negative heights in the height map that exceed a specific threshold height (e.g., corresponding to stairs leading to the bottom) and positive heights in the height map that exceed a specific threshold height (e.g., physical objects on the floor) from the continuous floor area. Reference is made herein to Figure 5 Block 506 in describes further details regarding determining the continuous floor area by determining the closed area of ​​the floor plane.

[0055] The exclusion identification module 440 can identify one or more areas of the continuous floor area determined by the continuous floor area determination module 438 that have a size below a threshold. In some embodiments, the size can be a width dimension (e.g., an "x-direction") of an area of ​​the floor plane that is closed on at least two sides and has one or more heights indicated by a height map, and the width dimension extends parallel to the floor plane. For example, when the XR system is facing the area of ​​the floor plane to be identified, a height dimension (e.g., a "y-direction") that extends laterally from the XR system and intersects the floor plane can be defined, a depth dimension (e.g., a "z-direction") that extends from the XR system along the floor plane and away from the XR system can be defined, and a width direction (e.g., an "x-direction") that is perpendicular to both the y-direction and the z-direction can be defined. Therefore, the exclusion identification module 440 can identify areas with a width below a threshold, such as a narrow corridor (e.g., having a width of less than 1 meter). The exclusion identification module 440 can also modify the continuous floor area to exclude the one or more identified areas. In this document, reference is made to Figure 5 Block 508 in describes further details regarding identifying one or more regions of the continuous floor region having a size below a threshold, and modifying the continuous floor region to exclude the identified one or more regions.

[0056] The boundary determination module 442 can determine the boundary of the XR environment as the continuous floor area determined by the continuous floor area determination module 438 and modified by the exclusion identification module 440. In some embodiments, the boundary determination module 442 can store the determined boundary for later rendering of the XR environment, and / or sharing with another XR system accessing the real-world environment. In some embodiments, the boundary determination module can also render the XR environment relative to the determined boundary. As further described herein, the boundary can define the location where a user of the XR system can access the XR environment without restriction. If the XR system approaches the boundary at a speed greater than a threshold speed within a threshold distance, and / or crosses the boundary, the XR system can take one or more preventive actions and / or proactive measures, such as displaying a warning, displaying a boundary, activating perspective, etc. Reference is made herein to Figure 5 Block 510 in describes further details regarding determining the boundaries of the XR environment.

[0057] Those skilled in the art will understand that the above described Figures 1 to 4 The multiple components shown in each of the multiple flow charts discussed above and below can be changed in various ways. For example, the order of the logic can be rearranged, multiple sub-steps can be performed in parallel, the logic shown can be omitted, other logic can be included, etc. In some embodiments, one or more of the multiple components described above can perform one or more of the multiple processes described below.

[0058] Figure 5 is a flow chart illustrating a process 500 for generating an automatic boundary of an XR environment provided by an artificial reality (XR) system in some embodiments of the present technology. As mentioned above, the term "boundary" may be used interchangeably herein with "protected area," as further defined herein. In some embodiments, process 500 may be performed upon activating the XR system or putting on the XR system. In some embodiments, process 500 may be performed upon launching an XR application on the XR system. In some embodiments, process 500 may be performed based on a user-level request, an application-level request, or a system-level request. In some embodiments, process 500 may be performed upon determining that a boundary of a real-world environment surrounding the XR system has not been previously established and / or stored.

[0059] In some embodiments, process 500 may be performed when the XR system fails to be identified and / or fails to be repositioned in the real-world environment. Process 500 may attempt to match the real-world space with a previously mapped real-world space (with previously established boundary data) by any suitable method. For example, process 500 may prompt the user to look around the room to generate a grid that can be compared to an existing room grid, and / or an XR space model that can be compared to an existing grid and / or XR space model. In another example, process 500 may use one or more cameras to capture one or more images of the real-world space, identify visual features of the real-world space (e.g., corners, edges, physical objects, etc.), and compare these visual features with visual features of previously captured known real-world spaces. In yet another example, process 500 may capture a positioning map including one or more spatial anchors of the real-world space, and determine whether the positioning map can be merged or matched to a pre-existing positioning map that includes one or more boundaries and / or other spatial data of the real-world space (e.g., spatial anchor data that provides a reference frame for the XR environment). However, it is contemplated that in some implementations, process 500 need not attempt to reposition in space.

[0060] In some implementations, part or all of process 500 may be performed by an XR system, which may include one or more XR devices, such as an XR head-mounted display (HMD) (e.g., Figure 2A XR HMD 200 and / or Figure 2B In some embodiments, part or all of process 500 may be performed by another XR device of an XR system including an XR HMD, such as an external processing component, one or more controllers (e.g., Figure 2CController 276A and / or 276B in the example embodiment) and / or other peripheral devices. In some embodiments, process 500 may be performed by Figure 1 In some embodiments, process 500 may be performed by the automatic boundary system 164 in Figure 4 Component 400 in is executed.

[0061] At block 502, process 500 may determine a floor plane in a real-world environment. In some embodiments, process 500 may automatically determine a floor plane, for example, by applying a machine learning model to data collected by an XR system. For example, process 500 may identify a floor plane based on a generated XR space model for a real-world environment. The XR space model (interchangeably referred to herein as a "room box") may indicate where walls, floors, and / or ceilings exist in the real-world space. In some embodiments, process 500 may automatically acquire the XR space model. For example, as described above, a user of an XR system may automatically identify one or more planes (e.g., walls, floors, ceilings) in the real-world space by moving and / or using the XR system, scanning the real-world space using one or more cameras and / or one or more depth sensors, and automatically identifying one or more planes (e.g., walls, floors, ceilings) in the real-world space using such image data and / or depth data. For example, process 500 may identify a plane by analyzing image data and / or depth data of a larger area that is the same color, continuously increases and / or decreases in depth relative to the XR system, and / or is specific in orientation (e.g., above the XR system, below the XR system, or around the XR system).

[0062] In some implementations, process 500 can be performed at least in part via one or more controllers detected (e.g., Figure 2CThe floor plane may be identified (e.g., via an XR space model) based on the position of controller 276A and / or controller 276B in the XR system and / or the position of a hand or other body part tracked (e.g., via one or more cameras of the XR system and / or external device). For example, a user of the XR system may move a controller or body part in the real-world environment to outline the floor, for example, with light projected from the controller or a finger. In another example, the user of the XR system may place a controller or body part on the floor to identify the floor based on the position of the controller or body part (e.g., as detected by one or more cameras on the XR system, such as via one or more sensors of an IMU). In some embodiments, process 500 may automatically identify the floor, and then the floor may be refined (if necessary) via one or more controllers or body parts, for example, by tapping the floor plane or pinching and dragging the floor plane toward or away from the user, or placing a controller or hand on the floor. Further details on identifying and realigning an XR room including a floor plane are described in U.S. patent application Ser. No. 18 / 346,379, filed on July 3, 2023, entitled “Artificial Reality Room Capture Realignment,” which is incorporated herein by reference in its entirety. Figure 6 An exemplary determined floor plane is shown and described.In some implementations, process 500 can similarly identify one or more walls and / or ceilings of a real-world environment.

[0063] In some embodiments, process 500 can automatically identify a floor plane based on a three-dimensional (3D) mesh generated by scanning a real-world environment. The mesh can be, for example, a three-dimensional (3D) model of each boundary of a real-world space, which includes one or more walls, ceilings, floors, and / or one or more physical objects, etc. In some embodiments, process 500 can generate a mesh using one or more cameras, one or more depth sensors, or any combination thereof. However, in some embodiments, it is considered that depth data does not need to be collected, but depth data can be predicted, for example, by a machine learning model based on one or more images. In some embodiments, process 500 can also perform post-processing on the mesh to refine and / or simplify the mesh. More details about generating and using XR space models and meshes are described in the following documents: U.S. patent application No. 18 / 454,349, entitled "Assisted Scene Capture for anArtificial Reality Environment", filed on August 23, 2023, which is incorporated herein by reference in its entirety.

[0064] At block 504, process 500 may generate a height map of the real-world environment by detecting a plurality of heights corresponding to a plurality of physical objects in the real-world environment, the plurality of heights being detected relative to a floor plane. In some embodiments, process 500 may detect a plurality of heights relative to a floor plane by using depth sensor measurements between the XR system and one or more identified walls. For example, process 500 may determine that a physical object exists 2 meters from the XR system (while the wall is 5 meters away); and that the physical object extends a certain distance (e.g., 1 meter) above the floor plane before the depth sensor no longer detects the physical object (but instead detects the wall at a distance of 5 meters). In some embodiments, process 500 may detect a plurality of heights relative to a floor plane by applying computer vision techniques to one or more images captured by the XR system. For example, for a set area (e.g., 1 square centimeter) captured by the XR system, process 500 may identify a physical object and determine the highest point of the physical object under the identified ceiling. In some embodiments, process 500 can determine the height of the physical object based on a known height of the floor plane (e.g., set to zero) and a known height of the ceiling, and / or based on a known height of the XR system and a distance from the XR system to the physical object. In some embodiments, process 500 can identify the physical object and / or determine the height of the physical object by applying machine learning techniques. Fig. 7A An exemplary generated height map is shown and described.

[0065] In some embodiments, when generating a height map, process 500 may perform one or more pre-processing steps before generating a boundary based on the height map. In some embodiments, process 500 may collapse one or more heights in the height map that are below a threshold distance from the floor plane to the floor plane. For example, process 500 may collapse detected heights below 3 centimeters to the floor plane in the height map, such as to ignore minor variations in the floor caused by carpets, floor mats, and other low objects that do not block user movement. In some embodiments, process 500 may exclude one or more heights from the height map that are above a threshold distance from the floor plane. For example, process 500 may exclude detected physical objects that are higher than 2.5 meters, such as to ignore low hanging chandeliers and other objects fixed to the ceiling that do not block user movement.

[0066] In some embodiments, process 500 can automatically generate the boundaries of the XR environment by performing at least blocks 506 to 510. At block 506, process 500 can determine a continuous floor area by determining an enclosed area of ​​the floor plane in which a threshold number of heights of a plurality of heights in the enclosed area are at the height of the floor plane. For example, process 500 can determine a continuous floor area by locating a continuous enclosed area of ​​the floor plane within the identified wall of the real-world environment (e.g., at zero height or at a height folded to the floor plane as described above). In other words, in some embodiments, the threshold number of heights of a plurality of heights within the enclosed area that are at the height of the floor plane can be all heights of the plurality of heights within the enclosed area. Thus, in some embodiments, when determining a continuous floor area, process 500 can exclude areas having one or more positive heights relative to the floor plane (e.g., physical objects at 0.5 meters, such as a coffee table), and / or areas having one or more negative heights relative to the floor plane (e.g., physical objects at -0.4 meters and below, such as stairs leading below). References herein to Figure 7B Exemplary determined contiguous floor regions are shown and described.

[0067] At block 508, process 500 may identify one or more regions of the continuous floor area that have a size below a threshold, and modify the continuous floor area to exclude the identified one or more regions. For example, process 500 may determine a region of the continuous floor area that has a narrower width relative to the XR system. Process 500 may remove such a region when generating the boundary so that if the user approaches or enters the narrower region, the boundary is not continuously triggered, which would otherwise affect the user experience. Figure 7C An exemplary modified continuous floor region is shown and described.

[0068] In some embodiments, process 500 may reduce the continuous floor area by a predetermined amount ( Figure 5 For example, to add an additional level of safety to the user, process 500 may reduce one or more portions of the boundary or the entire continuous floor area by a set amount (e.g., 0.1 meters) and / or by a set percentage (e.g., 5%). Fig.7D An exemplary reduced floor area is shown and described.

[0069] In some embodiments, process 500 may, when the continuous floor region encompasses one or more detected heights above a threshold among the plurality of detected heights, trim one or more portions from the continuous floor region so that the trimmed continuous floor region does not encompass any height above the threshold. Figure 5 ). For example, for a physical object having a threshold height (e.g., at least 15 cm, or just above the floor plane) and being surrounded by the floor plane, process 500 may remove an area from the continuous floor area so that the physical object is no longer surrounded by the floor plane. Fig. 7E An exemplary reduced continuous floor area is shown and described.

[0070] At block 510, process 500 may determine the boundary of the XR environment as the determined continuous floor region further modified by any combination of one or more of the above steps. In some implementations, process 500 may render the XR environment on the XR system relative to the boundary corresponding to the determined continuous floor region. Figure 7F An exemplary generated boundary of an XR environment is shown and described. In some implementations, process 500 can store the boundary corresponding to the determined continuous floor area for later rendering of the XR environment.

[0071] In some embodiments, the determined continuous floor area of ​​the real-world environment corresponding to the boundary of the XR environment can define a location where the user can view, access, and / or interact with the XR environment without restriction. However, if the XR system and / or one or more body parts of the user (e.g., as detected by one or more cameras) come within a threshold distance of the boundary and / or cross the boundary, the process 500 can implement one or more restrictions and / or automatically perform actions on the XR system. For example, when the user of the XR system comes within a threshold distance of the boundary and / or crosses the boundary with the XR system and / or one or more body parts detected, the process 500 can take one or more actions. In some embodiments, these actions can include activating a warning overlaid on the XR environment (e.g., "You are too close to the wall!") or an instruction to move back into the protected area (e.g., "Back off!"). In some embodiments, these actions can include pausing at least one of the following: executing the XR environment, updating the XR environment, rendering the XR environment, or any combination thereof. For example, if the user leaves the protected area and enters a room unknown to the XR system and / or application, process 500 can stop executing and / or rendering the XR environment. In some embodiments, these actions can include removing one or more virtual objects from the XR environment (e.g., stopping rendering virtual objects associated with the same application or a different application outside the protected area). In some embodiments, such as when approaching a boundary in VR mode, these actions can include activating at least partial perspective on the XR system. In some embodiments, these actions can include displaying the protected area on the XR system so that the user can visualize the location of the boundary and move away from it.

[0072] Figure 6is a conceptual diagram of an example stereogram 600 illustrating a determined floor plane 604 in a real-world environment 602 from an artificial reality (XR) system. In some implementations, the XR system can automatically predict the floor plane 604 and overlay a view of the predicted floor plane 604 on the floor in the real-world environment 602, for example, by using one or more images and / or depth measurements (e.g., via one or more cameras and / or depth sensors) and applying computer vision techniques and / or machine learning techniques. In some implementations, a user of the XR system can manually indicate the location of the floor to set the floor plane 604, for example, by placing a controller (not shown) on the floor in the real-world environment 602 and / or by placing a hand on the floor as detected via a hand tracking system. In some implementations, once the floor plane 604 is determined and overlaid on the view of the real-world environment 602, the user can manually adjust the floor plane 604 to correspond to the floor if the floor plane is incorrect (e.g., by clicking or grabbing the displayed floor plane 604 and moving the hand or controller closer to or farther away from the user). Although shown and described as having floor plane 604 displayed on the XR system, it is contemplated that in some implementations the XR system may determine floor plane 604 without displaying floor plane 604 on the XR system.

[0073] Fig. 7A 700A is a conceptual diagram showing an example overhead view 700A of a height map 702 corresponding to a real-world environment (e.g., a room). Height map 702 may be a mapping of heights of physical objects 706-714 within the real-world environment detected by the XR system relative to a floor plane 704. Floor plane 704 may be set to be at an altitude of 0 meters. In this example, physical object 706 may be detected at an altitude of 0.05 meters relative to floor plane 704; physical object 708 may be detected at an altitude of 1.7 meters relative to floor plane 704; physical object 710 may be detected at an altitude of 1.5 meters relative to floor plane 704; physical object 712 may be detected at an altitude of -0.2 meters relative to floor plane 704; and physical object 714 may be detected at an altitude of 1.2 meters relative to floor plane 704.

[0074] In some embodiments, before determining the continuous floor area, the automatic boundary system described herein can fold the physical object 706 onto the floor plane 704 in the height map 702 because its height is below a threshold distance (e.g., below 0.1 meters). For example, the physical object 706 can be a floor mat whose height difference relative to the floor plane 704 can be ignored. Similarly, if the physical object is detected to be above the floor plane 704 by a threshold distance (e.g., 2 meters and above), the automatic boundary system can remove the physical object from the height map 702 before determining the continuous floor area (not shown). Such a physical object can be, for example, a low-hanging chandelier that does not block the user's movement in the XR environment.

[0075] Figure 7B 700B is a conceptual diagram showing an example top view 700B of a determined continuous floor area 716 corresponding to a real-world environment. The automatic boundary system described herein can determine the continuous floor area 716 by determining an enclosed area of ​​the floor plane 704 in which a threshold number of the plurality of heights in the enclosed area are at the height of the floor plane 704. In some embodiments, all portions of the enclosed area of ​​the floor plane 704 have a height at the floor plane 704. Therefore, the continuous floor area 716 can exclude the area 728A (corresponding to Fig. 7A Physical object 708 in, area 730A (corresponding to Fig. 7A Physical object 710 in, area 732A (corresponding to Fig. 7A physical object 712 in) and region 734A (corresponding to Fig. 7A In some embodiments, physical object 706 may be collapsed onto height map 702 during preprocessing as described above, such that continuous floor region 716 does not exclude regions corresponding to physical object 706. However, in some embodiments where physical object 706 is not collapsed onto height map 702 during preprocessing, the automatic boundary system may similarly determine the following: Figure 7B The continuous floor area 716 is shown because a threshold number (eg, 95%) of the enclosed heights are at the height of the floor plane 704 .

[0076] Figure 7C 700C is a conceptual diagram showing an example top view 700C of a modified continuous floor region 718 corresponding to a real-world environment. In some embodiments, the automatic boundary system described herein can identify a region 720 having a size (e.g., a width) below a threshold value (e.g., 1 meter) between regions 728A and 730A (e.g., corresponding to physical objects 708 and 710, respectively). Thus, in some embodiments, the automatic boundary system described herein can generate a modified continuous floor region 718 (relative to Figure 7B The determined continuous floor area 716 in the figure is used to exclude area 720.

[0077] Fig.7D 700D is a conceptual diagram showing an example top view 700D of a contiguous floor region 722 corresponding to a real-world environment after being zoomed out. In some embodiments, the automatic boundary system described herein can construct a boundary in the contiguous floor region 722 relative to the real-world environment by removing additional amounts of floor space around regions 728A to 734A (corresponding to physical objects 708 to 714, respectively). Figure 7B 716. For example, the automatic boundary system may increase the size of the excluded areas 728A to 734A by 10%, so that the reduced continuous floor area 722 has excluded areas 728B to 734B. In this example, the excluded areas 728B to 734B may therefore have a size 10% larger than the physical objects 708 to 714, thereby providing an additional safety margin.

[0078] Fig. 7E 7 is a conceptual diagram showing an example top view 700E of a contiguous floor area 724 corresponding to a real-world environment after being trimmed. In some embodiments, in the case where the contiguous floor area 718 surrounds the excluded area 734A, the automatic boundary system described herein can be based on determining that the physical object 714 (corresponding to the excluded area 734A) has a height above a threshold value (e.g., 0.15 meters). Figure 7B The portion 726 is cut out from the determined continuous floor region 716 in FIG. The automatic boundary system can therefore generate a cut-off continuous floor region 724 that does not surround the region 734A corresponding to the physical object 714 .

[0079] Considered, this article refers to FIG. 7B to FIG. 7E Any of the techniques described may be applied alone or in any combination to generate a continuous floor region corresponding to the boundaries of an artificial reality (XR) environment. Figure 7F is shown by applying the reference 7A to 7E 700F of an example top view of a continuous floor region 736 corresponding to a boundary of an XR environment generated by all of the techniques described herein. For example, the continuous floor region 736 has excluded a region 720 where the continuous floor region 736 would otherwise have a width below a threshold (e.g., reference 700F). Figure 7C In addition, the continuous floor area 736 has a Figure 7B The continuous floor area 716 in the embodiment is 10% larger than the excluded areas 728B to 734B, thereby increasing the continuous floor area 736 relative to Figure 7B The continuous floor area 716 in the Fig.7D In addition, the continuous floor region 736 has a cut-back portion 726 such that the continuous floor region 736 does not surround the excluded region 734B (e.g., see Fig. 7E In some implementations, a continuous floor area 736 can be established as a boundary that can be used to render an XR environment.

[0080] References in this specification to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be rendered by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0081] As used herein, being higher than a threshold value means that the value of a comparison item is higher than other values ​​specified, the comparison item is located in a certain specified number of items with a maximum value, or the comparison item has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of a comparison item is lower than other values ​​specified, the comparison item is located in a certain specified number of items with a minimum value, or the comparison item has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of a comparison item is between two other values ​​specified, the comparison item is located in a middle specified number of items, or the comparison item has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" may be understood to mean selecting a connection with a value corresponding to a connection speed assigned thereto that is higher than a threshold value.

[0082] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of the following: A, B, C or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items (e.g., A and A; B, B and C; A, A, B, C and C, etc.).

[0083] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and acts described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except as set forth in the appended claims.

[0084] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for providing an automatic boundary of an artificial reality environment provided by an artificial reality system, the method comprising: Determine floor planes in real-world environments; generating a height map of the real-world environment by detecting a plurality of heights corresponding to a plurality of physical objects in the real-world environment, the plurality of heights being detected relative to the floor plane; and The boundary of the artificial reality environment is automatically generated by the following steps: determining a continuous floor region by determining an enclosed area of ​​the floor plane in which a threshold number of a plurality of heights in the enclosed area are at a height of the floor plane; identifying one or more regions of the continuous floor region having a size below a threshold, and modifying the continuous floor region to exclude the one or more identified regions; reducing the modified continuous floor area by a predetermined amount; In the case that the reduced continuous floor area surrounds one or more detected heights higher than a threshold among the plurality of detected heights, cutting off one or more portions from the reduced continuous floor area so that the reduced continuous floor area does not surround any height higher than the threshold; as well as The boundary of the artificial reality environment is determined as the cut-back continuous floor area.

2. The method according to claim 1, wherein: Generating the height map includes folding one or more heights in the height map below a first threshold distance from the floor plane to the floor plane.

3. The method according to claim 1, wherein: Generating the height map includes excluding from the height map one or more heights above a second threshold distance from the floor plane.

4. The method according to claim 1, further comprising: The artificial reality environment is rendered on the artificial reality system relative to the boundary.

5. The method according to claim 1, wherein: The floor plane is determined by applying a machine learning model to one or more images of the real-world environment.

6. The method according to claim 1, wherein: Determining the continuous floor area excludes areas having one or more positive heights relative to the floor plane and excludes areas having one or more negative heights relative to the floor plane.

7. The method according to claim 1, further comprising: displaying, by the artificial reality system, the boundary overlaid on a view of the real-world environment; as well as Input is received via the artificial reality system to apply a manual adjustment to the boundary.

8. The method according to claim 7, wherein: The manual adjustment is to move the boundary to a position closer to the artificial reality system, or to move the boundary to a position farther away from the artificial reality system.

9. The method according to claim 1, wherein: The threshold number of heights of the plurality of heights in the enclosed area is all heights of the plurality of heights in the enclosed area.

10. A computer-readable storage medium storing instructions for providing an automatic boundary of an artificial reality environment provided by an artificial reality system, the instructions, when executed by a computing system, causing the computing system to: Determine floor planes in real-world environments; generating a height map of the real-world environment by detecting a plurality of heights corresponding to a plurality of physical objects in the real-world environment, the plurality of heights being detected relative to the floor plane; and The boundary of the artificial reality environment is automatically generated by the following steps: determining a continuous floor region by determining an enclosed area of ​​the floor plane in which a threshold number of heights of a plurality of heights in the enclosed area are within a threshold height from the floor plane; identifying one or more regions of the continuous floor region having a size below a threshold, and modifying the continuous floor region to exclude the one or more identified regions; as well as The boundary of the artificial reality environment is determined as the modified continuous floor area.

11. The computer-readable storage medium of claim 10, wherein: Automatically generating the boundary further includes: reducing the modified continuous floor area by a predetermined amount, The boundary is determined as the modified and reduced continuous floor area.

12. The computer-readable storage medium according to claim 10, in, Automatically generating the boundary further comprises: in a case where the modified continuous floor region surrounds one or more detected heights among the plurality of detected heights that are higher than a threshold, cutting off one or more portions from the modified continuous floor region so that the cut-off continuous floor region does not surround any height higher than the threshold; and The boundary is determined as the modified and reduced continuous floor area.

13. The computer-readable storage medium of claim 10, wherein: Generating the height map includes folding one or more heights in the height map below a first threshold distance from the floor plane to the floor plane.

14. The computer-readable storage medium of claim 10, wherein: Generating the height map includes excluding from the height map one or more heights above a second threshold distance from the floor plane.

15. The computer-readable storage medium of claim 10, wherein: The floor plane is determined by applying a machine learning model to one or more images of the real-world environment.

16. The computer-readable storage medium of claim 10, wherein: Determining the continuous floor area excludes areas having one or more positive heights relative to the floor plane and excludes areas having one or more negative heights relative to the floor plane.

17. A computing system for providing an automatic boundary of an artificial reality environment provided by an artificial reality system, the computing system comprising: one or more processors; as well as one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to: Determine floor planes in real-world environments; generating a height map of the real-world environment by detecting a plurality of heights corresponding to a plurality of physical objects in the real-world environment, the plurality of heights being detected relative to the floor plane; and The boundary of the artificial reality environment is automatically generated by the following steps: determining a continuous floor region by determining an enclosed area of ​​the floor plane in which a threshold number of heights of a plurality of heights in the enclosed area are within a threshold height from the floor plane; identifying one or more regions of the continuous floor region having a size below a threshold, and modifying the continuous floor region to exclude the one or more identified regions; as well as The boundary of the artificial reality environment is determined as the modified continuous floor area.

18. The computing system of claim 17, wherein: Automatically generating the boundary further includes: reducing the modified continuous floor area by a predetermined amount; and In the case that the reduced continuous floor area surrounds one or more detected heights higher than a threshold among the plurality of detected heights, one or more portions are cut out from the reduced continuous floor area so that the cut-out continuous floor area does not surround any height higher than the threshold, The boundary is determined as the modified, reduced and trimmed continuous floor area.

19. The computing system of claim 17, wherein: The instructions, when executed by the one or more processors, further cause the computing system to: displaying, by the artificial reality system, the boundary overlaid on a view of the real-world environment; and Input is received via the artificial reality system to apply a manual adjustment to the boundary.

20. The computing system of claim 19, wherein: The manual adjustment is to move the boundary to a position closer to the artificial reality system, or to move the boundary to a position farther away from the artificial reality system.

Images