Mapping real-world rooms for shared artificial reality environment

Abstract

A room manager may generate a mapping of real-world rooms that supports a shared XR environment. For example, a real-world room may include real-world objects and real-world surfaces, such as one or more desks, one or more chairs, one or more walls, one or more doors, one or more windows, etc. A room manager may generate an XR object definition based on received information about a real-world room, one or more real-world objects, and one or more real-world surfaces. For example, a room manager may implement a process that directs a user equipped with an XR system to provide information for XR object definition, such as a real-world surface mapped to one or more XR objects, boundaries (e.g., boundaries on the real-world surface) (e.g., measured using components of the XR system), semantic information (e.g., semantic information on the real-world surface), or semantic information (e.g., semantic information on the real-world surface). ) at the XR table, the number of seats allocated at the XR table, the size of the XR object, etc. Embodiments generate previews of a shared XR environment, such as local previews and remote previews.

Description

Technical Field

[0001] The present disclosure relates to mapping real-world rooms to configure a shared artificial reality environment. Background Art

[0002] As global markets expand and challenges such as distributed teams and health issues that require social distance become more prevalent, remote work and collaboration are becoming more common. Remote work typically involves a variety of technologies, such as remote access to shared documents, various text-based communication services (e.g., email, instant messaging, text messaging, etc.), telephone communications, and video calls. Such remote work provides many benefits, such as reduced travel time, improved health and safety, and increased flexibility. However, remote workers face many challenges that their colleagues in the office do not experience. For example, video calls are still a pale imitation of face-to-face interactions. It may be difficult to understand body language and context using only a two-dimensional ("2D") representation of the sender. In addition, limiting video calls to flat-panel displays introduces an intrusive layer of technology that may distract from communication and reduce the perception of face-to-face communication. Remote workers often complain about missing opportunities for "water cooler" conversations and often feel forgotten by their colleagues. Furthermore, collaboration between teams in remote locations using traditional voice or 2D communication channels is not as effective as an in-person experience, such as using document and whiteboard spaces where various participants can view and work simultaneously.

[0003] Artificial reality, extended reality, or hyperreality (collectively referred to as "XR") is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Various XR environments exist that allow representations of multiple users to move around and talk to each other. However, such XR environments are not effectively created to facilitate collaboration and are not integrated into the workflow of remote workers. Summary of the invention

[0004] In a first aspect, a method for generating a real-world room configured with a shared artificial reality is provided. A method for mapping a shared XR (XR) room, the method comprising: receiving first and second mappings corresponding to first and second virtual objects from a user located in a real-world room, the first mapping comprising a boundary on a horizontal surface representing a real-world location of the first virtual object, and the second mapping comprising a boundary on a vertical surface representing a real-world location of the second virtual object, wherein at least a portion of the boundary is identified using one or more physical XR devices; receiving first semantic information defining the first virtual object as an XR table and second semantic information defining the second virtual object as an XR pad; and causing the XR system to display first and second previews of the shared XR room to the user, wherein the first preview comprises a pass-through display of the real-world room augmented with the first virtual object as the XR table at a location corresponding to the first mapping and augmented with the second virtual object as the XR pad at a location corresponding to the second mapping, and the second preview comprises a virtual reality room comprising the first virtual object as the XR table and the second virtual object as the XR pad, the virtual reality room comprising a geometry corresponding to the geometry of the real-world room, and relative locations of the first and second virtual objects in the virtual reality room corresponding to the first and second mappings.

[0005] In embodiments of the first aspect, one or more of the following may be used:

[0006] The method may also include storing a mapping definition for a shared XR room, the mapping definition including first semantic information, a first mapping, second semantic information, and a second mapping, wherein the stored configuration definition is used to implement a shared XR room for one or more local users using a transparent display of a real-world room including a first virtual object and a second virtual object, and to implement a shared XR room for one or more remote users using a virtual reality version of the real-world room including the first virtual object and the second virtual object.

[0007] The method may also include automatically generating a plurality of seat assignments for the plurality of users relative to the first virtual object, the plurality of seat assignments comprising a defined area within a boundary on a flat horizontal surface from the first mapping, wherein the generated plurality of seat assignments enhance semantic information of the first virtual object. During implementation of the shared XR room, the one or more local users and the one or more remote users may each be assigned one of the plurality of seat assignments at the first virtual object.

[0008] The method may also include determining that the geographic location of the real-world room does not include a previously stored configuration definition that maps the real-world room to the shared XR room, and prompting the user for the first mapping and the second mapping in response to the determination.

[0009] The method may further include verifying, prior to storing the configuration definition, that the received first semantic information and second semantic information include semantic information of at least one XR table and at least one XR board.

[0010] The method may also include receiving, from a user located within the real-world room, a third mapping corresponding to a third virtual object and a fourth mapping corresponding to a fourth virtual object, the third mapping including a boundary on the flat horizontal surface representing the real-world location of the third virtual object.

[0011] The method may also include receiving third semantic information defining the third virtual object as an XR chair and fourth semantic information defining the fourth virtual object as an XR door.

[0012] The method may also include verifying, before storing the configuration definition, that the received first semantic information, second semantic information, third semantic information, and fourth semantic information include semantic information of at least one XR table, at least one XR board, at least one XR chair, and at least one XR door.

[0013] In a second aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing instructions, which, when executed by a computing system, causes the computing system to perform a process for generating a mapping of a real-world room configured with a shared artificial reality (XR) room, the process comprising: receiving a first mapping and a second mapping corresponding to a first virtual object and a second virtual object from a user located in the real-world room, the first mapping comprising a boundary on a horizontal surface representing a real-world positioning of the first virtual object, and the second mapping comprising a boundary on a vertical surface representing a real-world positioning of the second virtual object, wherein at least a portion of the boundary is identified using one or more physical XR devices; receiving first semantic information defining the first virtual object as an XR table and the second semantic information defining the second virtual object as an XR table. The virtual object is defined as second semantic information of the XR board; and the XR system displays a first preview and a second preview of the shared XR room to the user, wherein the first preview includes a transparent display of the real-world room, which is augmented with a first virtual object as an XR table at a position corresponding to the first mapping and augmented with a second virtual object as the XR board at a position corresponding to the second mapping, and the second preview includes a virtual reality room, which includes the first virtual object as the XR table and the second virtual object as the XR board, and the virtual reality room includes a geometry corresponding to the geometry of the real-world room, and relative positions of the first virtual object and the second virtual object in the virtual reality room corresponding to the first mapping and the second mapping.

[0014] In embodiments of the second aspect, one or more of the following features may be used:

[0015] The process may also include storing a mapping definition for a shared XR room, the mapping definition including first semantic information, a first mapping, second semantic information, and a second mapping, wherein the stored configuration definition is used to implement a shared XR room for one or more local users using a pass-through display of a real-world room including a first virtual object and a second virtual object, and to implement a shared XR room for one or more remote users using a virtual reality version of the real-world room including the first virtual object and the second virtual object.

[0016] The process may also include automatically generating a plurality of seat assignments for the plurality of users relative to the first virtual object, the plurality of seat assignments comprising a defined area within a boundary on a flat horizontal surface from the first mapping, wherein the generated plurality of seat assignments enhance semantic information of the first virtual object. During implementation of the shared XR room, the one or more local users and the one or more remote users may each be assigned one of the plurality of seat assignments at the first virtual object.

[0017] The process may also include determining that the geographic location of the real-world room does not include a previously stored configuration definition that maps the real-world room to the shared XR room, and prompting the user for the first mapping and the second mapping in response to the determination.

[0018] The process may also include, before storing the configuration definition, verifying that the received first semantic information and second semantic information include semantic information of at least one XR table and at least one XR board.

[0019] The process may also include receiving, from a user located within the real-world room, a third mapping corresponding to a third virtual object and a fourth mapping corresponding to a fourth virtual object, the third mapping including a boundary on the flat horizontal surface representing the real-world location of the third virtual object.

[0020] The process may also include receiving third semantic information defining the third virtual object as an XR chair and fourth semantic information defining the fourth virtual object as an XR door.

[0021] The process may also include, before storing the configuration definition, verifying that the received first semantic information, second semantic information, third semantic information, and fourth semantic information include semantic information of at least one XR table, at least one XR board, at least one XR chair, and at least one XR door.

[0022] In a third aspect, a computing system for generating a mapping of a real-world room is provided. The computing system configures a shared artificial reality (XR) room, the computing system comprising one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the edge computing system to perform a process, the process comprising: receiving a first mapping and a second mapping corresponding to a first object and a second object, the first mapping comprising a boundary representing a real-world location of the first object, and the second mapping comprising a boundary representing a real-world location of the second object; receiving first semantic information defining the first object as an XR table and second semantic information defining the second object as an XR board; causing the XR system to display a first preview and a second preview, the first preview comprising a pass-through display of the real-world room, the pass-through display being enhanced with the first object according to the first mapping, and being enhanced with the second object according to the second mapping, the second preview comprising a virtual reality room that mimics the first preview; and storing the first semantic information, the first mapping, the second semantic information, and the second mapping in association with a geographic location of the real-world room.

[0023] In an embodiment of the third aspect, the process may also include storing a mapping definition for a shared XR room, the mapping definition comprising first semantic information, a first mapping, second semantic information, and a second mapping, wherein the stored configuration definition is used to implement a shared XR room for one or more local users using a transparent display of the real-world room including the first object and the second object, and to implement a shared XR room for one or more remote users using a virtual reality version of the real-world room including the first object and the second object.

[0024] It will be understood that any features described herein that are suitable for incorporation into one or more aspects or embodiments of the present disclosure are intended to be generalizable to any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and illustrative only, and are not intended to limit the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram showing an overview of a device on which some embodiments of the present technology may operate.

[0026] Figure 2A is a line diagram illustrating a virtual reality headset that may be used in some embodiments of the present technology.

[0027] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0028] Figure 2C is a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold in one or both hands to interact with an artificial reality environment.

[0029] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0030] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0031] Figure 5 is a conceptual diagram illustrating a representation of a real-world room used for mapping in some implementations.

[0032] Figure 6 is a conceptual diagram showing room boundaries defined for a real-world room.

[0033] Figure 7is a conceptual diagram showing the table surface definition of a table object located in a real-world room being mapped.

[0034] Figure 8 , Fig. 9 and Fig.10 is a conceptual diagram illustrating the use of an XR device to measure table boundaries for a table object located in a real-world room being mapped.

[0035] Fig.11 is a conceptual diagram illustrating alignment verification for a mapped table object located in a mapped real-world room.

[0036] Fig.12 is a conceptual diagram illustrating local and remote users within a shared XR environment including an XR table and an XR board.

[0037] Figures 13 to 17 is a conceptual diagram showing real-world movement and XR board interaction of a local user in a real-world room that is part of a shared XR environment.

[0038] Fig.18 is a conceptual diagram showing a remote user's avatar movement and XR board interaction within a virtual reality room that is part of a shared XR environment.

[0039] Fig.19 is a conceptual diagram showing real-world movement of a local user within a real-world room that is part of a shared XR environment.

[0040] Fig. 20 is a conceptual diagram illustrating an example of creating an artificial reality work environment linked to multiple real-world spaces.

[0041] Figure 21 to Figure 26 is a schematic diagram showing multiple versions of a shared XR environment in which a user approaches and interacts with an XR pad.

[0042] Fig.27A is a conceptual diagram showing a display error of text in a pass-through version of an augmented shared XR environment.

[0043] Fig.27B is a conceptual diagram illustrating an occlusion mask that mitigates display errors in a pass-through version of a shared XR environment.

[0044] Fig.28A is a conceptual diagram illustrating display errors of virtual objects in a pass-through version of a shared XR environment.

[0045] Fig.28B is a conceptual diagram illustrating an occlusion mask that mitigates display errors in a pass-through version of a shared XR environment.

[0046] Fig.29 is a flow chart illustrating a process for mapping a real-world room to enable a shared XR room in some embodiments of the present technology.

[0047] Fig.30 is a flow chart illustrating a process for creating an artificial reality work environment having one or both of desk space and / or standing space in some embodiments of the present technology.

[0048] Fig.31 is a flow chart illustrating a process for providing a shared XR environment including a local version for one or more local users and a remote version for one or more remote users in some embodiments of the present technology.

[0049] Fig.32 is a flow chart illustrating a process for configuring an occlusion mask for an avatar in an XR environment in some embodiments of the present technology.

[0050] Fig.33 is a flow chart illustrating a process for configuring occlusion masks for virtual objects in an artificial reality (XR) environment in some embodiments of the present technology.

[0051] The various techniques described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to the same elements or functionally similar elements. DETAILED DESCRIPTION

[0052] Various aspects of the present disclosure relate to mapping a real-world room to configure a shared artificial reality room. In some embodiments, the real-world room may contain one or more real-world: one or more tables, one or more chairs, one or more doors, one or more walls, one or more windows, and other suitable room elements. An embodiment of the room manager may map the real-world room by capturing the room (e.g., the three-dimensional volume of the room) using one or more cameras, instructing the user to navigate the room when equipped with an XR system, and by any other suitable mapping technology. In addition, an embodiment of the room manager may map the exact position of one or more real-world objects (e.g., a table) in the real-world room. The accurate position may provide a mapping of one or more XR objects (e.g., an XR table) that configures the position of the one or more XR objects in the XR version of the room. For example, the room manager may instruct the user to provide the boundaries of an object when the user is wearing an XR system (e.g., a head-mounted display and a handheld controller that can physically specify a location).

[0053] The user may also provide semantic information to the room manager that defines at least a first horizontal flat surface in the real-world space (e.g., a flat surface of a table) as an XR table, one or more second horizontal flat surfaces in the real-world space (e.g., a flat surface of a chair) as one or more XR chairs, a vertical flat surface in the real-world space (e.g., a portion of a wall in a room) as an XR board, etc. In some embodiments, the semantic information includes multiple seat assignments at the XR table. Once the real-world room is mapped, the room manager may store the mapping definition in association with the room. In some embodiments, the room manager may compare the mapping definition for the room with one or more validation criteria that define the minimum requirements for mapping the room (e.g., one XR table, one XR board, one XR door, and one XR window). If the mapping definition does not meet the validation criteria, one or more additional mappings may be generated (e.g., to add additional XR objects) and / or one or more existing mappings may be edited. Once it is determined that the mapping definition meets the validation criteria, the mapping definition may be saved in association with the real-world room to support a shared XR environment.

[0054] In some embodiments, before saving the mapping definition for reuse, the room manager can display previews of multiple versions of the shared XR environment configured by the mapping definition to the user performing the mapping. For example, the multiple previews can be a local version of the shared XR environment of the real-world room according to the generated mapping definition, and a remote version of the shared XR environment of the real-world room according to the generated mapping definition. For example, the local version of the preview can be a pass-through display or mixed reality vision of the real-world room enhanced with XR objects according to the mapping definition (e.g., enhanced with an XR table at a location corresponding to the XR table mapping definition, and enhanced with an XR board at a location corresponding to the XR board mapping definition). In some embodiments, the user performing the mapping displayed as a preview of the local version can be physically located in the real-world room so that the pass-through / mixed reality vision of the real-world room can be enhanced with XR objects.

[0055] In some embodiments, the remote version of the preview may be a virtual reality room that includes virtual objects corresponding to the mapping definition (e.g., a virtual object version of the XR table and a virtual object version of the XR pad). For example, the virtual reality room may include geometry corresponding to the geometry of the real-world room, and relative positions of the XR table and the XR pad within the virtual reality room corresponding to positions defined in the mapping definition for the XR table and the XR pad. In some embodiments, the remote version of the preview may be a virtual reality room that mimics the local version of the preview. In some embodiments, after viewing the preview, the user may complete the mapping process for the real-world room and store the mapping definition for use in supporting a shared XR environment.

[0056] In some embodiments, the room manager can use the mapping and semantic information to generate a shared XR environment corresponding to the real-world room. For example, the mapping definition can be reused by multiple users to implement multiple shared XR environments hosted at the real-world room. The stored mapping definition of the real-world room can also be edited, for example to update the location of the mapping of the XR object and / or configure the mapping according to a new layout. In some embodiments, the room manager can coordinate multiple XR systems of multiple users to: A) render a local version of the shared XR environment to a local user (e.g., a user physically located in the real-world room) (e.g., the local version displays a pass-through video capture of the real-world room or a mixed reality vision of the real-world room); and B) render a virtual reality (VR) version of the shared XR environment to a remote user (e.g., a user located in a physical space away from the real-world room) (e.g., the VR version displays a virtual room corresponding to the real-world room). For example, the shared XR environment can be an XR representation of the main real-world room that supports interactions between multiple users (e.g., multiple local users) located at the same location and interactions between multiple users (e.g., at least one remote user) that are far away from each other.

[0057] In a local version of a shared XR environment (e.g., rendered by an XR system to a local user), a local user may be represented as a pass-through video capture or mixed reality vision of the user, while a remote user may be represented as an avatar (which may be a computer-generated drawing or video or holographic representation of the remote user). Since the local user is physically located in the real-world room, the pass-through video capture or mixed reality vision of the real-world room inherently supports the local user representation. On the other hand, the remote user does not physically exist in the real-world room, so the pass-through video capture or mixed reality vision of the real-world room is augmented with an avatar to represent the remote user. In a remote version of a shared XR environment (e.g., rendered by an XR system to a remote user), both the local user and the remote user may be represented by an avatar. For example, a room manager renders a virtual room that corresponds to the real-world room in the remote version, so the local user representation in the captured pass-through video or mixed reality vision is not available.

[0058] In the local version, the local user's XR system can: A) selectively mute the local user's audio and allow the real-world audio in the real-world room to be passed through; and B) output the remote user's audio as spatial audio relative to the remote user's position in the shared XR environment. In the remote version, the remote user's XR system can output both the local user and the remote user's audio as spatial audio relative to the users' positions in the shared XR environment.

[0059] In some embodiments, both the remote version and the local version of the shared XR environment include an XR table and an XR board. For example, a user can sit at an XR table, and the user can interact with the XR board (e.g., draw on the XR board) so that input at the XR board is rendered to other users in the shared XR environment. In the local version, the local user can stand up from the XR table (e.g., a table in a real-world room that is mapped to the XR table) and physically approach the XR board (e.g., a flat vertical surface in the real-world room that is mapped to the XR board) to interact with the XR board. On the other hand, the remote user is not physically located in the real-world room and therefore cannot interact with the XR board in the same way.

[0060] In some embodiments, the remote user is located in a space remote from the real-world room that is accessible to the remote user, such as a home office or any other suitable space. Additional mapping of the XR pad can be used to allow the remote user to interact with the XR pad in a space remote from the real-world room. For example, the XR pad can be mapped to a vertical surface at a remote space accessible to the remote user. In this example, the remote user can physically walk to the mapped vertical surface to interact with the XR pad.

[0061] The room manager may also generate occlusion masks for virtual objects and / or artificial reality (XR) user representations in the XR environment. Occlusion masks may be virtual objects that, while not having a visual appearance themselves, when they are placed in the XR environment between a user's viewpoint and a portion of a virtual object, the portion of the virtual object is not rendered to the user. The XR environment may correspond to a captured pass-through video of a real-world room or a mixed reality vision of the real-world room, and user presence in the XR environment (e.g., local user presence) may be represented by a pass-through video (or mixed reality vision) of the user in the real-world room. For multiple users (i.e., local users) physically co-located in the real-world room, the XR environment may be displayed in a pass-through mode or a mixed reality mode, where the user may view the real-world room with virtual object overlays. The position and posture of the avatar corresponding to the local user may be tracked by each user's XR system, for example, to manage the interaction between the user presence in the XR environment and one or more virtual objects / user avatars in the XR environment.

[0062] Avatar position and pose data can be shared with other XR systems rendering the XR environment. In some embodiments, the receiving XR system can generate and attach an occlusion mask based on the received avatar position and position data (even when the local user's avatar is not displayed by the XR system in the XR environment). Consider two local users, each of whose XR systems shares avatar tracking information and pose data with the other. In some embodiments, a first XR system of a first local user can generate an occlusion mask (in the XR environment rendered to the first user) and fix the occlusion mask to the position of the second local user. In this example, portions of the virtual object that are behind the second user (from the perspective of the first user) are selectively not displayed due to the occlusion mask.

[0063] In some embodiments, an XR board (e.g., a virtual whiteboard) with virtual writing can be presented / rendered (e.g., rendered as an augmented body) in an XR environment. Consider two local users with an XR system, if the first local user moves between the second local user and the virtual writing on the XR board, the first local user's pass-through video presentation or mixed reality visual presentation may cause rendering errors. For example, without the disclosed occlusion mask, when the first local user (from the perspective of the second local user) is in front of the XR board, the virtual writing may be incorrectly rendered in the presence of the first local user. In this example, the second user's XR system can use the tracking information and pose data of the first user's avatar shared by the first user's XR system (although the avatar is not displayed in the XR environment) to generate and implement the occlusion mask. For example, the second user's XR system can generate an occlusion mask based on the shared avatar information and fix the occlusion mask to the position of the first user, so that parts of the virtual writing (e.g., parts behind the occlusion mask from the perspective of the second user) are not rendered in the XR environment. Thus, the occlusion mask may prevent rendering of virtual writing from an XR board behind the user's position.

[0064] In some embodiments, an occlusion mask can be applied to an object in the XR environment (e.g., an XR table, an XR chair, etc.). For example, when the local user's XR system renders an avatar of a non-local (i.e., "remote") user in a pass-through or mixed reality display, an occlusion mask applied to the object can cause portions of the remote user's avatar to disappear as if those portions were behind the object. An occlusion mask can be applied to any suitable element in the XR environment to mitigate rendering errors.

[0065] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0066] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images (e.g., by adding virtual objects) as they pass through the system. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially consists of light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.

[0067] Some traditional remote support systems can generate shared XR experiences for users, however these systems cannot support collaboration between local and remote users. For example, some existing systems generate a virtual room where each participant is an avatar. However, pure virtual experiences reduce the collaboration that is possible with real-world displays and do not support collaboration between local users in the same location. Video calls and video conferences can allow for real-world representations of users. However, these experiences are limited to two-dimensional representations and also cannot support collaboration between users in the same location.

[0068] The various embodiments disclosed herein generate multiple versions of a shared XR environment: a local experience that allows collaboration between users in the same location and a remote experience that allows remote users to participate in the collaboration. For example, because the various embodiments use pass-through video capture or mixed reality vision of the real-world room in the local version of the shared XR environment, local users can interact with the real-world versions of other local users. In addition, remote users can participate using a remote version of the shared XR environment that renders a virtual room corresponding to the real-world room occupied by the local user. This virtual copy of the real-world room brings the local user closer to the remote user and allows a high level of interaction with the remote user. In addition, the split experience for remote and local users means that local users can take advantage of the collaborative opportunities brought about by being in the same location without excluding remote users.

[0069] Embodiments also improve mapping techniques for mapping real-world rooms to support shared XR environments. For example, validation criteria are incorporated into the mapping process to ensure that a received mapping definition will support a valid shared XR environment. Additionally, given a mapping definition, embodiments display previews of both the local experience and the remote experience. These previews enable the user performing the mapping to fully understand how the mapped real-world room will support the shared XR environment from the perspective of both the local user and the remote user.

[0070] Embodiments also generate occlusion masks for the local version of the shared XR experience, which can mitigate rendering issues. For example, a pass-through video capture or mixed reality vision of a real-world room augmented with virtual objects and / or remote user avatars may encounter rendering issues related to the depth of the rendered augmented volume when using traditional techniques. However, embodiments utilize tracking of the non-displayed user avatar performed by the XR system worn by the user to generate and place occlusion masks, thereby mitigating these rendering issues.

[0071] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1is a block diagram showing an overview of devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of a computing system 100 that map real-world rooms to configure a shared artificial reality room. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted viewer and other processing operations are offloaded to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted view device is described. In some implementations, location data and environmental data may be collected solely by sensors incorporated into the head mounted view device, while in other implementations, one or more of the plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.

[0072] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0073] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits the information to the processors 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0074] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, or a wireless connection, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with a hardware controller for each device (e.g., for a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen, a light emitting diode (LED) display screen, a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0075] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, GPS units, LiDAR or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment and track the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0076] The computing system 100 may include a communication device that can communicate wirelessly or wired with other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using a Transmission Control Protocol / Internet Protocol (TCP / IP) protocol. The computing system 100 can use the communication device to distribute operations across multiple network devices.

[0077] Processor 110 may access memory 150, which may be included in one of the computing devices of computing system 100, or may be distributed across the computing devices of computing system 100, or across multiple other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, tape drive, etc. Memory is not a propagation signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160 that stores programs and software, such as operating system 162, room manager 164, and other applications 166. The memory 150 may also include a data store 170 which may include, for example, stored room maps, XR object definitions, semantic information, user data, avatar tracking and pose data, occlusion mask data, configuration data, settings, user options or preferences, and the like, which may be provided to the program memory 160 or any element in the computing system 100.

[0078] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0079] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a band 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. Position sensor 220, IMU 215, and computing unit 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensor 220, and locator 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locator 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shape and position of real objects around the HMD 200.

[0080] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), etc.), some other display, or some combination thereof.

[0081] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC can use the HMD in conjunction with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0082] Figure 2B 2 is a line diagram of a mixed reality HMD system 250 that includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0083] The projector can be coupled to the see-through display 258, for example, via an optical element, to display media to the user. The optical element can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for directing light from the projector to the user's eyes. The image data can be transmitted from the core processing component 254 to the HMD 252 via a link 256. The controller in the HMD 252 can convert the image data into a plurality of light pulses from the projector, which can be transmitted to the user's eyes via the optical element as output light. The output light can be mixed with the light passing through the display 258, thereby allowing the output light to present the following virtual objects: These virtual objects appear to exist in the real world.

[0084] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0085] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held in one or both hands by a user to interact with an artificial reality environment presented by HMD 200 and / or HMD 250. Controllers 270 can communicate with the HMD directly or via an external device (e.g., core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit light points farther away. The HMD 200 or HMD 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., tracking the controllers with 3DoF or 6DoF). The computing unit 230 or core processing component 254 in the HMD 200 can combine the IMU output and the position output and use this tracking to monitor the position and movement of the user's hands. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A-274B) that a user may actuate to provide input and interact with objects.

[0086] In various embodiments, HMD 200 or HMD 250 may also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor indications of user interaction and intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or HMD 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate one or both eyes of the user, and HMD 200 or HMD 250 can use eye-facing cameras to collect reflections of light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes and determine gaze direction.

[0087] Figure 3 305B may be a HMD 200 or a HMD system 250. The plurality of client computing devices 305 may be operated in a network environment using logical connections to one or more remote computers (e.g., server computing devices) through a network 330.

[0088] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, each of these server computing devices may be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0089] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0090] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.

[0091] Figure 44 is a block diagram showing a plurality of components 400 that can be used in a system that employs the disclosed technology in some embodiments. Each component 400 can be included in a device of the computing system 100, or can be distributed on multiple devices of the computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, a system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., in memory 315 or 325) or other network memory accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0092] Middleware 420 may include components that communicate resources between hardware 410 and a plurality of specialized components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

[0093] The plurality of specialized components 430 may include software or hardware configured to perform operations for mapping a real-world room to configure a shared artificial reality room. The plurality of specialized components 430 may include: a room mapping coordinator 434; a shared XR room controller 436; an XR object controller 438; a local manager 440; a remote manager 442; an occlusion mask controller 444; and components and application programming interfaces (APIs) (e.g., interface 432) that may be used to provide a user interface, transfer data, and control specialized components. In some implementations, each component 400 may be in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more specialized components 430. Although each specialized component 430 is depicted as a separate component, the specialized components may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0094] The room mapping coordinator 434 can generate a mapping of a real-world room that supports a shared XR room environment. For example, the real-world room may include real-world objects and real-world surfaces, such as one or more tables, one or more chairs, one or more walls, one or more doors, one or more windows, and other suitable objects or surfaces. The room mapping coordinator 434 can generate an XR object definition based on the information received about the real-world room, one or more real-world objects, and one or more real-world surfaces. For example, the room mapping coordinator 434 can implement a process that guides a user equipped with an XR system to provide information for an XR object definition, such as real-world surfaces mapped to one or more XR objects, boundaries (e.g., boundaries on real-world surfaces) mapped to XR objects (e.g., measured using components of the XR system), semantic information (e.g., the number of seat assignments at the XR table, the size of the XR object, etc.), and other suitable information. Reference Figures 5 to 11 , Fig. 20 , Fig.29 and Fig.30 The description further describes the functionality of the room coordinator 434.

[0095] The shared XR room controller 436 can manage the shared XR room environment. For example, a shared XR room can be hosted by a mapped real-world room according to a mapping definition, and the shared XR room controller 436 can retrieve the stored mapping of the real-world room to launch the shared XR room environment. An embodiment of the shared XR room controller 436 can, for example, launch a shared XR room environment between multiple users (e.g., remote users and local users) in response to a request from a user. The shared XR room controller 436 can also manage users participating in the shared XR environment, such as user invitations, privacy settings, security protocols, and other suitable functions. In some embodiments, the shared XR room controller 436 can interact with the XR object controller 438 to manage XR objects within the shared XR room environment. In some embodiments, the shared XR room controller 436 may interact with a local manager 440 to manage a local version of the shared XR room (e.g., a version displayed to local users physically present in a real-world main room) and with a remote manager 442 to manage a remote version of the shared XR room (e.g., a version displayed to remote users physically present in a space away from the real-world main room).

[0096] The XR object controller 438 can manage the display, tracking, and interaction of XR objects in the shared XR environment. For example, the XR object controller 438 can configure XR objects based on the object definition of the stored real-world room mapping. Example XR objects include one or more XR tables, one or more XR boards, one or more XR chairs, one or more XR windows, one or more XR doors, one or more XR screens, and other suitable XR objects. The mapping definition for the XR object may include the mapped real-world surface, the boundaries at the mapped real-world surface, semantic information, and other suitable information for configuring the display and functionality of the XR objects in the shared XR environment. In some embodiments, the shared XR object controller 438 can interact with the local manager 440 to manage the local version of the shared XR room (e.g., the version displayed to the local user who is physically present in the real-world main room) and interact with the remote manager 442 to manage the remote version of the shared XR room (e.g., the version displayed to the remote user who is physically present in the space away from the real-world main room). Reference Figures 12 to 17 , Figure 21 to Figure 26 as well as Fig.31 The description further describes the functionality of the XR object controller 438 .

[0097] The local manager 440 can manage one or more local versions of a shared XR environment, such as versions that render a pass-through vision (e.g., captured video) or mixed reality vision of a real-world room that hosts the shared XR environment augmented with XR objects. For example, the local manager 440 can render an overlay on an XR table as a pass-through video captured in the real-world room of the table augmented with one or more augmentations or XR objects (e.g., mapped to the XR table). In another example, the local manager 440 can render an XR board as a pass-through video captured in the real-world room of a wall augmented with one or more augmentations or XR objects (e.g., mapped to the XR table). Implementations of the local manager 440 render local users as captured pass-through vision or mixed reality vision of users within the real-world room itself, and render remote users as avatars (e.g., as computer-generated drawings or captured and displayed images). In some implementations, the local manager 440 renders the local version of the shared XR environment using a real-world background of the real-world room. Reference Figures 12 to 17 , Fig.19 , Figure 21 to Figure 26 as well as Fig.31 The description proceeds to further describe the functionality of the local manager 440 .

[0098] The remote manager 442 can manage one or more remote versions of the shared XR environment, such as rendering a virtual room corresponding to a main real-world room and rendering XR objects as versions of virtual objects according to stored object definitions. For example, the remote manager 442 can render an XR table as a virtual object such that the relative position of the rendered XR table in the virtual room corresponds to the relative position of the real-world mapping defined in the stored object definition for the XR table in the real-world room. In another example, the remote manager 442 can render an XR board as a virtual object such that the relative position of the rendered XR board in the virtual room corresponds to the relative position of the real-world mapping defined in the stored object definition for the XR board in the real-world room. Implementations of the remote manager 442 render remote users as avatars. In some implementations, the remote manager 442 renders the remote version of the shared XR environment using a virtual background. Reference Figures 12 to 17 , Fig.18 , Figure 21 to Figure 26 as well as Fig.31 The description proceeds to further describe the functionality of the remote manager 442 .

[0099] The occlusion mask controller 444 can generate, manage, and render occlusion masks, such as occlusion masks that are sized and positioned based on the user's determined avatar pose / position (where the mask can be displayed regardless of whether the avatar is displayed) or occlusion masks attached to one or more virtual objects. For example, various embodiments render virtual augmentations (e.g., XR objects) in conjunction with pass-through vision (e.g., a captured video of a real-world room) or mixed reality vision. Due to this heterogeneous environment, the relative depth of the rendered elements may cause display errors, such as: A) a virtual object located behind a user via pass-through rendering is rendered as covering the user; or B) a user's avatar located behind a virtual object via pass-through rendering is rendered as covering the virtual object. The occlusion mask controller 444 can generate one or more occlusion masks that are fixed to non-displayed avatar positions and / or virtual objects to achieve rendering that accurately reflects the depth of the rendered elements. Reference Fig.27A , Fig.27B , Fig.28A , Fig.28B , Fig.32 and Fig.33 The description proceeds to further describe the functionality of the occlusion mask controller 446 .

[0100] Figure 5is a conceptual diagram illustrating a representation of a real-world room that is mapped in some embodiments. Schematic diagram 500 illustrates a real-world room 502 that contains a table, a window, a wall, and two occupants. Embodiments may map the real-world room 502 and store the generated XR room configuration definition. The stored XR room configuration definition may then support a shared XR environment using the real-world room 502 that includes a pass-through version for a local user and an XR version for a remote user.

[0101] For example, room boundaries (eg, “guardians”) may be mapped within the real-world room 502 . Figure 6 600 illustrates a real-world room 502 and a room boundary 602. The room boundary 602 represents the space within the real-world room 502 that participates in the shared XR environment. For example, in the shared XR environment, virtual objects, avatars, other suitable versions of the user's presence, and / or any other XR elements can be located within the room boundary 602. In some embodiments, the room scale stream can be used, for example, by a user equipped with an XR device or XR system (e.g., an HMD, a controller, etc.) to measure out the room boundary 602. Any other suitable technique can be used to receive the room boundary 602.

[0102] In some implementations, mapping the real-world room 502 includes receiving selections and semantic information defining one or more configurations for the XR table. Figure 7 700 shows a user 702 and a surface 704. In the example shown, the user 702 is wearing an XR system (e.g., an HMD and one or more handheld controllers) 706, which can be used to draw the boundaries of the surface 704. Figure 8 , Fig. 9 , Fig.10 and Fig.11 1000 , 1102 , and 1103 are conceptual diagrams illustrating measuring table boundaries for a table object located in a mapped real-world room using an XR system. Diagrams 800 , 900 , and 1000 illustrate a real-world room 802 , a real-world surface 804 , an XR device 806 , conceptual instructions 808 , 902 , and 1002 , and an interface element 1102 .

[0103] In some embodiments, the XR system can instruct the user (e.g. Figure 7802 ) uses the XR system (e.g., XR device 806) to perform one or more actions to measure the boundaries of the table object being configured. In schematic 800, as shown by conceptual instructions 808, the user is instructed to use the XR device 806 to tap the corner of the real-world surface 804 that is closest to the door in the real-world room 802. In schematic 900, as shown by conceptual instructions 902, the user is instructed to use the XR device 806 to tap the corner of the real-world surface 804 that is at the other end of the real-world room 802. In schematic 1000, as shown by conceptual instructions 1002, the user is instructed to use the XR device 806 to tap the surface of the real-world surface 804.

[0104] In some embodiments, using signals received from the XR device 806 and conditions sensed within the real-world room 802 (e.g., vision captured by one or more cameras), a boundary and alignment can be generated for the table object. For example, as shown in schematic 1100, the XR system can display the generated boundary and alignment for the table object to the user, and the user can compare the boundary and alignment with the real-world surface 804. When the boundary and alignment are acceptable, the user can provide confirmation using interface element 1102. Otherwise, the user can restart and execute the processes shown in schematics 800, 900, and 1000 until acceptable boundaries and alignment are generated for the table object.

[0105] In some embodiments, once verification is received from a user of boundaries and alignment indicating placement of a table object in the real-world room 802 (e.g., along the real-world surface 804), semantic information defining the table object as an XR table can be received from the user. For example, the table object can be defined as a multi-person XR table, and one or more seat assignments can be generated for the defined XR table. For example, a user can enter a number of seat assignments for the XR table, and the XR table can be automatically divided to generate the seat assignments (e.g., same-sized arrangements, predefined sizes or size ranges, etc.). In another example, a user can measure the seat assignments using an XR device 806. The semantic information for the XR table can include the defined seat assignment divisions.

[0106] In some embodiments, a user may use elements of the XR system (e.g., XR device 806) to map other surfaces in the real-world room to generate configuration definitions for other suitable objects, and semantic information defining these objects as XR room elements may be received, such as one or more XR boards, one or more XR screens, one or more XR chairs, one or more XR doors, one or more XR windows, or other suitable XR elements. For example, the XR device may be used to measure boundaries on flat vertical surfaces in the real-world room (e.g., real-world walls, real-world boards, real-world screens, etc.) to define configuration definitions for board objects. A user may provide semantic information defining a board object as an XR board, and configuration definitions for the XR board may be stored.

[0107] In another example, an XR device can be used to measure boundaries on a flat vertical surface in a real-world room to define a configuration definition for a screen object (e.g., a television, a computer screen, etc.). A user can provide semantic information that defines a screen object as an XR screen, and the configuration definition for the XR screen can be stored. In some embodiments, the XR screen can be automatically placed when the XR table is mapped. For example, the XR screen can be automatically mapped to one end of the XR table relative to the seating division of the XR table. In some embodiments, the XR screen can be repositioned / remapped by the user.

[0108] In another example, an XR device may be used to measure boundaries on vertical surfaces (e.g., walls, real-world panels, real-world doors, etc.) in a real-world room to define a configuration definition for a door object. A user may provide semantic information defining a door as an XR door, and the configuration definition for the XR door may be stored. In another example, an XR device may be used to measure boundaries on vertical surfaces (e.g., real-world walls, real-world panels, real-world doors, real-world windows, etc.) in a real-world room to define a configuration definition for a window object. A user may provide semantic information defining a window as an XR window, and the configuration definition for the XR window may be stored. In some embodiments, the defined one or more XR doors and one or more XR windows correspond to one or more real-world doors and one or more real-world windows in a real-world room. In this example, the XR doors and XR windows displayed to the remote user in the VR version of the shared XR environment may include virtual doors and virtual windows corresponding to the real-world doors and real-world windows displayed to the local user in the pass-through version of the shared XR environment.

[0109] In another example, an XR device may be used to measure boundaries on a horizontal surface in a real-world room (e.g., a chair, bench, etc.) to define a configuration definition for a chair object (e.g., a seat portion of a chair). A user may provide semantic information defining the chair object as an XR chair, and the configuration definition for the XR chair may be stored. In some embodiments, a virtual representation of XR elements defined for a real-world room mapping is displayed to a remote user in a VR version of the shared XR environment, while the XR elements are represented using a pass-through / mixed reality vision of the real-world elements in a pass-through / local version of the shared XR environment.

[0110] In some embodiments, the coordinator can automatically identify objects mapped from a real-world room (e.g., by a user wearing an XR system) and automatically generate semantic information defining these objects as XR elements. For example, the coordinator can automatically mark a table object (e.g., a horizontal flat surface that is at a threshold size or meets a size standard) as an XR table. In another example, the coordinator can automatically mark a board object (e.g., a vertical flat surface that is at a threshold size or meets a size standard) as an XR board. In other examples, door objects (e.g., vertical surfaces that meet door size standards and placement (e.g., adjacent to the floor)), window objects (e.g., vertical surfaces that meet window size and placement standards (e.g., floating / not adjacent to the floor)), chair objects (e.g., horizontal surfaces that meet chair size and placement standards (e.g., adjacent to a table object / XR table)), and other suitable objects can be automatically marked with semantic information defining them as XR elements.

[0111] Configuration definitions for each of these XR elements may be stored as part of a mapping definition for a real-world room. In some implementations, the mapping definition for a real-world room may be compared to a validation standard before completing the mapping process. The validation standard may include a minimum set of XR elements, such as: a) one XR table, one XR board; b) one XR table, one XR board, one XR screen; c) one XR table, one XR board, one XR door, one XR window; d) one XR table, one XR board, at least two XR chairs; e) any combination thereof or any other suitable standard.

[0112] When the mapping definition for the real-world room meets the validation criteria, the real-world room mapping definition can be stored for reuse. For example, the stored mapping definition for the real-world room can be used to generate a shared XR environment (e.g., a pass-through version) for a local user and a shared XR environment (e.g., a VR version) for a remote user. In some embodiments, multiple users can use the stored mapping definition for the real-world room to launch a shared XR environment. In some embodiments, the mapping definition for the real-world room can be edited. For example, a user can remap an existing definition (e.g., semantic information, boundaries, etc.), delete a previous mapping, add a new mapping, and the like. In this example, the edited mapping definition can then be stored for reuse, for example to align the mapping of the real-world room with the reconfiguration of the furniture.

[0113] In some embodiments, a mapping definition for a real-world room can limit elements in the real-world room that are rendered in a local version of a shared XR environment hosted in the real-world room. For example, the local version renders the real-world room using the captured pass-through or mixed reality vision, but elements in the real-world room can be selectively displayed in the pass-through / mixed reality vision. For example, elements in the real-world room that are not explicitly included can be excluded from rendering in the local version. In another example, certain groups, categories, or collections of room elements (e.g., elements on a wall) can be excluded from rendering in the local version. In another example, a user can explicitly select room elements to be excluded from the local version.

[0114] Fig.12 is a conceptual diagram showing local users and remote users in a shared XR environment including an XR table and an XR board. Figures 5 to 11 The disclosed mapping process can be used to generate a mapping definition for a real-world room. The mapping definition can include configuration definitions for several XR elements that define how to place the XR elements in the real-world room, as well as semantic information for the XR elements. Schematic diagram 1200 shows a shared XR environment 1202, seat assignments 1204, remote users 1206, local users 1208, an XR board 1210, and an XR screen 1212.

[0115] The shared XR environment 1202 is a pass-through / local version. For example, the shared XR environment 1202 can be displayed to a local user using an XR system. The perspective displayed in the shared XR environment 1202 can be the perspective of the local user. The local user 1208 is physically present in the real-world room and is represented in the shared XR environment 1202 using the local user's pass-through real-world vision (e.g., video capture) or the local user's mixed reality vision. In another example, the remote user 1206 is represented as an avatar version in the shared XR environment 1202 because the remote user is not physically present in the real-world room.

[0116] The XR table 806 mapped and defined when mapping the real-world room includes six seat placements. For example, two local users and one remote user are seated at the XR table 806. Seat placement 1204 is vacant, so a new user (e.g., a local user or a remote user) joining the shared XR environment 1202 can select seat placement 1204. The real-world mapping definition for the real-world room implementing the shared XR environment 1202 also includes configuration definitions for the XR board 1210 and the XR screen 1212.

[0117] For example, the XR screen 1212 is located on (e.g., mapped to) a real-world wall in the shared XR environment 1202. In some embodiments, during the shared XR environment 1202, the XR screen 1212 can display video of one or more video users (e.g., users who are away from the real-world room and are not rendered by the remote / VR version of the shared XR environment). For example, the video users can view the shared XR environment 1202, but may be present in limited capabilities (e.g., unable to sit at the XR table 806, unable to interact with the XR board 1210). The XR screen 1212 can display any other suitable content, such as videos, websites, presentations, documents, etc.

[0118] The XR board 1210 is located on (e.g., mapped to) a real-world wall in the shared XR environment 1202. The XR board 1210 can be an XR whiteboard that receives input (e.g., sensed handwriting input) from one or more local users or remote users and renders the input to the users of the shared XR environment 1202. In some embodiments, multiple users can be located at the XR board 1210 at the same time. The XR board 1212 can display any other suitable content, such as a video, a website, a presentation, a document, etc. In some embodiments, one or both of the local users 1208 and 1212 can physically approach the real-world wall to which the XR board 12010 is mapped and trigger an interaction with the XR board 1210.

[0119] Fig.13 , Fig.14 , Fig.15 , Fig.16 and Fig.17 1302, an XR board 1304, an XR table 1306, an interface panel 1402, an XR board proximity area 1502, an interface panel 1602, and an XR board assembly 1702.

[0120] Schematic 1300, 1400, 1500, 1600, and 1700 represent perspectives of local users (e.g., a pass-through version of a shared XR environment from the perspective of one of the local users). For example, the XR system displays the perspectives represented by schematic 1300, 1400, 1500, 1600, and 1700 to the local users. Schematic 1300 shows a shared XR environment 1302 including an XR board 1304 and an XR table 1306. In the shared XR environment 1302, the local users are seated in a seating arrangement at the XR table 1306 and can view the XR board 1304 from the seating arrangement. The XR board 1304 displays handwriting input.

[0121] In some embodiments, the local user can interact with the XR board 1304 by standing up (e.g., standing up) from the assigned seat and physically approaching the XR board 1304 (e.g., walking up to the board) to trigger the interaction. In schematic 1400, the XR system detects that the local user is moving (e.g., when the local user stands up or moves out of the predefined position of the local user's assigned seat), or the local user indicates movement using an interface component (e.g., pressing a button that starts a movement flow). When the movement flow is started / detected, the XR system can display an interface component 1402 to the local user, which signals the local user that the user is free to move to another seat at the XR table 1306 or to the XR board 1304.

[0122] In the example shown, the local user moves toward the XR pad 1304. For example, in diagram 1500, the user has left the XR table 1306 and is approaching the XR pad 1304. In some embodiments, once the local user is within a threshold distance from the XR pad 1304 (e.g., within the pad proximity region 1502), the XR system can trigger interaction with the XR pad 1304. In diagram 1600, the user has reached the XR pad 1304 (e.g., within the pad proximity region 1502), and the XR system displays the interface panel 1602 to the user. For example, the user can initiate an input mode at the XR pad 1304 by continuing via the interface panel 1602.

[0123] In diagram 1700, the user has initiated input mode, causing the XR system to sense user movement and render input corresponding to the user movement at XR pad 1304. For example, once input mode is initiated, the user may draw, write alphanumeric characters, and otherwise render indicia on XR pad 1304. XR pad component 1702 may be used to select a configuration of the input at XR pad 1304, such as the color of the rendered input, and other suitable configurations.

[0124] In some embodiments, a first XR system renders a shared XR environment as a local version (e.g., a pass-through of captured visuals or mixed reality) to a local user, and a second XR system renders the shared XR environment as a remote version (e.g., a virtual room) to a remote user. For example, a local user may be physically present in a real-world room hosting the shared XR environment, while a remote user may be located elsewhere (e.g., in a home office, a remote office, or any other suitable remote location). In some embodiments, both the local user and the remote user present in the shared XR environment may interact with the displayed XR board.

[0125] Fig.18 18 is a conceptual diagram illustrating avatar movement and XR board interaction of a remote user within a virtual reality room that is part of a shared XR environment. Schematic diagram 1800 illustrates movement examples 1802, 1804, and 1806 for the remote user. In some embodiments, the remote user is remote from the real-world room hosting the shared XR environment. A remote version of the XR environment (e.g., a virtual room corresponding to a real-world room with one or more XR objects) is displayed to the remote user by the XR system. The depicted XR environment includes an XR table with divided seating assignments and an XR board located on one of the walls of the virtual room.

[0126] In schematic 1802, the remote user is seated in a seating assignment at the XR table. Because the user is remote and a remote version of the shared XR environment (e.g., a virtual room version) is displayed to the user, the user cannot move freely in the room. To move to the XR board, the user selects an interface component (not shown) that initiates progression from the seating assignment at the XR table to the XR board. In schematic 1804, the XR system is executing / loading the remote user's progression to the XR board, as indicated by the remote user's closed eyes. In schematic 1806, the XR system has caused the remote user to progress to the XR board so that the remote user can initiate input mode with the XR board.

[0127] Fig.19 19 is a conceptual diagram illustrating real-world movement of a local user within a real-world room that is part of a shared XR environment. Schematic diagram 1900 illustrates movement examples 1902, 1904, and 1906 for a local user. In some embodiments, a local user is located in a real-world room hosting a shared XR environment. A local version of the XR environment (e.g., a pass-through vision of the real-world room or a mixed reality with augmented XR objects) is displayed to the local user by the XR system. The depicted XR environment includes an XR table with divided seating assignments and an XR board located on one of the walls of the pass-through / mixed reality room.

[0128] In diagram 1902, the local user is seated in the seating assignment at the XR table. Since the user is local and the user is shown a local version of the shared XR environment, the user can move freely around the room. To move to the XR board, the user can stand up and walk to the XR board. In diagram 1904, the local user has stood up and walked to the XR board. In diagram 1906, the local user has reached the XR board and can initiate input mode with the XR board.

[0129] Fig.18 and Fig.19 A movement flow is distinguished for how a remote user and a local user interact with an XR board during a shared XR environment. A local user may interact with an XR board that is mapped to a real-world room in which the local user is located, however, the remote user is not in the real-world room and therefore cannot interact with the XR board in the same manner. In some embodiments, the XR board may be mapped to a predefined location in a room / space occupied by (or accessible to) the remote user. For example, a remote user may have a mapped workspace that includes at least a mapping of the XR board and, in some embodiments, a mapping of a desk area.

[0130] Fig. 20is a conceptual diagram illustrating an example 2000 of creating an artificial reality work environment for a remote user that is linked to one or more real-world spaces. Fig. 20 Includes a desk area 2002 and a whiteboard area 2010. User 2012 can move between multiple real-world areas to move into corresponding linked artificial reality work environment areas and perform actions specified for those areas. Desk area 2002 includes a dedicated space 2004 that displays content items such as content item 2006. When standing up and moving to whiteboard area 2010, whiteboard 2008 is active. In some embodiments, whiteboard 2008 can be part of a shared XR environment so that each content item is automatically populated from the XR board as part of the shared XR environment.

[0131] For example, user 2012 may be a remote user participating in a shared XR environment hosted in a real-world room. The XR system may display a remote version of the XR environment to user 2012, such as a virtual room with XR objects corresponding to an enhanced version of the real-world room (e.g., a captured vision of the real-world room enhanced with XR objects). In some embodiments, the real-world room may have an XR board mapped to a physical location in the real-world room, so that a local user physically located in the real-world room can approach the XR board and interact with the XR board. Whiteboard 2008 may be linked to an XR board in a shared XR environment. For example, content drawn by one or more local users physically present in the real-world room may be displayed at whiteboard 2008. In addition, content drawn by user 2012 at whiteboard 2008 may be displayed to a local user in a local version of the shared XR environment.

[0132] In some embodiments, the shared XR environment may include an XR table at which one or more remote users and local users are seated. When user 2012 transitions from the XR table to the XR board in the shared XR environment, user 2012 may physically move from desk area 2002 to whiteboard area 2010. In these examples, user 2012 (e.g., a remote user) may interact with the XR board in the shared XR environment using a mapping of the physical real-world space (e.g., whiteboard area 2010) of the XR board at a remote room (e.g., a room in which user 2012 is located or accessible). In some embodiments, for each remote user participating in the shared XR environment, a different mapping to the physical real-world space is used to implement the XR board in several different remote rooms (e.g., one room for each remote user).

[0133] Figure 21 to Figure 262 is a schematic diagram showing multiple versions of a shared XR environment in which users approach and interact with an XR pad. Schematic diagrams 2100, 2200, 2300, 2400, 2500, and 2600 show a local user perspective 2102, a remote user perspective 2104, and a camera view 2106. For example, users present in the shared XR environment may include local users 2112, 2114, and 2116 and remote user 2202. In some implementations, video user 2204 is a limited participant in the shared XR environment.

[0134] The local user perspective 2102 is the perspective of the local user 2112 (e.g., a local version of the shared XR environment displayed by the XR system to the user 2112). For example, the local user perspective 2102 is a pass-through vision of a real-world room augmented with XR objects (e.g., XR board 2110). The remote user perspective 2104 is the perspective of the remote user 2202 (e.g., a remote version of the shared XR environment displayed by the XR system to the user 2202). For example, the remote user perspective 2102 is a virtual room representation of the real-world room with displayed XR objects (e.g., XR board 2110, an XR table corresponding to a real-world table). The camera view 2106 is a video captured by a camera (e.g., a fixed camera) located in the real-world room hosting the shared XR environment. For example, the camera can be part of the XR system, or any other suitable imaging device. Diagrams 2100 , 2200 , 2300 , 2400 , 2500 , and 2600 illustrate the local user perspective 2102 , the remote user perspective 2104 , and the camera perspective 2106 as a sequence that progresses forward in time.

[0135] Schematic 2100 shows a camera view 2106 showing local users 2112, 2114, and 2116 sitting at chairs and tables physically present in a real-world room hosting a shared XR environment. The local user perspective 2102 depicts XR augmentations, such as an XR board 2110 (which displays written text) and an XR panel 2108, in the visual of the real-world room displayed in the local version of the shared XR environment, which the user 2112 can use to contribute during the shared XR environment (e.g., send messages to other users, etc.). The local user perspective 2102 in the schematic 2100 includes a real-world visual depiction of the user 2116 (e.g., pass-through vision and / or mixed reality vision).

[0136] Diagram 2100 also shows a remote user perspective 2104 or remote user 2202 (in Fig. 22 But in Fig.21The remote version of the shared XR environment displays avatars of local users 2112, 2114, and 2116, which are located at the XR table in seats corresponding to the physical seats of these local users in the real-world room.

[0137] Schematic 2200 shows a camera view 2106 where local users 2112 and 2114 are still sitting in chairs at the physical table, however local user 2112 has stood up and repositioned to a standing position. Schematic 2200 also shows a local user perspective 2102 after local user 2112 has stood up and repositioned, which includes the avatar of remote user 2202 (sitting at the XR table, to the left of the local user's 2112 seat, which is not occupied) and the video user 2204 displayed on the XR screen 2204. The difference between the camera view 2106 and the local user perspective 2102 in schematic 2200 shows the augmentations of the pass-through vision / mixed reality vision added to the local version of the shared XR environment, such as the avatar of the remote user 2202, the XR screen 2204, the XR panel 2108, and the XR board 2110. Diagram 2200 also shows the remote user perspective 2104, which continues to display the avatars of local users 2114 and 2116 in seats at the XR table, which seats correspond to the physical seats of these local users in the real-world room (the avatar of local user 2112 is not visible based on the head positioning / gaze of remote user 2202).

[0138] Schematic 2300 shows camera view 2106 with local users 2112 and 2116 seated in chairs at a physical table, however local user 2114 has stood up and is repositioning (moving to XR pad 2110 shown in local user perspective 2102). Schematic 2300 also shows local user perspective 2102 (after local user 2112 has been seated at the physical table), which includes the avatar of remote user 2202. Schematic 2300 also shows remote user perspective 2104, which continues to show the avatar of local user 2116 in a seat at the XR table that corresponds to the local user's physical seat in the real-world room, however the avatar of local user 2114 has closed its eyes to indicate that local user 2114 is repositioning in the real-world room.

[0139] Schematic 2400 shows camera view 2106, where local users 2112 and 2116 are sitting in chairs at a physical table, and local user 2114 is located at a wall mapped to XR board 2110. Schematic 2400 also shows local user perspective 2102, which continues to display the avatar of remote user 2202, however the avatar has closed its eyes to indicate that remote user 2202 is repositioning. Schematic 2400 also shows remote user perspective 2104, which has transitioned from a view of the remote version of the shared XR environment to a pass-through vision of the user's real-world environment. For example, user 2202 is physically repositioned to a real-world space mapped to XR board 2110, which is located in a real-world room occupied by (or accessible to) user 2202. The XR system of user 2202 detects the repositioning (e.g., by detecting that user 2202 has moved away from a predefined desk space) and displays interface element 2402, which user 2202 can use to confirm the repositioning.

[0140] Schematic 2500 shows camera view 2106 with local user 2116 sitting in a chair at a physical table, local user 2114 at a wall mapped to XR board 2110, and local user 2112 standing. Schematic 2500 also shows local user perspective 2102, which shows local user 2114 interacting with XR board 2110 (e.g., writing on the XR board). Schematic 2500 also shows remote user perspective 2104, which continues to display a pass-through vision of the user's real-world environment. User 2202 has moved to a real-world space mapped to XR board 2110, located in a real-world room occupied by user 2202. User 2202's XR system detects that user 2202 is approaching the real-world space mapped to XR board 2110, and the XR system displays interface element 2502, which user 2202 can use to trigger interaction with XR board 2110. The remote user perspective 2104 in diagram 2500 also shows an outline of an avatar of user 2114 interacting with the XR board 2110 .

[0141] Schematic 2600 shows a camera view 2106 with a local user 2116 sitting in a chair at a physical table, a local user 2114 at a wall mapped to an XR board 2110, and a local user 2112 standing. Schematic 2600 also shows a local user perspective 2102 showing the local user 2114 at the XR board 2110 and an avatar of a remote user 2202 interacting with the XR board 2110. Schematic 2600 also shows a remote user perspective 2104 that has been converted from a pass-through vision of the user 2202's environment back to a remote version of the shared XR environment. Specifically, the remote user perspective 2104 in schematic 2600 shows the remote user 2202 interacting with the XR board 2110. In the example shown, different real-world spaces are mapped to a shared XR board so that representations of users (different representations, such as avatars and pass-through vision / mixed reality vision) located in different physical spaces can all interact with the XR board.

[0142] Fig.27A 2700A shows a shared XR environment rendered to at least two users using a first XR system and a second XR system. For example, the shared XR environment may include two local users: a first user not depicted in schematic 2700A and user 2702. Schematic 2700A may be a local version of the shared XR environment displayed to the first user via the first XR system (e.g., the shared XR environment from the perspective of the first user). User 2702 may also be a local user to whom the local version of the shared XR environment is displayed via the second XR system. The shared XR environment may be hosted in a real-world room (e.g., with a stored mapping for the XR environment), and the local users may be physically present in the main real-world room.

[0143] Schematic 2700A shows that the local version of the shared XR environment includes XR object augmentations, such as an XR board 2704, an XR table 2706, and augmented text 2708. For example, the local version of the shared XR environment displayed from the perspective of the first user can be a pass-through video capture or mixed reality vision displayed to the user via the first XR system. The XR board 2704 can be a pass-through video capture of a real-world surface mapped to the XR board 2704 and augmented by one or more augmentations (e.g., augmented text 2708) displayed through the XR board 2704. A display error that may occur in this setting is to display the augmented text 2708 on top of the user 2702, even though the augmented text is behind the user 2702 from the perspective of the first user. For example, because the XR board 2704 and the augmented text 2708 are virtual elements (e.g., virtual objects) of the shared XR environment, and the user 2702 is a pass-through element of the shared XR environment, the first XR system may encounter problems with depth estimation. Embodiments may use the avatar data to generate an occlusion mask to mitigate this error.

[0144] Fig.27B 2700B includes a user 2702, an XR board 2704, an XR table 2706, augmented text 2708, and an occlusion mask 2710. For example, the occlusion mask 2710 can be attached to the determined position of the avatar of the user 2702 (even though the avatar is not displayed in the shared XR environment) to support effective occlusion of virtual objects relative to the user 2702. In some embodiments, the position tracking and pose data of the avatar of the user 2702 (within the shared XR environment) can be tracked by the second XR system (of the user 2702) and transmitted to the first XR system (of the first user). The first XR system can then generate the occlusion mask 2710 (e.g., based on the geometry of the avatar) and fix the mask to the non-displayed avatar position of the user 2702 based on the tracking and pose data. In some implementations, an occlusion mask 2710 can be attached to the position and pose data of the avatar of the user 2702 in the XR environment. For example, as depicted in diagram 2700B, the occlusion mask 2710 can prevent portions of the XR board 2704 and portions of the augmented text 2708 from being displayed in the shared XR environment.

[0145] In some cases, the occlusion mask may always be attached to the avatar position. In other cases, the first XR system may fix the occlusion mask 2710 to the undisplayed avatar of user 2702 when it is detected that the rendering of the user 2702 and the XR board 2704 overlap. For example, the first XR system receives tracking information and pose data for the undisplayed avatar of user 2702, and the first XR system stores the position of the XR board 2704 within the local version of the shared XR environment. When it is determined based on the tracking information and pose data for the undisplayed avatar of user 2702 that the user 2702 will overlap with the XR board 2704 (from the perspective of the first user), the first XR system may attach the occlusion mask 2710 to the undisplayed avatar of user 2702.

[0146] Although the avatar of user 2702 is not displayed in the shared XR environment, the data available through the position tracking of the avatar's pose data allows for the implementation of the occlusion mask 2710. In some embodiments, the occlusion mask 2710 is a structured shape (e.g., an ellipse, a square, other suitable polygon, etc.) or is shaped according to the geometry of the avatar of user 2702. In some embodiments, the occlusion mask 2710 is larger than the avatar of user 2702, for example to support an occlusion footprint that is larger than the avatar so that virtual objects / augmented volumes are not incorrectly rendered on user 2702.

[0147] In some embodiments, the first XR system can determine the speed of the user 2702 based on the tracking position and posture data received from the second XR system. The size of the occlusion mask 2710 can be adjusted based on the determined movement speed. For example, when the determined movement speed exceeds a speed threshold (e.g., 6 feet per second (ft / s), 8ft / s, 10ft / s, 12ft / s, etc.), the occlusion mask can be enlarged (e.g., 105%, 110%, 115%, 120%, etc.). In another example, when the determined movement speed is above the threshold, the size of the occlusion mask is set relative to the determined movement speed until the occlusion mask reaches a maximum size. Because the tracked position and pose data of user 2702's avatar does not completely correspond to user 2702's actual presence in the shared XR environment (e.g., a pass-through video capture of user 2702 is displayed instead of the avatar), the enlarged occlusion mask 2710 when user 2702 reaches a threshold speed can further mitigate the risk of virtual objects / augmentations being incorrectly rendered as covering user 2702.

[0148] In some embodiments, the opacity / transparency of the occlusion mask 2710 can be non-uniform, where the transparency of the occlusion mask increases near its edges. For example, when the occlusion mask is enlarged (e.g., based on the determined movement speed of the user 2702), the edges of the enlarged occlusion mask can have increased transparency to reduce the prominence of the enlarged occlusion mask. In some embodiments, the occlusion mask can also be generated for virtual objects displayed in a shared XR environment.

[0149] Fig.28A 2800A shows a shared XR environment rendered to at least two users using a first XR system and a second XR system. For example, the shared XR environment may include a remote user 2802 and a local user (a user not depicted in schematic 2800A). Schematic 2800A may be a local version of a shared XR environment displayed to a local user via a first XR system (e.g., a shared XR environment from the perspective of a local user). A remote version of the shared XR environment may be displayed to a remote user 2802 via a second XR system. The shared XR environment may be hosted in a real-world room (e.g., with a stored mapping for the XR environment), the user 2802 may be remote from the real-world room, and the local user may be physically located in the real-world room.

[0150] Schematic 2800A shows that the local version of the shared XR environment includes XR object augmentations, such as an XR board 2804, an XR table 2806, and a virtual object 2808. For example, the local version of the shared XR environment displayed from the perspective of the local user can be a pass-through video capture or mixed reality vision displayed to the local user via the first XR system. In some embodiments, the user 2802 can be an avatar representation, such as an avatar sitting in an assigned seat at the XR table 2806. In the example shown, the virtual object 2808 is located above the XR table 2806, and from the perspective of the local user, the virtual object and the XR table are both in front of the user 2802. However, rendering errors may occur that cause the user 2802 (e.g., the avatar representation) to occlude portions of the virtual object 2808. Various embodiments may generate an occlusion mask for the virtual object to mitigate this error.

[0151] Fig.28B2800B is a conceptual diagram illustrating an occlusion mask for mitigating display errors in a pass-through version of a shared XR environment. Schematic 2800B includes a user 2802, an XR board 2804, an XR table 2806, a virtual object 2808, and an occlusion mask 2810. For example, the occlusion mask 2810 can be attached to the virtual object 2808 to support effective occlusion relative to the virtual object. In some embodiments, the virtual object position and geometry can be tracked by the first XR system. The first XR system can then generate the occlusion mask 2810 (e.g., based on the tracked position) and fix the mask to the virtual object 2808. As depicted in schematic 2800B, the occlusion mask 2810 can cause portions of the avatar of the user 2802 to not be displayed in the shared XR environment.

[0152] In some implementations, when an overlap is detected between the avatar of user 2802 and the rendering of virtual object 2808, the first XR system can fix the occlusion mask 2810 to the virtual object 2808. For example, the first XR system receives tracking information and pose data for the avatar of user 2802, and the first XR system tracks the position of the virtual object 2808 within the local version of the shared XR environment. When it is determined that (from the perspective of the first user) the virtual object 2808 will overlap with the avatar of user 2802, the first XR system can fix the occlusion mask 2810 to the virtual object 2808.

[0153] In some embodiments, the occlusion mask 2810 is a structured shape (e.g., an ellipse, a square, other suitable polygon, etc.) or is shaped according to the geometry of the virtual object 2808. In some embodiments, the occlusion mask 2810 is larger than the virtual object 2808, for example to support a larger occlusion footprint than the virtual object so that the avatar / augmented body / virtual object is not incorrectly rendered on the virtual object 2808. In some embodiments, the opacity / transparency of the occlusion mask 2810 can be non-uniform, where the transparency of the occlusion mask increases near its edges.

[0154] It will be understood by those skilled in the art that the above Figures 1 to 26 , Fig.27A , Fig.27B , Fig.28A as well as Fig.28B The components shown in the flowcharts and the components shown in each of the flowcharts discussed below can be changed in various ways. For example, the logical order can be rearranged, multiple sub-steps can be performed in parallel, the logic shown can be omitted, other logic can be included, etc. In some embodiments, one or more of the above components can perform one or more of the processes described below.

[0155] Fig.292900 is a flow chart illustrating a process for mapping a real-world room to enable a shared artificial reality (XR) room in some embodiments of the present technology. In some embodiments, process 2900 may begin when a user puts on an XR device and / or when a user activates an artificial reality work environment application on their XR device. In other embodiments, process 2900 may begin when a user activates a setup or calibration process within an artificial reality work environment application.

[0156] At block 2902, process 2900 receives a request to implement a shared XR room using a real-world room. For example, a mapping of the real-world room configured with the shared XR room can be generated. In some implementations, the mapping can include one or more object definitions, semantic information for the XR object, boundaries of the XR object, and the like.

[0157] At block 2904, process 2900 can determine whether the room has a previously stored mapping. For example, a mapping definition can be stored for a real-world room that has been previously mapped. The mapping definition can be associated with a geographic location of the real-world room (e.g., an address and location in a building, a room number and building identifier, etc.). In some implementations, the geographic location of the real-world room can be used to search for previously stored mappings to determine whether a previously stored mapping exists.

[0158] When the room has a previously stored mapping, process 2900 may proceed to block 2906. When the room does not have a previously stored mapping, process 2900 may proceed to block 2908. At block 2906, process 2900 may retrieve the previously stored mapping. For example, when there is a previously stored mapping for the room, the user may: a) use the previously stored mapping to support a shared XR room environment; and / or b) edit the previously stored mapping to change semantic information of a previously stored object definition, edit a predefined boundary on a surface for a previously stored object definition, add a new object definition (e.g., using a process similar to blocks 2908 to 2914), etc.

[0159] At block 2908, process 2900 may select an object to be mapped. For example, the objects that may be selected for mapping include one or more tables, one or more boards, one or more chairs, one or more doors, one or more windows, one or more screens, and other suitable objects. The user may select one of the available objects to map, and follow the process of mapping the selected object via blocks 2910 and 2912, for example, as described in reference to FIG. Figures 7 to 11 The flow of the XR table described.

[0160] At block 2910, process 2900 may receive semantic information for the selected object. For example, the semantic information may include: a label identifying an XR object (e.g., an XR table, an XR board, an XR chair, an XR window, an XR door, etc.), the number of chairs / arrangements of the XR table, etc. In some implementations, a user with an XR system located in a real-world room may provide the semantic information.

[0161] In some embodiments, semantic information of at least one XR table and an XR board is received. For example, a first map and a second map corresponding to the XR table and the XR board can be received. The first map can be a boundary on a horizontal surface representing the real-world positioning of the XR table (e.g., a real-world map / anchor point), and the second map can be a boundary on a vertical surface representing the real-world positioning of the XR board (e.g., a real-world map / anchor point). In some embodiments, at least a portion of the boundary is identified by a user using an XR system (e.g., a handheld controller, an HMD, etc.).

[0162] In some embodiments, semantic information for an XR object may be automatically generated. For example, a seating division for users at an XR table may be automatically generated based on the number of seats. The seating assignment may be a defined area within a boundary (according to the semantic information for the XR table) on a flat horizontal surface. In some embodiments, the generated seating assignment (e.g., division) may enhance the semantic information for the XR table.

[0163] At block 2912, process 2900 may determine whether another object has been selected for mapping. For example, the user may select additional objects (e.g., one or more chairs, one or more doors, one or more screens, one or more windows, one or more additional tables, one or more additional boards, etc.) for mapping. When another object has not been selected for mapping, process 2900 may proceed to block 2914. When another object has been selected for mapping, process 2900 may loop back to block 2908. For example, blocks 2908 and 2910 may be repeated for the objects selected for mapping until another object is not selected at block 2912.

[0164] At block 2914, process 2900 may determine whether the generated mapping definition for the shared XR room satisfies validation criteria. For example, a mapping definition may be generated for each mapped XR object of the shared XR room / real-world room. The mapping definition may store semantic information of the received XR object.

[0165] In some embodiments, the mapping definition can be compared to the validation criteria to determine whether the mapping definition is sufficient to implement a shared XR room environment. Example validation criteria include: a) one XR table, one XR board; b) one XR table, one XR board, one XR screen; c) one XR table, one XR board, one XR door, one XR window; d) one XR table, one XR board, at least two XR chairs; e) any combination thereof or any other suitable criteria. In some embodiments, the example validation criteria include criteria specific to certain XR objects. Example validation criteria specific to certain XR objects may include: an XR table with at least a minimum number of seat assignments, an XR table with a minimum size, an XR board with a minimum size, any combination thereof, or any other suitable criteria.

[0166] When the generated mapping definition satisfies the validation criteria, process 2900 may proceed to box 2918. When the generated mapping definition does not meet the validation criteria, process 2900 proceeds to box 2916. At box 2916, process 2900 may display prompts for one or more missing XR objects and / or prompts for editing one or more defined XR objects. For example, the validation criteria that the generated mapping definition does not meet may be displayed to the user (e.g., the XR board does not meet the minimum size, the XR door and XR window are missing, the XR table does not include a minimum number of seat assignments, etc.). From box 2916, process 2900 may loop back to box 2908 so that the missing one or more XR objects may be mapped and / or the existing one or more XR objects may be edited. Boxes 2908, 2910, and 2914 may be repeated until the generated mapping definition meets the validity criteria at box 2914.

[0167] At block 2918, process 2900 can preview a local version of the shared XR room hosted by the real-world room according to the generated mapping definition, and preview a remote version of the shared XR room hosted by the real-world room according to the generated mapping definition. For example, the preview of the local version can be a pass-through display of the real-world room augmented with an XR table at least at locations corresponding to the XR table object definition and augmented with an XR board at locations corresponding to the XR board object definition. In some implementations, the preview of the local version can be displayed to a user physically located in the real-world room such that the pass-through vision of the real-world room can be augmented with the XR objects.

[0168] In some implementations, the preview of the remote version may be a virtual reality room including the XR table and the XR pad. For example, the virtual reality room may include a geometry corresponding to the geometry of the real-world room, and relative positions of the XR table and the XR pad within the virtual reality room corresponding to positions as defined in the object definitions for the XR table and the XR pad.

[0169] At block 2920, process 2900 may store the generated mapping definition. For example, the stored mapping definition is used to implement a shared XR room for one or more local users using a pass-through display of a real-world room including at least an XR table and an XR pad, and to implement a shared XR room for one or more remote users using a virtual reality version of the real-world room including at least an XR table and an XR pad.

[0170] In some embodiments, the remote user can also map one or more XR objects to a physical space accessible to the remote user. For example, a desk space and a whiteboard space can be mapped for the remote user. In some embodiments, the mapped whiteboard space can support the remote user to interact with an XR board that is part of a shared XR room environment. In this example, the remote user can interact with the XR board in a physical space away from the real-world room hosting the shared XR room environment.

[0171] Fig.30 is a flow chart showing a process for creating an artificial reality work environment with one or both of desk space and / or standing space for a remote user in some embodiments of the present technology. Fig. 20 An example showing a desk space and a whiteboard space is provided. In some embodiments, process 3000 may begin when a user puts on an XR device and / or when a user activates an artificial reality work environment application on their XR device. In other embodiments, process 3000 may begin when a user activates a setup or calibration process within an artificial reality work environment application.

[0172] At box 3002, process 3000 can establish a desk space by identifying the outline of a real-world desk to which the desk space will be linked. In some embodiments, the height desk outline can be manually specified by the user, for example, by placing a hand (the XR device is tracking the hand) or a tracked controller on the desk surface to specify the top height of the desk. The user can also use a hand, a controller, or light projected from a hand / or a controller to track an area on the real-world desk or mark the corner of an area on the real-world desk (for example, when the XR device is in a transparent mode so that the user can see the real-world desk). In some embodiments, process 3000 can automatically identify some or all of the multiple desk outlines, for example, by using computer vision (for example, using a conventional camera and / or a depth camera) to identify a flat surface and its height; and / or using a height determined for a tracked object (for example, a tracked keyboard, a tracked mouse, a tracked controller, etc.) placed on the real-world desk.

[0173] At block 3004, the process 3000 can determine a dedicated space on the real-world desk. As with the desk outline, this dedicated space can be set manually by the user (e.g., by defining a rectangle on the top of the desk using the user's hand or controller) or automatically (e.g., by using computer vision to identify an open area of ​​sufficient size on the desk). In some embodiments, the automatically identified dedicated space can be determined relative to the location of the tracked keyboard or other tracked device on the real-world desk. In some embodiments, a minimum amount of desk space can be required to establish an artificial reality work environment, and a warning can be issued if the designated dedicated space is not large enough.

[0174] At block 3006, process 3000 may set an XR anchor / map relative to the dedicated space identified at block 3004. The XR anchor / map may be a specific location within the dedicated space for a specific corresponding content item. For example, a screen anchor / map may be set for a streamed version of a user's computer screen; anchors / maps may be set for various user interface (UI) controls to be placed around the dedicated space; various grids or surface patterns may be applied to the dedicated space to determine where the content item will appear when the content item is added to the artificial reality work environment, etc. In some implementations, the XR anchor / map may be initially set to a default location, and the XR anchor / map may be updated by the user to a new desired location.

[0175] At box 3008, process 3000 can set up a real-world space for the XR whiteboard that is physically separated from the desk area. The user can select a whiteboard space from the physical area around the desk. The whiteboard space can have a specified occupied space (e.g., 1.5 meters by 2 meters), and process 3000 can instruct the user to clear other objects from the space. In some cases, process 3000 can warn the user if there is not enough space or if the selected space has objects that can endanger the user. In some embodiments, the user can establish multiple such whiteboard spaces adjacent to each other. This can allow the user to move between multiple areas to use larger virtual whiteboards or different whiteboards, thereby keeping the user's avatar centered in one of these areas in the artificial reality work environment without having to employ processing for fine-grained user position tracking.

[0176] In some embodiments, the whiteboard space is required to be adjacent to a wall or other flat vertical surface, allowing the user to have the physical service of writing where writing would appear on the whiteboard in the artificial reality work environment. In other embodiments, without having a physical surface for the user to write on, the user can specify a virtual plane near a designated whiteboard area where the whiteboard will appear in the artificial reality work environment. In some embodiments, the whiteboard space can be associated with multiple virtual whiteboards that can be activated or switched between to facilitate different conversations. For example, different projects can each have a dedicated whiteboard, or different teams can have a shared team whiteboard.

[0177] In some embodiments, at least one whiteboard may correspond to an XR board that is part of a shared XR room environment. In various embodiments, a user can switch between their desk area and the whiteboard area by: activating a control; and / or standing up and physically moving between corresponding real-world areas. In some cases, if the user stands up or moves their eyes away from the dedicated space, and / or the user's virtual computer screen, or other virtual objects in the artificial reality work environment, the XR device may activate a pass-through mode. In some embodiments, this transition can be used to enable movement from an XR table in a shared XR room environment to an XR board, so that a remote user can interact with the XR board via the whiteboard space mapped for the user.

[0178] Fig.313100 and 3102 for providing a shared XR environment including a local version for one or more local users and a remote version for one or more remote users in some embodiments of the present technology. In some embodiments, processes 3100 and 3102 can begin when a user launches a shared XR room environment or application. In some embodiments, process 3100 is performed by a first XR system that displays a local version of the shared XR environment to a local user, and process 3102 is performed by a second XR system that displays a remote version of the shared XR environment to a remote user.

[0179] At block 3104, process 3100 may render the room as a pass-through of captured visuals of the real-world room augmented with XR objects or a mixed reality augmented with XR objects for the local version. For example, during a shared XR environment, the first XR system may render the local version to the first local user. In some implementations, at least a portion of the real-world room is displayed in a pass-through manner by the first XR system in the local version using a video capture of the real-world room augmented with an XR panel displayed at a location of a predefined boundary mapped onto a vertical surface within the real-world room.

[0180] At block 3106, process 3100 may render the local user using the captured pass-through visuals of the user in the real-world room and render the remote user as an avatar for the local version. For example, since the local user is physically present in the real-world room, the pass-through visuals may be used to represent the local user. On the other hand, the remote user is not physically present in the real-world room, so an avatar is used to represent the remote user.

[0181] At block 3108, process 3100 may selectively mute the local user and unmute the remote user. For example, the local user's audio is selectively muted at the first XR system (e.g., the local user may hear the real-world pass-through audio in the room), and the remote user's audio is output by the first XR system. Since the local users are co-located in the real-world room, the real-world audio may be heard between the local users. Therefore, the local user's audio from the first XR system is unnecessary and may cause confusion.

[0182] In some embodiments, the first XR system outputs spatial audio based on the relative position of each user in the shared XR environment. For example, a directional audio filter can be applied to the audio from a given remote user to adjust the frequencies heard by each ear so that the audio can sound like its source is the relative position of the given user in the shared XR environment. Any other suitable technology can be used to implement spatial audio.

[0183] At block 3110, process 3100 may track a local user located at a real-world location in a real-world room. For example, because the local user is physically present in the real-world room hosting the shared XR environment, the local user may move freely within the room. In some implementations, two local users may be co-located in a real-world room that includes a first local user, and the local version of the shared XR environment displayed to the second local user may display the first local user's pass-through vision at any location in the real-world room that the first local user chooses to move to.

[0184] At block 3112, process 3100 may determine whether the first local user is in proximity to an XR board that is displayed as an augmented XR object in the local version. For example, in the local version, the XR board is displayed at a location that maps to a predefined boundary on a vertical surface within a real-world room. When the first local user is physically within a threshold distance of the XR board, the first local user may be detected to be in proximity to the XR board. When the local user is in proximity to the XR board, process 3100 may proceed to block 3114. When the local user is not in proximity to the XR board, process 3100 may loop back to block 3110 until the local user is in proximity to the XR board.

[0185] At box 3114, process 3100 can trigger a local user XR board interaction for the first local user. For example, when the interaction with the XR board for the local user is triggered, the first local user can draw on the XR board using the detected body movement. In some embodiments, the detected body movement corresponds to the input rendered at the XR board. For example, the input can be rendered on the XR board for all users (e.g., local users and remote users). In some embodiments, the input from the first local user is rendered at the XR board when the first local user is physically close to a predefined boundary within the real-world room that is mapped to the vertical surface of the XR board and performs a detected body movement corresponding to the rendered input.

[0186] Moving to process 3102, at block 3116, process 3102 can render the room as a virtual room for the remote version. For example, the virtual room corresponding to the real-world room can be displayed by the second XR system to the remote user. In some embodiments, the display of the virtual room includes dynamically rendering an XR board that corresponds to the detected body movement of the first local user. For example, in the remote version, the second XR system can display the XR board at a relative position in the virtual room that corresponds to the relative position of the XR board in the pass-through video capture of the real-world room.

[0187] At block 3118, process 3102 may render the local user and the remote user as avatars (e.g., computer-generated drawings, portions of captured video depicting the users, reconstructed holograms of the users, etc.) for the remote version. Since the remote user is not physically located in the real-world room, pass-through vision of the real-world room is not available. Thus, both the remote user and the local user are represented as avatars in the remote version of the shared XR environment.

[0188] At block 3120, process 3102 may output audio for both the local user and the remote user. For example, the second XR system may output audio from both the local user and the remote user in the remote version, since the remote user cannot hear real-world audio in the real-world room. In some implementations, the second XR system outputs spatial audio based on the relative positions of the users in the shared XR environment. For example, a directional audio filter may be applied to audio from a given user (e.g., a remote user or a local user) to adjust the frequencies heard by each ear so that the audio may sound as if its source is the relative position of the given user in the shared XR environment. Any other suitable technique may be used to implement spatial audio.

[0189] At block 3122, process 3102 may track a remote user at a predefined location in the virtual room. For example, since the remote user is not physically present in the real-world room, an avatar representing the remote user may be constrained to a predefined location. In contrast to a local user, in some embodiments, the remote user cannot move freely in the real-world room. In some embodiments, the predefined location includes an XR table (e.g., a seating assignment at the XR table), an XR board, or any other suitable XR object with which the remote user can interact.

[0190] At block 3124, process 3102 may determine whether the remote user has selected to proceed to the XR board. In various embodiments, this may be done by selecting the XR board in the UI or by moving to a physical space designated in the remote user's area for interacting with the XR board. In some embodiments, the remote user provides input to the second XR system to move at a predefined location. For example, the remote user may interact with a panel displayed by the second XR system to confirm movement to the XR table (e.g., an assigned seat at the XR table) and / or the XR board. When the remote user has selected to proceed to the XR board, process 3102 may proceed to block 3126. When the remote user has not selected to proceed to the XR board, process 3102 may loop back to block 3122 until the remote user selects to proceed to the XR board.

[0191] At block 3126, process 3102 may trigger a remote user XR board interaction. For example, when a remote user interaction with the XR board is triggered, the remote user may draw on the XR board using the detected body motion. In some embodiments, the detected body motion corresponds to input rendered at the XR board. For example, input may be rendered on the XR board for all users (e.g., local users and remote users).

[0192] The remote user is located in a physical space away from the real-world room hosting the shared XR environment. In some embodiments, an additional predefined boundary on another vertical surface at the remote physical space is mapped to the XR board. For example, input from the remote user can be dynamically rendered by the XR board when the remote user physically approaches the additional predefined boundary on another vertical surface within another physical space mapped to the XR board and performs a detected body movement corresponding to the rendered input.

[0193] In some embodiments, the local version and the remote version of the shared XR environment include an XR table, and the XR table can be displayed using a pass-through video capture of the real-world table in the local version and displayed as a virtual object in the remote version. For example, one or more local users and one or more remote users include a seating assignment at the XR table in the local version and the remote version. In some embodiments, the local version and the remote version display an XR screen relative to the XR table, and the XR screen displays a video capture of a video user of the shared XR environment.

[0194] In some implementations, a second local version of the shared XR environment is rendered by the third XR system to the second local user. The first local user may be represented by a pass-through video capture of the first local user while sitting at the XR table in the second local version displayed by the third XR system; and the first local user may be represented by an avatar representation of the first local user while sitting at the XR table in the remote version displayed by the second XR system.

[0195] In some embodiments, a volumetric video of a real-world room (e.g., rather than a virtual representation of the real-world room) can be rendered to a remote user during a shared XR environment. For example, a volumetric video of a mapped real-world room (e.g., including objects in the real-world room and local users) can be captured and rendered to a remote user during a shared XR environment. In some embodiments, the volumetric video can be a three-dimensional representation of the real-world room and people (e.g., local users) and objects (e.g., room objects) in the room. The volumetric video can be generated using video from several synchronized cameras (e.g., XR devices located in the real-world room) that capture the room from different perspectives. In some embodiments, the fields of view of the several cameras include overlapping portions.

[0196] In some embodiments, images from several cameras may be processed by a machine learning service (e.g., a pipeline of machine learning models) that estimates depth (e.g., pixel depth information, mesh models, etc.) based on video images. In another example, one or more of the cameras may be depth sensing cameras that collect depth information. Any other suitable techniques may be used to sense, estimate, or otherwise generate depth information for the collected video.

[0197] In some embodiments, captured video from a camera (e.g., video with depth information, mesh models, etc.) can be stitched together to generate a three-dimensional representation of a real-world room. For example, the generated depth-enhanced video or mesh model can each represent a portion of a real-world room, and these portions can be stitched together in three dimensions to estimate a three-dimensional capture of the real-world room. In some embodiments, one or more machine learning models can use the depth-enhanced video / mesh model (e.g., a three-dimensional capture of a personal portion of a real-world room) to generate a combined three-dimensional room capture. Example machine learning models that can process the video capture of a camera to generate volumetric video can include deep neural networks, convolutional neural networks, encoder-decoder networks, variational autoencoder networks, and any other suitable models.

[0198] In some embodiments, position and pose information of one or more remote users' positions within a shared XR environment may be received (e.g., from the remote users' XR systems). Each user's XR system may track the position and pose of an avatar corresponding to the remote user, for example, to manage the interaction between the user's presence in the XR environment and one or more virtual objects in the XR environment. In some embodiments, the generated volumetric video of the real-world room may be processed based on the remote users' position and pose information to generate an XR rendering from the perspective of each remote user. For example, the overall volumetric video of the real-world room may be filtered based on the perspective of the user's position information and pose (e.g., position and gaze in the shared XR environment). The filtered volumetric video (of each remote user) may represent a rendering of the real-world room from the perspective of each remote user. The rendering may be streamed to the remote users' XR systems so that a volumetric video of the real-world room from the perspective of the remote user may be rendered to the remote users during the shared XR environment. In the XR rendering from the perspective of each remote user, the local users may be represented as captured video of their real-world selves.

[0199] In some embodiments, the volumetric video rendered to the remote user may include one or more XR objects or augmentations. For example, the volumetric video may be processed to include an XR table, an XR board (with writing), avatars of other users, or other suitable augmentations.

[0200] Fig.32 3200 is a flow chart illustrating a process for configuring an occlusion mask for an avatar in an artificial reality (XR) environment in some embodiments of the present technology. In some embodiments, process 3200 may begin when a user launches a shared XR environment or application. In some embodiments, process 3200 is performed by a first XR system that displays a local version of the shared XR environment to a first local user.

[0201] At block 3202, process 3200 may render a shared XR environment. For example, a first XR system may render the shared XR environment to a first user. In some implementations, the shared XR environment may correspond to a real-world room, where at least two users, including the first user, are local users in the real-world room (e.g., co-located in the real-world room). In some implementations, the local user views at least a portion of the shared XR environment in A) mixed reality or B) pass-through using video capture of the real-world room.

[0202] At box 3204, process 3200 may receive tracking position and pose data for an avatar in the XR environment. For example, tracking position and pose data may be received from a second XR system of a second user among the local users. For example, the second XR system may track the position and pose data of the avatar using one or more sensors, captured video, or any other suitable technology used by the XR system to track user position and pose. In some embodiments, the first XR system does not render the avatar of the second user in the shared XR environment. For example, when two users are local, both users see each other using pass-through vision or mixed reality vision, so their devices do not need to render each other's avatars.

[0203] At block 3206, process 3200 can generate an occlusion mask for the avatar. For example, the occlusion mask can be attached relative to the position and pose data of the second user (e.g., as indicated by the avatar data of the second user) in the XR environment. In some implementations, the occlusion mask causes portions of the virtual object behind the occlusion mask to not be rendered from the viewpoint of the first user.

[0204] At block 3208, process 3200 may render a virtual object in an XR environment, wherein portions of the virtual object are not rendered due to an occlusion mask. For example, the generated occlusion mask causes at least a portion of the virtual object to be hidden from display by the first XR system. In some embodiments, the generated occlusion mask is a structured shape (e.g., an ellipse, a square, other suitable polygons, etc.), or is shaped according to the geometry of the second user's avatar (i.e., matching the user's outline and / or having a body shape and posture that matches the avatar's body shape and posture). In some embodiments, the generated occlusion mask is larger than the second user's avatar, e.g., the shape of the avatar but increased in size by 15%. For example, this may be to support a larger occlusion footprint than the avatar so that the virtual object / augmented body is not incorrectly rendered on the second user.

[0205] At block 3210, process 3200 may determine a speed of movement of the second user based on the received tracking and position data. For example, the second XR system tracks an avatar of the second user based on the sensed movement of the second user (e.g., within a real-world room). The received tracking and position data for the avatar of the second user may be used to determine the speed of movement of the second user.

[0206] At block 3212, process 3200 may determine whether the determined speed of the second user meets a speed criterion. For example, the speed criterion may be a speed threshold (e.g., 6 ft / s, 8 ft / s, 10 ft / s, 12 ft / s, etc.), and the determined speed of the second user may be compared to the speed threshold. When the determined speed of the second user meets the speed criterion, process 3200 may proceed to block 3214. When the determined speed of the second user does not meet the speed criterion, process 3200 may loop back to block 3208, where one or more virtual objects and / or avatars may continue to be rendered according to the occlusion mask.

[0207] At block 3214, process 3200 can change the size of the occlusion mask. For example, when the second user's movement speed meets the speed criteria, the occlusion mask can be enlarged (e.g., 105%, 110%, 115%, 120%, etc.). In some implementations, the size of the occlusion mask increases as the determined movement speed increases.

[0208] In some embodiments, a first occlusion mask may be generated for a user avatar, and a second occlusion mask may be generated for a virtual object. For example, a second virtual object may be rendered by a first XR system in a shared XR environment. In some embodiments, a second occlusion mask may be generated that is attached relative to the position and geometry of the second virtual object in the XR environment. For example, from the viewpoint of the first user, the second occlusion mask may cause portions of the user avatar or virtual object that are behind the second occlusion mask to not be rendered. In some embodiments, the generated second occlusion mask causes at least a portion of a virtual object or a rendered user avatar (e.g., of a remote user) to not be rendered.

[0209] Fig.33 3300 is a flowchart illustrating a process for configuring an occlusion mask for a virtual object in an artificial reality (XR) environment in some embodiments of the present technology. In some embodiments, process 3300 may begin when a user launches a shared XR room environment or application.

[0210] At block 3302, process 3300 may render a shared XR environment. For example, a first XR system may render the shared XR environment to a first user. In some embodiments, the shared XR environment may correspond to a real-world room, where at least the first user is a local user (e.g., physically located in the real-world room) and the second user is a remote user (e.g., physically located away from the real-world room). In some embodiments, the local user may view at least a portion of the shared XR environment in A) mixed reality or B) pass-through using video capture of the real-world room. In some embodiments, the remote user is represented as a rendered avatar in the shared XR environment rendered to the local user.

[0211] In some implementations, virtual objects may be rendered at specific locations in a shared XR environment. For example, a first XR system may render a virtual object in a shared XR environment (to a first user) at a specific location. Virtual objects may be rendered as an augmentation of a pass-through video capture of a real-world room or a mixed reality vision of a real-world room.

[0212] At block 3304, process 3200 may generate an occlusion mask for the virtual object. For example, the occlusion mask may be attached relative to the position of the virtual object in the XR environment. In some embodiments, the occlusion mask causes portions of the second user's avatar that are behind the occlusion mask to not be rendered from the viewpoint of the first user. For example, each tracked avatar and virtual object may have a position in the artificial reality environment, and when the positions indicate that the occlusion mask is between the viewing user and another virtual object, portions occluded by the occlusion mask are not rendered.

[0213] At block 3306, process 3200 may render an avatar of a second user in the XR environment, wherein portions of the avatar of the second user are not rendered due to an occlusion mask. For example, the generated occlusion mask causes at least a portion of the representation of the second user to be hidden from display by the first XR system. In some embodiments, the generated occlusion mask is a structured shape (e.g., an ellipse, a square, other suitable polygons, etc.), or is shaped according to the geometry of the virtual object. In some embodiments, the generated occlusion mask is larger than the virtual object, e.g., to support an occlusion footprint that is larger than the virtual object, such that the user representation or other virtual objects are not incorrectly rendered on the virtual object.

[0214] References in this specification to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be rendered by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0215] As used herein, being higher than a threshold value means that the value of a comparison item is higher than other values ​​specified, the comparison item is located in a certain specified number of items with a maximum value, or the comparison item has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of a comparison item is lower than other values ​​specified, the comparison item is located in a certain specified number of items with a minimum value, or the comparison item has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of a comparison item is between two other values ​​specified, the comparison item is located in a middle specified number of items, or the comparison item has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" may be understood to mean selecting a connection with a value corresponding to a connection speed assigned thereto that is higher than a threshold value.

[0216] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of the following: A, B, C or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items (e.g., A and A; B, B and C; A, A, B, C and C, etc.).

[0217] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The above-mentioned specific features and actions are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except in the appended claims.

Claims

1. A method for generating a mapping of a real-world room configured with a shared artificial reality (XR) room, the method comprising: receiving, from a user located in the real-world room, first and second mappings corresponding to first and second virtual objects, the first mapping comprising a boundary on a horizontal surface representing a real-world location of the first virtual object, and the second mapping comprising a boundary on a vertical surface representing a real-world location of the second virtual object, wherein at least a portion of the boundary is identified using one or more physical XR devices; receiving first semantic information defining the first virtual object as an XR table and second semantic information defining the second virtual object as an XR board; and causing the XR system to display a first preview and a second preview of the shared XR room to the user, wherein: The first preview includes a pass-through display of the real-world room augmented with the first virtual object being an XR table at a location corresponding to the first mapping and augmented with the second virtual object being an XR board at a location corresponding to the second mapping, and The second preview includes a virtual reality room including the first virtual object as an XR table and the second virtual object as an XR pad, the virtual reality room including a geometry corresponding to the geometry of the real-world room, and relative positions of the first virtual object and the second virtual object within the virtual reality room corresponding to the first mapping and the second mapping.

2. The method according to claim 1, further comprising: A mapping definition for the shared XR room is stored, the mapping definition comprising the first semantic information, the first mapping, the second semantic information, and the second mapping, wherein the stored configuration definition is used to implement the shared XR room for one or more local users using a pass-through display of the real-world room including the first virtual object and the second virtual object, and to implement the shared XR room for one or more remote users using a virtual reality version of the real-world room including the first virtual object and the second virtual object.

3. The method according to claim 1 or 2, further comprising: A plurality of seat assignments relative to the first virtual object are automatically generated for a plurality of users, the plurality of seat assignments comprising a defined area within a boundary on a flat horizontal surface from the first mapping, wherein the generated plurality of seat assignments enhance semantic information of the first virtual object.

4. The method according to claim 3, wherein: During implementation of the shared XR room, one or more local users and one or more remote users are each assigned one of the plurality of seat assignments at the first virtual object.

5. The method according to any preceding claim, further comprising: Determining the geographic location of the real-world room does not include a previously stored configuration definition that maps the real-world room to the shared XR room, and prompting the user for the first mapping and the second mapping in response to the determining.

6. The method according to any preceding claim, further comprising: Before storing the configuration definition, verifying that the received first semantic information and the second semantic information include semantic information of at least one XR table and at least one XR board.

7. The method according to any preceding claim, further comprising: A third map corresponding to a third virtual object and a fourth map corresponding to a fourth virtual object are received from the user located within the real-world room, the third map including a boundary on a flat horizontal surface representing a real-world location of the third virtual object.

8. The method according to claim 7, further comprising: Third semantic information defining the third virtual object as an XR chair and fourth semantic information defining the fourth virtual object as an XR door are received.

9. The method according to claim 8, further comprising: Before storing the configuration definition, verifying that the received first semantic information, the second semantic information, the third semantic information, and the fourth semantic information include semantic information of at least one XR table, at least one XR board, at least one XR chair, and at least one XR door.

10. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for generating a mapping of a real-world room configured with a shared artificial reality (XR) room, the process comprising: receiving, from a user located in the real-world room, first and second mappings corresponding to first and second virtual objects, the first mapping comprising a boundary on a horizontal surface representing a real-world location of the first virtual object, and the second mapping comprising a boundary on a vertical surface representing a real-world location of the second virtual object, wherein at least a portion of the boundary is identified using one or more physical XR devices; receiving first semantic information defining the first virtual object as an XR table and second semantic information defining the second virtual object as an XR board; and causing the XR system to display a first preview and a second preview of the shared XR room to the user, wherein: The first preview includes a pass-through display of the real-world room augmented with the first virtual object being an XR table at a location corresponding to the first mapping and augmented with the second virtual object being an XR board at a location corresponding to the second mapping, and The second preview includes a virtual reality room including the first virtual object as an XR table and the second virtual object as an XR pad, the virtual reality room including a geometry corresponding to the geometry of the real-world room, and relative positions of the first virtual object and the second virtual object within the virtual reality room corresponding to the first mapping and the second mapping.

11. The computer-readable storage medium of claim 10, wherein: The process is a method according to any one of claims 2 to 9.

12. A computing system for generating a mapping of a real-world room configured with a shared artificial reality (XR) room, the computing system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the edge computing system to perform a process comprising: receiving a first map and a second map corresponding to a first object and a second object, the first map including a boundary representing a real-world location of the first object and the second map including a boundary representing a real-world location of the second object; receiving first semantic information defining the first object as an XR table and second semantic information defining the second object as an XR board; causing the XR system to display a first preview and a second preview, the first preview comprising a pass-through display of the real-world room augmented with the first object according to the first mapping and augmented with the second object according to the second mapping, the second preview comprising a virtual reality room that mimics the first preview; as well as The first semantic information, the first mapping, the second semantic information, and the second mapping are stored in association with a geographic location of the real-world room.

13. The computing system of claim 19, wherein: The process is a method according to any one of claims 2 to 9.

Images