Asymmetrical presentation of the environment

Abstract

This disclosure relates to techniques for collaboration in a multi-user communication environment. One such technique includes: receiving data at a first communication device associated with a multi-user communication session between a first user of the first communication device and a second user of a second communication device; presenting at the first communication device, based on the data associated with the multi-user communication session, the non-extended reality graphical user interface (GUI) including non-extended reality representations of virtual objects included in the multi-user communication session and a representation of the second user; and updating at the first communication device, in response to the data indicating an interaction between the second user's representation and the virtual objects, to show the interaction.

Description

Background Technology

[0001] This disclosure relates in its entirety to multi-user environments. More particularly, but not limitingly, this disclosure relates to techniques and systems for automatically determining the state of applications in extended reality (XR) environments.

[0002] Some devices are capable of generating and presenting XR environments (XREs). An XRE can include a fully or partially simulated environment that people perceive and / or interact with via electronic systems. In an XRE, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual elements simulated in the XRE are adjusted in a manner consistent with at least one physical law. Some XREs allow multiple users to interact with each other within the XRE. However, an improved technique is needed to determine the application state within the XRE. Attached Figure Description

[0003] Figure 1 A simplified system diagram according to one or more implementation schemes is shown in block diagram form.

[0004] Figure 2 A diagram illustrating an exemplary operating environment according to one or more implementation schemes is shown.

[0005] Figure 3 This is a block diagram illustrating an exemplary communication environment according to various aspects of this disclosure.

[0006] Figure 4A , Figure 4B , Figure 5A , Figure 5B , Figure 6A and Figure 6B A view of an XR communication (XRC) session according to various aspects of this disclosure is shown.

[0007] Figure 7 and Figure 8 This is a block diagram illustrating techniques for interaction in an XRC session according to various aspects of this disclosure.

[0008] Figure 9A and Figure 9B An exemplary system for use in various XR technologies is shown according to one or more embodiments. Detailed Implementation

[0009] This disclosure relates to techniques for enabling users to collaborate within an XRC session. A range of devices with various capabilities can access an XRC session. For example, some participating devices may be able to display an XR view of the XRC session. Other devices may not be able to display an XR view and may instead render a non-XR view. Generally, users can participate in an XRC session to interact with virtual objects and with each other. Non-XR views may include user interfaces that display virtual objects that can be interacted with to facilitate user interaction with these virtual objects or with other users in the XRC session. Therefore, the techniques described herein provide an improved user interface for interaction within an XRC session.

[0010] Humans can interact with and / or perceive the physical environment or physical world without the aid of electronic devices. The physical environment can include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest that includes physical plants and animals. Humans can directly perceive and / or interact with the physical environment through various means, such as hearing, vision, taste, touch, and smell. In contrast, humans can use electronic devices to interact with and / or perceive a fully or partially simulated extended reality (XR) environment. This XR environment can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and so on. Using an XR system, a person's physical movements, or some of their representations, can be tracked, and in response, the characteristics of virtual objects simulated in the XR environment can be adjusted in a manner consistent with at least one physical law. For example, the XR system can detect movement of the user's head and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). For example, the XR system can detect the movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) presenting the XR environment and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). In some cases, the XR system can adjust the characteristics of the graphical content in response to other inputs such as representations of physical motion (e.g., voice commands).

[0011] Many different types of electronic systems enable users to interact with and / or perceive XR environments. A non-exclusive list of examples includes head-up displays (HUDs), head-mounted systems, projection-based systems, windows or vehicle windshields with integrated display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses), head-mounted receivers / headsets, input systems with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have opaque displays and one or more speakers. Other head-mounted systems may be configured to accept opaque external displays (e.g., smartphones). Head-mounted systems may include one or more image sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light is directed to the user's eyes. Displays can utilize various display technologies, such as uLED, OLED, LED, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. Optical waveguides, optical reflectors, holographic media, optical combiners, or combinations thereof, or other similar technologies can be used as the medium. In some implementations, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects graphic images onto a user's retina. Projection systems can also project virtual objects into the physical environment (e.g., as holograms or onto physical surfaces).

[0012] For the purposes of this disclosure, an XR communication (XRC) session may be represented by an XR environment (XRE) in which two or more devices are participating.

[0013] For the purposes of this disclosure, a local XRC device refers to a current device described in an XRC session or controlled by the described user.

[0014] For the purposes of this disclosure, a co-located XRC device refers to two devices that share a physical environment and an XRC session, enabling users of the co-located devices to experience the same physical objects and XR objects.

[0015] For the purposes of this disclosure, a remote XRC device refers to an auxiliary device located in a physical environment isolated from the current local XRC device. In one or more embodiments, the remote XRC device may be a participant in an XRC session.

[0016] For the purposes of this disclosure, a shared virtual element refers to an XR object that is visible to participants in an XRC session or otherwise can be experienced by participants in an XRE.

[0017] For the purposes of this disclosure, an XRC compute environment (XRCE) refers to a compute environment or container capable of hosting an XRC session of an application. XRCE enables applications to run within an XRC session. In some cases, XRCE allows users of an XRC session to interact with the applications hosted within that XRC session.

[0018] For the purposes of this disclosure, an XRCE instance refers to the XRCE of the current device described or controlled by the described user. An XRCE instance allows a user to participate in an XRC session and run applications within that session.

[0019] For the purposes of this disclosure, a second XRCE instance refers to an auxiliary device in an XRC session other than the local XRCE instance, or an XRCE controlled by a second user. The second XRCE instance may be remote or co-located.

[0020] For the purposes of this disclosure, an XRCE application refers to an application that can be executed in the context of XRCE.

[0021] For the purposes of this disclosure, a second XRCE application refers to an auxiliary device in an XRC session other than a local XRCE instance, or an XRCE application controlled by a second user. The second XRCE application may be remote or co-located.

[0022] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, and / or to invoke the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with that implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring to all of the same implementation.

[0023] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0024] See Figure 1 This document depicts a simplified block diagram of an XR electronic device 100, communicatively connected via a network 105 to an additional XR electronic device 110 and a network storage device 115, according to one or more embodiments of the present disclosure. The XR electronic device differs from other electronic devices by displaying the XRE in a manner that allows the user of the XR electronic device to perceive the XRE as a three-dimensional (3D) interactive experience. In contrast, other electronic devices may be able to display a two-dimensional “window” that can display a 3D perspective projection into the XRE. The extended reality electronic device 100 may be part of a multi-functional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system as described herein. The extended reality electronic device 100, the additional XR electronic device 110, and / or the network storage device 115 may additionally or alternatively include one or more additional devices (such as server devices, base stations, accessory devices, etc.) that can accommodate various functions or may distribute various functions across these devices. Examples of networks such as network 105 include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). According to one or more embodiments, XR electronic device 100 is used to participate in XRC sessions. It should be understood that various components and functions within XR electronic device 100, additional XR electronic device 110, and network storage device 115 may be distributed differently on the device or may be distributed on the additional device. XR electronic device 100 may include a network interface 150 that interfaces with networking components such as radio, infrared, and / or visible light transceivers for communication with other devices. Network interface 150 may interface with wired or wireless networking components or both.

[0025] The extended reality electronic device 100 may include one or more processors 125, such as a central processing unit (CPU). Processor 125 may include a system-on-a-chip (such as a system-on-a-chip present in a mobile device) and may include one or more dedicated graphics processing units (GPUs). Additionally, processor 125 may include multiple processors of the same or different types. The extended reality electronic device 100 may also include memory 135. Memory 135 may include one or more different types of memory that can be used in conjunction with processor 125 to perform device functions. For example, memory 135 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 135 may store various programming modules for execution by processor 125, including XR module 165, XRCE module 170, and various other application programs 175. The extended reality electronic device 100 may also include a storage device 130. Storage device 130 may include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). According to one or more embodiments, storage device 130 may be configured to store content item 160.

[0026] The extended reality electronics device 100 may also include one or more cameras 140 or other sensors 145, such as depth sensors that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 140 may be a conventional RGB camera or a depth camera. Furthermore, the cameras 140 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc. The extended reality electronics device 100 may also include a display 155. The display device 155 may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection technology, which projects graphic images onto a person's retina. Projection systems may also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0027] According to one or more embodiments, memory 135 may include one or more modules comprising computer-readable code executable by processor 125 to perform functions. The memory may include, for example, an XR module 165 that can be used to provide an XR environment for a local XRC device. XRCE module 170 may generate an environment that supports the XR environment. For example, XRCE module 170 may provide service and / or application programming interfaces that can be used by the XR environment and / or hosted applications to interoperate with the operating system (OS) and / or hardware of extended reality electronic device 100. An XRC session may be a computing environment supporting a shared experience for extended reality electronic device 100 and attached extended reality electronic devices 110.

[0028] The memory 135 may also include an OS module 180 for supporting basic functions and managing the hardware of the XR electronic device 100. The OS module 180 provides an environment in which application 175 can execute. In some cases, the XRCE module 170 and XR module 165 may run in the context of the OS module 180. In other cases, the XRCE module 170 and XR module 165 may handle basic functions and manage the hardware of the XR electronic device 100 in parallel with or in place of the OS module 180. The XRCE module 170 also provides an environment in which application 175 can execute. Application 175 may include, for example, computer applications that can be experienced by multiple devices (such as XR electronic device 100 and additional XR electronic device 110) in an XRC session.

[0029] Although the XR electronic device 100 is depicted as including the aforementioned components, in one or more embodiments, the various components may be distributed across multiple devices. Therefore, while certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the functions of different distributions. Additionally, additional components may be used, and certain combinations of the functions of any components may be combined.

[0030] Figure 2 A diagram of an example operating environment according to one or more embodiments is shown. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the examples in the specific embodiments disclosed herein. Therefore, by way of non-limiting example, operating environment 240 includes a first physical environment, while operating environment 250 includes a second physical environment.

[0031] like Figure 2As shown, the first environment 240 includes a first user 220 utilizing the first electronic device 200, and the second environment 250 includes a second user 232 utilizing the second electronic device 210. In one or more embodiments, the first electronic device 200 and the second electronic device 210 include mobile devices, such as handheld devices, wearable devices, etc.

[0032] In one or more embodiments, the first electronic device 200 and the second electronic device 210 communicate with each other via a network 205. Examples of the network 205 may include, for example, the Internet, a wide area network (WAN), a local area network (LAN), etc. In one or more embodiments, the first electronic device 200 and the second electronic device 210 may participate in an XRC session.

[0033] Although electronic devices 200 and 210 can participate in a common XRC session, the virtual environment can be rendered differently on each device. As shown, electronic device 200 can depict the physical objects of environment 240. As shown, physical table 222 can be depicted as virtual table 224 on display 242. In one or more embodiments, display 242 can be a pass-through display, and virtual table 224 can simply be a view of physical table 222 through display 242.

[0034] The display 242 of the electronic device 200 may also include an avatar 226 corresponding to a user 232 in the physical environment 250. For the purposes of this disclosure, the avatar may include a virtual representation of the user. The avatar may depict real-time actions of the corresponding user 232, including movement, updated location, and / or interaction with various physical and / or virtual components within an XRC session.

[0035] According to one or more implementations, XRCE can be an XRE of other modules that support one or more XRCE applications or provide depictions of XR objects on all participating devices, such as electronic devices 200 and 210. As shown in display 242, presentation panel 230A is an example of a virtual object that is visible to all participating devices.

[0036] As an example, returning to environment 250, electronic device 210 includes display 252 on which a presentation panel virtual object 230B is depicted. It should be understood that in one or more embodiments, although the same virtual object may be visible on all participating devices, the virtual object may be rendered differently depending on the location of the electronic device, the orientation of the electronic device, or other physical or virtual characteristics associated with electronic devices 200 and 210 and / or the XRCE depicted within displays 242 and 252.

[0037] Returning to environment 250, another characteristic of the XRC session is that while virtual objects can be shared across participating devices, the physical world can appear different. Therefore, the physical chair 234 is depicted as a virtual chair 236. As described above, and in one or more embodiments, the display 252 can be a pass-through display, and the virtual chair 236 can be a view of the physical chair 234 through the pass-through display 252. Furthermore, the electronic device 210 depicts an avatar 238 corresponding to the user 220 within the physical environment 240.

[0038] According to one or more embodiments, virtual objects such as presentation panel 230 can be combined with XRCE application rendering. In one or more embodiments, multiple XRCE applications can be executed within an XRCE. For example, an XRCE can host an XRCE application as a virtual object within a virtual environment of the XRCE. Applications can be configured to run within or outside of an XRCE. For example, based on the operating environment or context in which the application is executing, the application can include multiple operating modes, such as XRCE mode and non-XRCE mode. It can be determined that an application is running in the context of an XRCE, such as through an application programming interface (API) and operating in XRCE mode. When an application runs outside of an XRCE, it can run in non-XRCE mode. As used herein, an XRCE application or hosted application can also refer to an application running in XRCE mode, as well as an application configured only to run in the context of an XRCE. An XRCE can provide containers or computing environments in which XRCE applications can run in a manner similar to an operating system. These containers can be presented to the user as virtual objects, making the XRCE application appear to be executing in a window, on a surface, or on another virtual object.

[0039] In some cases, XRCE applications execute within the context of a specific XRCE instance. For example, a first user associated with a first device may participate in an XRC session with a second user associated with a second device. In this example, the XRCE application executes on both the first and second devices. Based on input from the first user, the first device may execute a first XRCE application hosted in its local XRCE instance. The first XRCE application may share data with a second XRCE application hosted in a second XRCE instance and executing on the second device. Executing the hosted application within the context of a specific XRCE instance helps provide privacy for local users within the XRCE. In some cases, the XRCE instance may execute on the local device. In other cases, the XRCE instance may execute at least partially on the network or on the hosted device.

[0040] In some cases, XRCE users can share content from XRCE applications with other users. Examples of content include images, videos, audio, documents, spreadsheets, presentations, and so on. Depending on one or more implementations, one or more XRCE applications running within an XRCE can share content among participants in an XRC session. For example, a first user can use an XRCE application to share their content (such as photos, videos, music, etc.) with other users (such as a second device) active in the XRC session.

[0041] Figure 3 This is a block diagram illustrating an exemplary communication environment 300 according to various aspects of the present disclosure. Users can access XRC sessions using various devices with multiple capabilities for accessing XRC sessions. As an example, a first user can access an XRC session using a first communication device 302. The first communication device 302, such as a personal computing device like a laptop computer, may be able to access the XRC session using a non-XR graphical user interface (GUI) 304. A second user can access an XRC session using a second communication device 306. The second communication device 306, such as a head-mounted device, may be able to access the XRC session using an XR GUI 308. Other users can also access the XRC session using various devices such as smartphones, tablets, smartwatches, etc. In some cases, the non-XR GUI 304 and XR GUI 308 may be part of an XRCE. For example, the non-XR GUI 304 may execute in the context of a first XRCE instance 310 executing on the first communication device 302, and the XR GUI may execute in the context of a second XRCE instance 312 executing on the second communication device 306. In some cases, an XRCE instance provides services and / or access to an interface that can be used for XRC sessions. For example, an XRCE instance may provide data transfer, networking, audio, video, avatar processing, etc. As a more specific example, an XRCE instance may provide a data stream 314 for carrying XR data (such as first XR data 316 and second XR data 318) between a first communication device 302 and a second communication device 306. It is understood that although shown as a direct connection between the first communication device 302 and the second communication device 306, the data stream may be routed via any number of intermediaries (such as routers, hubs, switches, gateways, and servers capable of processing and / or coordinating the data stream).

[0042] In some cases, the data streams between devices participating in an XRC session can have a common format, allowing the same information to be available to each device, assuming adequate network bandwidth is available. For example, the data stream may include a scene graph that includes information describing shared virtual elements of the XRC session, such as avatars, virtual objects, spatial arrangement information, etc. This data stream can be common to devices capable of presenting the XRC session. While the data stream can be common, the ways in which different devices display the shared virtual elements of the XRC session may differ.

[0043] According to various aspects of this disclosure, the way a given device displays information describing shared virtual elements of an XRC session can be adjusted based on the device's attributes and capabilities. Figure 4A A non-XR first view 400 of an XRC session according to various aspects of this disclosure is shown, and Figure 4B A second XR view 450 of the same XRC session is shown. It is worth noting that, for clarity, the first view 400 is displayed by the first device associated with user 1 and presented from the point of view (POV) of user 1's avatar in the XRC session. The second view 450 is a user-independent view leading to the XRC session.

[0044] Generally, in an XRC session, the focus is on the content of the XRC session (e.g., shared virtual elements). To enhance participants' ability to interact with the content of the XRC session, a scene graph can encode the shared virtual elements and then describe the 3D relationships relative to them. In some cases, the same scene graph can be used to render a first view 400 and a second view 450. For example, assuming these views come from the same POV, a first device that cannot display an XR view can render a view as shown in the first view 400 based on the scene graph, while a device that can display an XR view can render a view as shown in the second view 450 based on the scene graph. As another example, the same scene graph may include one or more virtual objects, descriptions of the virtual objects used to display them, and their respective locations. A first device may display all or a subset of the virtual objects in a list in the virtual element UI 408, while a second device may display the virtual objects within the second view 450. Devices capable of participating in an XRC session may include a feature mapping layer to help the device determine how to render the elements described in the scene graph.

[0045] The feature mapping layer can be used to transform various aspects of the scene graph for a device. The feature mapping layer can be device-specific or device-type-specific and can define how the device renders shared virtual elements of the XRC session. The way shared elements are rendered on the device can be based on the device's capabilities. For example, a head-mounted system may be able to display a 3D representation of the XRC session, and the feature mapping layer can take this into account. The head-mounted system can then be configured to render a view of the XRC session using shared virtual elements arranged spatially around a 3D scene of the XRC session. Users of the head-mounted system can then be spatially maneuvering between different shared virtual elements (such as video element 402B, game element 454, user 1, user 2, and user 3) within the 3D scene. It is worth noting that additional users (user 4…user N) may also exist in the 3D scene but may not be visible in the second view 450. For example, those users may be behind the POV of the second view 450.

[0046] Certain devices, such as desktop or laptop computers, tablets, and telephones, may be able to display two-dimensional (2D), non-XR views (e.g., non-immersive) within a 3D setting of an XRC session, flattening the spatiality (e.g., depth) of the XRC session. A corresponding feature mapping layer may instruct such devices to internally render a 3D representation of the 3D setting and then display a 2D second view 450 via rasterization, ray tracing, or otherwise generating a 2D viewport 406 of the 3D setting. In this example, the position of user 1's viewport 406 corresponds to the position of user 1's avatar within the 3D setting of the XRC session. When user 1 interacts with (e.g., watches) a video element 402A in viewport 406, user 1's avatar in the 3D setting of the XRC is shown as viewing the corresponding video element 402B.

[0047] In some cases, a user's representation may be displayed as an avatar within the 3D context of an XRC session. The avatar may or may not simulate the user's physical characteristics and facial expressions. In some cases, the avatar may indicate device status. For example, as shown in second view 450, an avatar representing user 1 may include an indication (shown here in dashed outline) that user 1 is accessing an XRC session using a non-XR view. In some cases, this indication may be made using a 2D avatar, an image, a display icon, using a different avatar, a floating ball replacing the avatar, directional audio, etc. As discussed further below, users using a non-XR view may navigate the 3D context in ways that might appear unnatural when represented by a plain-looking avatar, and including an indication that the user is using a non-XR view can help mitigate such problems. In another example, when a user is wearing a headset, the avatar may be depicted as wearing a headset. In other cases, information collected by the user's device may be used to update the avatar. For example, if a user is wearing a headset that can detect whether one or more headsets in the headset have been removed (e.g., one or more earpieces have been removed) or a headset that can detect the removal of the headset, the user's avatar can indicate that the user's attention has been diverted elsewhere, such as by graying out the avatar, displaying a specific animation / icon, or switching to a default / inactive state.

[0048] In the XR second view, the user may be able to point to, highlight, or otherwise interact with shared virtual elements. In this example, user 2 is shown interacting with video element 402B. In some cases, interactions with virtual elements, such as those in an XRCE application, can be normalized to the image space relative to the virtual element. For example, the location where user 2 touches video element 402B can be determined relative to video element 402B, and this relative position can be encoded in the scene graph. By encoding interactions relative to virtual elements, interactions can be translated across devices, despite differences in resolution, aspect ratio, size, etc.

[0049] In some cases, non-XR views leading to an XRC session can be more spatially constrained than XR views, and when using non-XR views, it can be more difficult to see other users in the XRC session, as well as other users who may be interacting with the same virtual element. For example, in an XR view, other users can be visible from the periphery. To help address such issues, non-XR views can include UI elements, such as user bar 410, to indicate which other users are participating in the XRC session. While user bar 410 is shown as a list of avatar faces in this example, it is understood that any UI element or combination of UI elements can be used to convey the information described herein. In some cases, user bar 410 can indicate which other users are interacting with the same virtual element as the user associated with the non-XR view. As an example, as shown in the second view, both user 1 and user 2 are interacting with video elements 402A / 402B, while users 3, 4…N are not. In this example, the avatar face of the user interacting with the same virtual element (user 2 in this case) can be shown in user bar 410. Other users (users 3, 4...N) are shown turned away, indicating that those users are not interacting with video element 402 (e.g., detached from it). For example, user 3 is interacting with game element 404B, while users 4...N may be interacting with other elements (not shown). In other implementations, it may be desirable to make the avatar faces of users in user bar 410 visible, rather than turned away. In such cases, another indication of which users are not interacting with the same virtual element as users associated with non-XR views, or are interacting with the same virtual element as users associated with non-XR views, could be used, such as icons, user sorting, highlighting / summarizing certain users, graying out users, etc.

[0050] As indicated above, in certain situations, a user may be able to point to, highlight, or otherwise interact with a shared virtual element. In an XR view of an XRC session, such as shown in second view 450, a user, such as user 2, may be able to manipulate their avatar to reach out and touch video element 402B. The position where user 2 touches video element 402B may be determined based on a coordinate system relative to the virtual element and encoded into the scene graph. In certain situations, a device rendering a non-XR view of an XRC session may be able to display and / or transmit touch / highlight information. For example, interaction indicator 412 may be displayed in viewport 406 based on positional information encoded in the scene graph corresponding to the position where user 2 touches video element 402B. In some cases, the manner in which interaction indicator 412 is displayed may be based on the virtual element. For example, different XRCE applications may display interaction indicator 412 in different ways, depending on, for example, how the XRCE application handles such interactions, the mode of the XRCE application, the content being displayed, etc.

[0051] As indicated above, the focus of an XRC session is typically on shared virtual elements, and feature mapping may include alternative methods of presenting certain shared virtual elements rather than arranging them spatially around a 3D scene. In some cases, shared virtual elements that a user can interact with in an XRC session can be determined from the scene graph. These interactive shared virtual elements may be collected and displayed in a portion of the 2D second view 450. For example, UI elements such as Virtual Element UI 408 may display a list of interactive shared virtual elements. Although shown as a list, it is understood that interactive shared virtual elements may be displayed in any suitable UI, such as wheels, grids, dropdowns, menus, etc. By collecting and displaying interactive shared virtual elements together, users can more easily discover the virtual elements they can interact with in an XRC session. In some cases, an XRC session may include non-interactive virtual elements, such as decorative items, directional surfaces, backgrounds, etc., and such non-interactive virtual elements may be omitted from Virtual Element UI 408. Here, Virtual Element UI 408 includes video element 402A and game element 414. In some cases, virtual elements may include XRCE applications that users can interact with. For example, game element 414 may represent a game XRCE application that a user of an XRC session can play.

[0052] In some cases, users may interact with other users. In such cases, representations of other users can be displayed in the virtual element UI 408. This representation of other users in the virtual element UI 408 may correspond to the avatars of other users, or it may be another representation, such as a list of usernames, representative images, phone numbers, etc. In this example, when both User 1 and User 2 are interacting with the video element 402B, User 1 and User 2 can be considered to be interacting with each other. This interaction can be indicated in the user bar 410, where User 2's avatar face is shown, while other users not interacting with User 1 are shown as turned away. Although shown in the context of a non-XR view of the XRC session, it should be understood that the virtual element UI 408 and user bar 410 can be used in conjunction with other views that include the XR view of the XRC session.

[0053] Users can interact with the virtual element UI 408 by, for example, selecting video element 416 (shown as highlighted) within the virtual element UI 408. In some cases, selecting an interactive virtual element in the virtual element UI 408 can be achieved by moving the viewport 406 to face the corresponding interactive virtual element, in this case, video element 402A. In the 3D setting of the XRC session, the user's (in this example, user 1) viewport 406 can correspond to a representation of the user, such as an avatar of user 1. Therefore, moving the viewport 406 between interactive virtual elements can visualize the user's representation within the 3D setting of the XRC session by moving the viewport 406 between the interactive virtual elements. For example, user 1 can select game element 414 from the virtual element UI 408. Figure 5A As shown, viewport 506 in the first view 500 can be moved to display game element 504A, and game element 514 from virtual element UI 508 can be highlighted based on selection. Figure 5B In the second view 550 shown, User 1's avatar can move from video element 502B to game element 504B. In some cases, User 1's avatar can automatically move from video element 502B to game element 504B across the 3D scene, as if User 1 were moving their avatar. In other cases, User 1's avatar can move in other ways, such as being teleported, moving icons, etc. The user bar 510 can also be updated to indicate which other users are interacting with the same virtual element as that user. In this example, it indicates that User 3's avatar face is now shown, while the faces of Users 2, 4...N's avatars are now shown as turned around.

[0054] In some cases, a user can interact with another user using a virtual element UI. For example, referring to Figure 6, user 1 can select user 2 from virtual element UI 608. In response, viewport 606 shown in the first view 600 can be moved to face user 2, and the user element 616 corresponding to user 2 in the virtual element UI 608 can be highlighted based on the selection. In some cases, user 2 may be given the option to approve the interaction before moving viewport 606. In the second view 650, the avatar of user 1 can be moved to face the avatar of user 2.

[0055] According to various aspects of this disclosure, an avatar of any or both of the users can be animated based on motion data. Motion data can be provided from a variety of sensors, including wearable, handheld, or separate sensors from the user. As an example, the avatar's face can be animated based on the facial expressions of the user represented by the avatar, based on captured images of the user's face. The captured images can be provided by a variety of sensors, such as visible light cameras, dot projectors, non-visible light sensors, ultrasound, radar, etc. Sensors can, for example, be incorporated into participating devices, such as those described above. Figure 2 The electronic devices discussed. The movement of the avatar's head may also be based on these captured images or another sensor, such as an accelerometer in a head-mounted display or headset. Motion data may also be captured, for example, by a handheld controller, imaging device, or touch surface, and this motion is used to animate the avatar's body, limbs, or another part.

[0056] Different motion data can be captured for each user. For example, the first user may participate in an XRC session via a tablet device including a camera and / or dot projector. Images captured by the camera and / or dot projector can be analyzed to generate motion data of the user, such as facial motion, via the tablet device. Participating devices can be configured to collect information from multiple sensors and generate motion data in a common format based on that information. That is, participating devices process sensor information to generate common motion data. In some cases, the first user's body / limbs may be hidden from the camera and / or dot projector, and motion data corresponding to those parts of the avatar may be unavailable. Available motion data can be transmitted, for example, in the scene graph to other participants in the XRC session. Another participating device can receive the motion data and use it to animate the avatar corresponding to the first user. In the event that the avatar's motion data is unavailable, the corresponding parts of the avatar may remain in a default or non-animated position (e.g., an arm resting to the side, not moving relative to the rest of the avatar).

[0057] In some cases, the interaction can be initiated by a user accessing the XRC session via the XR view. For example, user 2 can attempt to interact with user 1 by turning their avatar to face user 1, pressing a button, or attempting to touch user 1's avatar. When user 2 initiates an interaction with user 1, user 1's viewport 606 can be moved to face user 2, and the user element 616 corresponding to user 2 in the virtual element UI 608 can be highlighted. In some cases, user 1 may be presented with the option to approve the interaction initiated by user 2.

[0058] In some cases, the specific arrangement of shared content items or the view from a particular POV may be important for communication from one user to another. In such cases, a user may be able to temporarily move viewport 606 or an avatar of another user to a specific location. In other cases, instead of moving viewport 606 or an avatar of another user, a third-person view, such as an over-the-shoulder view, can be used to indicate to other users that another user may be temporarily controlling the view.

[0059] Figure 7 A first technique 700 for interaction in a multi-user communication session according to various aspects of this disclosure is illustrated. At block 702, a first communication device receives data associated with a multi-user communication session between a first user of the first communication device and a second user of a second communication device, wherein the first data associated with the multi-user communication session is configured to be presented as an XR graphical user interface (GUI) or a non-immersive GUI, such as a two-dimensional (2D) GUI. For example, a non-XR device displaying a 2D image (such as a laptop computer, desktop computer, tablet computer, smartphone, etc.) can participate in and receive data from the XRC session. In some cases, the difference between a non-immersive GUI and an XR GUI may be that the non-immersive GUI includes conventional 2D elements (e.g., planar elements) or 2.5D elements, wherein the elements may have shadows, relative sizes, perspective shortening, etc., to simulate a three-dimensional effect on a conventional planar (e.g., 2D) display. 2.5D elements help to present perspective views on a conventional planar display and can be considered a type of 2D image used for display on a conventional planar display. In some cases, the data may include a scene graph describing a virtual environment and elements of the XRC session. At block 704, the first communication device determines that it can render a non-immersive GUI. For example, a non-XR device may determine that it supports a non-XR view of an XRC session, or that it does not support an XR view of an XRC session. At block 706, the first communication device presents a non-XR graphical user interface (GUI) based on data associated with a multi-user communication session, the non-XR GUI including non-XR representations of virtual objects included in the multi-user communication session and representations of second users. For example, the non-XR device may render a non-XR view of an XRC session. The non-XR view may include virtual element UIs indicating virtual elements, such as XRCE applications, other users, and other interactive virtual elements that the user can interact with. Figure 4A , Figure 5A and Figure 6AThe virtual element UIs described in the corresponding elements 408, 508, and 608 can be displayed in any suitable UI, such as wheels, grids, dropdowns, menus, etc. At box 708, the first communication device updates the non-XR GUI to show the interaction between the second user's representation and the virtual object in response to data indicating such interaction. For example, the first user (such as the user associated with viewport 506 of FIG. 5) can select a virtual element from the virtual element UI to interact with. The user's viewport can be updated based on this selection, for example, to display the selected virtual element.

[0060] In some cases, the first technology 700 may further include receiving second data associated with a multi-user communication session at a second communication device. For example, an XR device, such as a head-mounted system, may participate in and receive data from an XRC session. In some cases, the data may include a scene graph describing the virtual environment and elements of the XRC session. The second communication device may present an XR GUI based on the second data, the XR GUI including XR representations of virtual objects and an avatar of the first user in the second user's environment, such as in combination. Figure 4B , Figure 5B and Figure 6B The corresponding elements 450, 550, and 650 are described. The second communication device may update the XR GUI to show the second interaction in response to a second data indication between the avatar and the virtual object. For example, the XR GUI may present a view of the environment in which the first user is interacting with the virtual object based on data from the first user.

[0061] In some cases, image data may be captured at a first communication device. For example, one or more cameras of the first device may capture image data. The first communication device may generate motion data describing the movement of a first user based on the image data. For example, images captured by a camera may be analyzed to generate motion data. The first communication device may transmit the motion data to a second communication device. The second communication device may receive the motion data and, based on the motion data, display an avatar of the first user in an animated manner. In some cases, the virtual object includes an application. In some cases, the virtual object includes a depiction of a third user in a multi-user communication session.

[0062] In some cases, the first communication device may update the non-XR GUI to show a representation of the second user detached from the virtual object in response to an indication that the second user is interacting with a different virtual object. For example, as Figures 5A to 6B As shown, if the second user stops interacting with the object that the first user is interacting with, the user bar in the non-XR view can be updated to indicate that the second user is no longer interacting with the object.

[0063] In some cases, the first and second data include scene graphs describing virtual objects and users in a multi-user communication session. For example, the scene graph may encode the content of an XRC session, including users and shared virtual elements. In some cases, the scene graph is configured to be rendered in an XR GUI or a non-immersive GUI. For example, the same scene graph may be used to render the first view 400 and the second view 450 of Figure 4.

[0064] In some cases, non-immersive GUIs are scene-graph based and include viewports that present views into a multi-user communication session, a list of virtual objects in the multi-user communication session, and instructions for other users in the multi-user communication session, such as combined with... Figure 4A , Figure 5A and Figure 6A The corresponding first views 400, 500, and 600 are described. In some cases, the XR GUI is based on a scene graph and includes a 3D view leading to a multi-user communication session, where virtual objects and other users are presented within the environment of a second user, such as in combination with... Figure 4B , Figure 5B and Figure 6B The corresponding second views 450, 550 and 650 are described.

[0065] In some cases, the first communication device may receive a selection of a second virtual object from a list of virtual objects. For example, as described above with respect to video element 416 in Figure 4, a virtual object may be selected in the virtual element UI. In some cases, first avatar movement data describing the movement of the first user to the second virtual object may be generated. For example, based on the selection of the virtual object, the first communication device may, for example, describe the movement of the first user's viewport and avatar toward the selected virtual object in the scene graph. In some cases, the first motion data is transmitted to the second communication device. In some cases, the second communication device may receive the first avatar movement data and move the first user's avatar based on the first avatar movement data. For example, the second communication device may update the view to show that the first user's avatar is moving toward the selected virtual object, such as... Figure 4B and Figure 5B As shown. In some cases, avatar movement can be represented as if the user is moving the avatar, being teleported, or moving in other ways.

[0066] In some cases, the second communication device may receive an instruction to move its avatar to a virtual object that the first user is interacting with. For example, a second user of an XR device may move their avatar to a virtual object that a first user of a non-XR device is interacting with. The second communication device may generate second avatar movement data describing the movement of the second user to the virtual object and transmit this second movement data to the first communication device. In some cases, the first communication device may receive the second avatar movement data and update the instructions of other users to indicate that the second user is interacting with the first user. For example, as... Figure 4A As shown, the user field 410 can be updated to indicate that another user is interacting with the first user while another user is interacting with the same virtual object.

[0067] Figure 8 A second technique 800 for interaction in a multi-user communication session according to various aspects of this disclosure is illustrated. In some cases, the second technique 800 can extend the first technique 700. At block 802, a second communication device receives second data associated with the multi-user communication session. For example, the first communication device may transmit XRC data associated with the XRC session to other participants in the XRC session. In some cases, the first communication device may be a non-XR device. This data may be received by other participants. At block 804, the second communication device renders an XR GUI based on the second data, the XR GUI including extended reality representations of virtual objects and an avatar of the first user in the second user's environment. For example, the XRC data may include a scene graph, and other participating devices may render an XR view of the XRC session based at least in part on the XRC data from the first user. At block 806, the second communication device updates the extended reality GUI to show the second interaction in response to the second data indicating a second interaction between the avatar and the virtual object. For example, the first user may select a virtual element to interact with, and the XRC data from the first user may include information describing the selection.

[0068] Figure 9A and Figure 9B An exemplary system 900 for use in various XR technologies is depicted. In some examples, such as... Figure 9A As shown, system 900 includes device 900a. Device 900a includes various components such as processor 902, RF circuitry 904, memory 906, image sensor 908, orientation sensor 910, microphone 912, position sensor 916, speaker 918, display 920, and touch-sensitive surface 922. These components optionally communicate via communication bus 950 of device 900a.

[0069] In some examples, components of system 900 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop computer), and other components of system 900 are implemented in a second device (e.g., a head-mounted device). In some examples, device 900a is implemented in either the base station device or the second device.

[0070] like Figure 9B As shown, in some examples, system 900 includes two (or more) devices that communicate via wired or wireless connections. A first device 900B (e.g., a base station device) includes a processor 902, RF circuitry 904, and memory 906. These components optionally communicate via a communication bus 950 of device 900b. A second device 900c (e.g., a head-mounted device) includes various components such as processor 902, RF circuitry 904, memory 906, image sensor 908, orientation sensor 910, microphone 912, position sensor 916, speaker 918, display 920, and touch-sensitive surface 922. These components optionally communicate via a communication bus 950 of device 900c.

[0071] System 900 includes processor 902 and memory 906. Processor 902 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 906 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 902 to perform the techniques described below.

[0072] System 900 includes RF circuitry 904. RF circuitry 904 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 904 optionally includes circuitry for using near-field communication and / or short-range communication (such as... The circuit used for communication.

[0073] System 900 includes a display 920. Display 920 may have an opaque display. Display 920 may also have a transparent or translucent display, which may be combined with a substrate through which light representing an image is directed to an individual's eye. Display 920 may incorporate LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The light-transmitting substrate may be an optical waveguide, an optical combiner, a light reflector, a holographic substrate, or any combination of these substrates. In one example, the transparent or translucent display may selectively switch between an opaque state and a transparent or translucent state. Other examples of display 920 include head-up displays, automotive windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, tablet computers, smartphones, and desktop or laptop computers. Alternatively, system 900 may be designed to receive an external display (e.g., a smartphone). In some examples, System 900 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical setting (e.g., onto a physical surface or as a hologram).

[0074] In some examples, system 900 includes a touch-sensitive surface 922 for receiving user input, such as tap and swipe input. In some examples, display 920 and touch-sensitive surface 922 form a touch-sensitive display.

[0075] System 900 includes an image sensor 908. Image sensor 908 optionally includes one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, operable to acquire images of physical elements from a physical scene. One or more image sensors also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the physical scene. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical scene. Image sensor 908 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical scene. Image sensor 908 also optionally includes one or more depth sensors configured to detect the distance between physical elements and system 900. In some examples, system 900 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical scene around system 900. In some examples, image sensor 908 includes a first image sensor and a second image sensor. The first and second image sensors are optionally configured to capture images of physical elements in a physical setting from two different perspectives. In some examples, system 900 uses image sensor 908 to receive user input, such as gestures. In some examples, system 900 uses image sensor 908 to detect the position and orientation of system 900 and / or display 920 in the physical setting. For example, system 900 uses image sensor 908 to track the position and orientation of display 920 relative to one or more fixed elements in the physical setting.

[0076] In some examples, system 900 includes microphone 912. System 900 uses microphone 912 to detect sound from a user and / or the user's physical set. In some examples, microphone 912 includes a microphone array (including multiple microphones) that optionally cooperate to identify ambient noise or locate sound sources in the space of the physical set.

[0077] System 900 includes an orientation sensor 910 for detecting the orientation and / or movement of system 900 and / or display 920. For example, system 900 uses orientation sensor 910 to track changes in the position and / or orientation of system 900 and / or display 920, such as relative to physical elements in a physical setting. Orientation sensor 910 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0078] The technologies defined in this document take into account options for obtaining and utilizing users' personal information. For example, one aspect of this technology is automatically determining whether a particular device is capable of displaying an XR view of an XRC session. However, in cases where such personal information is collected, it should be obtained with the user's informed consent, enabling the user to understand and control the use of their personal information.

[0079] Parties authorized to access personal information will use it solely for lawful and reasonable purposes and will comply with privacy policies and practices that at least meet appropriate laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and deemed to meet or exceed government / industry standards. In addition, personal information will not be distributed, sold, or otherwise shared outside of any reasonable and lawful purpose.

[0080] However, users can limit the extent to which parties can access personal information. The processes and devices described herein can allow for changes to settings or other preferences that enable users to control access to their personal information. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. As an example, a user's personal information can be masked or otherwise generalized so that it does not identify the specific user from whom it is obtained.

[0081] It should be understood that the above description is intended to be exemplary and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide that material in the context of a particular embodiment, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, Figure 7 and Figure 8 The specific arrangement of the steps or actions shown or Figure 1 To Figure 6 and Figures 9A to 9B The arrangement of the elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms “comprising” and “therein” are used as common English equivalents of the corresponding terms “including” and “wherein”.

Claims

1. A method comprising: At a first communication device, first data associated with a multi-user communication session between a first user of the first communication device and a second user of the second communication device is received, wherein the first communication device supports a non-immersive graphical user interface (GUI) and wherein the second communication device supports an extended reality (GUI). At the first communication device, the non-immersive GUI is presented based on the first data associated with the multi-user communication session. The non-immersive GUI includes graphical representations of at least some of the multiple elements of the multi-user communication session, including virtual objects and a representation of the second user. Receive a first instruction for selecting the virtual object in the non-immersive GUI; as well as At the first communication device, in response to the first instruction of the selection, the non-immersive GUI is updated to present a view of the virtual object within the multi-user communication session.

2. The method according to claim 1, further comprising: Send a second instruction to the second communication device regarding the selection of the virtual object in the non-immersive GUI. The second communication device updates the extended reality GUI to show a second interaction between the representation of the first user and the virtual object.

3. The method according to claim 2, further comprising: Capture image data at the first communication device; At the first communication device, motion data describing the motion of the first user is generated based on the image data; as well as The motion data is transmitted from the first communication device to the second communication device.

4. The method of claim 3, wherein the second communication device displays the representation of the first user in an animated manner based on the motion data.

5. The method according to any one of claims 1 to 3, wherein the virtual object includes an application.

6. The method according to any one of claims 1 to 3, wherein the virtual object includes a depiction of a third user of the multi-user communication session.

7. The method according to any one of claims 1 to 3, further comprising: At the first communication device, a second instruction is received regarding the interaction between the second virtual object and the second user within the multi-user communication session; and At the first communication device, the non-immersive GUI is updated to show the representation of the second user interacting with the second virtual object.

8. The method of claim 7, wherein the second instruction for the second user to interact with the virtual object includes a scene diagram describing the spatial relationship between the virtual object and the user in the multi-user communication session.

9. The method of claim 8, wherein the scene graph is configured to render a representation of the virtual object and the user via the extended reality GUI or the non-immersive GUI.

10. The method of claim 8, wherein the extended reality GUI is based on the scene graph and includes a three-dimensional view leading to the multi-user communication session, wherein the virtual objects and representations of other users are presented within the environment of the second user.

11. The method of claim 8, wherein the non-immersive GUI is presented based on the scene graph and includes a list of the plurality of elements.

12. The method of claim 11, further comprising: At the first communication device, a selection of a third virtual object from the list of the plurality of elements is received; At the first communication device, first representation movement data describing moving the representation of the first user to the third virtual object is generated; as well as The first indication of mobile data is transmitted from the first communication device to the second communication device.

13. The method of claim 12, wherein the representation of the first user is moved at the second communication device based on the first representation mobility data.

14. The method according to claim 12, wherein: In response to the second user approaching the second virtual object, second representation movement data describing the movement of the second user to the virtual object is received.

15. The method according to claim 14, further comprising: Update the representation of other users to indicate that the second user is interacting with the first user.

16. A computer-readable storage device for storing instructions, said instructions being executable by one or more processors to perform the method according to any one of claims 1-15.

17. A system comprising: A first communication device, comprising: One or more processors; and A memory that stores instructions that can be executed by the one or more processors to perform the method according to any one of claims 1-15.

18. A computer program product comprising computer-readable code for performing the method according to any one of claims 1-15.

Images