Combining virtual portal and electronic device functionality for user interaction in virtual portal space of extended reality environment

Abstract

An XR management device determines a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment Operations render a virtual electronic device in the second virtual portal space. Operations identify the participant has selected a first virtual user interface element for an operational capability. When the selected operational capability is in the first set of operational capabilities, the operations trigger performance of the selected operational capability by operation of the physical electronic device. In contrast, when the selected operational capability is included in the second set of operational capabilities, the operations trigger performance of the selected operational capability provided for the second virtual portal space.

Description

COMBINING VIRTUAL PORTAL AND ELECTRONIC DEVICE FUNCTIONALITY FOR USER INTERACTION IN VIRTUAL PORTAL SPACE OF EXTENDED REALITY ENVIRONMENTTECHNICAL FIELD

[0001] The present disclosure relates to rendering extended reality (XR) environments and associated XR management devices, and more particularly to rendering objects in immersive XR environments for display on XR participant devices.BACKGROUND

[0002] Immersive extended reality (XR) environments have been developed which provide a myriad of different types of user experiences for gaming, on-line meetings, cocreation of products, etc. One type of immersive XR environments (also referred to as "XR environments") is virtual reality (VR) environments where human users only see computer generated graphical renderings. Another type of XR environment is augmented reality (AR) environments where users see a combination of computer-generated graphical renderings overlaid on a view of the physical real-world through, e.g., see-through display screens. Accordingly, the term "XR" is used herein to refer to VR and / or AR.

[0003] Various types of XR management devices can be used to manage XR environments rendered through XR display devices for viewing by participants (users) in the XR environment. The XR management device can be separate from the XR display device or may be at least partially integrated therein. Example XR management devices include, without limitation, XR environment servers, XR head-mounted devices (HMDs), gaming consoles, smartphones running an XR application, and tablet / laptop / desktop computers running an XR application. Meta Quest 2 and Varjo XR-3 are example VR display devices and Hololens is an example AR display device.

[0004] Users can interact with virtual objects that are rendered in some immersive XR environments, such as by controlling a user's avatar to manipulate virtual objects. Immersive XR environments presently support a limited interface between physical real -world objects and virtual objects that are rendered.

[0005] In a two-person holographical communication session, there may be two physical realities to manage during a holographic communication session and each person may be rendered through virtual over-layering onto another person's physical reality and vice versa. There are ongoing efforts to develop interactive tactile true-3D XR displays that integrate thecomplete light field of a 3D XR space, and which would enable parts of a rendered XR environment scene to be directly available for interaction through a non-contact tactile system that allows users to feel virtual objects in mid-air. For example, it would be desirable for the XR management device to enable users to reach into the viewing volume to virtually interact with (e.g., touch) virtual objects, and preferably feel gradients of haptic feedback as the hands touch or interact with (e.g., manipulate pose of) the virtual objects.

[0006] Apple ARKit AR software enables a smartphone application to gather information about an AR participant's physical environment by processing a data feed from the smartphone camera and / or Lidar scanning the physical environment, and which is processed to provide a multi-user AR experience in which one AR participant may learn more about another AR participant's environment.

[0007] Depth maps obtained from one or more cameras and / or Lidars can be used with item modeling to create a virtual representation of a physical object. For example, adding a depth-map for a single image using a deep learning-based method to direct infer depth.

[0008] Further enhancements are desired to provide XR environments support more advanced interactions between XR participants and physical real -world objects through virtual objects that are rendered in the XR environments.SUMMARY

[0009] Some embodiments of the present disclosure are directed to an XR management device for managing an immersive XR environment which is displayable through an XR display device to a participant in the immersive XR environment. The XR management device includes at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations include to determine a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment. The operations generate a rendering of a virtual electronic device in the second virtual portal space for display by the XR display device, where the rendering of virtual electronic device includes a combined virtual user interface (VUI) with a combined set of VUI elements that the participant can interact with in the second virtual portal space. The operations identify an interaction of the participant with a first VUI element of the combined VUI that satisfies a capability trigger rule. Based on the capability trigger rule becoming satisfied, the operations identify a selected operationalcapability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set. When the selected operational capability is in the first set of operational capabilities of the physical electronic device, the operations trigger performance of the selected operational capability by operation of the physical electronic device. In contrast, when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, the operations trigger performance of the selected operational capability provided for the second virtual portal space.

[0010] In some further embodiments, the operations can interconnect one or more operational capabilities in the first set of operational capabilities with one or more operational capabilities in the second set of operational capabilities. In one example embodiment, the operations determine which of the first set of operational capabilities of the physical electronic device and which of the second set of operational capabilities provided for the second virtual portal space are operationally compatible with each other. The operational compatibility is determined to occur when an operational capability of the first set has an application programming interface (API) that is configured to interface to an API of an operational capability of the second set. The operations then select operational compatible ones of the first and second set of operational capabilities to include in the combined set of operational capabilities. In a further embodiment, the operations operationally interconnect at least part of the API of the operational capability of the first set and at least part of the API of the operational capability of the second set.

[0011] Some other embodiments are directed to a corresponding method performed by an XR management device for managing an immersive XR environment displayable through an XR display device to a participant in the immersive XR environment. The method includes determining a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment. The method further includes generating a rendering of a virtual electronic device in the second virtual portal space for display by the XR display device, where the rendering of virtual electronic device includes a combined VUI with a combined set of VUI elements that the participant can interact with in the second virtual portal space. The method further includes identifying an interaction of the participant with a first VUI element of the combined VUI that satisfies acapability trigger rule. Based on the capability trigger rule becoming satisfied, the method identifies a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set. When the selected operational capability is in the first set of operational capabilities of the physical electronic device, the method triggers performance of the selected operational capability by operation of the physical electronic device. In contrast, when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, the method triggers performance of the selected operational capability provided for the second virtual portal space.

[0012] Some other embodiments are directed to a corresponding computer program product including a non-transitory computer readable medium storing instructions executable by at least one processor of an XR management device for managing an immersive XR environment displayable through an XR display device to a participant in the immersive XR environment. The instructions are executed by the at least one processor to perform operations. The operations include to determine a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment. The operations generate a rendering of a virtual electronic device in the second virtual portal space for display by the XR display device, where the rendering of virtual electronic device includes a combined virtual user interface (VUI) with a combined set of VUI elements that the participant can interact with in the second virtual portal space. The operations identify an interaction of the participant with a first VUI element of the combined VUI that satisfies a capability trigger rule. Based on the capability trigger rule becoming satisfied, the operations identify a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set. When the selected operational capability is in the first set of operational capabilities of the physical electronic device, the operations trigger performance of the selected operational capability by operation of the physical electronic device. In contrast, when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, the operations trigger performance of the selected operational capability provided for the second virtual portal space.

[0013] Potential advantages that may be provided by one or more of these embodiments include that participants in immersive XR environments are able to view and interact with virtual renderings of a combination operational capabilities of the physical electronic device and the virtual portal space. A participant can thereby select to trigger an operational capability of the physical electronic device, trigger an operational capability of the second virtual portal space, and may trigger a new type of operational capability that is created through a combination of at least one operational capability of the physical electronic device and at least one operational capability of the second virtual portal space. For example, a new type of operational capability may be created by operationally interconnecting APIs of the physical electronic device and the second virtual portal space. Moreover, portal-specific policies can be defined such that the same virtual portal space can be used by different devices and different participants, and which are granted different access rights to different sets of operational capabilities. Virtualizing operational capabilities of both the physical electronic device and the virtual portal space and enabling portal-specific access rights will provide participants with new, more advanced, and intuitive ways of interacting and experiencing XR environments.

[0014] Other XR management device, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional XR management device, methods, and computer program products be included within this description and protected by the accompanying claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0016] Figure 1 illustrates an XR system that includes a physical electronic device and at least one participant device that communicates through networks with an XR management device to operate in accordance with some embodiments of the present disclosure;

[0017] Figure 2 illustrates the XR management device rendering a virtual smartphone or other virtual electronic device in a second virtual portal space of the immersive XR environment as a physical smartphone is passed into the first virtual portal space, in accordance with some embodiments of the present disclosure;

[0018] Figure 3 illustrates a virtual user interface that can be interacted with by the second participant in the second virtual portal space to trigger performance of a selectedoperational capability by operation of the physical smartphone, by a selected operational capability of the second virtual portal space, or by a combination of the physical smartphone and the second virtual portal space, in accordance with some embodiments of the present disclosure;

[0019] Figures 4A and 4B illustrate a combined flowchart of operations that can be performed by an XR management device in accordance with some embodiments of the present disclosure;

[0020] Figures 5-12 illustrate a series of operational scenarios that can be performed with the XR management device and the XR display device in accordance with some embodiments of the present disclosure;

[0021] Figure 13 illustrate a more general flowchart of operations that can be performed by an XR management device in accordance with some embodiments of the present disclosure;

[0022] Figure 14 illustrates a block diagram of components of an XR management device that are configured to operate in accordance with some embodiments of the present disclosure; and

[0023] Figure 15 illustrates a block diagram of components of a physical electronic device that are configured to operate in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0024] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0025] As summarized above, an XR management device is provided for managing an immersive XR environment displayable through an XR display device to a participant in the immersive XR environment. The XR management device can be configured to determine a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device in a first virtual portal space of the immersive XR environment andfurther includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment.

[0026] The term virtual portal space is used herein to refer to a displayable space within the computer-rendered immersive XR environment. In contrast, physical space is used to refer to the real space in which the human participant resides.

[0027] Access privileges may control which operational capabilities of the physical electronic device and which operational capabilities provided for the second virtual portal space can be viewed by and / or interacted with by the second participant. The access privileges may be defined based which portal is being accessed and by which device and / or by which participant. Thus, for example, the access privileges may be determined based on the identity of the participant, identity of the physical electronic device, available operational capabilities of the physical electronic device, available operational capabilities provided for the virtual portal space, etc.

[0028] Figure 1 illustrates an XR system that includes a physical electronic device 210 and at least one participant XR display device 110 that communicates through networks 120 with an XR management device 100 to operate in accordance with some embodiments of the present disclosure. The XR management device 100, the XR display devices 110, or another device of the system operates to render the XR environment for display through the XR display devices 110, including rendering the virtual electronic device which is viewable from various perspectives of virtual poses of human participants in the XR environment through display screens of the various participant XR display devices 110.

[0029] In the example scenario illustrated in Figure 1, XR headsets 110 can be worn by participants to view and navigate through the XR environment. The XR display devices 110 are not limited to XR headsets and, instead, may be a personal computer, tablet computer, smartphone, gaming device, vehicle (e.g., automobile), or other electronic device having operational capabilities that can be virtualized for viewing and interaction with by a participant in the XR environment. The participant may have an associated avatar which is rendered in the XR environment to represent poses (e.g., location, body assembly orientation, etc.) of the participant relative to a coordinate system of the XR environment.

[0030] The physical electronic device 210, also called “electronic device”, has operational capabilities at least some of which may be controlled through a user interface (UI). The physical electronic device 210 can be any type of electronic device. Non-limiting examples of physical electronic devices include smartphones, tablet computers, laptop computers, smart watches, gaming devices, smart appliances, vehicle (e.g., automobile), etc.Non-limiting examples of user interfaces include touch screens and other display devices that can display user selectable UI elements (e.g., selectable indicia in the form of buttons, menus, etc.), and physical buttons, etc.

[0031] As will be explained in further detail below, a virtual representation of the physical electronic device 210 is rendered in the XR environment as a virtual electronic device with a virtual representation of a user interface of the physical electronic device 210. A participant in the XR environment can interact with the virtual user interface (VUI) in order to control various electronic operational capabilities of the physical electronic device 210 and operational capabilities provided for the virtual portal space.

[0032] Although the XR management device 100 is illustrated in Figure 1 as being a centralized network computing server separate from one or more of the XR display devices 110, in some other embodiments the XR management device 100 is implemented as a component of one or more of the XR display devices 110. For example, one of the XR display devices 110 may be configured to perform operations of the XR management device 100 in a centralized manner to provide a combined set of operational capabilities of the physical electronic device 210 and the virtual portal space. Alternatively, each of the XR display devices 110 may be configured to perform at least some of the operations of the XR management device 100 in a distributed decentralized manner with coordinated communications being performed between the distributed XR display devices 110 (e.g., between software instances of XR management devices). A non-limiting example of an XR management device is the cloud computing environment provided by Meta which includes numerous networked computing devices which host VR gaming environments serving VR headsets worn by remote users.

[0033] Some embodiments of the present disclosure are directed to the XR management device 100 enabling a first participant (Participant 1) in an immersive XR environment to pass a physical electronic device (e.g., real smartphone) through a first virtual portal space located in physical space (e.g., space along a tabletop), to trigger the XR management device 100 to render a virtual electronic device (e.g., virtual smartphone) representing the physical electronic device (real smartphone) in the immersive XR environment.

[0034] In an illustrative example, the Participant 1 reaches a smartphone into a first virtual portal space to trigger the XR management device 100 to render a computer-generated virtual representation of the smartphone in a second virtual portal space of the immersive XR environment. Participant 1 and another second participant (Participant 2) may simultaneously view the virtual smartphone. The Participant 2 may interact with the virtualsmartphone in a second virtual portal space, e.g., by controlling an avatar representing Participant 2 and / or through computer-tracking of the Participant 2’s physical hand or other object interacting with the VUI of the virtual smartphone. The first and second participant may be provided corresponding feedback of the virtual interaction with the virtual smartphone, e.g., as tactile feedback initiated by the XR management device 100 and provided through a tactile device, such as gloves or smart fabric, worn by the first and second participants. In some embodiments, only one participant is involved in the operations, where the participant puts the physical electronic device within the portal and then interacts with the virtual representation of the virtual electronic device. As will be appreciated in view of the description provided below, the type of physical electronic device that can be passed into or otherwise scanned into a virtual portal space for rendering and possible interaction with by participants in the immersive XR environment is not limited to any of the example electronic devices described herein.

[0035] Whether a virtual electronic device remains rendered on a display and accessible to a participant for interaction after the physical smartphone has been removed from the virtual space, and how long such interaction is permitted and what operational functionality of the physical smartphone is allowed to be controlled, can depend on access rules and / or recipient-sender trust relations. These and other related operations relating to persistence after the physical smartphone or other physical electronic device has been removed from the first virtual portal space are described in further detail below.

[0036] As will be explained in further detail below, the XR management device 100 includes an operational capability handler 102 which operate to provide and / or control access to (use of) a set of operational capabilities of at least on virtual portal space, and to control performance of selected ones of the capabilities.

[0037] Figure 2 illustrates the XR management device 100 rendering a virtual smartphone 300 or other virtual electronic device in a second virtual portal space of the immersive XR environment as a physical smartphone 210 is passed into the first virtual portal space, in accordance with some embodiments of the present disclosure. Although some embodiments are described herein in the example context of a physical smartphone, any type of physical electronic device may be used herewith.

[0038] Referring to Figure 2, Participant 1 and Participant 2 can be participants involved in an immersive XR environment rendered by the XR management device 100. Participant 1 and Participant 2 may simultaneously view virtual objects that are rendered by the XR management device 100 in their respective virtual spaces, such as through XR headsets 110.The XR management device 100 is communicatively coupled to at least one sensor 230 that tracks location of the physical smartphone 210 or other physical electronic device relative to a boundary of a virtual portal 200 having a location defined in physical space and / or relative to a pose (2D or 3D location and / or angular orientation, such as 3 axial directions and 3 rotational angles about axes or 6 degree-of-freedom) within the virtual portal space. The sensor 230 may be one or more cameras mounted to an XR headset and facing forward in the wearer's eye gaze direction, or may be separate from the XR headset. The sensor 230 may additionally or alternatively be other types of sensors able to track location of physical objects, such as LIDAR sensors, ultrasound sensors, etc. In some embodiments, two cameras may be used to track the location of the physical smartphone 210 in three-dimensional (3D) space or a single LIDAR sensor may be used. LIDAR and one or more cameras may be used together to track and determine location, shape, texture, temperature, tactile notification, and color of the physical smartphone 210. Alternatively or additionally, the XR headset or other device transported by a participant may contain a sensor(s) that can track pose, such as an inertial measurement unit (e.g., multi-axis accelerometers), and report the pose (e.g., location, angular orientation, etc.) to the XR management device 100.

[0039] In some embodiments, the XR management device 100 defines a location, size, and shape of an entrance boundary of virtual portal 200 and / or the virtual portal space. The location of the entrance boundary of virtual portal space 200 is defined in physical space, such as relative to a feature on a physical table in a room (e.g., tabletop), a distance away from Participant 1, relative to an object worn by Participant 1 (smartwatch, smart glasses, etc.), etc. A computer-generated representation of the virtual portal space 200 may be displayed in the immersive XR environment viewed by Participants 1 and 2.

[0040] In some embodiments, the XR management device 100 defines a location of an entrance boundary of the virtual portal 200 relative to a reference system in physical space where the participant resides and which is tracked by one or more sensors 230. The entrance boundary may be a single point, may extend along a two-dimensional (2D) plane (e.g., flat rectangle or flat circle), such as illustrated in Figure 2, or extend in a three-dimensional (3D) shape (e.g., sphere or box). The entrance boundary may be defined to be a fixed distance and / or a fixed angle (fixed pose offset) relative to the participant or relative to a defined physical object (e.g., tabletop) which is in the physical space. The participant and / or the defined physical object may move within the physical space, and the entrance boundary location in the reference system of physical space may be dynamically updated to track changes in location of the participant and / or the defined physical smartphone 210 or otherphysical electronic device. In some embodiments, any pre-existing physical electronic devices, placed within the physical space where the virtual portal space is rendered, are scanned as a part of the initialization of the first virtual portal.

[0041] Figure 3 illustrates a virtual user interface (VUI) 310 that can be interacted with by Participant 2 in the second virtual portal space to trigger performance of a selected operational capability by operation of the physical smartphone 200, by a selected operational capability provided for the second virtual portal space, or by a combination of the physical smartphone 200 and the second virtual portal space, in accordance with some embodiments of the present disclosure. The operational capability handler 102 can be configured to control which operational capabilities of the physical smartphone 200 and the second virtual portal space are made accessible to Participant 2, and to control performance of selected ones of the capabilities.

[0042] Figure 13 illustrate a flowchart of operations that can be performed by the XR management device 100, e.g., the operational capabilities handler 102 therein, in accordance with some embodiments of the present disclosure.

[0043] Referring to Figures 3 and 13, some of these embodiments are now described in the context of the non-limiting example above where Participant 1 has passed a physical smartphone 210 across the entrance boundary of the virtual portal, which has triggered the XR management device 100 to render a virtual smartphone 300 in the immersive XR environment to be viewable by Participants 1 and 2 through their respective XR headsets.

[0044] The XR management device 100 obtains a first set of operational capabilities 312 of the physical smartphone 210 (or other physical electronic device) in a first virtual portal space of the immersive XR environment. The XR management device 100 obtains a second set of operational capabilities 314 provided for a second virtual portal space of the immersive XR environment.

[0045] The XR management device 100 determines 1300 a combined set of operational capabilities that includes the first set of operational capabilities of the smartphone 210 and further includes the second set of operational capabilities provided for the second virtual portal space. The first set of operational capabilities of the smartphone 210 may be a subset of the operational capabilities that are available to be virtualized for interaction within the second virtual portal space. Similarly, the second set of operational capabilities provided for second virtual portal space may be a subset of the operational capabilities that are available to be virtualized for interaction within the second virtual portal space.

[0046] The XR management device 100 generates 1302 a rendering of a virtual smartphone 300 in the second virtual portal space for display by the XR display device 110. The rendering of virtual smartphone 300 includes a combined VUI with a combined set of VUI elements that Participant 2 can interact with in the second virtual portal space.

[0047] The XR management device 100 identifies 1304 an interaction of Participant 2 with a first VUI element of the combined VUI that satisfies a capability trigger rule.

[0048] Based on the capability trigger rule becoming satisfied, the XR management device 100 identifies 1306 a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set.

[0049] When the selected operational capability is in the first set of operational capabilities of the physical smartphone 210, the XR management device 100 triggers performance of the selected operational capability by operation of the physical smartphone 210. In contrast, when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, the XR management device 100 triggers performance of the selected operational capability provided for the second virtual portal space.

[0050] When identifying 1304 the interaction, the XR management device 100 may track interaction of Participant 2's avatar hand, physical hand, virtual / physical pointer, etc. with the virtual smartphone 300 and may change the rendered pose of the virtual smartphone 300 responsive to Participant 2's interaction. For example, the XR management device 100 may rotate the virtual smartphone 300 to track movement of Participant 2 avatar hand or physical hand rotating while virtually holding the virtual smartphone. In this manner, Participant 2 can rotate the virtual smartphone 300 to view the display device and VUI 310 associated therewith. Participant 1 may visually observe Participant 2's interaction with the virtual smartphone 300.

[0051] Figures 4A and 4B illustrate a combined flowchart of operations that can be performed by the XR management device 100, via the operational capability handler 102, in accordance with some embodiments of the present disclosure. Figures 5-12 illustrate a series of operational scenarios that can be performed with the XR management device 100 and the XR display device in accordance with some embodiments of the present disclosure.

[0052] Referring now to these figures, the XR management device 100 determines 400 that Participant 2 has entered an area where the first virtual portal space of the immersive XRenvironment is available. Participant 2 activates 402 XR features of the physical smartphone 210, i.e., allowing the physical smartphone 210 to enter a virtual portal, e.g., by starting an XR application on the physical smartphone 210 or causing Participant 1 to start the XR application. Participant 2 sends 404 a request to the XR management device 100 to render the first and second virtual portal spaces. The XR management device 100 validates that an access rights policy is satisfied.

[0053] In one embodiment, the XR management device 100 obtains participant information (e.g., Participant 2's ID, biometrics identifying Participant 2, login credentials, etc.), and selects based on the participant information and based on the access rights policy which of the first set of operational capabilities of the physical smartphone 210 and which of the second set of operational capabilities provided for the second virtual portal space to include in the combined set of operational capabilities. As explained above, the access rights policy can be specific to the second virtual portal space, the participant, the physical electronic device, or any combination thereof.

[0054] In another embodiment, a security rule controls based on the participant information whether to generate the rendering in the second virtual portal space of the combined VUI with the combined set of VUI elements that Participant 2 can interact with in the second virtual portal space. In an example thereof, the XR management device 100 obtains the participant information. Responsive to determining the security rule is satisfied by the participant information, the XR management device 100 generates the rendering in the second virtual portal space of the combined VUI with the combined set of VUI elements that Participant 2 can interact with in the second virtual portal space. The combined set of VUI elements are mapped to the combined set of operational capabilities.

[0055] The XR management device 100, via the operational capability handler 102, may control what interaction Participant 2 is permitted to have with the combined set of VUI elements in the second virtual portal space to control operational functions of the combined set, based on the participant information.

[0056] The participant information may include, but is not limited to, at least one of an identifier of Participant 2 and / or Participant 1 (e.g., owner of smartphone), access credentials provided by Participant 2 and / or Participant 1, access credential or other information provided by the physical smartphone 210 and / or a device controlled by Participant 2 and / or Participant 1.

[0057] The XR management device 100 triggers 408 rendering of the coupled first (input) and second (output) virtual portal spaces by the XR display device(s) 110.

[0058] As shown in Figures 5 and 4A, the XR management device 100 determines 410 (e.g., views through forward-facing camera on XR display device) that Participant 2 or Participant 1 has entered the physical smartphone 210 into the first (input) virtual portal space.

[0059] As shown in Figures 6 and 4A, the XR management device 100 determines 412 the identity and characteristics of the physical smartphone 210. For example, a surface of the physical smartphone 210 may be scanned by the sensor 230 (e.g., camera) and identified (recognized), a device ID transmitted by the physical smartphone 210 may be identified, etc. The XR management device 100 validates 414 that the physical smartphone 210 has a first set of operational capabilities that are permitted to be used in the second virtual portal space, e.g., by Participant 2 who is viewing the second (output) virtual portal space through XR headset 110.

[0060] The XR management device 100 identifies 416 the second set of operational capabilities provided for the second virtual portal space. The operational capabilities may include, but are not limited to, hardware processing and / or communication resources, local software resources (e.g., locally hosted software services), networked software resources (e.g., over-the-top services provided by a remote server, etc.) which are available to be used by Participant 2 through the second virtual portal space. Examples of these and other operational capabilities are described below.

[0061] As shown in Figures 7 and 4 A, the XR management device 100 generates 418 a rendering of a virtual smartphone 300 which includes the combined VUI 310 with a combined set of VUI elements that Participant 2 and / or another participant can interact with in the second (output) virtual portal space. In the example illustrated in Figure 7, the UI of the physical smartphone 210 has UI elements "1", "2", "3", and "4" which can be individually selected through the physical UI to trigger four corresponding operational capabilities of the physical smartphone 210. The XR management device 100 has selected, based on an access rights policy and / or security rule, only operational capabilities "1" and "4" for inclusion in the combined set of operational capabilities that are displayed as VUI elements in the combined VUI 310 of the virtual smartphone 300 in the second (output) virtual portal space for selection by Participant 2.

[0062] As shown in Figure 8 and 4A, the XR management device 100 has selected, based on the access rights policy and / or security rule, only operational capabilities "A" and "B" provided for the second (output) virtual portal space for inclusion in the combined set of operational capabilities that are displayed as VUI elements in the combined VUI 310 of thevirtual smartphone 300 in the second (output) virtual portal space for selection by Participant 2.

[0063] As shown in Figures 9 and 4B, the XR management device 100 identifies 420 an interaction of Participant 2 and / or another participant with a first VUI element of the combined VUI 310 that satisfies a capability trigger rule. Based on the capability trigger rule becoming satisfied, the XR management device 100 identifies 422 a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI 310 and the operational capabilities of the combined set.

[0064] When the selected operational capability is in the first set of operational capabilities of the physical smartphone 210, the XR management device 100 triggers 424 performance of the selected operational capability by operation of the physical smartphone 210. For example, as shown in Figure 9, in the scenario where the selected operational capability is in the first set of operational capabilities of the physical smartphone 210, the XR management device 100 sends a message to the physical smartphone 210 to cause it to perform the selected operational capability.

[0065] In an example embodiment, the operation of the XR management device 100 to trigger the performance of the selected operational capability by operation of the physical smartphone 210, includes to trigger execution of an application and / or an operational state of the application by instructions executable by at least one processor of the physical smartphone 210.

[0066] The XR management device 100 may operate to receive a message from the physical smartphone 210 responsive to triggering performance of the selected operational capability by operation of the physical smartphone 210. The message can indicate what UI content is to be updated responsive to the performance of the selected operational capability. The XR management device 100 can responsively update the combined VUI to contain content that is based on the UI content indicated by the message.

[0067] In contrast, when the selected operational capability is in the second set of operational capabilities provided for the second virtual portal space, the XR management device 100 triggers 426 performance of the selected operational capability provided for the second virtual portal space. For example, as shown in Figure 10, in the scenario where the selected operational capability is in the second set of operational capabilities provided for the second (output) virtual portal space, the XR management device 100 sends a message to theXR headset 110 to cause it to perform the selected operational capability for the second (output) virtual portal space.

[0068] In a first example embodiment, the operation of the XR management device 100 to trigger the performance of the selected operational capability provided for the second virtual portal space, includes to provide the participant, through the second virtual portal space, access through at least one communications network to an over-the-top service having a subscription accessed using credentials associated with the second virtual portal space.

[0069] In a second example embodiment, the operation of the XR management device 100 to trigger the performance of the selected operational capability provided for the second virtual portal space, includes to provide the participant, through the second virtual portal space, access to use communication network resources using credentials associated with the XR display device (110) displaying XR objects in the second virtual portal space.

[0070] In a third example embodiment, the operation of the XR management device 100 to trigger the performance of the selected operational capability provided for the second virtual portal space, includes to provide the participant control of how user output of an application performed by the physical electronic device is visually rendered when displayed by the XR display device (110) in the second virtual portal space for viewing by the participant.

[0071] In a fourth example embodiment, the operation of the XR management device 100 to trigger the performance of the selected operational capability provided for the second virtual portal space, includes to trigger local storage in memory of the XR display device (110), participant selected program data, video, images, and / or audio which is processed by the XR display device (110) in the second virtual portal space.

[0072] The XR management device 100 updates 428 the combined VUI 310 to indicate ongoing status or the result of performance of the selected operation capability.

[0073] Responsive to detecting 430 that the physical smartphone 210 has been removed from the first virtual portal space, the XR management device 100 updates the combined set of operational capabilities to remove at least part of the force of artificial capabilities of physical smartphone 210. For example the XR managing device 100 rendering of or terminate access to the combined VUI 310 and / or to one or more VUI elements thereof.

[0074] The XR management device 100 may store 432 an operational state of the first set of operational capabilities, the second set of operational capabilities, and or the combined set of operational capabilities for future use when rendering the virtual smartphone 300 corresponding to the physical smartphone 210.

[0075] Referring to Figure 11, different first and second sets of operational capabilities may be associated with different physical spaces (e.g., rooms) and / or virtual spaces, associated with different participants, and / or associated with different physical devices (e.g., different device IDs, different types of devices, etc.).

[0076] For example, a virtual portal space boundary may be defined by a doorway, and different rooms may defined to provide different sets of operational functionality for the associated virtual portal spaces for use by physical electronic devices that are entering and / or leaving respective ones of the virtual portal spaces.

[0077] Thus, the location of the physical smartphone 210 in the first virtual portal space (e.g., tabletop in living room, desktop in home office, within car, etc.) controls selection by the XR management device 100 of which of the first set of operational capabilities of the physical smartphone 210 and which of the second set of operational capabilities provided for the second virtual portal space are included in the combined set of operational capabilities and made available for access to Participant 2.

[0078] In an example embodiment, the XR management device 100 determines location of the physical smartphone 210 in the first virtual portal space. The XR management device 100 selects based on the location of the physical smartphone 210 in the first virtual portal space which of the first set of operational capabilities of the physical smartphone 210 and which of the second set of operational capabilities provided for the second virtual portal space to include in the combined set of operational capabilities and made available for access to Participant 2.

[0079] Compatibility between operational functions of the physical smartphone 210 and of the second virtual portal space can determine which operational capabilities are included in the combined set of operational capabilities. In an example embodiment, the XR management device 100 determines which of the first set of operational capabilities of the physical smartphone 210 and which of the second set of operational capabilities provided for the second virtual portal space are operationally compatible with each other. Operational compatibility is determined to occur when the operational capability of the first set has an application programming interface (API) that is configured to interface to an API of an operational capability of the second set. The XR management device 100 selects operational compatible ones of the first and second set of operational capabilities to include in the combined set of operational capabilities and made available for access to Participant 2.

[0080] For example, the XR management device 100 may operationally interconnect at least part of the application programming interface of the operational capability of the first setand at least part of the application programming interface of the operational capability of the second set.

[0081] A certain room may be associated with a selected (e.g., policy-based) first and second subsets of operational capabilities to be used by participant(s) in that room (virtual portal space). For example, some defined person(s) (e.g., parents) may be defined by rule to be allowed use of first subsets Al of operational capabilities in a room within a home, and use of a second subset Bl of operational capabilities anywhere within the home. In contrast, other defined person(s) (e.g., children) may be defined by the rule to be allowed use of first subset A2 and second subset B2 of the operational capabilities (where Al > A2, Bl > B2, etc.) when located in the respective virtual portal spaces. Other undefined person(s), e.g., guests, may be defined by the rule to be allowed use of even less operational capabilities corresponding to first and second subsets A3 and B3, where A2 > A3, B2 > B3, or may instead not be granted access to any of second subset’s operational capabilities.

[0082] In some embodiments, Al and A2 may be defined by a parent and a child having two different devices. In other, Al and A2 may originate from the same device but instead depend on the identified user of the device.

[0083] These and other embodiments can enable innovative use cases. In one example use case, the XR management device 100 limits access by a remote service technician to a limited subset of the combined set of VUI elements in the second virtual portal space to control a corresponding limited subset of the operational functions of the combined set. The limited subset of the operational functions of the combined set enables the technician to control and view output responses of a subset of operational functions of the electronic device.

[0084] One application of this use case is directed to enabling a remote service technician access to a defined subset of electronic functionality of a vehicle, such as an automobile. The limited subset of the operational functions may allow the technician to view and input values through the vehicle UI(s), view and reset recorded vehicle onboard diagnostics (OBD) data, and to provide input to and view output of certain electronic components of the vehicle. However, the technician may be prohibited from viewing saved vehicle camera videos, prohibited from viewing and / or resetting certain recorded OBD data, prohibited from moving the vehicle, prohibited from moving the vehicle more than a define distance, prohibited from starting the vehicle, prohibited from increasing an internal combustion engine revolutions- per-minute (RPM) beyond a defined RPM limit, etc. In this case, the first virtual portal may be defined as a physical enclosure where the car is located, e.g., a garage, or as a part of thecar, e.g., the driver’s seat. The second virtual portal may be placed in the office of the remote service technician.

[0085] Another application of this use case is directed to enabling a remote service technician access to a defined subset of electronic functionality of a smart appliance, such as a home Heating, Ventilation, and Air Conditioning (HVAC) system. The limited subset of the operational functions may allow the technician to view and input values through the HVAC UI (e.g., thermostat) and to provide input to and view output of certain electronic components of the HVAC. However, the technician may be prohibited from changing the temperature beyond a defined range, prohibited from changing a user-defined schedule of temperature and time-of-day settings, etc.

[0086] The virtual smartphone 300 and the second virtual portal space may be associated with a policy that includes an authorization part and a persistence part. The authorization part defines what operational capabilities of the physical smartphone 210 and the second virtual portal space, that Participant 2 is allowed to utilize, e.g. apps, settings, features, interfaces, files etc., and how changes to such features should be transferred to the physical smartphone 210 and / or to the state of and rendering of the second virtual portal space. The persistence part defines what happens to those operational capabilities when the physical smartphone 210 is no longer present in the first virtual portal space, e.g., do all of the combined operational capabilities stay updated or do they revert to an operational state prior to the physical smartphone 210 becoming present in the first virtual portal space.

[0087] The authorization policy may be defined by a user of the system beforehand and may be associated with Participant 2, e.g. “when I share my device with Participant X, he may access applications A, B, C”. The policy may also be decided ad hoc during the sharing, e.g. by issuing a voice comment “allow Participant X to access apps A, B, C ” Each operational capability may have its own specified access rules, where the attributes in these access rules includes, but are not limited to, read, write, delete, export, import, on-behalf-of.

[0088] The XR management device 100 may make decisions based on one or more access policies as to which of the displayed combined VUI elements are selectable by Participant 2 or other participants in the immersive XR environment. The access policy(ies) may define which identified persons are permitted to select which combined VUI elements or otherwise access defined operational functions of the physical smartphone 210 and / or the first virtual portal space. For example, Participant 2 may have different access privileges than another participant for selecting which of the combined VUI element and / or accessing defined operational functions of the physical smartphone 210 and / or the first virtual portalspace. VUI elements which are permitted by the access policy to be selected by Participant 2 may be displayed in a visually different way (e.g., color, shading, font, etc.) than other VUI elements which are not permitted by the access policy to be selected by Participant 2. The access policies may be visualized by grayed out options, or by something indicating the allowed actions to Participant 2. Upon Participant 2 pressing a grayed-out (not-yet approved option), the XR management device 100 may identify the associated optional operational interaction and provide Participant 1 with an option to either allow or deny the interaction attempted or requested.

[0089] The persistence policy controls how interactions with the virtual smartphone 300 will affect the operational capabilities of the physical smartphone 210, affect the operational capabilities of the second virtual portal space, and for how long. Figure 12 illustrates the XR management device 100 determining that the physical smartphone 210 is no longer present in the first virtual portal space, e.g., based on processing video provided by a forward facing camera of the XR headset 110.

[0090] Options include temporary or permanent control of settings and / or operations of the physical smartphone 210 and the second virtual portal space, per action or until end of session, automatic or manual approval by Participant 1 who possesses the physical smartphone 210. For example, when the above-example physical smartphone 210 is withdrawn from the first virtual portal space, the XR management device 100 and / or an application on the physical smartphone 210 can operate to return the physical smartphone 210 and / or the second virtual portal space to its initial operational state existing immediately before time when the physical smartphone 210 entered the first virtual portal space. In one embodiment, a journal of operations is created which logs a sequence of operations that are performed on the physical smartphone 210 and / or changes in operational state(s) of the second virtual portal space while the physical smartphone 210 is within the first virtual portal space, e.g., changes occurring responsive to Participant 2 manipulating the combined VUI 310. In response to the physical smartphone 210 being withdrawn from the first virtual portal space, the sequence of operations and / or operational state(s) logged in the journal can be operationally undone, e.g., reversed, to return the physical smartphone 210 and / or the second virtual portal space to its initial operational state existing immediately before time when the physical smartphone 210 entered the first virtual portal space.

[0091] In some embodiments, removal of the persistence policy may trigger the persistence policy to terminate rendering of the virtual smartphone 300 representation in theimmersive XR environment, including terminating the rendering of the combined VUI in the second virtual portal space.

[0092] Example XR management device Configuration.

[0093] Figure 14 illustrates a block diagram of components of an XR management device 100 that are configured to operate in accordance with some embodiments of the present disclosure. The XR management device 100 can include at least one processing circuit 1400 ("processor"), at least one memory circuit 1410 ("memory"), at least one network interface 1420 ("network interface:), and may include a display device 1430. The processor 1400 is operationally connected to these various components. The memory 1410 stores executable instructions 1412 that are executed by the processor 1400 to perform operations. The executable instructions 1412 can be configured to perform the functionality described herein for the operational capability handler 102. The processor 1400 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 1400 is configured to execute the instructions 1412 in the memory 1410, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for an XR management device 100. As explained herein, the XR management device 100 may be separate from and communicatively connected to the participant devices or may be at least partially integrated within one or more of the participant devices.

[0094] Figure 15 illustrates a block diagram of components of a physical electronic device 210 that are configured to operate in accordance with some embodiments of the present disclosure. The electronic device 210 can include at least one processing circuit 1500 ("processor"), at least one memory circuit 1510 ("memory"), at least one network interface 1520 ("network interface:), and a display device 1530. The processor 1500 is operationally connected to these various components. The memory 1510 stores executable instructions 1512 and an XR module 1514 that are executed by the processor 1500 to perform operations. The processor 1500 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 1500 is configured to execute the instructions 1512 and XR module 1514 in the memory 1510, described below as a computer readable medium, to perform some or all of theoperations and methods for one or more of the embodiments disclosed herein for an electronic device.

[0095]

[0096] Further Definitions and Embodiments:

[0097] In the above-description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0098] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0099] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0100] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions butdoes not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id est," may be used to specify a particular item from a more general recitation.

[0101] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0102] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0103] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / actsinvolved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0104] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts are to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. An extended reality, XR, management device (100) for managing an immersive XR environment displayable through an XR display device (110) to a participant in the immersive XR environment, the XR management device comprising: at least one processor (1400); and at least one memory (1410) storing instructions executable by the at least one processor (1400) to perform operations to: determine a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device (210) in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment; generate a rendering of a virtual electronic device in the second virtual portal space for display by the XR display device (110), the rendering of virtual electronic device including a combined virtual user interface, VUI, with a combined set of VUI elements that the participant can interact with in the second virtual portal space; identify an interaction of the participant with a first VUI element of the combined VUI that satisfies a capability trigger rule; based on the capability trigger rule becoming satisfied, identify a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set; when the selected operational capability is in the first set of operational capabilities of the physical electronic device (210), trigger performance of the selected operational capability by operation of the physical electronic device (210); and when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, trigger performance of the selected operational capability provided for the second virtual portal space.

2. The XR management device (100) of Claim 1, wherein the instructions executable by the at least one processor (1400) further perform operations to: obtain participant information; and select based on the participant information and based on an access rights policy which of the first set of operational capabilities of the physical electronic device (210) and which of the second set of operational capabilities provided for the second virtual portal space to include in the combined set of operational capabilities.

3. The XR management device (100) of any of Claims 1 to 2, wherein the instructions executable by the at least one processor (1400) further perform operations to: obtain participant information; and responsive to determining a security rule is satisfied by the participant information, generate the rendering in the second virtual portal space of the combined VUI with the combined set of VUI elements that the participant can interact with in the second virtual portal space, wherein the combined set of VUI elements are mapped to the combined set of operational capabilities.

4. The XR management device (100) of any of Claims 2 to 3, wherein the instructions executable by the at least one processor (1400) further perform operations to: control what interaction the participant is permitted to have with the combined set of VUI elements in the second virtual portal space to control operational functions of the combined set, based on the participant information.

5. The XR management device (100) of any of Claims 2 to 3, wherein the physical electronic device (210) comprises an electronic device the instructions executable by the at least one processor (1400) further perform operations to: limit access by a remote service technician to a limited subset of the combined set of VUI elements in the second virtual portal space to control a corresponding limited subset of the operational functions of the combined set, wherein the limited subset of the operational functions of the combined set enable the technician to control and view output responses of a subset of operational functions of the electronic device .

6. The XR management device (100) of any of Claims 2 to 5, wherein the participant information comprises at least one of an identifier of the participant and access credentials provided by the participant or by a device controlled by the participant.

7. The XR management device (100) of any of Claims 1 to 6, wherein the instructions executable by the at least one processor (1400) further perform operations to: determine location of the physical electronic device in the first virtual portal space; and select based on the location of the physical electronic device in the first virtual portal space which of the first set of operational capabilities of the physical electronic device (210) and which of the second set of operational capabilities provided for the second virtual portal space to include in the combined set of operational capabilities.

8. The XR management device (100) of any of Claims 1 to 7, wherein the instructions executable by the at least one processor (1400) further perform operations to: determine which of the first set of operational capabilities of the physical electronic device (210) and which of the second set of operational capabilities provided for the second virtual portal space are operationally compatible with each other, wherein operational compatibility is determined to occur when an operational capability of the first set has an application programming interface that is configured to interface to an application programming interface of an operational capability of the second set; and select operational compatible ones of the first and second set of operational capabilities to include in the combined set of operational capabilities.

9. The XR management device (100) of Claim 8, wherein the instructions executable by the at least one processor (1400) further perform operations to: operationally interconnect at least part of the application programming interface of the operational capability of the first set and at least part of the application programming interface of the operational capability of the second set.

10. The XR management device (100) of any of Claims 1 to 9, wherein to trigger the performance of the selected operational capability provided for the second virtual portalspace, the instructions executable by the at least one processor (1400) further perform operations to: provide the participant, through the second virtual portal space, access through at least one communications network to an over-the-top service having a subscription accessed using credentials associated with the second virtual portal space.

11. The XR management device (100) of any of Claims 1 to 10, wherein to trigger the performance of the selected operational capability provided for the second virtual portal space, the instructions executable by the at least one processor (1400) further perform operations to: provide the participant, through the second virtual portal space, access to use communication network resources using credentials associated with the XR display device (110) displaying XR objects in the second virtual portal space.

12. The XR management device (100) of any of Claims 1 to 11, wherein to trigger the performance of the selected operational capability provided for the second virtual portal space, the instructions executable by the at least one processor (1400) further perform operations to: provide the participant control of how user output of an application performed by the physical electronic device is visually rendered when displayed by the XR display device (110) in the second virtual portal space for viewing by the participant.

13. The XR management device (100) of any of Claims 1 to 12, wherein to trigger the performance of the selected operational capability provided for the second virtual portal space, the instructions executable by the at least one processor (1400) further perform operations to: trigger local storage in memory of the XR display device (110), participant selected program data, video, images, and / or audio which is processed by the XR display device (110) in the second virtual portal space.

14. The XR management device (100) of any of Claims 1 to 13, to trigger performance of selected operational capability by operation of the physical electronic device (210), the instructions executable by the at least one processor (1400) further perform operations to:trigger execution of an application and / or an operational state of the application by instructions executable by at least one processor (1400) of the physical electronic device (210).

15. The XR management device (100) of any of Claims 1 to 14, wherein the instructions executable by the at least one processor (1400) further perform operations to: receive a message from the physical electronic device (210) responsive to triggering performance of the selected operational capability by operation of the physical electronic device (210), wherein the message indicates what UI content is to be updated responsive to the performance of the selected operational capability; and update the combined VUI to contain content that is based on the UI content indicated by the message.

16. The XR management device (100) of any of Claims 1 to 15, wherein the XR management device (100) comprises an XR server connected through at least one network to at least one XR display device.

17. The XR management device (100) of any of Claims 1 to 15, wherein: the XR management device (100) comprises a plurality of processors and memories; one of the processors and one of the memories reside in the XR display device (110); and the instructions executable the processor residing in the XR display device (110) perform operations to display on a display device of the XR display device (110) the rendering of the virtual electronic device in the second virtual portal space, and when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, trigger performance of the selected operational capability provided for the second virtual portal space.

18. The XR management device (100) of any of Claims 1 to 17, wherein the instructions executable by the at least one processor further perform operations to: responsive to determining the physical electronic device (210) has been removed from the first virtual portal space, updating the combined set of operationalcapabilities to remove at least part of the first set of operational capabilities of the physical electronic device (210).

19. A method performed by an extended reality, XR, management device for managing an immersive XR environment displayable through an XR display device to a participant in the immersive XR environment, the method comprising: determining (1300) a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment; generating (1302) a rendering of a virtual electronic device in the second virtual portal space for display by the XR display device, the rendering of virtual electronic device including a combined virtual user interface, VUI, with a combined set of VUI elements that the participant can interact with in the second virtual portal space; identifying (1304) an interaction of the participant with a first VUI element of the combined VUI that satisfies a capability trigger rule; based on the capability trigger rule becoming satisfied, identifying (1306) a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set; when the selected operational capability is in the first set of operational capabilities of the physical electronic device, triggering (1308) performance of the selected operational capability by operation of the physical electronic device; and when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, triggering (1310) performance of the selected operational capability provided for the second virtual portal space.

20. A computer program product comprising a non-transitory computer readable medium storing instructions executable by at least one processor of an extended reality, XR,management device (100) for managing an immersive XR environment displayable through an XR display device (110) to a participant in the immersive XR environment, the instructions executed by the at least one processor to perform operations comprising to: determine a combined set of operational capabilities that includes a first set of operational capabilities of a physical electronic device (210) in a first virtual portal space of the immersive XR environment and further includes a second set of operational capabilities provided for a second virtual portal space of the immersive XR environment; generate a rendering of a virtual electronic device in the second virtual portal space for display by the XR display device (110), the rendering of virtual electronic device including a combined virtual user interface, VUI, with a combined set of VUI elements that the participant can interact with in the second virtual portal space; identify an interaction of the participant with a first VUI element of the combined VUI that satisfies a capability trigger rule; based on the capability trigger rule becoming satisfied, identify a selected operational capability of the combined set of operational capabilities that is mapped to the first VUI element by a defined mapping between the VUI elements of the combined VUI and the operational capabilities of the combined set; when the selected operational capability is in the first set of operational capabilities of the physical electronic device (210), trigger performance of the selected operational capability by operation of the physical electronic device (210); and when the selected operational capability is included in the second set of operational capabilities provided for the second virtual portal space, trigger performance of the selected operational capability provided for the second virtual portal space.

Images