Radio relay of physical electronic device communications in extended reality environment

Abstract

An XR rendering device renders a first virtual electronic device representation in the XR environment of a first physical electronic device for display through an XR display The XR rendering device communicates traffic through a first communication channel from the first operational function to the first physical electronic device to control an operational function of the first physical electronic device responsive to a defined interaction of a participant with the first virtual user interface of the first virtual electronic device. The XR rendering device establishes a second communication channel between a second physical electronic device and a second operational function of the first virtual electronic device responsive to another defined interaction with the first virtual user interface. The XR rendering device communicates traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display and the second physical electronic device to control an operational function of the second physical electronic device.

Description

RADIO RELAY OF PHYSICAL ELECTRONIC DEVICE COMMUNICATIONS IN EXTENDED REALITY ENVIRONMENTTECHNICAL FIELD

[0001] The present disclosure relates to rendering extended reality (XR) environments and associated XR rendering devices, and more particularly to communications associated with immersive XR environments displayed on XR display devices.BACKGROUND

[0002] Immersive extended reality (XR) environments have been developed which provide a myriad of different types of user experiences for gaming, on-line meetings, cocreation of products, etc. One type of immersive XR environments (also referred to as "XR environments") is virtual reality (VR) environments where human users only see computer generated graphical renderings. Another type of XR environment is augmented reality (AR) environments where users see a combination of computer-generated graphical renderings overlaid on a view of the physical real-world through, e.g., see-through display screens.

[0003] Example XR environment rendering devices (XR rendering devices) include, without limitation, XR environment servers, XR head-mounted devices (HMDs), gaming consoles, smartphones running an XR application, and tablet / laptop / desktop computers running an XR application. XR rendering devices render the immersive XR environment for display through an XR display device for viewing by a participant in the XR environment. The XR rendering device can be separate from the XR display device, such as when an XR server renders an XR environment for display on a network connected HMD worn by a participant. Meta Quest 2 and Varjo XR-3 are example VR display devices and Hololens is an example AR display device.

[0004] XR meeting applications are tools for experiencing online immersive meetings and are useful as a thinking and planning space for oneself as well as among a group of people. Some XR meeting applications support AR devices, browsers, and VR devices. A participant using a browser may join via desktop, tablet-PC or smartphone and share their views using a front faced camera or a web camera. Some XR meeting solutions have mobile application versions, e.g., Android and iOS, which allow a user to navigate in the virtual space on the screen or activate an AR mode to display the meeting in their own surroundings. The XR meeting solutions introduce new features to online meetings that allow for new waysto share and create content etc. Today’s commonly and commercially available XR devices typically include an HMD and a pair of hand controllers, sometimes with more advanced solutions including “foot controllers” and / or tactile gloves.

[0005] Immersive XR environments, such as gaming environments and meeting environments, are often configured to display computer generated avatars which represent poses of human users in the immersive XR environments. A user may select and customize an avatar, such as gender, clothing, hair style, etc. to represent that user for viewing by other users participating in the immersive XR environment. Users can interact with virtual objects that are rendered in the immersive XR environment, such as by controlling their avatars to manipulate virtual objects. Immersive XR environments presently support little if any interface between real -world physical objects and their corresponding rendering as virtual objects in the XR environment.SUMMARY

[0006] Some embodiments disclosed herein are directed to an XR rendering device for rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment. The XR rendering device includes at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations include to render a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device for display through the XR display device for viewing by the first participant. The rendering of the first virtual electronic device includes a first virtual user interface (UI) that the first participant views through the XR display device and can interact with in the immersive XR environment. The operations also include establishing a first communication channel between the first physical electronic device and a first operational function of the first virtual electronic device. The operations include communicating traffic through the first communication channel from the first operational function to the first physical electronic device to control an operational function of the first physical electronic device responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device. The operations include establishing a second communication channel between a second physical electronic device and a second operational function of the first virtual electronic device responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device. The operations include communicating traffic from the second operationalfunction through the second communication channel and a wireless communication relay between the XR display device and the second physical electronic device to control an operational function of the second physical electronic device.

[0007] Some other embodiments are directed to a corresponding method by an XR rendering device for rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment. The method includes rendering a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device for display through the XR display device for viewing by the first participant. The rendering of the first virtual electronic device includes a first virtual user interface that the first participant views through the XR display device and can interact with in the immersive XR environment. The method includes establishing a first communication channel between the first physical electronic device and first operational function of the virtual electronic device. The method includes communicating traffic through the first communication channel from the first operational function to the first physical electronic device to control an operational function of the first physical electronic device responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device. The method includes establishing a second communication channel between a second physical electronic device and a second operational function of the first virtual electronic device responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device. The method includes communicating traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display device and the second physical electronic device to control an operational function of the second physical electronic device.

[0008] Some other embodiments are directed to a non-transitory computer readable medium storing instructions executable by at least one processor of an XR rendering device rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment. The instructions executed by the at least one processor perform operations including render a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device for display through the XR display device for viewing by the first participant. The rendering of the first virtual electronic device comprises a first virtual user interface that the first participant views through the XR display device and can interact with in the immersive XR environment. The instructions also include establish a first communication channel betweenthe first physical electronic device and a first operational function of the virtual electronic device. The instructions also include communicate traffic through the first communication channel from the first operational function to the first physical electronic device to control an operational function of the first physical electronic device responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device. The instructions also include establish a second communication channel between a second physical electronic device and a second operational function of the first virtual electronic device responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device. The instructions also include communicate traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display device and the second physical electronic device to control an operational function of the second physical electronic device.

[0009] Some potential advantages of these embodiments are that they enable an XR display device (e.g., HMD) to operate as a communication relay for communications between the XR rendering device (e.g., XR server) and one or more physical electronic devices which are communicatively connected to the XR display device. An XR environment participant can interact with the first virtual user interface that has been rendering for the first physical electronic device (e.g., smartphone) to cause the XR rendering device (e.g., XR server) to control the XR display device (e.g., HMD) to discover presence of other physical electronic devices (e.g., smartwatch, wireless audio headphones, etc.), and enable the participant to interact with virtual UI interfaces rendered for the other physical electronic devices to establish communication channel(s) with the participant-selected ones of the other physical electronic devices and control operational functions of the other physical electronic devices.

[0010] Other XR rendering devices, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional XR rendering devices, methods, and computer program products be included within this description and protected by the accompanying claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0012] Figure 1 illustrates an XR system that includes a physical electronic device and a plurality of XR display devices that communicate through networks with an XR rendering device to operate in accordance with some embodiments of the present disclosure;

[0013] Figure 2 illustrates the XR rendering device rendering through an XR display device a virtual representation of the physical electronic device (virtual electronic device) in the first participant’s physical space while the physical electronic device is being passed through a virtual portal, in accordance with some embodiments of the present disclosure;

[0014] Figure 3 illustrates a system in which an XR rendering server displays virtual electronic devices through a head mounted display device and enables a participant to interact with virtual user interfaces and of the virtual electronic devices, which is operationally associated with the user interface of the physical electronic device, in accordance with some embodiments of the present disclosure;

[0015] Figure 4 illustrates a flowchart of operations of an XR rendering server, XR display device (e.g., HMD), first physical electronic device (e.g., smartphone), and second physical electronic device (e.g., smartwatch) in accordance with some embodiments of the present disclosure;

[0016] Figure 5 is a flowchart of operations that can be performed by an XR rendering device in accordance with some embodiments of the present disclosure;

[0017] Figure 6 is a block diagram of components of an XR rendering device that are configured to operate in accordance with some embodiments of the present disclosure; and

[0018] Figure 7 is a block diagram of components of a physical electronic device that are configured to operate in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0019] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0020] Figure 1 illustrates an XR system that includes a physical electronic device 210 and a plurality of XR display devices 110a and 110b that communicate through wired andwireless communication networks 120 with an XR rendering device 100 to operate in accordance with some embodiments of the present disclosure. The XR rendering device 100 is configured to generate a graphical representation of an immersive XR environment (also called an "XR environment" for brevity), through operation of a computer generated XR environment module 102 which is viewable from various perspectives of virtual poses of human participants in the XR environment through display screens of the various XR display devices HOa-b.

[0021] For example, the illustrated XR display devices 1 lOa-b of Figure 1 include HMDs which can be worn by participants to view and navigate through the XR environment. The XR display devices are not limited to HMDs, as they may be a personal computer, tablet computers, smartphone, or other electronic device which can be operated by a participant to view and navigate through the XR environment. Participant interaction with objections rendered in the XR environment may be tracked by the XR display device itself via built-in motion tracking sensors (e.g., operative to track hands, hand controllers, etc.) or may be tracked via external motion tracking sensors. XR environment participants may have associated avatars which are rendered in the XR environment to represent poses (e.g., location, body assembly orientation, etc.) of the participants relative to a coordinate system of the XR environment.

[0022] A physical electronic device 210, also called "physical object" and “electronic device,” has electronic operational functions which can be controlled through a user interface. The physical electronic device 210 can be any type of electronic device and may include a user interface, e.g., touchscreens of a smartphone and smartwatch, other user interfaces of televisions, wireless audio headphone, etc. Non-limiting examples of physical electronic devices include smartphones, personal computers, table computers, smart appliances, etc. Non-limiting examples of user interfaces include touchscreens and other display devices that can display user selectable indicia and / or user selectable menus, and physical buttons, touch sensor interfaces, etc.

[0023] As will be explained in further detail below, at least a portion of a physical electronic device 210 is rendered in the XR environment for viewing through an XR display device as a virtual electronic device with a virtual user interface representation of at least a portion of the user interface of the physical electronic device 210. A participant in the XR environment can interact with the virtual user interface in order to control various electronic operational function(s) of the physical electronic device 210.

[0024] Although the XR rendering device 100 is illustrated in Figure 1 as being a centralized network computing server separate from one or more of the XR display devices, in some other embodiments the XR rendering device 100 is implemented as a component of one or more of the XR display devices. For example, one of the XR display devices may be configured to perform operations of the XR rendering device in a centralized manner controlling rendering for or by other ones of the XR display devices. Alternatively, each of the XR display devices may be configured to perform at least some of the operations of the XR rendering device in a distributed decentralized manner with coordinated communications being performed between the distributed XR rendering devices (e.g., between software instances of XR rendering devices).

[0025] Some embodiments of the present disclosure are directed to an XR rendering device (e.g., XR server) which enables one or more participants in an immersive XR environment to interact with a digital virtualized representation of a physical electronic device (e.g., smartphone). A second participant can possess the physical electronic device and a first participant can possess the XR display device (e.g., HMD) which renders the virtual electronic device representation of the physical electronic device. The first participant can view and interact with the virtual electronic device through the XR display device (e.g., HMD) to control operational functions of the physical electronic device through communications provided by the XR rendering device (e.g., XR server).

[0026] An XR environment participant can interact with a first virtual user interface that has been rendering for the physical electronic device (e.g., smartphone) to cause the XR rendering device (e.g., XR server) to control the XR display device (e.g., HMD) to discover presence of other physical electronic devices (e.g., smartwatch, wireless audio headphones, etc.), and enable the participant to interact with virtual UI interfaces rendered for the other physical electronic devices to establish communication channel(s) (e.g., Bluetooth or WiFi links) with the participant-selected ones of the other physical electronic devices and control operational functions of the other physical electronic devices. In this manner, the XR display device (e.g., HMD) can operate as a communication relay for communications between the XR rendering device (e.g., XR server) and one or more other physical electronic devices (e.g., smartwatch, wireless audio headphones, etc.) which are communicatively connected to the XR display device (e.g., HMD).

[0027] Figure 5 is a flowchart of operations that can be performed by an XR rendering device in accordance with some embodiments of the present disclosure.

[0028] Referring to Figures 1 and 5, an XR rendering device 100 is provided for rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment. The XR rendering device 100 may include at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations comprise to render 500 a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device 210 for display through the XR display device 110a for viewing by the first participant. The rendering of the first virtual electronic device comprises a first virtual user interface that the first participant views through the XR display device 110a and can interact with in the immersive XR environment. The operations also comprise to establish 502 a first communication channel between the first physical electronic device 210 and a first operational function of the first virtual electronic device. The operations also comprise, responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, to communicate 504 traffic through the first communication channel from the first operational function to the first physical electronic device 210 to control an operational function of the first physical electronic device 210. The operations also comprise, responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, to establish 506 a second communication channel between a second physical electronic device (e.g., smartwatch 320 in Figure 3) and a second operational function of the first virtual electronic device. The operations also comprise to communicate 508 traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display device 110a and the second physical electronic device 320 to control an operational function of the second physical electronic device 320.

[0029] The XR rendering device 100 may be a single computing device or may be a distributed system of computing devices, e.g., computer servers, connected via one or more networks, such as a cloud computing environment. A non-limiting example of a XR rendering device is the cloud computing environment provided by VR headsets which includes numerous networked computing devices which host VR gaming environments serving VR headsets worn by remote users.

[0030] The first participant can interact with the virtual user interface in the immersive XR environment by controlling an avatar representing the first participant, or hands of the first participant interacting with the virtual electronic device. The first participant may additionally or alternatively interact with the virtual user interface through the computertracking, e.g., via sensors built-in an HMD and / or external sensors, of the first participant’s physical hand interacting with the virtual user interface of the virtual electronic device.

[0031] The rendering of the virtual electronic device or virtual object may be provided to a device of the participant and / or one or more other participants for viewing and possible interaction with. HMDs can be worn by participants in the immersive XR environment to provide an intuitive way for the participants to view the rendered virtual object and possibly also view a rendering of the virtual portal, as described in further detail below.

[0032] Some potential advantages of these embodiments are that they enable an XR display device (e.g., HMD) to operate as a communication relay for communications between the XR rendering device (e.g., XR server) and one or more physical electronic devices which are communicatively connected to the XR display device. An XR environment participant can interact with a virtual user interface that has been rendering for a physical electronic device (e.g., smartphone) to cause the XR rendering device (e.g., XR server) to control the XR display device (e.g., HMD) to discover presence of other physical electronic devices (e.g., smartwatch, wireless audio headphones, etc.), and enable the participant to interact with virtual UI interfaces rendered for the other physical electronic devices to establish communication channel(s) with the participant-selected ones of the other physical electronic devices and control operational functions of the other physical electronic devices.

[0033] The XR rendering device operates through the XR display device to discover and communicate through a radio interface with physical electronic devices that are proximately located to the participant who is viewing the XR environment through, e.g., a HMD, and which may be external to a virtual portal (e.g., Figure 2). The XR rendering device may operate to synchronize the operational state(s) of the physical electronic device with its virtual representation as a virtual electronic device, which can operate as an intermediate party for communications between the virtual and physical electronic device, enabling asynchronous updates in both directions.

[0034] Figure 2 is an example illustration of the XR rendering device 100 rendering a virtual representation (virtual electronic device 220) of the physical electronic device 210 in the first participant’s physical space while the physical electronic device 210 is being passed through a virtual portal 200, in accordance with some embodiments of the present disclosure. Referring to Figure 2, Participant 1 and Participant 2 are present participants in an immersive XR environment rendered by the XR rendering device 100. Participant 1 and Participant 2 can simultaneously view virtual objects that are rendered by the XR rendering device 100 in their respective virtual spaces through XR display device, e.g., HMDs worn by theparticipants. The XR rendering device 100 is communicatively coupled to at least one sensor 230 that tracks location of a physical electronic device 210, e.g., a smartphone, relative to the boundary of a virtual portal 200 having a location defined in physical space. The sensor 230 may be one or more cameras mounted to an XR headset and facing forward in the wearer's eye gaze direction. The sensor 230 may additionally or alternatively be other types of sensors able to track location of physical objects, such as LIDAR sensors, ultrasound sensors, etc. In some embodiments, two cameras may be used to track location of the physical electronic device 210 in three-dimensional (3D) space or a single LIDAR sensor may be used. LIDAR and one or more cameras may be used together to track and determine location, shape, texture, temperature, tactile notification, and color of the physical electronic device 210.

[0035] Alternatively or additionally, the XR headset or other device transported by a participant may contain a sensor(s) that can track pose of the XR headset or other device, such as an inertial measurement unit, and report the pose (e.g., location, head orientation, etc.) to the XR rendering device 100.

[0036] A virtual user interface 240 may represent a physical electronic device’s user interface 212. The first participant can interact with the virtual user interface 240 in the immersive XR environment by, for example, controlling a hand of an avatar or using a real hand that is tracked in 3D space of the immersive XR environment via a camera or other position tracking sensor 230.

[0037] In some embodiments, the rendering of the virtual user interface 240 includes an arrangement of graphical objects that are selectable by the first participant, e.g., by an avatar, in the immersive XR environment and / or arrangement of menu items that are selectable by the first participant in the immersive XR environment.

[0038] Graphical objects may be icons or any virtual object which the participant is able to press or interact with.

[0039] In one illustrative example, Participant 2 allows Participant 1 to use the Participant 2's real-world physical smartphone through interaction with a virtual representation as a virtual smartphone in the XR environment so that Participant 1 can generate and send a text message or email to another person using the physical smartphone, via the virtual user interface rendered for Participant 1. Participant 2 pushes the physical smartphone through the virtual portal, which triggers rendering of a virtual smartphone in the XR environment, and rendering of a virtual representation (virtual UI 310) of a user interface (UI) 240 for the physical smartphone in the XR environment. Participant 1 interacts with the virtual user interface 240, displayed within or associated with, e.g., adjacent to, the virtualsmartphone to. The XR environment interaction with the virtual UI 240 of the virtual smartphone may correspond to Participant 1 selecting a text message icon in the virtual UI 240, entering (e.g., type-in, speech input, etc.) a message destination address (e.g., Participant 3 or other person having device having messaging capability) and message content, and initiating sending of the message. The message destination address and message content are communicated from the XR rendering device 100 (e.g., XR server) to the physical smartphone and passed through an application programming interface to a messaging application to trigger creation of a message directed to the message destination address (e.g., Participant 3) and containing the message content. Participant 1 may further interact with the user interface displayed on the virtual smartphone to trigger sending of the created message by the physical smartphone to the message destination address.

[0040] In another example, a user, also called "participant," needs help from an online help center for operation of a smartphone. The user (participant) establishes a first communication channel between the first physical electronic device 210 and a first operational function of the first virtual electronic device. This establishment of the first communication channel triggers rendering of the virtual smartphone in an XR help center environment which includes a help center participant, or the virtual smartphone may already be rendered in the XR help center environment. A help center participant interacts with a virtual representation of a user interface displayed on (or associated with) the virtual smartphone. In one example, the help center participant selects a "Settings" icon displayed in the XR environment, which triggers communication of that selection to the physical smartphone and passed to the operating system (OS) to cause display of Settings sub-menu items on the physical smartphone and corresponding rendering of a representation of the Settings sub-menu items on or adjacent to the virtual smartphone in the XR environment. The help center participant can further select among the rendered hierarchy of displayed submenu items and / or set operational modes and / or values through the rendered user interface of the virtual smartphone to cause communication of corresponding selections and / or set operational modes and / or values to the physical smartphone and corresponding control operation of the physical smartphone.

[0041] Figure 3 illustrates a system where an XR rendering server 100 displays virtualized electronic devices on a head mounted display device (XR display device) 110a and enables a participant to interact with virtual user interfaces 310 and 322 of the virtualized electronic devices in accordance with some embodiments of the present disclosure. The XRrendering server 100 communicates with the head mounted display device 110a through one or more wired or wireless networks 120.

[0042] In the example of Figure 3, three physical devices are virtualized and controllably by the participant through interactions in the XR environment. The first physical electronic device 210 (physical smartphone) has a first user interface 212 which is illustrated as an arrangement of icons that are user selectable. The physical smartphone may have been earlier entering into the virtual portal of Figure 2 and virtualized into the XR environment through operation of the XR rendering server 100. The second physical electronic device 320 is illustrated as a smartwatch that includes a second user interface 322 which is illustrated as an arrangement of icons that are user selectable. The third physical electronic device 330 is illustrated as physical wireless audio headphones, which may have another UI or operationally associated with one of the other icons rendered in the XR environment. The physical smartwatch 320 and physical wireless audio headphones 330 can be outside of any virtual portal, yet can be communication connectable with and operationally controllable by the XR rendering server 100 through local communication links that can be established through between the HMD 110a and the physical smartwatch 320 and wireless audio headphones 330. The HMD 110a relays communications between the XR rendering server 100 and addressed ones of the physical smartwatch 320 and wireless audio headphones 330.

[0043] In the example of Figure 3, the XR rendering server 100 can render a virtual representation of each of these physical electronic devices in the immersive XR environment that is displayed through the HMD (XR rendering server) 110a. More particularly, the XR rendering server 100 can display a rendering of a first virtual user interface 310 that corresponds to at least a portion of the first user interface 212. The XR rendering server 100 can also display a second virtual user interface that corresponds to at least a portion of the second user interface 322 of the physical smartwatch 320. As will be described in further detail below, the HMD 100 may also stream audio to the wireless audio headphones 330.

[0044] Figure 4 illustrates a flowchart of further example operations of the XR rendering server 100, XR display device 110a (e.g., HMD), first physical electronic device 210 (e.g., smartphone), and second physical electronic device 320 (e.g., smartwatch), in accordance with some embodiments of the present disclosure.

[0045] Referring to Figure 4, the XR rendering server 100 and HMD 110a can communicate to establish an XR session for a participant through the HMD 110a. The XR rendering server 100 establishes 400 an XR session through the HMD 110a for the participant. Then the XR rendering server 100 renders 402 a virtual smartphonerepresentation of at least a portion of smartphone, including a first virtual user interface. The HMD 110a displays 404 the rendering. The XR rendering server 100 establishes 406 a first communication channel between the virtual smartphone (including the first virtual UI) and a first operational function of the smartphone 210. The HMD 110a identifies 408 a defined interaction of the participant with the first virtual UI and sends a notification to the XR rendering server 100. The XR rendering server 100 communicates 410 traffic to the smartphone 210 to control the first operational function. The smartphone 210 may operate the first operational function 412 responsive to the control input triggered by the participant interaction with the virtual smartphone. The HMD 110a identifies 414 another defined interaction of the participant with the first virtual UI and sends a notification to the XR rendering server 100. The XR rendering server 100 communicatively connects 416 the HMD 110a and the smartwatch 320, and establishes a second communication channel between the smartphone 210 and a second operational function of the virtual smartphone. The XR rendering server 100 renders 418 a virtual smartwatch representation of at least a portion of smartwatch 320, including a second virtual user interface. The HMD 110a displays 420 a rendering and notifies the XR rendering server 100 of participant interaction with the second virtual user interface. The XR rendering server 100 selectively routes 422 traffic between the smartphone 210 and smartwatch 320 via the relay through the HMD 110a. The smartwatch 320 may operate 424 through communications with the virtual smartwatch and the virtual smartphone performed via relay through the XR rendering server 100 and the HMD 110a.

[0046] In some embodiments, operations by the XR rendering server 100 further include to, based on establishing the second communication channel between the second physical electronic device 320 and the second operational function of the first virtual electronic device, render a second virtual electronic device representation in the immersive XR environment of at least a portion of the second physical electronic device 320 for display through the HMD 110a for viewing by the first participant. The rendering of the second virtual electronic device includes a second virtual user interface that the first participant views through the HMD 110a and can interact with in the immersive XR environment. The second virtual user interface is displayed as part of or separate from the first virtual user interface.

[0047] In further embodiments, the first virtual user interface is rendered to display to the first participant through the HMD 110a a first set of user selectable items that corresponds to a set of user selectable items that are displayed on a display of the first physical electronic device 210, and the defined interaction corresponds to the first participant selecting one of theuser selectable items in the first set. The second virtual user interface is rendered to display to the first participant through the HMD 110a a second set of user selectable items that corresponds to a set of user selectable items that are displayed on a display or physical control or communicated by a haptic display of the second physical electronic device 320. Responsive to the first participant selecting one of the user selectable items in the second set, operations by the XR rendering server 100 trigger communication of the traffic from the second operational function through the second communication channel and the wireless communication relay between the HMD 110a and the second physical electronic device 320 to control the operational function of the second physical electronic device 320.

[0048] In some embodiments, operations by the XR rendering server 100 further include to control what interaction the first participant is permitted to have with the first virtual user interface of the first virtual electronic device and the second virtual user interface of the second virtual electronic device based on at least one security mechanism.

[0049] For example, a first participant may interact with the virtual user interface through tracked motion of a hand, a stylus, a touch-pen, or by controlling an avatar representation of the first participant in the immersive XR environment.

[0050] The XR rendering server (XR rendering device) 100 may be controlled based on one or more access rules as to which of the displayed virtual user interface elements are selectable by the first participant or other participants in the immersive XR environment. The access rule(s) may define which identified persons are permitted to select which virtual user interface elements or otherwise access defined operational functions of the physical electronic device. For example, the first participant may have different access privileges than another participant for selecting virtual user interface element(s) and / or accessing defined operational functions of the physical electronic device. Virtual user interface elements which are permitted by the access rule(s) to be selected by the first participant may be displayed in a visually different way (e.g., color, shading, font, etc.) than other virtual user interface elements which are not permitted by the access rule(s) to be selected by the first participant. The access rules may be visualized by grayed out options, or by something indicating the allowed actions to the second participant. Upon the first participant pressing a grayed-out (not-yet approved option), the managing application may identify the associated optional operational interaction and provide the second participant with an option to either allow or deny the interaction attempted or requested.

[0051] In some further embodiments, the at least one security mechanism defines operational functions of the first physical electronic device 210 and the second physical electronic device 320 that the first participant is permitted to interact with.

[0052] In some embodiments, operations further include to determine the at least one security mechanism based on at least one of:• a characteristic of the operational function of the first physical electronic device and / or the second physical electronic device attempted to be controlled by the first participant;• an identifier of the first participant, an identifier of the first physical device and / or an identifier of the second physical device;• a proof of possession of a digital token or cryptographic key of the first participant which is used for communications with the first physical electronic device 210 and / or the second physical electronic device 320; and• whether the first physical electronic device 210 is presently being operated by person other than the first participant.

[0053] There are other scenarios where the first participant may at least initially have sufficient access privileges to the physical electronic device. In these scenarios, Participant 2 may interact with the virtual user interface to enable or prohibit access to certain operational functions of the physical electronic device for access by the first participant. For example, the first participant may interact with the virtual user interface, e.g., type in an unlock code, to unlock the user interface screen of the physical electronic device for access by the first participant.

[0054] Accordingly, in some embodiments, operations by the XR rendering server 100 further include to determine that the first participant has interacted with the second virtual user interface of the second virtual electronic device to unlock the operational function of the second physical electronic device 320. The operations also further include communicating a message to the second physical electronic device 320 indicating the second physical electronic device 320 is to unlock the operational function.

[0055] In some embodiments, the second physical electronic device 320 includes at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations of the second physical electronic device 320 include an unlock function to allow all operational functionality of the second physical electronic device 320 to be accessible to the first participant. The operations of the secondphysical electronic device 320 also include a defined set of function access privileges to restrict access by the first participant to operational functions of the second physical electronic device 320 allowed by at least one security mechanism.

[0056] In some embodiments, operations of the XR rendering device 100 further include receiving a message from the second physical electronic device 320 indicating what content is displayed on a display device or physical control or communicated by a haptic display of the second physical electronic device 320. The operations also include generating the second virtual user interface to contain content that is based on what content is indicated by the message as being displayed on the second physical electronic device 320.

[0057] In some embodiments, the first participant interacting with a virtual electronic device, associated to a physical electronic device that sends / receives data on behalf of a virtual user may result in one of the following:• mirroring updates to a user interface of the physical electronic device and the virtual electronic device;• parts of the user interface of the physical electronic device are replicated to the virtual electronic device based on defined rules; or• an icon present on physical electronic device indicating association to a virtual electronic device and updating the virtual user interface only on the virtual electronic device.

[0058] In further embodiments, the message indicates at least one of• a listing of icons for communication in the second virtual user interface;• a listing of menu items for communication in the second virtual user interface;• a screen shot of content displayed on the second physical electronic device 320 for display in the second virtual user interface;• an auto-rendered “text-to-audio” stream of a listing of icons and / or menus; and• a live screen mirroring between what is displayed on the second physical electronic device 320 and what is to be displayed in the second virtual user interface.

[0059] The listing of icons and listing of menu items may be displayed by a display device, displayed by a physical control, such as a physical button, or communicated by a haptic display through haptic feedback such as vibration.

[0060] Some embodiments are directed to the XR rendering device 100 establishing a communicative connection with the second physical electronic device to update information rendered.

[0061] In some embodiments, operations further include to discover presence of the second physical electronic device 320 responsive to a message received from the HMD 110a indicating receipt of broadcast signaling which identifies the second physical electronic device 320. The operations also further include to, based on discovering the presence of the second physical electronic device 320, update the rendering of the first virtual user interface of the first virtual electronic device to indicate presence of the second physical electronic device 320. The operations also further include to, responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, communicate a connection command to the HMD 110a to initiate wireless communication connection between the HMD 110a and the second physical electronic device 320.

[0062] In further embodiments, the connection command identifies the second physical electronic device 320 and initiates completion of a pairing procedure of the HMD 110a to the second physical electronic device 320. The operation to establish the second communication channel between the second physical electronic device 320 and the second operational function of the first virtual electronic device is further responsive to receipt of a confirmation message from the HMD 110a indicating completion of the pairing procedure of the HMD 110a to the second physical electronic device 320. The pairing procedure may be any communication protocol, such as Bluetooth pairing, Bluetooth Low Energy pairing, or WiFi pairing.

[0063] In some embodiments, operations by the XR rendering server 100 further include to, following the update of the rendering of the first virtual user interface of the first virtual electronic device to indicate presence of the second physical electronic device 320, further update the rendering of the first virtual user interface of the first virtual electronic device to remove the indication of presence of the second physical electronic device 320 responsive to at least one of the following: determining that a threshold time has expired since traffic was received from the second physical electronic device 320 through the wireless communication relay between the HMD 110a and the second physical electronic device 320; receiving a disconnect message; or determining the first physical electronic device was disconnected from the first virtual electronic device by the second participant.

[0064] Some embodiments are directed to the XR rendering device 100 automatically pairing to physical electronic devices.

[0065] In some embodiments, operations by the XR rendering server 100 further include to discover presence of a third physical electronic device 330 responsive to a message received from the HMD 110a indicating receipt of broadcast signaling which identifies thethird physical electronic device 330. The operations also further include obtaining credentials for accessing the third physical electronic device 330. The operations further include communicating a connection command containing the credentials to the HMD 110a to initiate wireless communication connection between the HMD 110a and the third physical electronic device 330 using the credentials.

[0066] In some embodiments, operations by the XR rendering server 100 further include to receive traffic from the first operational function of the first physical electronic device 210 responsive to the traffic communicated through the first communication channel from the first operational function to the first physical electronic device 210. The operations include to route the traffic through the second communication channel to the second physical electronic device 320 via the wireless communication relay between the HMD 110a and the second physical electronic device 320 to further control the operational function of the second physical electronic device 320.

[0067] In some embodiments, as a part of relaying communication, the XR rendering device may define a separate address (e.g., IP and / or MAC address) for the relayed traffic, to differ between traffic from the HMD itself and traffic from XR rendering server, e.g., originating from user interactions with the first virtual electronic device.Furthermore, if several different virtual electronic devices are present in the first participants XR environment, each of the virtual electronic devices may receive its own address from the HMD. In this manner, the second electronic device is given the impression that it is communicating with several different physical electronic devices, making it possible to interact with a second electronic device being oblivious of virtual electronic devices.

[0068] In some embodiments, a participant electronic device is connected to a network, e.g., a LAN, and may hold a register of services available in the network. The services, e.g., camera feeds, a network-connected printer, or network-connected scanner, may be made available through the UI of the virtual electronic device. Any usage request of such services is routed through the participant electronic device. For example, the user selects a document available on the first physical electronic device, using the virtual UI of the virtual device in the XR environment, and selects said document for printing. The participant interaction is received by the XR rendering server which requests the document from the first physical electronic device. The document and the printing instruction is routed to the participant’s electronic device, which sends the document to the network-connected printer.

[0069] In some embodiments, the first physical electronic device 210 includes one of a smartphone, a tablet computer, and a laptop computer. The second physical electronic device 320 includes one of a smartwatch, a wireless audio device, and a keyboard.

[0070] In some embodiments, the XR rendering device 100 comprises an XR rendering server or a component of the XR display device.

[0071] In some embodiments, the XR display device is not limited to an HMD device and can include other types of display devices.

[0072] Example XR Rendering Device Configuration.

[0073] Figure 6 is a block diagram of components of an XR rendering device 100 that are configured to operate in accordance with some embodiments of the present disclosure. The XR rendering device 100 can include processing circuitry, at least one network interface 620 (network interface), and a display device 630. The processing circuitry may include at least one processor circuit 600 (processor), at least one memory 610 (memory). The processor 600 is operationally connected to these various components. The memory 610 stores executable instructions 612 that are executed by the processor 600 to perform operations. The processor 600 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 600 is configured to execute the instructions 612 in the memory 610, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for an XR rendering device. As explained above, the XR rendering device may be separate from and communicatively connected to the XR display devices or may be at least partially integrated within one or more of the XR display devices.

[0074] Figure 7 is a block diagram of components of a physical electronic device 210 that are configured to operate in accordance with some embodiments of the present disclosure. The physical electronic device 210 can include processing circuitry, at least one network interface 720 (network interface), and a display device 730. The processing circuitry may include at least one processor circuit 700 (processor), at least one memory 710 (memory). The processor 700 is operationally connected to these various components. The memory 710 stores executable instructions 712 and an XR module 714 that are executed by the processor 700 to perform operations. The processor 700 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 700 is configured to execute the instructions 712 and XRmodule 714 in the memory 710, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for an electronic device.

[0075] Further Definitions and Embodiments:

[0076] In the above-description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0077] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0078] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0079] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions butdoes not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id est," may be used to specify a particular item from a more general recitation.

[0080] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0081] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0082] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / actsinvolved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0083] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts are to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. An extended reality, XR, rendering device (100) for rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment, the XR rendering device comprising: at least one processor; and at least one memory storing instructions executable by the at least one processor to perform operations to: render a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device (210) for display through the XR display device for viewing by the first participant, wherein the rendering of the first virtual electronic device comprises a first virtual user interface that the first participant views through the XR display device and can interact with in the immersive XR environment; establish a first communication channel between the first physical electronic device (210) and a first operational function of the first virtual electronic device; responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, communicate traffic through the first communication channel from the first operational function to the first physical electronic device (210) to control an operational function of the first physical electronic device (210); responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, establish a second communication channel between a second physical electronic device (320) and a second operational function of the first virtual electronic device; and communicate traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display device and the second physical electronic device (320) to control an operational function of the second physical electronic device (320).

2. The XR rendering device (100) of Claim 1, wherein the operations further comprise to: based on establishing the second communication channel between the second physical electronic device (320) and the second operational function of the first virtual electronic device, render a second virtual electronic device representation in the immersive XR environment of at least a portion of the second physical electronic device (320) for display through the XR display device for viewing by the first participant, wherein the rendering of the second virtual electronic device comprises a second virtual user interface that the first participant views through the XR display device and can interact with in the immersive XR environment, and wherein the second virtual user interface is displayed as part of or separate from the first virtual user interface.

3. The XR rendering device (100) of any of Claim 2 wherein: the first virtual user interface is rendered to display to the first participant through the XR display device a first set of user selectable items that corresponds to a set of user selectable items that are displayed on a display of the first physical electronic device (210), and the defined interaction corresponds to the first participant selecting one of the user selectable items in the first set; and the second virtual user interface is rendered to display to the first participant through the XR display device a second set of user selectable items that corresponds to a set of user selectable items that are displayed on a display or physical control or communicated by a haptic display of the second physical electronic device (320); and responsive to the first participant selecting one of the user selectable items in the second set, triggering communication of the traffic from the second operational function through the second communication channel and the wireless communication relay between the XR display device and the second physical electronic device (320) to control the operational function of the second physical electronic device (320).

4. The XR rendering device (100) of any of Claims 2 to 3, wherein the operations further comprise to:control what interaction the first participant is permitted to have with the first virtual user interface of the first virtual electronic device and the second virtual user interface of the second virtual electronic device based on at least one security mechanism.

5. The XR rendering device (100) of Claim 4, wherein the at least one security mechanism defines operational functions of the first physical electronic device (210) and the second physical electronic device (320) that the first participant is permitted to interact with.

6. The XR rendering device (100) of any of Claims 4 to 5, wherein the operations further comprise to determine the at least one security mechanism based on at least one of a characteristic of the operational function of the first physical electronic device and / or the second physical electronic device attempted to be controlled by the first participant; an identifier of the first participant or an identifier of the second device; a proof of possession of a digital token or cryptographic key of the first participant which is used for communications with the first physical electronic device (210) and / or the second physical electronic device (320); and whether the first physical electronic device (210) is presently being operated by person other than the first participant.

7. The XR rendering device (100) of any of Claims 2 to 6, wherein the operations further comprise to: determine that the first participant has interacted with the second virtual user interface of the second virtual electronic device to unlock the operational function of the second physical electronic device (320); and communicate a message to the second physical electronic device (320) indicating the second physical electronic device (320) is to unlock the operational function.

8. The XR rendering device (100) of any of Claim 7, wherein the operational function of the second physical electronic device (320) comprises one of an unlock function to allow all operational functionality of the second physical electronic device (320) to be accessible to the first participant; anda defined set of function access privileges to restrict access by the first participant to operational functions of the second physical electronic device (320) allowed by at least one security mechanism.

9. The XR rendering device (100) of any of Claims 2 to 8, wherein the operations further comprise to: receive a message from the second physical electronic device (320) indicating what content is displayed on a display device or physical control or communicated by a haptic display of the second physical electronic device (320); and generate the second virtual user interface to contain content that is based on what content is indicated by the message as being displayed on the second physical electronic device (320).

10. The XR rendering device (100) of Claim 9, wherein the message indicates at least one of a listing of icons for communication in the second virtual user interface; a listing of menu items for communication in the second virtual user interface; a screen shot of content displayed on the second physical electronic device (320) for display in the second virtual user interface; and a live screen mirroring between what is displayed on the second physical electronic device (320) and what is to be displayed in the second virtual user interface.

11. The XR rendering device (100) of any of Claims 1 to 10, wherein the operations further comprise to: discover presence of the second physical electronic device (320) responsive to a message received from the XR display device indicating receipt of broadcast signaling which identifies the second physical electronic device (320); based on discovering the presence of the second physical electronic device (320), update the rendering of the first virtual user interface of the first virtual electronic device to indicate presence of the second physical electronic device (320); and responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, communicate a connection command to the XR display device to initiate wireless communicationconnection between the XR display device and the second physical electronic device (320).

12. The XR rendering device (100) of Claim 11 wherein: the connection command identifies the second physical electronic device (320) and initiates completion of a pairing procedure of the XR display device to the second physical electronic device (320); and the operation to establish the second communication channel between the second physical electronic device (320) and the second operational function of the first virtual electronic device is further responsive to receipt of a confirmation message from the XR display device indicating completion of the pairing procedure of the XR display device to the second physical electronic device (320).

13. The XR rendering device (100) of any of Claims 11 to 12, wherein the operations further comprise to: following the update of the rendering of the first virtual user interface of the first virtual electronic device to indicate presence of the second physical electronic device (320), further update the rendering of the first virtual user interface of the first virtual electronic device to remove the indication of presence of the second physical electronic device (320) responsive to at least one of the following: determining that a threshold time has expired since traffic was received from the second physical electronic device (320) through the wireless communication relay between the XR display device and the second physical electronic device (320); receiving a disconnect message; or determining the first physical electronic device was disconnected from the first virtual electronic device by the second participant.

14. The XR rendering device (100) of any of Claims 1 to 13, wherein the operations further comprise to: discover presence of a third physical electronic device (330) responsive to a message received from the XR display device indicating receipt of broadcast signaling which identifies the third physical electronic device (330);obtaining credentials for accessing the third physical electronic device (330); and communicate a connection command containing the credentials to the XR display device to initiate wireless communication connection between the XR display device and the third physical electronic device (330) using the credentials.

15. The XR rendering device (100) of any of Claims 1 to 14, wherein the operations further comprise to: receive traffic from the first operational function of the first physical electronic device (210) responsive to the traffic communicated through the first communication channel from the first operational function to the first physical electronic device (210); and route the traffic through the second communication channel to the second physical electronic device (320) via the wireless communication relay between the XR display device and the second physical electronic device (320) to further control the operational function of the second physical electronic device (320).

16. The XR rendering device (100) of any of Claims 1 to 15 wherein: the first physical electronic device (210) comprises one of a smartphone, a tablet computer, and a laptop computer; and the second physical electronic device (320) comprises one of a smartwatch, a wireless audio device, and a keyboard.

17. The XR rendering device (100) of any of Claims 1 to 16 wherein: the XR rendering device (100) comprises an XR rendering server or a component of the XR display device.

18. The XR rendering device (100) of any of Claims 1 to 17 wherein: the XR display device comprises a head mounted display device.

19. A method performed by an extended reality, XR, rendering device for rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment, the method comprising: rendering (500) a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device for displaythrough the XR display device for viewing by the first participant, wherein the rendering of the first virtual electronic device comprises a first virtual user interface that the first participant views through the XR display device and can interact with in the immersive XR environment; establishing (502) a first communication channel between the first physical electronic device and a first operational function of the virtual electronic device; responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, communicating (504) traffic through the first communication channel from the first operational function to the first physical electronic device to control an operational function of the first physical electronic device; responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, establishing (506) a second communication channel between a second physical electronic device and a second operational function of the first virtual electronic device; and communicating (508) traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display device and the second physical electronic device to control an operational function of the second physical electronic device.

20. A computer program product comprising a non-transitory computer readable medium storing instructions executable by at least one processor of an extended reality, XR, rendering device rendering an immersive XR environment on an XR display device for viewing by a first participant in the immersive XR environment, the instructions executed by the at least one processor perform operations comprising to: render a first virtual electronic device representation in the immersive XR environment of at least a portion of a first physical electronic device for display through the XR display device for viewing by the first participant, wherein the rendering of the first virtual electronic device comprises a first virtual user interface that the first participant views through the XR display device and can interact with in the immersive XR environment; establish a first communication channel between the first physical electronic device and a first operational function of the virtual electronic device;responsive to a defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, communicate traffic through the first communication channel from the first operational function to the first physical electronic device to control an operational function of the first physical electronic device; responsive to another defined interaction of the first participant with the first virtual user interface of the first virtual electronic device, establish a second communication channel between a second physical electronic device and a second operational function of the first virtual electronic device; and communicate traffic from the second operational function through the second communication channel and a wireless communication relay between the XR display device and the second physical electronic device to control an operational function of the second physical electronic device.

Images