Mapping a virtual electronic device in a virtual portal in an extended reality environment to a physical backend

Abstract

An XR rendering device enabling rendering an immersive XR environment on display devices for viewing by participants in the immersive XR environment. The XR rendering device performs operations to define a virtual portal at a location in physical space of a first participant. Operations determine a set of virtual electronic device characteristics of a virtual electronic device, and render a virtual electronic device representation in the immersive XR environment of the virtual electronic device based on the set of virtual electronic device characteristics. Operations identify a characteristic of an interaction of the first participant or a second participant with a virtual user interface of the virtual electronic device, and communicate with an XR managing server to control an operational function of the virtual electronic device based on the identified characteristic of the interaction.

Description

MAPPING A VIRTUAL ELECTRONIC DEVICE IN A VIRTUAL PORTAL IN AN EXTENDED REALITY ENVIRONMENT TO A PHYSICAL BACKENDTECHNICAL FIELD

[0001] The present disclosure relates to rendering extended reality (XR) environments and associated XR rendering devices, and more particularly to rendering objects in immersive XR environments for display on XR participant devices.BACKGROUND

[0002] Immersive extended reality (XR) environments have been developed which provide a myriad of different types of user experiences for gaming, on-line meetings, cocreation of products, etc. One type of immersive XR environments (also referred to as "XR environments") is virtual reality (VR) environments where human users only see computergenerated graphical renderings. Another type of XR environment is augmented reality (AR) environments where users see a combination of computer-generated graphical renderings overlaid on a view of the physical real-world through, e.g., see-through display screens.

[0003] Example XR environment rendering devices include, without limitation, XR environment servers, XR headsets, gaming consoles, smartphones running an XR application, and tablet / laptop / desktop computers running an XR application. The Oculus Quest and the Apple Vision Pro are examples of VR device with AR functionality and the Google Glass is an example AR device.

[0004] Users can interact with virtual objects that are rendered in some immersive XR environments, such as by controlling a user's avatar to manipulate virtual objects. Immersive XR environments presently support little if any interface between physical objects and virtual objects that are rendered.SUMMARY

[0005] Some embodiments disclosed herein are directed to an XR rendering device for rendering an immersive XR environment on display devices for viewing by participants in the immersive XR environment. The XR rendering device includes at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations include to define a virtual portal at a location in physical space of a first participant. The operations also include to determine a set of virtual electronicdevice characteristics of a virtual electronic device. The operations also include to render a virtual electronic device representation in the immersive XR environment of the virtual electronic device based on the set of virtual electronic device characteristics. The virtual electronic device comprises a virtual user interface that the first participant or a second participant can interact with in the immersive XR environment. The operations also include to identify a characteristic of an interaction of the first participant or second participant with a virtual user interface of the virtual electronic device. The operations also include to communicate with an XR managing server to control an operational function of the virtual electronic device based on the identified characteristic of the interaction.

[0006] Some other embodiments are directed to an XR managing server for managing an immersive XR environment on display devices for viewing by participants in the immersive XR environment. The XR managing server includes at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations include to determine a set of virtual electronic device characteristics of a virtual electronic device from a virtual electronic database. The operations also include to transmit the set of virtual electronic device characteristics of a virtual electronic device to an XR rendering device. The operations also include to receive a characteristic of an interaction of a first participant or a second participant with a virtual user interface of the virtual electronic device from the XR rendering device. The operations also include to control an operational function of the virtual electronic device based on the identified characteristic of the interaction.

[0007] Some other embodiments are directed to a corresponding method performed by an XR rendering device for rendering an immersive XR environment on display devices for viewing by participants in the immersive XR environment. The method includes defining a virtual portal at a location in physical space of a first participant. The method also includes determining a set of virtual electronic device characteristics of a virtual electronic device. The method also includes rendering a virtual electronic device representation in the immersive XR environment of the virtual electronic device based on the set of virtual electronic device characteristics. The virtual electronic device comprises a virtual user interface that the first participant or a second participant can interact with in the immersive XR environment. The method also includes identifying a characteristic of an interaction of the first participant or second participant with a virtual user interface of the virtual electronic device. The method also includes communicating with an XR managing server to control an operational function of the virtual electronic device based on the identified characteristic of the interaction.

[0008] Some other embodiments are directed to a corresponding method performed by an XR managing server for managing an immersive XR environment on display devices for viewing by participants in the immersive XR environment. The method includes determining a set of virtual electronic device characteristics of a virtual electronic device from a virtual electronic database. The method also includes transmitting the set of virtual electronic device characteristics of a virtual electronic device to an XR rendering device. The method also includes receiving a characteristic of an interaction of a first participant or a second participant with a virtual user interface of the virtual electronic device from the XR rendering device. The method also includes controlling an operational function of the virtual electronic device based on the identified characteristic of the interaction.

[0009] Potential advantages that may be provided by one or more of these embodiments can include that participants in immersive XR environments are able to view and interact with virtual representation renderings of the operational functionality provided by a single physical electronic device or a combination of two or more physical electronic devices characterized in a database. Participants can thereby interact with a virtual representation of one or more physical electronic devices, and can be provided a combined user interface that maps to a combination of operational functionality residing in two or more physical electronic devices. Moreover, as will be explained in further detail below, the combined user interface and mapped combination of operational functions can include new operational functions which do not reside in any of physical electronic devices individually, but which instead is created by combining individual functions residing on two or more different electronic devices into the new operational function. Virtualizing physical electronic devices in an XR environment can provide a more easily displayable rendering and provide participants with more effective and intuitive ways of interacting with physical electronic devices.

[0010] Additional potential advantages that may be provided by one or more of these embodiments may include enabling participants in virtual meetings to on-demand obtain access to additional functions and capabilities existing in physical electronic device(s) that are otherwise not available to those participants.

[0011] Other XR rendering devices, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional XR rendering devices, methods, and computer program products be included within this description and protected by the accompanying claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0013] Figure 1 illustrates an XR system that includes a physical electronic device and a plurality of participant devices that communicate through networks with an XR rendering device to operate in accordance with some embodiments of the present disclosure;

[0014] Figure 2 illustrates an example operation of the XR rendering device with a virtual user interface being projected from the virtual electronic device in the virtual space of the immersive XR environment, in accordance with some embodiments of the present disclosure;

[0015] Figure 3 illustrates a flowchart of operations that can be performed by an XR rendering device and XR managing server in accordance with some embodiments of the present disclosure;

[0016] Figure 4 illustrates a flowchart of operations that can be performed by an XR rendering device and XR managing server in accordance with some embodiments of the present disclosure;

[0017] Figure 5 illustrate flowcharts of operations that can be performed by an XR rendering device to perform operations for determining a set of virtual electronic device characteristics of a virtual electronic device in accordance with some embodiments of the present disclosure;

[0018] Figure 6 illustrate flowcharts of operations that can be performed by an XR managing server to perform operations in accordance with some embodiments of the present disclosure;

[0019] Figure 7 illustrates a block diagram of components of an XR rendering device that are configured to operate in accordance with some embodiments of the present disclosure;

[0020] Figure 8 illustrates a block diagram of components of an electronic device that are configured to operate in accordance with some embodiments of the present disclosure; and

[0021] Figure 9 illustrates a block diagram of components of an XR managing server that are configured to operate in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0022] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0023] Figure 1 illustrates an XR system that includes a physical electronic device 210 and a plurality of participant devices 110a and 110b that communicate through networks 120 with an XR rendering device 100 to operate in accordance with some embodiments of the present disclosure. The XR rendering device 100 is configured to generate a graphical representation of an immersive XR environment (also called an "XR environment" for brevity) with a computer-generated virtual object representation module 102, and which is viewable from various perspectives of virtual poses of human participants in the XR environment through display screens of the various participant devices 1 lOa-b and / or a physical electronic device 210.

[0024] For example, the illustrated devices of Figure 1 include XR headsets 110a and 110b which can be worn by participants to view and navigate through the XR environment. The participant devices are not limited to XR headsets, as they could be a personal computer, tablet computers, smartphone, or other electronic device which can be operated by a participant to view and navigate through the XR environment. The participants may have associated avatars which are rendered in the XR environment to represent poses (e.g., location, body assembly orientation, etc.) of the participants relative to a coordinate system of the XR environment.

[0025] A physical electronic device 210, also called "physical object" and “electronic device”, has electronic operation(s) which can be controlled through a user interface. The physical electronic device 210 can be any type of electronic device that has a user interface. Non-limiting examples of physical electronic devices include smartphones, tablet computers, laptop computers, smart watches, gaming devices, smart appliances, etc. Non-limiting examples of user interfaces include touch screens and other display devices that can display user selectable indicia and / or user selectable menus, and physical buttons, touch sensor interfaces, etc.

[0026] As will be explained in further detail below, a representation of the physical electronic device 210 is rendered in the XR environment as a virtual electronic device with a virtual representation of a user interface of the physical electronic device 210. A participant in the XR environment can interact with the virtual user interface in order to control various electronic operational function(s) of the physical electronic device 210.

[0027] Although the XR rendering device 100 is illustrated in Figure 1 as being a centralized network computing server separate from one or more of the participant devices, in some other embodiments the XR rendering device 100 is implemented as a component of one or more of the participant devices. For example, one of the participant devices may be configured to perform operations of the XR rendering device in a centralized manner controlling rendering for or by other ones of the participant devices. Alternatively, each of the participant devices may be configured to perform at least some of the operations of the XR rendering device in a distributed decentralized manner with coordinated communications being performed between the distributed XR rendering devices (e.g., between software instances of XR rendering devices).

[0028] Additionally, an XR managing server 130 may communicate with the XR rendering device 100 through networks 120. The XR managing server 130 may be tasked with managing a virtual electronic database 104 containing the virtual electronic devices, the characteristics of the virtual electronic devices, and the current states of virtual electronic devices being interacted with by participants using the XR rendering devices 100.

[0029] Figure 2 illustrates an example operation of the XR rendering device with a virtual user interface being projected from the virtual electronic device in the virtual space of the immersive XR environment, in accordance with some embodiments of the present disclosure.

[0030] As illustrated in Figure 2, the XR rendering device 100 may track location of a physical electronic device 210 in the first participant’s physical space while the physical electronic device 210 is being passed through a virtual portal 200 and rendered in the virtual space of the immersive XR environment, in accordance with some embodiments of the present disclosure. Referring to Figure 2, Participant 1 and Participant 2 are present participants in an immersive XR environment rendered by the XR rendering device 100. Participant 1 and Participant 2 can simultaneously view virtual electronic device 220s that are rendered by the XR rendering device 100 in their respective virtual spaces, such as through XR headsets worn by the participants. The XR rendering device 100 is communicatively coupled to at least one sensor 230 that tracks location of a physical electronic device 210, e.g., a smart phone, relative to the boundary of a virtual portal 200 having a location definedin physical space. The sensor 230 may be one or more cameras mounted to an XR headset and facing forward in the wearer's eye gaze direction. The sensor 230 may additionally or alternatively be other types of sensors able to track location of physical electronic device 210, such as LIDAR sensors, ultrasound sensors, etc. In some embodiments, two cameras may be used to track location of the physical electronic device 210 in three-dimensional (3D) space or a single LIDAR sensor may be used. LIDAR and one or more cameras may be used together to track and determine location, shape, texture, temperature, tactile notification, and color of the physical electronic device 210. Alternatively or additionally, the XR headset or other device transported by a participant may contain a sensor(s) that can track pose, such as an inertial measurement unit, and report the pose (e.g., location, head orientation, etc.) to the XR rendering device 100.

[0031] In Figure 2, Participant 2 can interact with the virtual user interface 222 in the immersive XR environment by, for example, controlling a hand of an avatar or using a real hand or other object that is tracked in 3D space of the immersive XR environment via a camera or other position tracking sensor. However, Participant 1 may also interact with the virtual user interface 222 in the immersive XR environment and is not just limited to interaction by Participant 2.

[0032] The term virtual space is used herein to refer to the displayable space within the computer rendered immersive XR environment. In contrast, physical space is used to refer to the real space in which the human participant resides.

[0033] In the example illustration of Figure 2, the virtual electronic device 220 has predefined physical counterpart in the physical electronic device 210. However, in some embodiments, the virtual electronic device 220 does not have a pre-defined physical counterpart in the physical electronic device 210.

[0034] Some XR environment solutions have discussed mapping a virtual electronic device with a physical electronic device having digital state and creating a digital representation of the physical electronic device, or an XR environment solutions which assumes a virtual portal where a physical electronic device is first entered into the virtual portal and then rendered for the receiving participant as virtual electronic device. There have not been XR environment solutions for creating a virtual electronic device and attaching a physical backend with matching capabilities.

[0035] The various embodiments below provide operations that enable participants in an XR communication session to interact with a digital representation of a specific physicalelectronic device (e.g., as a hologram) associated to a virtual electronic device in an XR managing server.

[0036] Some embodiments are related to a virtual portal solution where a first participant having a first virtual portal creates a virtual electronic device with a set of capabilities, A, and a digital representation of the virtual electronic device holding internal digital states (and associated operations and operational capabilities), associated with capability set A, in the virtual portal.

[0037] In some of these embodiments, the virtual electronic device connected to the device-as-a-service device is a part of a virtual portal. In this embodiment, first participant / device may request capabilities / resources in a virtual portal session context, but where other devices in the session cannot provide requested capabilities and managing server may request the resources from external servers and associated devices.

[0038] The embodiment further comprising requesting access to at least one physical electronic device in a second virtual portal, comprising a set of capabilities B where A c B. When receiving response from a second virtual portal, connecting the digital representation to the physical electronic device, the digital representation now being populated with the states of the capabilities B corresponding to the requested capability set A. Finally, the embodiment establishes a session between the first and second virtual portal.

[0039] In some embodiments, prioritization is performed among several virtual portals with adequate physical electronic devices.

[0040] In some embodiments, previously present electronic devices are requesting to be inserted into the second virtual portal.

[0041] Potential advantages that may be provided by one or more of these embodiments can include allowing creation of a virtual electronic device, with a defined set of capabilities, but without an existing defined mapping to a physical electronic device. This enables seamless creation of virtual electronic devices where a participant can temporarily utilize and manipulate physical electronic devices, having at least the requested capabilities, through the virtual portal. By allowing the participant’s virtual portal to search for a second virtual portal with a physical electronic device which has the capabilities matching the virtual electronic device, the participant can choose from a database of virtual electronic devices without having a previous connection to a second virtual portal. The participant does not have to be aware of the identity or geographical placement of the physical electronic device.

[0042] Figure 5 illustrate flowcharts of operations that can be performed by an XR rendering device to perform operations in accordance with some embodiments of the presentdisclosure such as determining a set of virtual electronic device characteristics of a virtual electronic device.

[0043] Some embodiments of the present disclosure are directed to an XR rendering device 100 enabling rendering an immersive XR environment on display devices for viewing by participants in the immersive XR environment. The XR rendering device 100 performs operations to define 500 a virtual portal at a location in physical space of a first participant. The operations also include to determine 502 a set of virtual electronic device characteristics of a virtual electronic device. The operations also include to render 504 a virtual electronic device representation in the immersive XR environment of the virtual electronic device 220 based on the set of virtual electronic device characteristics. The virtual electronic device comprises a virtual user interface that the first participant or a second participant can interact with in the immersive XR environment. The operations also include to identify 506 a characteristic of an interaction of the first participant or second participant with a virtual user interface 222 of the virtual electronic device. The operations also include to communicate 508 with an XR managing server 130 to control an operational function of the virtual electronic device 220 based on the identified characteristic of the interaction.

[0044] In some embodiments, the set of virtual electronic device characteristics include at least one of: a defined virtual shape (e.g., form factor); a defined software requirement of the virtual electronic device; a defined hardware requirement of the virtual electronic device; a defined virtual user interface, UI, of the virtual electronic device; a type of device represented by the virtual electronic device; a manufacturer of a physical electronic device represented by the virtual electronic device; a geographic location of the XR rendering device; a geographic location of a rendered virtual electronic device; and a security mechanism of the virtual electronic device.

[0045] In some embodiments, the first participant or second participant could also interact with the user interface of the physical electronic device. This may then control the virtual electronic device in the same way.

[0046] In other words, in some embodiments, the operations by the XR rendering device further include to receive an input from the physical electronic device indicating an interaction of the first participant or second participant with a user interface of a physical electronic device 210. The operations by the XR rendering device also further include to communicate with the XR managing server 130 to control the operational function of the virtual electronic device 220 based on the input.

[0047] In some embodiments, the XR rendering device 100 does not have the virtual electronic device characteristics of the virtual electronic device, so the XR rendering device 100 must obtain the virtual electronic device characteristics from the XR managing server 130.

[0048] In other words, in these embodiments, the operation to determine 502 the set of virtual electronic device characteristics of the virtual electronic device includes requesting the set of virtual electronic device characteristics from a virtual electronic device database 104 from the XR managing server 130.

[0049] Some embodiments are directed to determining a physical electronic device to connect to virtual electronic devices if the physical electronic device is compatible.

[0050] In these embodiments, the XR rendering device 100 performs further operations to retrieve a set of physical electronic device characteristics of at least one physical electronic device 210. Further operations also include to determine if the set of virtual electronic device characteristics and the set of physical electronic device characteristics satisfy a device compatibility rule. The further operations also include to, responsive to the device compatibility rule being satisfied, identifying a characteristic of the interaction of the first participant or second participant with the virtual user interface 222 of the virtual electronic device 220. The further operations also include to, responsive to the device compatibility rule being satisfied, communicate with the physical electronic device 210 to control an operational function of the physical electronic device 210 based on the identified characteristic of the interaction.

[0051] In some embodiments, the set of physical electronic device characteristics includes at least one of: a defined physical shape (e.g., form factor) of the least one physical electronic device; a defined software requirement of the least one physical electronic device; a defined hardware requirement of the least one physical electronic device; a defined virtual user interface, UI, of the least one physical electronic device; a type of device to be represented by the least one physical electronic device; a manufacturer of the least one physical electronic device; at least one network connectivity property of the at least one physical electronic device; a geographic location of the least one physical electronic device; and a security mechanism of the least one physical electronic device.

[0052] In some embodiments, the previously discussed operation to retrieve the set of physical electronic device characteristics of the at least one physical electronic device 210 further includes to, selecting physical electronic device characteristics based on informationreceived from the first participant and / or second participant, and / or received from the physical electronic device.

[0053] In some embodiments, the XR rendering device 100 does not have the physical electronic device characteristics of the physical electronic device, so the XR rendering device 100 must obtain the physical electronic device characteristics from the XR managing server 130.

[0054] In these embodiments, the operation to retrieve the set of physical electronic device characteristics of the at least one physical electronic device 210 includes to send a message to the XR managing server 130 requesting a list of physical electronic device characteristics for an available at least one physical electronic device and receive the list of physical electronic device characteristics for the available at least one physical electronic device from the XR managing server 130.

[0055] The device compatibility rule is used to determine if the virtual electronic device will be able to be implemented on the physical electronic device.

[0056] In some embodiments, a physical electronic device is not in the virtual electronic device database and must be scanned and characterized to determine if it is suitable for implementing the virtual electronic device before being interacted with through a virtual portal. In these embodiments, operations further include tracking a location of a physical electronic device 210 relative to the location of an entrance boundary of the virtual portal. Responsive to determining the physical electronic device 210 has entered the virtual portal, the XR rendering device 100 scans the physical electronic device 210 to determine the set of physical electronic device characteristics of the physical electronic device 210. The XR rendering device 100 also determines if the set of virtual electronic device characteristics and the set of physical electronic device characteristics satisfy a device compatibility rule. The device compatibility rule is used to determine if the virtual electronic device will be able to be implemented on the physical electronic device. Responsive to the device compatibility rule being satisfied, the XR rendering device identifies a characteristic of the interaction of the first participant or second participant with the virtual user interface 222 of the virtual electronic device 220. The XR rendering device also communicates with the physical electronic device 210 to control an operational function of the physical electronic device 210 based on the identified characteristic of the interaction. Alternatively, if the device compatibility rule is not satisfied, then a new virtual electronic device satisfying the device compatibility rule may be generated.

[0057] Other embodiments of the present disclosure focus on the XR managing server 130.

[0058] Figure 6 illustrates flowcharts of operations that can be performed by an XR managing server to perform operations in accordance with some embodiments of the present disclosure.

[0059] Some embodiments are directed to an XR managing server 130 managing an immersive XR environment on display devices for viewing by first and second participants in the immersive XR environment. The XR managing server 130 performs operations including to determine 600 a set of virtual electronic device characteristics of a virtual electronic device from a virtual electronic database 104. The operations also include to transmit 602 the set of virtual electronic device characteristics of a virtual electronic device to an XR rendering device. The operations also include to receive 604 a characteristic of an interaction of the first participant or second participant with a virtual user interface 222 of the virtual electronic device from the XR rendering device. The operations also include to control 606 an operational function of the virtual electronic device 220 based on the identified characteristic of the interaction.

[0060] In some embodiments, it is possible that the XR managing server’s created characteristics could be used in addition to the characteristics of a physical electronic device. For example, the physical electronic device is used for most capabilities related to the characteristic, but a missing characteristic could be created and managed by the XR managing server.

[0061] Next, an example illustration of operations of the XR rendering device and the XR managing server is discussed. Figure 3 illustrates a flowchart of operations that can be performed by an XR rendering device and XR managing server in accordance with some embodiments of the present disclosure.

[0062] In step 300, the first participant, using an XR rendering device, initiates a first virtual portal volume (VPV). The term “virtual portal volume” is also referred to as a virtual portal space or virtual portal since the virtual portal may correspond to any dimensional space.

[0063] In step 302, the first participant populates the first virtual portal with a virtual electronic device(s) (also called virtual object(s)) having a specified form factor (shape).

[0064] In step 304, the first participant associates the virtual electronic device(s) with an individual set of capabilities A (e.g., Ai, A2, ... An). The first participant can select a set of virtual electronic device capabilities, set A, for the virtual electronic device from among setsof defined capabilities provided by a database managed by a managing server or other computing platform.

[0065] A capability may define both software and hardware requirements for the physical device. Example capabilities include for example applications (operational functionality of the applications), sensor resources, hardware components and processing resources.

[0066] Optionally, in step 306, the first device indicates additional requirements for the physical electronic device. An additional requirement may include certain type of electronic device, certain manufacturer, geographic location of physical electronic device, trust relationship with owner of physical electronic device, etc.

[0067] In step 308, the device of the first participant requests the managing server to set up a virtual electronic device able to represent a physical electronic device with the selected capability set A.

[0068] In step 308, the managing server sets up virtual electronic device with capability set A.

[0069] In step 310, the managing server sets up the virtual electronic device (also referred to as a digital twin) in the virtual portal by selecting at least one virtual portal which is active and fulfills the additional requirements, for example, in terms of geographical location and / or trust relationship with the owner of the virtual portal.

[0070] In step 312, the managing server sends request to the selected virtual portal(s), the request containing the set of capabilities, set A. In one example, three virtual portals are selected and belong to a second, third and fourth participant respectively.

[0071] In step 314, the second, third, fourth participant’s XR device responds to the managing server with an indication of if a physical electronic device which has a set of capabilities B which contains at least the set of capabilities A.

[0072] If no such electronic device is available, the managing server may send a request to additional virtual portal(s) or may request the participant to alter the capabilities and / or additional requirements.

[0073] In some embodiments discussed later, the method returns to step 304.

[0074] In this example, the second participant’s XR device responds with an identification and description of the physical electronic device while the third and fourth participants XR device informs that there is no suitable physical electronic device available.

[0075] Managing server sends a request to set up a virtual meeting session with the second participant’s virtual portal.

[0076] In step 316, a physical electronic device corresponding to the virtual electronic device is found, so the managing server populates the state of the virtual electronic device with the state of the physical electronic device. E.g., if the virtual electronic device is a smartphone, the virtual electronic device is updated to represent the UI of the physical smartphone present in the second participant’s virtual portal.

[0077] As another example, if one of the requested capabilities is a camera, the camera application and gallery are set up in the virtual electronic device.

[0078] Access controls and synchronization between the virtual electronic device and the physical electronic device is also set up in this step.

[0079] In step 318 and 320, the virtual electronic device in the first participant’s virtual portal is now mapped to the physical electronic device in the second participant’s virtual portal and the participant may interact with the virtual electronic device.

[0080] If a physical electronic device meeting the device compatibility rule are not found, then, in step 322, the participant may be asked to reduce the required virtual electronic device characteristics and / or additional requirements.

[0081] Optionally, if a physical electronic device meeting the device compatibility rule are not found, then, in step 324, the XR rendering device or XR managing server may request a second or other previously used physical electronic device to be placed into the virtual portal to be used.

[0082] Further embodiments of the steps described above are now discussed herein.

[0083] In some embodiments, the method returns to step 304 from step 314 and requests a previously present electronic device in the virtual portal. In these embodiments, step 314 fails to produce a physical electronic device with an adequate capability set and / or additional requirements, the managing server may send a request to a participant which has previously inserted an electronic device in the virtual portal which fulfills the first participant’s requirements. In some of these embodiments, the XR managing server may issue a request for the missing functionality outside the set of devices currently in the session.

[0084] In some embodiments, if multiple participants respond to the XR managing server in step 314, the XR managing server applies a policy to prioritize between multiple available physical electronic devices which can be used for the virtual electronic device. Prioritization may be based on trust for participant, device type, processing power of physical device, remaining battery life for physical electronic device, etc.

[0085] In similar embodiment, if multiple participants in a virtual portal session request to XR managing server on a set of capabilities (at the same time, in timespan T, etc.), the XRmanaging server may apply a policy to prioritize between multiple requests of a “same” capability (i.e., having potentially same associated device and / or hardware).

[0086] In some embodiments, if multiple participant’s respond to the XR managing server in step 314, the XR managing server applies a policy to prioritize between multiple available physical electronic devices which can be used for the virtual electronic device. Prioritization may be based on combining different capabilities from multiple participants by comparing capability A, B, C etc with device 1-n and select e.g., Devicel-A, Device2-B, etc.

[0087] In some embodiments, the second participant’s virtual portal contains multiple physical electronic devices, for example device X which has the capabilities desired by the first device and a different device Y. In such embodiments, the XR managing server may request to split the second participant’s virtual portal into two different virtual portal’s, one containing device X and one containing device Y. By doing so, device X becomes a part of the session between the first and second participant while device Y is not exposed to the first participant. When the session is finished, the split virtual portal may be re-merged into a single virtual portal again.

[0088] Other embodiments operations for a participant to on-demand request for a virtual object or virtual device, and its associated capabilities. Some embodiments are directed to provide a virtual-device-as-a-service based on a participant request and to combine cloud resources and available participant XR rendering device to provide the service.

[0089] In these embodiments, an XR rendering device capable of rendering a digital representation of a function / capability with associated input / output (I / O) for participant control of the function / capability. The XR rendering device forwards a participant request for an additional function / capability to an XR managing server and receives a response, which contains data used for rendering.

[0090] Some potential advantages of these embodiments include enabling participants in virtual meetings to on demand get access to additional functions and capabilities otherwise not available.

[0091] In one embodiment, the virtual electronic device is a part of another type of XR communication session having access to a device-as-a-service offering, and the embodiment may be considered as one single participant active in general non-virtual portal XR communication session.

[0092] Various embodiments provide a solution that enables a first participant, using XR rendering device, to request for access to a particular function / capability, e.g., a virtual electronic device. A virtual-device-as-a-service provides a virtual electronic device andvirtual user interface, or graphical user interface (GUI), with I / O capabilities visible via XR rendering device associated to certain cloud resources. The cloud resources, based on the participant request, like hardware components, relevant operating system and applications are configured to respond to the participant I / O actions via the virtual electronic device virtual user interface. Additionally, the participant may request the service to include associations to additional local components like devices, sensors, etc. For example, adding an association to a local camera which adds a camera capability to the virtual-device-as-a-service offering.

[0093] In other words, when the XR managing server virtualizes a physical electronic device as a virtual electronic device, the XR managing server can store the virtual electronic device along with the operational capabilities of the virtual electronic device. The XR server can maintain a database of the virtual electronic devices and their operational capabilities, which can be made available as a service to users. For example, operational capabilities that can be characterized in the stored information and made available for associated virtual electronic devices can include: processing capabilities (number of processing units, manufacturer and model, available accelerators), camera and characteristics, e.g., number of cameras, resolution, video stream format (e.g., MP4, etc.), still picture format, microphone and characteristics, e.g., number of microphones, audio stream format (e.g., raw, MP3, etc.); sensor(s) and characteristics, e.g., temperature, accelerometers, satellite location, etc.; rotational and / or linear motors (number, controllable actions, e.g., axial rotation about identified axes, linear motion along identified directional axes, etc.)

[0094] In some embodiments, the operations performed by the XR managing server 130 further include to receive a request from the XR rendering device 100 for creating additional functions to a server not in the virtual electronic database. The operations also further include, responsive to determining the set of virtual electronic device characteristics of the virtual electronic device not in the virtual electronic database 104 based on the requested additional functions, to store the set of virtual electronic device characteristics of the virtual electronic device in the virtual electronic database 104. The operations also further include to send the set of virtual electronic device characteristics of the virtual electronic device not in the virtual electronic database 104 based on the requested additional functions to the XR rendering device 100.

[0095] Next, an example illustration of operations of the XR rendering device and the XR managing server is discussed. Figure 4 illustrates a flowchart of operations that can be performed by an XR rendering device and XR managing server in accordance with some embodiments of the present disclosure.

[0096] In step 400, a participant, using an XR rendering device, sends a request for creating additional function(s) or capabilities to a virtual electronic device not in the virtual electronic database 104.

[0097] In step 402, the server receives the user request. A mechanism operative in the XR managing server, assess its available resources related to the received participant requests.

[0098] In step 404, the XR managing server configures the necessary (hardware and / or software) set of virtual electronic device characteristics and associated interfaces required to comply with the participant request.

[0099] In step 406, the participant receives a response containing rendering data, interface data, etc., or a delta notification.

[0100] In step 408, the participant XR rendering device renders the added functions and / or capabilities.

[0101] In step 410, the participant interaction with the digital representation of the virtual electronic device with the additional functions / capabilities are processed and responded to via the XR managing server.

[0102] Example XR Rendering Device Configuration.

[0103] Figure 7 is a block diagram of components of an XR rendering device 100 that are configured to operate in accordance with some embodiments of the present disclosure. The XR rendering device 100 can include processing circuitry, at least one network interface 720 (network interface), and a display device 730. The processing circuitry may include at least one processor circuit 700 (processor), at least one memory 710 (memory). The processor 700 is operationally connected to these various components. The memory 710 stores executable instructions 712 that are executed by the processor 700 to perform operations. The processor 700 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 700 is configured to execute the instructions 712 in the memory 710, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for an XR rendering device. As explained herein, the XR rendering device may be separate from and communicatively connected to the participant devices or may be at least partially integrated within one or more of the participant devices.

[0104] Figure 8 is a block diagram of components of an electronic device 210 that are configured to operate in accordance with some embodiments of the present disclosure. The electronic device 210 can include processing circuitry, at least one network interface 820(network interface), and a display device 830. The processing circuitry may include at least one processor circuit 800 (processor), at least one memory 810 (memory). The processor 800 is operationally connected to these various components. The memory 810 stores executable instructions 812 and an XR module 814 that are executed by the processor 800 to perform operations. The processor 800 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 800 is configured to execute the instructions 812 and XR module 814 in the memory 810, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for an electronic device.

[0105] Figure 9 is a block diagram of components of an XR managing server 130 that are configured to operate in accordance with some embodiments of the present disclosure. The XR managing server 130 can include processing circuitry, at least one network interface 920 (network interface), and a display device 930. The processing circuitry may include at least one processor circuit 900 (processor), at least one memory 910 (memory). The processor 900 is operationally connected to these various components. The memory 910 stores executable instructions 912 and a virtual electronic device database 104 that are executed by the processor 900 to perform operations. The processor 900 may include one or more data processing circuits, such as a general purpose and / or special purpose processor (e.g., microprocessor and / or digital signal processor), which may be collocated or distributed across one or more data networks. The processor 900 is configured to execute the instructions 912 and XR module 914 in the memory 910, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for an electronic device.

[0106] Further Definitions and Embodiments:

[0107] In the above-description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined incommonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0108] When an element is referred to as being “connected”, “coupled”, “responsive”, or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected”, “directly coupled”, “directly responsive”, or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, “coupled”, “connected”, “responsive”, or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term “and / or” includes any and all combinations of one or more of the associated listed items.

[0109] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0110] As used herein, the terms “comprise”, “comprising”, “comprises”, “include”, “including”, “includes”, “have”, “has”, “having”, or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation “e.g ”, which derives from the Latin phrase “exempli gratia,” may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation “i.e ”, which derives from the Latin phrase “id est,” may be used to specify a particular item from a more general recitation.

[0111] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the blockdiagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0112] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.

[0113] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0114] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All suchvariations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts are to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. An extended reality, XR, rendering device (100) for rendering an immersive XR environment on display devices for viewing by participants in the immersive XR environment, the XR rendering device comprising: at least one processor (700); and at least one memory (710) storing instructions executable by the at least one processor (700) to perform operations to: define a virtual portal at a location in physical space of a first participant; determine a set of virtual electronic device characteristics of a virtual electronic device; render a virtual electronic device representation in the immersive XR environment of the virtual electronic device (220) based on the set of virtual electronic device characteristics, wherein the virtual electronic device comprises a virtual user interface that the first participant or a second participant can interact with in the immersive XR environment; identify a characteristic of an interaction of the first participant or second participant with a virtual user interface (222) of the virtual electronic device; and communicate with an XR managing server (130) to control an operational function of the virtual electronic device (220) based on the identified characteristic of the interaction.

2. The XR rendering device (100) of Claim 1, wherein the set of virtual electronic device characteristics comprise at least one of: a defined virtual shape; a defined software requirement of the virtual electronic device; a defined hardware requirement of the virtual electronic device; a defined virtual user interface, UI, of the virtual electronic device; a type of device represented by the virtual electronic device; a manufacturer of a physical electronic device (210) represented by the virtual electronic device; a geographic location of the XR rendering device; a geographic location of a rendered virtual electronic device; anda security mechanism of the virtual electronic device.

3. The XR rendering device (100) of Claim 1, wherein the operations further comprise to: receive an input from a physical electronic device (210) indicating an interaction of the first participant or second participant with a user interface of the physical electronic device (210); and communicate with the XR managing server (130) to control the operational function of the virtual electronic device (220) based on the input.

4. The XR rendering device (100) of Claim 1, wherein the operation to determine the set of virtual electronic device characteristics of the virtual electronic device comprises to: send a request to the XR managing server (130) requesting a set of virtual electronic device characteristics; and receive the set of virtual electronic device characteristics from the XR managing server (130).

5. The XR rendering device (100) of any of Claims 1 to 4, wherein the operations further comprise to: retrieve a set of physical electronic device characteristics of at least one physical electronic device (210); determine if the set of virtual electronic device characteristics and the set of physical electronic device characteristics satisfy a device compatibility rule; and responsive to the device compatibility rule being satisfied, identifying a characteristic of the interaction of the first participant or second participant with the virtual user interface (222) of the virtual electronic device (220); and communicate with the physical electronic device (210) to control an operational function of the physical electronic device (210) based on the identified characteristic of the interaction.

6. The XR rendering device (100) of Claim 5, wherein the set of physical electronic device characteristics comprise at least one of: a defined physical shape of the at least one physical electronic device; a defined software requirement of the at least one physical electronic device;a defined hardware requirement of the at least one physical electronic device; a defined virtual user interface, UI, of the at least one physical electronic device; a type of device to be represented by the at least one physical electronic device; a manufacturer of the at least one physical electronic device; at least one network connectivity property of the at least one physical electronic device; a geographic location of the at least one physical electronic device; and a security mechanism of the at least one physical electronic device.

7. The XR rendering device (100) of any of Claims 5 to 6, wherein the operation to retrieve the set of physical electronic device characteristics of the at least one physical electronic device (210) comprises to: send a message to the XR managing server (130) requesting a list of physical electronic device characteristics for an available at least one physical electronic device; and receiving the list of physical electronic device characteristics for the available at least one physical electronic (210) device from the XR managing server (130).

8. The XR rendering device (100) of any of Claims 5 to 6, wherein the operation to retrieve the set of physical electronic device characteristics of the at least one physical electronic device (210) further comprises to: selecting physical electronic device characteristics based on information received from the first participant and / or second participant, and / or received from the physical electronic device.

9. The XR rendering device (100) of any of Claims 1 to 8, wherein the operations further comprise to: track a location of a physical electronic device (210) relative to the location of an entrance boundary of the virtual portal; responsive to determining the physical electronic device (210) has entered the virtual portal, scan the physical electronic device (210) to determine the set of physical electronic device characteristics of the physical electronic device (210); anddetermine if the set of virtual electronic device characteristics and the set of physical electronic device characteristics satisfy a device compatibility rule; and responsive to the device compatibility rule being satisfied, identify a characteristic of the interaction of the first participant or second participant with the virtual user interface (222) of the virtual electronic device (220); and communicate with the physical electronic device (210) to control an operational function of the physical electronic device (210) based on the identified characteristic of the interaction.

10. An extended reality, XR, managing server (130) for managing an immersive XR environment on display devices for viewing by participants in the immersive XR environment, the XR managing server (130) comprising: at least one processor (900); and at least one memory (910) storing instructions executable by the at least one processor (900) to perform operations to: determine a set of virtual electronic device characteristics of a virtual electronic device from a virtual electronic database (104); transmit the set of virtual electronic device characteristics of a virtual electronic device to an XR rendering device (100); receive a characteristic of an interaction of a first participant or a second participant with a virtual user interface (222) of the virtual electronic device from the XR rendering device (100); and control an operational function of the virtual electronic device (220) based on the identified characteristic of the interaction.

11. The XR managing server (130) of Claim 10, wherein the operations further comprise to: receive a request from the XR rendering device (100) for creating additional functions to a server not in the virtual electronic database. responsive to determining the set of virtual electronic device characteristics of the virtual electronic device not in the virtual electronic database (104) based on the requested additional functions, store the set of virtual electronic devicecharacteristics of the virtual electronic device in the virtual electronic database (104); and send the set of virtual electronic device characteristics of the virtual electronic device not in the virtual electronic database (104) based on the requested additional functions to the XR rendering device (100).

12. A method performed by an extended reality, XR, rendering device for rendering an immersive XR environment on display devices for viewing by participants in the immersive XR environment, the method comprising: defining (500) a virtual portal at a location in physical space of a first participant; determining (502) a set of virtual electronic device characteristics of a virtual electronic device; rendering (504) a virtual electronic device representation in the immersive XR environment of the virtual electronic device based on the set of virtual electronic device characteristics, wherein the virtual electronic device comprises a virtual user interface that the first participant or a second participant can interact with in the immersive XR environment; identifying (506) a characteristic of an interaction of the first participant or second participant with a virtual user interface of the virtual electronic device; and communicating (508) with an XR managing server to control an operational function of the virtual electronic device based on the identified characteristic of the interaction.

13. The method of Claim 12, wherein the set of virtual electronic device characteristics comprise at least one of: a defined virtual shape; a defined software requirement of the virtual electronic device; a defined hardware requirement of the virtual electronic device; a defined virtual user interface, UI, of the virtual electronic device; a type of device represented by the virtual electronic device; a manufacturer of a physical electronic device represented by the virtual electronic device; a geographic location of the XR rendering device; a geographic location of a rendered virtual electronic device; anda security mechanism of the virtual electronic device.

14. The method of Claim 12, further comprising: receiving an input from a physical electronic device indicating an interaction of the first participant or second participant with a user interface of the physical electronic device; and communicating with the XR managing server to control the operational function of the virtual electronic device based on the input.

15. The method of Claim 12, wherein determining (502) the set of virtual electronic device characteristics of the virtual electronic device comprises: sending a request to the XR managing server requesting a set of virtual electronic device characteristics; and receiving the set of virtual electronic device characteristics from the XR managing server.

16. The method of any of Claims 12 to 15, further comprising: retrieving a set of physical electronic device characteristics of at least one physical electronic device; determining if the set of virtual electronic device characteristics and the set of physical electronic device characteristics satisfy a device compatibility rule; and responsive to the device compatibility rule being satisfied, identifying a characteristic of the interaction of the first participant or second participant with the virtual user interface of the virtual electronic device; and communicating with the physical electronic device to control an operational function of the physical electronic device based on the identified characteristic of the interaction.

17. The method of Claim 16, wherein the set of physical electronic device characteristics comprise at least one of: a defined physical shape of the at least one physical electronic device; a defined software requirement of the at least one physical electronic device; a defined hardware requirement of the at least one physical electronic device;a defined virtual user interface, UI, of the at least one physical electronic device; a type of device to be represented by the at least one physical electronic device; a manufacturer of the at least one physical electronic device; at least one network connectivity property of the at least one physical electronic device; a geographic location of the at least one physical electronic device; and a security mechanism of the at least one physical electronic device.

18. The method of any of Claims 16 to 17, wherein retrieving the set of physical electronic device characteristics of the at least one physical electronic device comprises to: send a message to the XR managing server requesting a list of physical electronic device characteristics for an available at least one physical electronic device; and receiving the list of physical electronic device characteristics for the available at least one physical electronic device from the XR managing server.

19. The method of any of Claims 16 to 17, wherein retrieving the set of physical electronic device characteristics of the at least one physical electronic device further comprises to: selecting physical electronic device characteristics based on information received from the first participant and / or second participant, and / or received from the physical electronic device.

20. The method of any of Claims 12 to 19, further comprising: tracking a location of a physical electronic device relative to the location of an entrance boundary of the virtual portal; responsive to determining the physical electronic device has entered the virtual portal, scanning the physical electronic device to determine the set of physical electronic device characteristics of the physical electronic device, and determining if the set of virtual electronic device characteristics and the set of physical electronic device characteristics satisfy a device compatibility rule; and responsive to the device compatibility rule being satisfied,identifying a characteristic of the interaction of the first participant or second participant with the virtual user interface of the virtual electronic device, and communicating with the physical electronic device to control an operational function of the physical electronic device based on the identified characteristic of the interaction.

21. A method performed by an extended reality, XR, managing server for managing an immersive XR environment on display devices for viewing by participants in the immersive XR environment, the method comprising: determining (600) a set of virtual electronic device characteristics of a virtual electronic device from a virtual electronic database; transmitting (602) the set of virtual electronic device characteristics of a virtual electronic device to an XR rendering device; receiving (604) a characteristic of an interaction of a first participant or a second participant with a virtual user interface of the virtual electronic device from the XR rendering device; and controlling (606) an operational function of the virtual electronic device based on the identified characteristic of the interaction.

22. The method of Claim 21, further comprising: receiving a request from the XR rendering device for creating additional functions to a server not in the virtual electronic database. responsive to determining the set of virtual electronic device characteristics of the virtual electronic device not in the virtual electronic database based on the requested additional functions, storing the set of virtual electronic device characteristics of the virtual electronic device in the virtual electronic database; and sending the set of virtual electronic device characteristics of the virtual electronic device not in the virtual electronic database based on the requested additional functions to the XR rendering device.

Images