Technology that enables collaboration between devices, especially XR devices

Abstract

The present invention provides a solution, including a system, a method, and a device, for enabling collaboration between multiple augmented reality (XR) devices (XR1, XR2, 100, 200). The system comprises a first XR device (XR1, 100) having a first set of functions, a second XR device (XR2, 200) having a second set of functions, and a hub device (XRH, 300) communicatively connected to the first XR device (XR1, 100) and / or the second XR device (XR2, 200), wherein the first XR device (XR1, 100) is configured to provide the first set of functions, and the second XR device (XR2, 200) is configured to use at least one of the functions defined in the first set of functions and / or the functions of the hub device (XRH, 300).

Description

[Technical Field]

[0001] In particular, the present invention relates to a solution including a system, a method and a device that enables collaboration between multiple Extended Reality (XR) devices. [Background technology]

[0002] Today, XR devices, such as augmented reality (AR), virtual reality (VR), or mixed reality (MR) devices, are increasingly being used in scenarios where collaboration between two or more people in AR, VR, and / or MR spaces (hereinafter also referred to as collaboration spaces) is common and, in most cases, essential. Most collaborative use cases require participants to have similar levels of affordance for successful collaboration. However, collaboration parties may have devices with different capabilities, features, and / or affordances. As newer, more powerful devices are used alongside older devices in the same collaborative scenario, the gap in capabilities, particularly the difference in their respective hardware performance, grows over time. Therefore, a solution is needed to level the gap between devices, particularly by enhancing the capabilities of lower-end devices within the collaboration space. Summary of the Invention

[0003] According to the invention, this is achieved by utilizing the capabilities of other devices present in the collaboration space, in particular the devices of the participants present in the collaboration space. The solution according to the invention allows, for example, a device with low computing power (low-end device) to outsource computations or parts of computations to a more powerful device (high-end device) during such collaboration.

[0004] To address this problem, the present invention proposes a system for enabling collaboration between multiple augmented reality (XR) devices, comprising a first XR device (XR1) having a first set of functions, a second XR device (XR2) having a second set of functions, and a hub device (XRH) communicatively connected to the first XR device (XR1) and / or the second XR device (XR2), whereby the first XR device (XR1) is configured to provide the first set of functions in particular to the hub device (XRH), and the second XR device (XR2) is configured to use at least one of the functions defined in the first set of functions and / or advantageously the functions of the hub device (XRH).

[0005] The solution according to the invention is characterized in the claims.

[0006] Regarding a method for enabling collaboration between devices, in particular between multiple Extended Reality (XR) devices, the present invention provides: a first device, in particular a first XR device (XR1), having a first set of functions; a second device, in particular a second XR device (XR2), having a second set of functions; There, The communication connection is Established between each of the first device, in particular the first XR device (XR1) and the second device, in particular the second XR device (XR2), and a hub device (XRH) or a hub interface (XRH) of a hub device, and preferably initiated by each of the first device, in particular the first XR device (XR1) and the second device, in particular the second XR device (XR2); The established communication connection is used to provide the first function set of the first device, in particular the first XR device (XR1), and the second function set of the second device, in particular the second XR device (XR2), respectively, to the hub device (XRH) or the hub interface of the hub device (XRH) and / or to the second device, in particular the second XR device (XR2), and the first device, in particular the first XR device (XR1), respectively; The established communication connection is used to receive from the hub device (XRH) or a hub interface (XRH) of the hub device, or from each of the first device, in particular the first XR device (XR1) and the second device, in particular the second XR device (XR2), at least one function defined in a first function set and at least one function defined in a second function set of the first device, in particular the first XR device (XR1), respectively, and / or advantageously functions of the hub device (XRH), The second device, in particular the second XR device (XR2), is configured to utilize at least one received function defined in the first function set and / or preferably a function of the hub device (XRH), and respectively the first device, in particular the first XR device (XR1), is configured to utilize at least one received function defined in the second function set and / or preferably a function of the hub device (XRH).

[0007] The method is advantageously designed and / or adapted for the system according to the invention.

[0008] Regarding devices with a feature set, in particular XR devices (XR1, XR2), the present invention provides: Establishing a communication connection with at least one other device, in particular an XR device (XR2, XR1), and / or establishing a communication connection with a hub device (XRH) and a hub interface of the hub device (XRH), respectively; Using the established communication connection, provide a function set of the XR device (a first function set of XR1 and a second function set of XR2, respectively) to a hub device (XRH) and a hub interface of the hub device (XRH), respectively; receiving at least one function defined in the function set of the other device, in particular the other XR device (the first function set of XR1 in XR2 and the second function set of XR2 in XR1, respectively) and / or advantageously in the function set of the hub device (XRH); and / or utilize the received at least one function defined in the function set of the other device, in particular the other XR device (the first function set of XR1 in XR2 and the second function set of XR2 in XR1, respectively) and / or advantageously in the function set of the hub device (XRH), It is proposed that the device be designed, configured and / or set up so that

[0009] Advantageously, the device, in particular the XR device (XR1, XR2), is designed and / or adapted to perform the steps performed by the device, in particular the XR device (XR1, XR2), of the method according to the invention and / or is designed and / or adapted for the system according to the invention.

[0010] An XR device according to the present invention is any device that actively participates in an XR context without necessarily being worn on the body of a person / user. It could be a sensor attached to a wall or the landscape, or a drone that maps the terrain. A set of such devices, which may be body-worn, is defined as an XR wearable in the sense of the present invention. An XR wearable is a subset of the XR device category. An XR wearable advantageously comprises a single sensor and / or stimulation device, or a collection of sensors and / or stimulation devices, each contributing to providing or supporting an XR experience for a user, in particular an end user. An XR wearable typically has the components shown in the table of FIG. 4.

[0011] An XR wearable may be a single device worn on the head, such as a head-mounted display (HMD), and assisted by handheld controllers, or it may be a collection of devices worn on the body. For example, an XR wearable may be an integrated HMD or a combination of XR glasses and sensors that may not be located in the same physical structure / housing as the glasses. An example of an external sensor would be a sensor that monitors muscle movements in the fingers, in which case the sensor patch / device would be worn as a glove or a wristband / watch. Another example of an external sensor would be a head-worn BCI (brain-computer interface) or a separate depth camera located away from the body. In essence, an XR device can be thought of as a set of different electronic components whose purpose is to participate in the XR experience by gathering input, presenting stimuli to the user, or supporting other components. These components themselves do not need to be housed in a single physical container.

[0012] The hub device (XRH) according to the present invention (hereinafter also referred to as HyperReality Hub) advantageously enables the collaboration of one or more XR devices. This solution can advantageously be embodied in the form of a purpose-built device or implemented as an interface to existing XR wearables / devices.

[0013] The hub device (XRH) and hub interface, respectively, according to the invention advantageously enable: 1. Distributed computation across participating XR devices Each participating XR device advantageously advertises its capabilities and spare capacity to the other XR devices, either directly (through an instance of the hub device (XRH) interface) or through a physical instance of an already connected hub device (XRH). Even the least powerful XR device can now advantageously participate in the XR experience by utilizing the computational power of other XR devices around it. For example, if an XR device must reduce its computational power due to low power or thermal constraints, the participating XR device can refresh its capabilities and advertise them to the quorum. 2. Collaborative enhancements The primary scenario envisioned is that most XR wearables and devices offload computation and / or tracking (partially or fully) to a purpose-built physical instance of a hub device (XRH). Advantageously, when multiple XR devices participate in a single experience, some XR devices can enhance their capabilities by utilizing features on other XR devices participating in the same experience. For example, a user using a tablet device that lacks hand tracking can participate in the XR experience using hand tracking, if the other XR devices can provide hand tracking to the user using that tablet device, according to the present invention. This is advantageously done by the hub device (XRH) combining sensor data from other hand-tracking-enabled devices that may have the tablet device's hands within their field of view. This sensor stream advantageously allows the user to see their hands even if their own device does not support hand tracking. Participating XR devices can advantageously refresh their capabilities and notify the quorum. For example, an HMD may detect that its controllers are running out of power and become unusable. 3. Real-time communication channel between XR device and hub device (XRH) Advantageously, when connected to an XR device, an instance of the hub device (XRH) can initiate real-time communication and / or control protocols such as ROS (Robot Operating System), ExpressLRS, etc., which allow for control, telemetry, position, orientation, capabilities negotiation, notifications, etc., as well as communication with sensors and stimulators in real-time. 4. Collaborative SLAM (Simultaneous Location and Mapping) of XR device spatial position and orientation Advantageously, collaborative point cloud generation is provided, which may be static and / or dynamic. Advantageously, when multiple XR devices are present in the same physical space and connected to a hub device (XRH), typical challenges of XR occlusion can be solved by combining sensor inputs from all connected XR devices to form a point cloud using available algorithms such as RANSAC (Random Sample Consensus) or ICP (Iterative Closest Point). Advantageously, with a sufficient number of XR devices participating, this point cloud can become a dynamic point cloud, thereby generating a digital twin of the real world for virtual objects to dynamically interact with. The more XR devices that participate in the collaborative process, the higher the fidelity and resolution of the dynamic point cloud of the real world. 5. Position and orientation of permanent anchors Advantageously, the hub device (XRH) has access to volumetric data (preferably inclusive, not exclusive) of the physical space in which it resides. Anchors placed by one user / entity can be advantageously persistent, allowing other users visiting the physical space at different times to utilize the same anchor. For example, a doctor might place an anchor related to a patient's medication needs. This allows a nurse on call to access the anchor and administer the medication. In the same example, an anchor could display a patient's vital statistics. As another example, in a museum, digital anchors may be placed on or near exhibits for visitors to access as they pass by. Anchors may contain a lot of information, media, and related information that would otherwise require a larger physical space to display. In yet another example, a sorting robot may place anchors on incoming cargo overnight for human reference in a warehouse scenario. Some anchors, such as directional markers, need to maintain their position as well as their orientation, and the hub device (XRH) can advantageously provide this position and orientation if it has access to volumetric data. 6. Collaborative Occlusion Calculation The hub device (XRH) advantageously has the ability to dynamically generate and / or update volume data in a multi-user / entity scenario, allowing the hub device (XRH) to know where objects are located in space at a given time. This information can be advantageously used to calculate occlusion using existing algorithms such as ray tracing, which helps detect hidden object points and improves efficiency. The frequency of updates, determined based on the use case or user settings, allows the hub device (XRH) to further improve the efficiency of occlusion calculations using differential dynamic volume data. 7. Collaboration between multiple XRH instances The real-time protocol can advantageously employ a physical cable or a wireless interface such as Wi-Fi. The hub device (XRH) can advantageously be connected to other hub devices (XRH) via a physical cable such as optical cable, copper / metal / Ethernet, or a wireless interface such as Wi-Fi. Various connection configurations include, but are not limited to, daisy chain, star, mesh, and ring. Variations of the hub device (XRH) can advantageously provide a flexible platform for expansion through hardware or software updates. For example, a version of the hub device (XRH) can provide free slots for future expansion by adding compute cards to support more XR devices as needed. 8. Collaborative (static and dynamic) point cloud generation Point cloud generation can be time-consuming with a single XR device, especially when covering a large area. In such a scenario, compromises are usually made in terms of resolution and image quality. Advantageously, processes such as collaborative SLAM exist, which enable point cloud generation by combining volumetric and / or non-volume data from multiple sources, preferably simultaneously. The hub device (XRH) advantageously enables not only the collaborative SLAM approach but also collaborative dynamic SLAM, in which data from various XR devices collaboratively connected to the hub device (XRH) updates the volume data continuously, periodically, or in response to a trigger. For example, participating XR devices notify the hub device (XRH) of detected movement. 9. Object Tracking, Moving Anchors, and Tagging Anchors Advantageously, dynamic point cloud generation allows the hub device (XRH) to maintain up-to-date volume data and its associated properties in a time-domain independent manner, tracking objects, movements, and / or changes associated with each volume unit. For example, for points, the volume units are points. Advantageously, this functionality can be used to generate floating anchors (virtual anchors) that can be tagged / pinned to physical objects. In other words, the location of the anchor is determined by the location of the object to which it is tagged, rather than the anchor's original location in space. As in the hospital example above, if an anchor is tagged to a patient with details of the medication it requires, the anchor will follow the patient when the patient is moved to a different bed or location. In another example, anchors can be tagged to products in a supermarket, and as the products move, the anchors will move with the products thanks to object tracking and floating anchors.

[0014] According to the invention, XR devices (XR1, XR2) connecting to a hub device (XRH) advantageously go through an initial phase of initialization and registration before commencing normal operation.

[0015] Initialization advantageously begins with the XR devices (XR1, XR2) first being powered on at the user's command.

[0016] According to a preferred embodiment of the present invention, the first XR device (XR1) is configured to directly provide a first feature set, including the capabilities, functions, and / or affordances of the first XR device (XR1), to the hub device (XRH) and / or the second XR device (XR2). This advantageously allows data to be communicated between clients, i.e., the first XR device (XR1) and the second XR device (XR2), via the hub device (XRH) through a publisher / subscriber model (as an example). Here, the hub device (XRH) may advantageously function as a central hub for establishing communication between clients, i.e., the first XR device (XR1) and the second XR device (XR2). According to a further embodiment of the present invention, the hub device (XRH) also advantageously maintains the status of each client, i.e., the first XR device (XRI) and the second XR device (XR2), and updates other devices in the cohort accordingly. These statuses advantageously include, but are not limited to, a registered but out of range device, a device energy state, a device privacy state, a registered but in various sleep states, etc. Sleep states are advantageously provided to conserve energy and include, but are not limited to, transitioning the XR device into deeper sleep states over time or in response to user settings. Sleep states may advantageously progress from a reduced operating frequency to a complete shutdown of components such as the CPU, GPU, memory, sensors and / or stimulators.

[0017] Sleep state sleep-0 may advantageously trigger dimming of the display, shutting down sensors, and reducing the frequency of the CPU, GPU, and / or memory. Sleep state Sleep-1 may advantageously trigger further reductions in the frequency of the CPU, GPU, memory, and communications chipset, and / or reducing the brightness of the display. Sleep state Sleep-2 may advantageously trigger a "hibernate" state that shuts down various parts of the CPU, saves any volatile data, shuts down most components, and puts the system into a hibernate state. Sleep state Sleep-3 may advantageously complete the shutdown of the XR device.

[0018] According to a preferred embodiment of the present invention, the registration procedure is advantageously facilitated by near field communication (NFC), where the user taps the XR device (i.e., XR1 or XR2) to the hub device (XRH). This registration advantageously enables the hub device (XRH) to identify differences in functionality and to identify the spatial location of the XR device (i.e., XR1 or XR2), especially when multiple devices are registered.

[0019] Optionally or additionally, the registration process between the XR device (i.e., XR1 or XR2) and the hub device (XRH) may be accomplished by other communication means, including optical or wired information exchange, WiFi, Bluetooth, and / or RF protocols that enable information exchange.

[0020] According to a further embodiment of the present invention, an XR device (i.e., XR1 or XR2) that has already been initialized and registered with the hub device (XRH) and that has gone into sleep mode, for example as part of an energy saving function, advantageously reinitializes itself by sensing physical movement of the XR device (i.e., XR1 or XR2) itself, alternatively reinitialization of the XR device (i.e., XR1 or XR2) is triggered by an on-board sensor of the XR device (i.e., XR1 or XR2), or reinitialization of the XR device (i.e., XR1 or XR2) is triggered by a predetermined signal / message from the hub device (XRH). According to a further embodiment of the present invention, re-registration may advantageously, but not necessarily, be triggered by the hub device (XRH) depending on various factors, for example, the time elapsed since the last activity, changes in activity over time, etc.

[0021] According to a further preferred embodiment, an XR device (i.e., XR1 or XR2) registered with the hub device (XRH) is requested to perform a mapping of the physical environment and submit the results to the hub device (XRH). The hub device (XRH) then builds a volumetric representation of the physical surroundings, e.g., a point cloud, within which the registered XR device can be manipulated and tracked. A point cloud or point cloud map in the sense of the present invention is a discrete collection of data points in space, each point's location having a set of coordinates, i.e., Cartesian coordinates (x, y, z). These points may represent 3D shapes or objects. Point cloud is a well-defined term in the XR field. Advantageously, each of these points can have its own set of properties, including parameters such as color, brightness, transparency, etc. Each point may have associative and / or reactive functions associated with it, e.g., behavior based on neighboring points, reaction to interactions, etc. Thus, when devices, particularly multiple XR devices, are connected to a hub device (XRH) and reside within the same physical space, e.g., a room, for example, the challenge of XR occlusion can be advantageously addressed by advantageously combining sensor inputs from all devices connected to the hub device (XRH), particularly XR devices (i.e., XR1 or XR2), and / or the capabilities of the hub device (XRH) to create a point cloud map for each point cloud by the hub device (XRH). To this end, the hub device (XRH) advantageously utilizes algorithms such as RANSAC (Random Sample Consensus) or ICP (Iterative Closest Point). With the participation of a sufficient number of XR devices, e.g., a third XR device (XR3), a fourth XR device (XR4), and / or more XR devices (XRn), this point cloud can advantageously be a dynamic point cloud, thus advantageously generating a digital twin of the real world for virtual objects to interact with. The more XR devices (XR1, XR2, XRn) that participate in the collaborative process with the hub device (XRH), the higher the fidelity and resolution of the dynamic point cloud of the real world.

[0022] Advantageously, the virtual space is fixed to the physical space using an inside-out (markerless / markered) or outside-in tracker to determine the physical coordinates of the XR device, preferably via or using SLAM. When multiple XR devices are used in the same physical space and access the same virtual space, the following a) to g) are not guaranteed. a) Individual XR devices advantageously latch onto the same marker as other XR devices, especially in markerless inside-out systems. b) XR devices advantageously know the relative position and orientation (i.e., the direction the XR device is facing) of other XR devices that share the same physical space. c) Spatial anchors generated by XR devices advantageously do not drift over time, as explained in Examples 1-3 below. 1. If a spacecraft is orbiting around an astronaut using an XR device onboard or during a spacewalk, the spacecraft / ship may not provide sufficient range to secure the anchor. 2. When using XR devices in underwater exploration where visual markers are scarce, such as in underwater oil field training / maintenance 3. Outdoor situations where visual markers are scattered and scarce, such as military training in desert environments. d) The XR device can advantageously generate and detect anchors in low contrast or low light spaces / environments. e) XR devices can advantageously generate and detect anchors in dynamic spaces / environments such as crowded conference rooms, trade shows, outdoor gatherings, etc. f) The XR device is advantageously able to generate and detect anchors, preferably visually or otherwise, in repetitive / identical spaces / environments (e.g., most conference rooms in an office building may follow the same dimensions and interior design). g) XR devices advantageously minimize occlusion when many other people wearing XR devices are in the same physical space (e.g., a classroom full of students, a concert, an exhibition, or any gathering of people).

[0023] The same advantageously applies when multiple XR devices are used in different physical spaces to access the same virtual space, or when multiple XR devices are used in the same physical space to access different virtual spaces.

[0024] In a further embodiment of the invention, optionally, additional XR devices that register themselves with the hub device (XRH) are requested to also perform mapping, for example if the existing mapping information is determined to be insufficient in terms of completeness and / or resolution, which advantageously enables the hub device (XRH) to add point cloud map information to fill gaps, update existing point cloud maps, or increase the density or resolution of existing point cloud maps stored on the hub device (XRH).

[0025] In further embodiments of the present invention, the point cloud map is optionally represented or stored as polygons or voxels or a combination thereof. Preferably, the hub device (XRH) optionally provides the user of the XR device with the affordance of saving either / or / and model of the point cloud map. Each of these representations of the surrounding spatial features offers advantages / compromises depending on the nature of the surroundings, available storage space, and / or computational requirements. In either / or model, the hub device (XRH) advantageously decides where to store the map and / or associated data, preferably in local storage space within the physical environment or in cloud storage space. This decision is advantageously pre-configured, user-defined, dynamic, or AI-generated (AI). The decision of where the generated map is stored depends on various factors, including but not limited to the nature of the application, latency requirements, security requirements, privacy requirements, asset management, digital licenses, etc.

[0026] Advantageously, the hub device (XRH) serves as a functional transmission medium between the XR devices (i.e., XR1, XR2, ~XRn) participating in the collaborative process with the hub device (XRH), e.g., as the origin of the world coordinate system scanned by the capable XR device (i.e., XR1). The first XR device (XR1), which may be, e.g., a high-end device with higher hardware capabilities, taps into the hub device (XRH), registers itself with the hub device (XRH), and scans the environment. The second XR device (XR2), which may be, e.g., a low-end device with lower hardware capabilities, can also register itself with the hub device (XRH), but is constrained by its hardware, e.g., camera, LIDAR (Light Imaging Detection And Ranging), etc., and is limited to low-quality mapping. The hub device (XRH) acts as the origin of the point cloud map and therefore advantageously transfers a higher quality point cloud map obtained from / using the first XR device (XR1) to the second XR device (XR2), thereby advantageously enhancing the feature set of the second XR device (XR2) which may have lower hardware capabilities, e.g., in terms of camera, LIDAR, etc.

[0027] In a further embodiment of the present invention, the hub device (XRH) is advantageously configured to receive location-related information from the first XR device (XR1) and generate a point cloud map including the positions of the first XR device (XR1) and the second XR device (XR2). This is advantageous because, for example, a low-end device, i.e., the second XR device (XR2), may not have built-in location information and / or mapping functionality and, if it requires specific functionality in the collaboration space, it can communicate with the hub device (XRH) to obtain location-related information (particularly including the positions of the first XR device (XR1) and the second XR device (XR2) in the point cloud map) from the hub device (XRH). For example, if the second XR device (XR2) wants to present an object located at a specific spatial position to a user of the second XR device (XR2), it can query the hub device (XRH) using a currently captured image as input. In response to this query, the hub device (XRH) determines the location of the second XR device (XR2) based on the stored point cloud map and transmits the queried features (images) to the second XR device (XR2). This advantageously provides coordination and distribution. The point cloud map stored in the hub device (XRH) is a higher quality point cloud map because it was generated using sensor input from the first XR device (XR1). Optionally or additionally, the hub device (XRH) can store the point cloud map in a central repository or in a location outside the physical boundaries of the hub device (XRH) itself, but where the data can be accessed and recalled as needed.

[0028] According to a further preferred embodiment, the hub device (XRH) is further configured to: receive from the second XR device (XR2) images and image streams acquired by the second XR device (XR2); determine the position of the second XR device (XR2) based on the images and image streams; and provide the determined position to the second XR device (XR2). Thus, the second XR device (XR2) can determine its position without having corresponding functionality itself. When multiple high-end XR devices (e.g., the first XR device (XR1)) are present, all such XR devices can provide acquired point cloud map data to the hub device (XRH), which then uses them to generate a global point cloud map. The hub device can use various SLAM algorithms, such as ARKit, ARCore, and Orb SLAM3.

[0029] In further preferred embodiments of the present invention, LIDAR, SONAR, TOF, and / or other mechanisms and / or sensors may optionally or additionally be used advantageously as an alternative to and / or in combination with images or image streams for localization and / or mapping.

[0030] In a further preferred embodiment of the present invention, if the first XR device (XR1) and the second XR device (XR2) have similar capabilities, functions, and / or affordances, mapping data acquired from the second XR device (XR2) is used to update and / or enhance the point cloud map in terms of accuracy, density, and / or filling gaps that may have been missed in previous mapping. The map data can be advantageously enhanced by combining data generated by the first XR device (XR1) and the second XR device (XR2). Further, it can be advantageously enhanced using implementations such as OctoMap (Hornung, A., Wurm, K.M., Bennewitz, M. et al. OctoMap: an efficient probabilistic 3D mapping framework based on octrees. Auton Robot 34, 189-206 (2013)).

[0031] Advantageously, if there is a time difference between the maps generated by the first XR device (XR1) and the second XR device (XR2), the data difference is used to locate and identify the moved object. By appropriately increasing the frequency to suit the application, a dynamic volumetric map is achieved with dynamic mapping and prediction of object movement. Trajectory prediction implementations well known in the computer vision industry are advantageously further used, preferably kernel-based tracking and / or contour tracking.

[0032] In a further preferred embodiment of the present invention, when the first XR device (XR1) and the second XR device (XR2) have different sensing capabilities, mapping data acquired from the second XR device (XR2) is used to update and / or enhance the point cloud map in terms of accuracy, density, and / or filling gaps that may have been missed in previous mapping. This is advantageously implemented with widely available multi-sensor SLAM (Simultaneous Localization and Mapping) procedures and / or algorithms. Multi-sensor SLAM advantageously combines the output of various sensors, such as LIDAR, IR, UV, and visible spectrum cameras, and touch-based sensors, to generate spatial maps with higher fidelity and resolution than would be possible using a single sensor. This can advantageously be implemented using existing, well-known multi-sensor SLAM algorithms, such as LIDAR-IMU-based loosely and tightly coupled systems.

[0033] In a further preferred embodiment of the present invention, if the first XR device (XR1) is a high-end device with higher functionality and capabilities and the second XR device (XR2) is a low-end device with lower functionality and capabilities, and the second XR device (XR2) does not have hand tracking functionality, the hub device (XRH) can provide that functionality to the second XR device (XR2) as long as the user's hands of the second XR device (XR2) are within the field of view of the first XR device (XR2). This is advantageously made possible by the hub device (XRH) processing sensor data from the first XR device (XR1), using the second XR device (XR2) to determine the position of the user's hands, and providing hand tracking functionality to the second XR device (XR2). In this way, the system according to the present invention advantageously enables hand tracking functionality for the second XR device (XR2), which was previously not possible due to hardware limitations.

[0034] A further embodiment of the present invention is characterized in that the hub device (XRH) is configured to receive sensor data from the first XR device (XR1) and to provide the at least one function defined in the first function set to the second XR device (XR2).

[0035] Advantageously, improved quality and / or performance is achieved by combining multiple sensor data from all connected XR devices, allowing a user who is not within the tracking range of one XR device to be tracked if within the tracking range of another participating XR device, and allowing for collaborative compensation for occlusions.

[0036] Further advantageous embodiments of the present invention allow for the selection and processing of sensor data from only a small number of XR devices depending on capabilities, computational power requirements, energy optimization, range, user preferences, application needs, etc.

[0037] In another preferred embodiment of the present invention, the hub device (XRH) itself has a sensor or array of sensors to track changing conditions in the hub device's (XRH) physical environment and vicinity / range of operation and extend various functionality to all XR devices registered to the hub device (XRH), advantageously including changes in user posture, user facial movement, hand tracking, user body movement, XR device movement, light, sound, VOC, heat, humidity, gas composition and content, and / or RF (radio frequency) conditions in the vicinity / range of operation.

[0038] According to a preferred embodiment, when a second XR device (XR2) registers with a hub device (XRH), it is notified of a set of features that are not present on or with the second XR device (XR2), and the second XR device (XR2) can choose to enable those features.

[0039] In a further embodiment of the present invention, multiple functions that are not present in both XR devices (first XR device (XR1) and second XR device (XR2)) may be advantageously enabled by the system, particularly via the hub device (XRH).

[0040] A preferred application of such enhancements is facial expression tracking, which is realized as long as the user's face is within the range or field of view of one of the participating XR devices. Facial expression tracking, also referred to as face tracking, advantageously includes algorithms for detecting and / or tracking faces and their facial features (e.g., eyes, eyebrows, eyelids, pupils, nose, lips, and / or contours). The detection and / or tracking is performed on images and / or videos (e.g., video streams from cameras, video streams from sensors, or video files) acquired in color, grayscale, and / or infrared, millimeter-wave (mm), and especially near-infrared. This advantageously allows tracking of one or more faces and / or their facial features (e.g., 3D head pose, facial landmark locations, gaze direction, etc.). Thus, collaboration makes available functionality that was previously unavailable on any of the participating XR devices. Thus, all movement-related features can be tracked by the participating XR devices. This allows functionality such as real-time facial expression tracking without the need for additional sensors on each XR device.

[0041] Further preferred applications of such enhancements are body movements, postures, gestures, movements of other people, movements of objects, movements of other XR devices participating in the same physical space and connected to the hub device (XRH), movements of other XR devices participating in the same physical space but not attached / connected to the same hub device (XRH), basically any movements and / or changes that can be captured by other XR devices, including light, sound, VOCs, heat, humidity, gas composition and / or content, radio frequency (WiFi, Bluetooth, etc.).

[0042] According to another advantageous embodiment of the invention, the first XR device can provide the functions and features of the hub device (XRH). Thus, advantageously, all of the functions of the hub device XRH are implemented in the first XR device (XR1). Advantageously, the first XR device (XR1) also functions as the hub device (XRH) in addition to functioning as an XR device (i.e., XR1) that is part of the system according to the invention. Therefore, the solution according to the invention advantageously comprises at least two devices, in particular XR devices (XR1, XR2), one of which provides the hub interface or hub device (XRH). A further embodiment of the invention suggests that a reduced function set of the hub device (XRH) is implemented in the first XR device (XR1). This allows for the limited function set and advantages of the system according to the invention.

[0043] According to a preferred embodiment of the present invention, the first XR device (XR1) and the second XR device (XR2) are located in the same environment, e.g., physical and virtual space. This is particularly advantageous because the projection from the physical world to the digital world is identical, apart from the viewpoint, whether in VR / MR or AR.

[0044] According to another embodiment of the present invention, regardless of whether the first XR device (XR1) and the second XR device (XR2) are involved in different augmented reality (AR / VR / MR), the system can provide the above-mentioned features and advantages as long as the first XR device (XR1) and the second XR device (XR2) are present in the same physical space. This allows sensors on one XR device to monitor required actions / changes on the other XR device. These features may include providing one or more functions such as posture tracking, eye tracking, face tracking, hand tracking, and motion tracking.

[0045] In another embodiment, when multiple XR devices are used in the same physical space but access different virtual spaces, there is no guarantee that each XR device will latch to the same markers or anchors as the other XR devices, nor is there any guarantee that the XR devices are aware of the relative positions and orientations of the other XR devices that share the same physical space. In this case, the hub device (XRH) allows the XR devices that share the same physical space to have a common spatial reference / point cloud map that can operate regardless of whether the XR devices are engaged in different virtual spaces.

[0046] According to another preferred embodiment of the present invention, a spatial map of the physical space is dynamically updated in response to changes in the physical space, and the updated spatial map can be advantageously used by participating XR devices to reflect / project into their respective virtual spaces. As an example, assume that a first XR device (XR1) and a second XR device (XR2) are head-mounted devices (HMDs) operating in the same physical space, but that the users of the first XR device (XR1) operate in different virtual spaces. The user of the first XR device (XR1) moves a new physical object into the physical space where both the first XR device (XR1) and the second XR device (XR2) are operating. The new physical object must also be reflected / projected into the experience of the second XR device (XR2), preferably in a format appropriate for the experience in which the user of the second XR device (XR2) is immersed. As a result, the second XR device (XR2) can recognize the physical object. This reduces the risk of a user of an XR device unintentionally interacting with physical objects that dynamically appear in the physical environment, which may include other users of the XR device.

[0047] The hub device (XRH) is advantageously configured to provide a real-time communication channel towards the first XR device (XR1) and / or the second XR device (XR2). For example, an instance of the hub device, when connected to an XR device, can initiate a real-time communication and / or control protocol (ROS, ExpressLRS, etc.), which may enable real-time sensor and stimulator communication apart from control, telemetry, position, orientation, capability negotiation, notifications, etc.

[0048] According to another preferred embodiment, the first XR device (XR1) and / or the second XR device (XR2) are communicatively connected to the hub device (XRH) by at least one of a wireless connection, including Wi-Fi, Bluetooth, and / or NFC, and a wired connection, including optical fiber, metal, and / or Ethernet. For example, a real-time protocol may use a physical cable or a wireless interface, such as Wi-Fi. Furthermore, the hub device (XRH) may be connected to other hub devices (XRH) in various topology configurations, including, but not limited to, daisy chain, star, mesh, circular, etc., via either a physical cable or a wireless interface.

[0049] Variations of the hub device (XRH) advantageously allow for a flexible platform for expansion through hardware or software updates, for example, some versions of the hub device (XRH) may provide free slots for future expansion by adding compute cards to support additional XR devices as needed.

[0050] According to a further aspect of the present invention, the second XR device (XR2) is configured to provide a second function set, and the first XR device (XR1) is configured to use at least one of the functions defined in the second function set. This has the advantage that when both XR devices are high-end devices but have different specific functions, the functions can be made available to both XR devices. Thus, the hub device (XRH) can be used to exchange functions between the first XR device (XR1) and the second XR device (XR2).

[0051] The second XR device (XR2) is advantageously configured to provide a second set of capabilities to the hub device (XRH) and / or directly to the first XR device (XR1), thereby enabling the first XR device (XR1) and the second XR device (XR2) to subscribe to each other's capabilities.

[0052] Preferably, the first XR device (XR1) and / or the second XR device (XR2) are wearable devices. A wearable device in the sense of the present invention may be a single device worn on the head, such as a head-mounted display (HMD) assisted by a handheld controller, or a collection of multiple devices worn on the body. For example, a wearable may be an integrated HMD or a combination of XR glasses and sensors that may not be located in the same physical structure / housing as the glasses. An example of such an external sensor is a sensor that monitors muscle movements in the fingers, where the sensor patch / device is worn as a glove, a device held on the hand, or a wristband / watch. Another example of an external sensor could be a head-mounted BCI (Brain-Computer Interface) or another depth camera installed remotely from the body. Thus, such an XR device can be considered as a set of different electronic components aimed at collecting inputs and providing stimuli to the end user, although the components themselves may not be housed in a single physical container.

[0053] According to a further embodiment of the present invention, the first XR device (XR1) and / or the second XR device (XR2) is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device.

[0054] According to further embodiments, the XR devices connectable to the hub device (XRH) are not limited to wearable devices that can be worn by a human. These devices may advantageously be simple things like webcams, drones, home appliances, industrial equipment, sensors, monitors, intelligent structures, medical equipment, etc. Essentially, any physical object that needs to be represented in the XR world is considered an XR device.

[0055] Preferably, the hub device (XRH) includes a computing unit, a communications unit, and a platform unit. The computing unit is comprised of integrated computing elements such as a GCPU (general-purpose CPU (Central Processing Unit)), a GPU, an AI (artificial intelligence), and an ML (machine learning) component, providing an XR platform for supporting various pairs of XR devices and other non-body-worn devices (usually external sensors). The communications unit of the hub device (XRH) is broadly divided into a backend and a frontend. The backend communications unit connects the hub device (XRH) to the Internet as needed via various means (e.g., 5G connection via cellular base stations, satellite connection, Ethernet / optical connection to other hubs). The backend communications unit is advantageously also responsible for connecting to other hub devices (XRH) via various technologies, such as 5G, Ethernet, Wi-Fi, etc. The frontend communications unit connects to all XR devices and other related hardware required to provide an XR experience. The platform section advantageously comprises mechanisms for controlling and managing a number of functions, preferably including: - creating a computational model of a virtual or digital equivalent of a physical object spatially located in a physical area, the data for creating the computational model of the physical object may advantageously be obtained from an XR device connected to a hub device (XRH). -Ensuring synchronization of back-end and front-end communication depending on the use case. -Ensuring latency requirements for XR devices. -Providing an abstraction layer for developing and deploying applications (e.g., an operating system).

[0056] Further details, features and / or advantages of the invention are explained in more detail below with reference to embodiments of the invention illustrated in the drawings. [Brief explanation of the drawings]

[0057] [Figure 1]FIG. 1 is a flow chart illustrating one embodiment of interactions in a system including a first XR device (XR1), a second XR device (XR2), and a hub device (XRH). [Figure 2] FIG. 2 is a block diagram illustrating an embodiment utilizing a global SLAM located in a hub device (XRH). [Figure 3] FIG. 3 is a flow chart illustrating one embodiment of the initial stages of initializing and registering an XR device with a hub device (XRH) in accordance with the present invention. [Figure 4] FIG. 4 is a diagram illustrating the components that an XR device, in particular an XR wearable, according to the present invention comprises. DETAILED DESCRIPTION OF THE INVENTION

[0058] 1 illustrates one embodiment of interactions in a system including a first XR device 100, a second XR device 200, and a hub device 300. In a first step S11, the second XR device 200 (e.g., a low-end device) can subscribe to the hub device 300 to be notified about a first set of features that may be provided to the hub device 300 by the first XR device 100.

[0059] In step S12, when the first XR device 100 (e.g., a high-end device) provides its functionality to the hub device 300, the hub device 300 notifies the second XR device 200 of this based on a previous subscription. Alternatively, the first XR device 100 may directly notify the second XR device 200 of the available functionality.

[0060] According to one embodiment, particularly when there are three or more devices, each participating XR device may advertise its capabilities and spare capacity to the other XR devices directly (i.e., through an instance of the hub device interface) or through a physical instance of an already connected hub device, allowing even less powerful XR devices (e.g., low-end devices) to take advantage of the computing, tracking, and / or other capabilities of surrounding XR devices to participate in the XR experience.

[0061] Thereafter, the second XR device 200 may send a function request to the hub device 300 in step S14. The request is for a function that has been previously notified to the second XR device 200, i.e., a function that is only available on the first XR device 100 and is not available on the second XR device 200.

[0062] The hub device 300 then notifies the first XR device 100 in step S15 that its capabilities have been requested. For example, a user of a second XR device 200 wants to participate in an XR experience using hand tracking. Since the second XR device 200 cannot perform hand tracking itself, for this purpose the first XR device 100 lends itself (i.e., its corresponding capabilities) to provide hand tracking to the second XR device 200. According to step S16, this is performed by the hub device 300 combining sensor data of the first XR device 100 (which includes the hands of the user using the second XR device in its field of view). The corresponding sensory stream acquired by the first XR device 100 allows the user of the second XR device 200 to see their own hands, even though their own second XR device 200 is not capable of hand tracking.

[0063] Therein, the hub device 300 may include a computing portion 305 , a communication portion 310 , and a platform portion 315 .

[0064] The computing unit 305 may include an integrated general purpose central processing unit (GCPU), a graphics processing unit (GPU), an artificial intelligence (AI) unit, and machine learning (ML) components, and can provide an XR platform to service many different pairs of XR devices, i.e., a first XR device 100 and a second XR device 200 pair, as well as other non-body-worn devices such as off-board sensors.

[0065] The communications section 310 of the hub device 300 can be broadly divided into a backend and a frontend, where the backend ensures that the hub device 300 is connected to the internet, for example using a cellular network such as 5G, a connection to a satellite, another hub via an Ethernet / optical network, etc. The backend may also be responsible for connecting to other hub devices via various technologies including 5G, Ethernet, Wi-Fi, etc. The frontend, on the other hand, connects to all XR devices such as the first XR device 100 and the second XR device 200, as well as other associated hardware required to provide the XR experience.

[0066] Finally, the platform unit 315 comprises software that manages multiple functions including the creation of computational models, where data for creating such computational models of real physical objects may come from XR devices connected to the hub device 300. Furthermore, the platform unit 315 comprises software that ensures synchronization between back-end and front-end communications and meets the latency requirements of XR devices, for example by providing an abstraction layer for applications to be developed and deployed.

[0067] FIG. 2 illustrates an embodiment in which a global SLAM located on a hub device 300 is used to provide position data to a low-end device, i.e., a second XR device 200.

[0068] 2, a Robot Operating System (ROS) may be used as middleware to enable communication between the hub device 300 and the AR devices 100 / 200, as indicated by the dotted lines.

[0069] There, the hub device 300 may act as a ROS master to establish communication between clients, i.e., between a first XR device 100 and a second XR device 200. As explained above, data may be communicated between clients using a Publisher / Subscriber model.

[0070] 2, there is a global Simultaneous Localization and Mapping (SLAM) in the hub device 300, which stores maps from all devices that can provide corresponding data, i.e., the first XR device 100 (a high-end device). Thus, in step S21 of FIG. 2, the first XR device 100 provides such data to the hub device 300.

[0071] If the low-end device, i.e., the second XR device 200, needs a corresponding function, it communicates with the hub device 300 to obtain it. This is shown in step S22. For example, if the second XR device 200 wants to present some object in a specific world pose, it can query the hub device 300 for a global map using the current image, i.e., the image acquired by the second XR device 200, as input.

[0072] In response, the hub device 300 determines the location of the second XR device 200 based on a map created based on data obtained from other devices (i.e., high-end devices) such as the first XR device 100, and thereby transmits corresponding capabilities to the second XR device 200 through collaboration and distribution. Thus, the second XR device 200 can determine its own location despite not having its own SLAM capabilities.

[0073] The flowchart in Figure 3 illustrates one embodiment of the initial stages of initializing and registering an XR device to a hub device (XRH) according to the present invention. On the XR device side, the XR device taps to the hub device (XRH), registers itself with the hub device (XRH), and scans the environment. On the hub device (XRH) side, the tap is used as an initial alert. The hub device (XRH) determines whether the XR device was previously registered with the hub device (XRH). If not, the hub device (XRH) queries the XR device and causes it to exchange a first set of features (including, among other things, functionality and location) with the XR device. If an XR device is already registered with the hub device (XRH), the hub device (XRH) determines whether a map / point cloud map used for collaboration needs to be updated. If a map update is determined to be necessary by software or an application hosted on the hub device (XRH), the hub device (XRH) requests a map update from the registered XR device. This is typically a message notifying the user of the XR1, and in the example herein, the XR1 is referred to as the device with higher capabilities, XR1. The XR device scans its surroundings with the help of the user operating it and exchanges the scanned data with the hub device (XRH). If the map does not need to be updated or if the map has already been updated, the hub device (XRH) exchanges session data, parameters, and protocols to be used within the session, thereby completing the registration of the XR device to the hub device (XRH).

[0074] 4 shows a table of components of an XR device, in particular an XR wearable, according to the present invention, which are advantageously a collection of sensors and stimulators that provide an XR experience to the end user.

[0075] The embodiments shown in the drawings and described in connection therewith, in particular the example messages of FIG. 2, are provided only to illustrate the proposed solution according to the present invention and are not intended to limit the present invention. [Explanation of symbols]

[0076] 100 First XR device (XR1) 200 Second XR Device (XR2) 300 Hub Devices (XRH) 305 Hub Device (XRH) Calculation Section 310 Hub device (XRH) communication section 315 Hub Device (XRH) Platform Part S11 Method Step S12 Method Step S13 Method Step S14 Method Step S15 Method Step S16 Method Step S21 Method Step S22 Method Step S23 Method Step

Claims

1. 1. A system for enabling collaboration between multiple Extended Reality (XR) devices (XR1, XR2, 100, 200), comprising: a first XR device (XR1, 100) having a first feature set; a second XR device (XR2, 200) having a second set of features; a hub device (XRH, 300) communicatively connected to the first XR device (XR1, 100) and / or the second XR device (XR2, 200); Equipped with The system, wherein the first XR device (XR1, 100) is configured to provide the first set of functions, and the second XR device (XR2, 200) is configured to use at least one of the functions defined in the first set of functions and / or the functions of the hub device (XRH, 300).

2. 2. The system of claim 1, wherein the first XR device (XR1, 100) is configured to provide the first set of functions to the hub device (XRH, 300) and / or directly to the second XR device (XR2, 200).

3. 3. The system of claim 2, wherein the second XR device (XR2, 200) is configured to subscribe to the hub device (XRH, 300) to be notified of the first set of capabilities when the first set of capabilities is provided to the hub device (XRH, 300).

4. The system of any one of claims 1 to 3, wherein the hub device (XRH, 300) is configured to receive sensor data from the first XR device (XR1, 100) and provide the at least one function defined in the first function set to the second XR device (XR2, 200).

5. The system of claim 4 , wherein the first XR device (XR1, 100) and the second XR device (XR2, 200) are located in the same environment.

6. The system of any one of claims 1 to 5, wherein the hub device (XRH, 300) is configured to receive location related information from the first XR device (XR1, 100) and create a map, preferably a point cloud map, comprising the locations of the first XR device (XR1, 100) and the second XR device (XR2, 200).

7. 7. The system of claim 6, wherein the hub device (XRH, 300) is further configured to receive, from the second XR device (XR2, 200), images and / or an image stream acquired by the second XR device (XR2, 200), determine a location of the second XR device (XR2, 200) based on the images and / or image stream, and provide the determined location to the second XR device (XR2, 200).

8. The system of any one of claims 1 to 7, wherein the hub device (XRH, 300) is configured to provide a real-time communication channel towards the first XR device (XR1, 100) and / or the second XR device (XR2, 200).

9. 9. The system of any one of claims 1 to 8, wherein the first XR device (XR1, 100) and / or the second XR device (XR2, 200) are communicatively connected to the hub device (XRH, 300) by at least one of a wireless connection including Wi-Fi, Bluetooth, and / or NFC, and / or a wired connection including optical fiber, metal, and / or Ethernet.

10. The system of any one of claims 1 to 9, wherein the hub device (XRH, 300) is configured to control the first XR device (XR1, 100) and / or the second XR device (XR2, 200) to work collaboratively in point cloud generation.

11. The system of any one of claims 1 to 10, wherein the second XR device (XR2, 200) is configured to provide a second set of functions and the first XR device (XR1, 100) is configured to use at least one of the functions defined in the second set of functions.

12. 12. The system of claim 11, wherein the second XR device (XR2, 200) is configured to provide the second set of functions to the hub device (XRH, 300) and / or directly to the first XR device (XR1, 100).

13. The system according to any one of claims 1 to 12, wherein the first XR device (XR1, 100) and / or the second XR device (XR2, 200) are wearable devices.

14. The system according to any one of claims 1 to 13, wherein the first XR device (XR1, 100) and / or the second XR device (XR2, 200) is an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device.

15. The system of any one of claims 1 to 14, wherein the hub device (XRH, 300) includes a computing unit (305), a communication unit (310), and a platform unit (315).

16. The system according to one or more of claims 1 to 15, wherein the hub device (XRH, 300) itself has a sensor or an array of sensors to track changes in conditions in the physical environment and vicinity / range of operation of the hub device (XRH, 300) and to extend various functionalities to all the XR devices (XR1, XR2, 100, 200) registered with the hub device (XRH, 300).

17. The system according to one or more of claims 1 to 16, wherein the hub device (XRH, 300) provides facial expression tracking, in particular as long as the face of the user of the XR device is within range or field of view of one of the participating XR devices (XR1, XR2, 100, 200).

18. 20. The system of claim 17, including changes in user posture, user facial movement, hand tracking, user body movement, XR device movement, light, sound, VOC, heat, humidity, gas composition and content, and / or RF (radio frequency) conditions in the vicinity / area of ​​operation.

19. The system according to one or more of the preceding claims, wherein the first XR device (XR1, 100) is capable of providing the functions and features of the hub device (XRH, 300).

20. The system according to one or more of the preceding claims, wherein the functionality of the hub device (XRH, 300) is implemented in the first XR device (XR1, 100).

21. The system according to one or more of the preceding claims, wherein the first XR device (XR1, 100), in addition to being part of the system, also functions as the hub device (XRH, 300).

22. 1. A method for enabling collaboration between devices, in particular between multiple Extended Reality (XR) devices, comprising: a first device, in particular a first XR device (XR1, 100), having a first feature set; a second device, specifically a second XR device (XR2, 200), having a second feature set; The communication connection is Established between each of the first device, in particular the first XR device (XR1, 100) and the second device, in particular the second XR device (XR2, 200), and a hub device (XRH, 300) or a hub interface of a hub device, preferably initiated by each of the first device, in particular the first XR device (XR1, 100) and the second device, in particular the second XR device (XR2, 200); The established communication connection is used to provide the first set of functions of the first device, in particular the first XR device (XR1, 100), and the second set of functions of the second device, in particular the second XR device (XR2, 200), respectively, to the hub device (XRH, 300) or the hub interface of the hub device (XRH, 300) and / or to the second device, in particular the second XR device (XR2, 200), and the first device, in particular the first XR device (XR1, 100), respectively; The established communication connection used to receive, from the hub device (XRH, 300) or a hub interface of the hub device, or from each of the first device, in particular the first XR device (XR1, 100) and the second device, in particular the second XR device (XR2, 200), at least one function defined in a first function set and at least one function defined in a second function set of the first device, in particular the first XR device (XR1, 100), respectively, and / or functions of the hub device (XRH, 300); the second device, in particular the second XR device (XR2, 200), is configured to utilize the received at least one function defined in the first function set and / or functions of the hub device (XRH, 300); and the first device, in particular the first XR device (XR1, 100), respectively, is configured to utilize the received at least one function defined in the second function set and / or functions of the hub device (XRH, 300); method.

23. A method according to claim 22, designed and / or adapted for use with a system according to one or more of claims 1 to 21.

24. A hub device (XRH, 300) that enables collaboration between devices, in particular between multiple Extended Reality (XR) devices, comprising: a first device, in particular a first XR device (XR1, 100), having a first feature set; a second device, specifically a second XR device (XR2, 200), having a second feature set; enabling a communication connection to be established with said first device (XR1, 100) and / or said second device (XR2, 200); designed and / or adapted to receive information of at least a portion of the first set of functions of the first device (XR1, 100) and at least a portion of the second set of functions of the second device (XR2, 200), respectively; and / or designed and / or adapted to transmit information on at least a part of the first set of functions of the first device (XR1, 100) and at least a part of the second set of functions of the second device (XR2, 200), respectively, and / or information on at least a part of the functions of the hub device (XRH, 300); Hub device (XRH, 300).

25. The hub device (XRH, 300) of claim 24 configured to receive sensor data from the first device (XR1, 100) and provide at least one function defined in the first function set to the second device (XR2, 200).

26. A hub device (XRH) as described in claim 24 or claim 25, configured to receive location related information from the first device (XR1, 100) and create a map, preferably a point cloud map, including the locations of the first device (XR1, 100) and the second device (XR2, 200).

27. A hub device (XRH, 300) according to one or more of claims 14 to 26, configured to receive from the second device (XR2, 200) images and / or image streams acquired by the second device (XR2, 200), determine a position of the second device (XR2, 200) based on the images and / or image streams, and provide the determined position to the second device (XR2, 200).

28. A hub device (XRH, 300) according to one or more of claims 24 to 27, configured to provide a real-time communication channel towards the first device (XR1, 100) and / or the second device (XR2, 200).

29. A hub device (XRH, 300) according to one or more of claims 24 to 28, configured to control the first device (XR1, 100) and / or the second device (XR2, 200) to work collaboratively in point cloud generation.

30. The hub device (XRH, 300) according to one or more of claims 24 to 29, comprising a computing unit (305), a communication unit (310), and a platform unit (315).

31. The hub device (XRH, 300) of one or more of claims 24 to 30, wherein the hub device (XRH, 300) itself has a sensor or an array of sensors to track changes in conditions in the physical environment and vicinity / range of operation of the hub device (XRH, 300) and to extend various functions to all the XR devices (XR1, XR2, 100, 200) registered with the hub device (XRH, 300).

32. The hub device (XRH, 300) according to one or more of claims 24 to 31, wherein the hub device (XRH, 300) provides facial expression tracking, in particular facial expression tracking as long as the face of a user of the device, in particular an XR device, is within range or field of view of one of the participating devices, in particular XR devices (XR1, XR2, 100, 200).

33. The hub device (XRH, 300) of claim 32, including user posture, user facial movement, hand tracking, user body movement, XR device movement, changes in light, sound, VOCs, heat, humidity, gas composition and content, and / or RF (radio frequency) conditions in the vicinity / area of ​​operation.

34. A hub device (XRH, 300) according to one or more of claims 24 to 34, designed and / or adapted to perform the steps performed by the hub device (XRH, 300) of the method according to claim 22 or claim 23, and / or designed and / or adapted for use in a system according to one or more of claims 1 to 21.

35. A device, in particular an XR device (XR1, XR2, 100, 200), having a set of functions, Establishing a communication connection with at least one other device, in particular an XR device (XR2, XR1, 200, 100) and / or with a hub device (XRH, 300) and each of the hub interfaces of the hub device (XRH, 300); providing a set of XR device functions (a first set of XR1 and a second set of XR2, respectively) to a hub device (XRH, 300) and a hub interface of the hub device (XRH, 300), respectively, using the established communication connection; receiving at least one function defined in the function set of the other device, in particular the other XR device (the first function set of XR1 in XR2 and the second function set of XR2 in XR1, respectively) and / or advantageously in the function set of the hub device (XRH, 300); and / or Utilizing the received at least one function defined in the function set of the other device, in particular the other XR device (the first function set of XR1 in XR2 and the second function set of XR2 in XR1, respectively), and / or advantageously in the function set of the hub device (XRH, 300). A device, in particular an XR device (XR1, XR), designed, configured and / or configured to:

36. A device according to claim 35, in particular an XR device (XR1, XR2), designed and / or adapted to perform the steps performed by a device, in particular an XR device (XR1, XR2, 100, 200), of the method according to claim 22 or claim 23 and / or designed and / or adapted for a system according to one or more of claims 1 to 21.

37. A device according to claim 35 or claim 36, in particular an XR device (XR1, XR2), providing a hub interface for a hub device (XRH, 300) and a hub device (XRH, 300) according to one or more of claims 24 to 34, respectively.

38. 38. The device of claim 37, providing the functions and features of the hub device (XRH, 300).

39. The device according to one or more of claims 35 to 38, wherein the functionality of the hub device (XRH, 300) is implemented in the device (XR1, XR2, 100, 200).

40. A device according to one or more of claims 35 to 39, wherein the device (XR2, XR1, 200, 100) is part of a system according to one or more of claims 1 to 21 and also functions as a hub device (XRH, 300).

Images