Enabling collaboration between devices, in particular XR devices

Abstract

The invention provides a solution for enabling collaboration between a plurality of augmented reality (XR) devices (XR1, XR2, 100, 200), in particular comprising a system, method and device, the system comprising: a first XR device (XR1, 100) having a first set of features; a second XR device (XR2, 200) having a second set of features; and a hub device (XRH, 300) communicatively connected to the first XR device (XR1, 100) and / or the second XR device (XR2, 200), where the first XR device (XR1, 100) is configured to provide a first set of features, where the second XR device (XR2, 200) is configured to use at least one feature defined in the first set of features and / or a feature of the hub device (XRH, 300).

Description

[0001] The present invention relates to solutions for enabling collaboration among multiple extended reality (XR) devices, and particularly to systems, methods and apparatus for enabling collaboration among multiple extended reality (XR) devices.

[0002] Today, XR devices (such as augmented reality (AR), virtual reality (VR), or mixed reality (MR) devices) are increasingly used, and in most cases are necessary, in collaborative scenarios involving more than two people within AR, VR, and / or MR spaces (hereinafter also referred to as collaborative spaces). Most collaborative use cases guarantee that participants have a similar level of availability for successful collaboration. However, collaborators can have devices with different capabilities, features, and / or availability. In the same collaborative scenario, the difference in hardware capabilities will increase over time as devices with newer, more powerful capabilities are used in parallel with older devices. Therefore, a level playing field is needed, particularly by enhancing the features of lower-end devices in collaborative spaces.

[0003] According to the invention, this is achieved by utilizing the capabilities of other devices present in the collaborative space (particularly devices of the participants in the collaborative space). The solution according to the invention, for example, allows devices with lower computing power (lower-end devices) to outsource computation or part of computation to devices with higher capabilities (higher-end devices) during such collaboration.

[0004] To address this problem, the present invention proposes a system for enabling collaboration among multiple Extended Reality (XR) devices, comprising: a first XR device (XR1) having a first set of features; a second XR device (XR2) having a second set of features; and a hub device (XRH) communicatively connected to the first XR device (XR1) and / or the second XR device (XR2). The first XR device (XR1) is configured to provide the first set of features, in particular, to the hub device (XRH), and the second XR device (XR2) is configured to use at least one of the features defined in the first set of features and / or advantageously use the features of the hub device (XRH).

[0005] The solution according to the invention is characterized in the claims.

[0006] The present invention proposes a method for enabling collaboration between devices, particularly multiple extended reality (XR) devices.

[0007] A first device having a first set of features, particularly a first XR device (XR1),

[0008] as well as

[0009] A second device having a second set of features, particularly a second XR device (XR2),

[0010] in,

[0011] A communication connection is established between the first device, particularly the first XR device (XR1), or the second device, particularly the second XR device (XR2), and the hub interface of the hub device (XRH) or hub device (XRH, 300).

[0012] The communication connection is preferably initiated by a first device, particularly a first XR device (XR1), or a second device, particularly a second XR device (XR2).

[0013] The established communication connection is used to provide, respectively, a first set of features of the first device, particularly the first XR device (XR1), or a second set of features of the second device, particularly the second XR device (XR2), to the hub device (XRH) or the hub interface of the hub device (XRH) and / or to the second device, particularly the second XR device (XR2), or the first device, particularly the first XR device (XR1).

[0014] The established communication connections are respectively used at the second device, particularly the second XR device (XR2), to receive at least one of the features defined in the first feature set and / or the features of the hub device (XRH) from the hub device (XRH) or the hub interface of the hub device (XRH) or the first device, particularly the first XR device (XR1), or at the first device, particularly the first XR device (XR1), to receive at least one of the features defined in the second feature set and / or the features of the hub device (XRH, 300) from the hub device (XRH) or the hub interface of the hub device (XRH) or the second device, particularly the second XR device (XR2).

[0015] as well as

[0016] The second device, particularly the second XR device (XR2), is configured to utilize at least one feature received in the first feature set and / or the features of the hub device (XRH, 300), or the first device, particularly the first XR device (XR1), is configured to utilize at least one feature received in the second feature set and / or the features of the hub device (XRH, 300).

[0017] Advantageously, the method is designed and / or adapted to the system according to the invention.

[0018] Regarding a device having a set of features, particularly an XR device (XR1, XR2), the present invention therefore proposes that the device be designed, configured, and / or set up as

[0019] Establish a communication connection with at least one other device, particularly an XR device (XR2, XR1) and / or a hub device (XRH) or a hub interface of a hub device (XRH).

[0020] The established communication connection is used to provide the hub device (XRH) or the hub interface of the hub device (XRH) with its feature set (the first feature set of XR1 or the second feature set of XR2).

[0021] Receive at least one feature defined in the feature set of the other device, particularly another XR device (receiving the first feature set of XR1 at XR2 or receiving the second feature set of XR2 at XR1), and / or advantageously receive the hub device (XRH).

[0022] and / or

[0023] Utilize at least one feature defined in the feature set of the other device, particularly the other XR device (utilizing the first feature set of XR1 at XR2 or the second feature set of XR2 at XR1) and / or advantageously utilize the features of the hub device (XRH).

[0024] Advantageously, the equipment, particularly the XR equipment (XR1, XR2), is designed and / or adapted to perform the steps performed by the equipment, particularly the XR equipment (XR1, XR2), according to the method according to the invention, and / or is designed and / or adapted to the system according to the invention.

[0025] The XR device according to the invention is any device that actively participates in the XR context without necessarily being worn by a person / user. This could be a sensor attached to a wall or landscape, or a drone mapping terrain. In the sense of the invention, the collection of such devices that can be worn on the body is defined as XR wearable devices. XR wearable devices are a subset of the XR device category. XR wearable devices advantageously include a single sensor and / or stimulator, or a collection of sensors and / or stimulators, which deliver or assist in the XR experience to the user, particularly the end user. XR wearable devices typically have Figure 4 The components shown in the table.

[0026] XR wearable devices can be single devices worn on the head, such as head-mounted displays (HMDs) assisted by handheld controllers, or collections of devices worn on the body. For example, an XR wearable device can be an integrated HMD or a combination of XR glasses and sensors, such sensors and glasses may not coexist in a spatial physical structure / casing. Examples of non-airborne sensors could be sensors that monitor finger muscle movement, where sensor patches / devices are worn as gloves or wristbands / watches. Another example of non-airborne sensors could be a BCI (brain-computer interface) worn on the head, or even a separate depth camera placed at a distance from the body. Essentially, XR devices can be considered as different collections of electronic components intended to collect input and / or deliver stimuli to the user or to engage in an XR experience by supporting other components. The components themselves may not be housed in a single physical container.

[0027] The hub device (XRH) according to the invention – also referred to below as a hyper-realistic hub – advantageously enables collaboration of one or more XR devices. Advantageously, the solution can be an embodiment in the form of a custom device or implemented as an interface in existing XR wearable devices / devices.

[0028] The hub device (XRH) or hub interface according to the invention advantageously allows:

[0029] 1. Distributed computing across participating XR devices; advantageously, each participating XR device advertises its capabilities and margins to other XR devices directly (via an instance of a hub device (XRH) interface) or via a physical instance of a hub device (XRH) to which it is already connected. The least capable XR device can now advantageously utilize the computing power of other XR devices around it to participate in the XR experience. For example, when an XR device must reduce its computing power due to low energy or heat, a participating XR device can refresh its capabilities and announce it to the arbitrator;

[0030] 2. Enhanced Collaborative Features. The primary scenario envisioned is that most XR wearables and XR devices offload computation and / or tracking (partially or completely) to a custom physical instance of a hub device (XRH); advantageously, when multiple XR devices participate in a single experience, some XR devices can enhance their features by leveraging features from other XR devices in the same experience; for example, if other XR devices are suitable for providing hand tracking for users using tablet devices, then users using tablet devices that cannot perform hand tracking can participate in an XR experience with hand tracking according to the invention; this is advantageously accomplished by the hub device (XRH) by combining sensor data from other hand-tracking devices that may have a field of vision of the hands of users using tablet devices; this sensor stream will advantageously enable the user to see his / her / their hands, even though their own devices may not be able to perform hand tracking; participating XR devices can advantageously refresh their capabilities and declare them to an arbitrator, for example, the HMD can detect that the power in the controller is depleted and can no longer be used;

[0031] 3. Real-time communication channel between XR devices and hub devices (XRH); Advantageously, when connected to an XR device, instances of hub devices (XRH) will be able to initiate real-time communication and / or control protocols, such as ROS (Robot Operating System), ExpressLRS, etc., to enable real-time sensor and stimulator communication in addition to protocols such as control, telemetry, positioning, orientation, capability negotiation, and announcement.

[0032] 4. Collab SLAM (Simultaneous Localization and Mapping) for spatial localization and orientation of XR devices; advantageously, it provides collaborative point cloud generation, which can be static and / or dynamic; advantageously, when more than one XR device exists in the same physical space and is connected to a hub device (XRH), the typical challenge of XR occlusion can be addressed by combining sensor inputs from all connected XR devices to form a point cloud using available algorithms such as RANSAC (Random Sample Consensus) and ICP (Iterative Nearest Point); advantageously, with sufficient participation, this point cloud can be made into a dynamic point cloud, thereby dynamically creating a real-world digital twin of virtual objects; the more XR devices involved in the collaborative process, the better the fidelity and resolution of the real-world dynamic point cloud;

[0033] 5. Persistent Anchor Position and Orientation; Advantageously, the hub device (XRH) can access (preferably including but not limited to) volumetric data of the physical space in which it resides; an anchor placed by a user / entity can advantageously be persistent, so that the anchor is now available for other users accessing the physical space at different times. For example, a doctor can place an anchor related to a patient's medication needs. A nurse during his / her rounds can now approach the anchor and administer the medication. In the same example, the anchor can also display the patient's vital statistics. Another example is a museum, where digital anchors are placed on / near exhibits, and customers can now access them as they walk past the exhibits. Anchors can contain a large amount of information, media, and / or associated information that would otherwise require a larger physical space to display. In yet another example, a sorting robot can place anchors overnight in an inbound scenario for human reference. Some anchors (such as directional markers) also need to maintain their orientation and their position, which the hub device (XRH) can advantageously achieve due to its accessible volumetric data;

[0034] 6. Collaborative Occlusion Calculation; Advantageously, the hub device (XRH) has the ability to dynamically generate and / or update volumetric data in multi-user / entity scenarios. This helps the hub device (XRH) know the spatial location of objects at a given time. This information can be advantageously used to calculate occlusion using existing algorithms such as ray tracing, which advantageously helps to hide object points to be discovered, thus improving efficiency. Depending on the update frequency based on use case or user configuration, the hub device (XRH) can use differential dynamic volumetric data to further improve the efficiency of occlusion calculation;

[0035] 7. Collaboration between multiple XRH instances: Advantageously, the real-time protocol can be implemented via physical cables or other wireless interfaces, such as Wi-Fi; advantageously, the hub device (XRH) can be connected to other hub devices (XRH) in various configurations, including but not limited to daisy-chain, star, mesh, and circular topologies, via physical cables (e.g., fiber optic, copper / metal / Ethernet) or wireless interfaces (e.g., Wi-Fi); variations of the hub device (XRH) can also advantageously allow for a flexible platform that can be expanded via hardware or software updates; for example, a version of the hub device (XRH) can provide free time slots for future expansion by adding compute cards to support more XR devices on demand.

[0036] 8. Collaborative (Static and Dynamic) Point Cloud Generation; Point cloud generation for a single XR device can be time-consuming, especially when the area to be covered is large; in such scenarios, compromises are often made on resolution and / or quality. Advantageously, processes (such as cooperative SLAM) exist that can generate point clouds by combining, preferably simultaneously combining, volumetric and / or non-volumetric data from multiple sources. Advantageously, the hub device (XRH) implements not only cooperative SLAM methods but also cooperative dynamic SLAM, where data from various XR devices collaboratively connected to the hub device (XRH) is continuously or periodically updating volumetric data or triggering it. For example, participating XR devices notify the hub device (XRH) to perform motion detection; and / or

[0037] 9. Object Tracking, Anchor Movement, and Anchor Marking. Advantageously, dynamic point cloud generation enables hub devices (XRH) to be time-domain independent and keeps volumetric data and its associated attributes current, tracking objects, movements, and / or changes associated with each volume unit. For example, in the case of points, a volume unit can be a point. Advantageously, this feature can be used to generate floating anchors (virtual) that can be marked / fixed to physical objects. In other words, the anchor's positioning is now determined by the positioning of the object it marks, rather than the anchor's initial position in space. Based on a previous example from a hospital, if an anchor is marked to a patient with details of a required medication, and if the patient is moved to another bed or location, the anchor follows the patient. In another example, an anchor can be marked to a supermarket item, and when the item is moved, the anchor moves with the item due to object tracking and floating anchors.

[0038] According to the invention, XR devices (XR1, XR2) connected to a hub device (XRH) advantageously undergo an initial phase of initialization and registration before commencing regular operation.

[0039] Initialization advantageously involves the XR devices (XR1, XR2) first powering themselves on at the user's command.

[0040] According to a preferred embodiment of the invention, a first XR device (XR1) is configured to provide a first set of features, including the capabilities, features, and / or availability of the first XR device (XR1), to a hub device (XRH) and / or directly to a second XR device (XR2). Advantageously, by means of this, data can be communicated between clients (i.e., the first XR device (XR1) and the second XR device (XR2)) via the hub device (XRH) through a publisher / subscriber model (as an example), which advantageously acts as a central hub to establish communication between clients (i.e., the first XR device (XR1) and the second XR device (XR2)). According to another embodiment of the invention, the hub device (XRH) advantageously also maintains the state of each client (i.e., the first XR device (XR1) and the second XR device (XR2)) and updates other clients in the group accordingly. These states can advantageously include, but are not limited to, registered but out-of-scope devices, device energy states, device privacy states, devices in various states of being registered but in a dormant state, etc. Advantageously, sleep states can be provided to conserve energy, including but not limited to gradually introducing XR devices into increasingly deeper sleep states over time or according to each user's configuration. The range of sleep states advantageously ranges from a reduction in operating frequency to a gradual transition to the complete shutdown of components such as the CPU, GPU, memory, sensors, and / or stimulators.

[0041] Sleep state Sleep-0 can advantageously trigger display dimming, sensor shutdown, and reduction of CPU, GPU, and / or memory frequencies. Sleep state Sleep-1 can advantageously trigger further reductions in the frequencies of the CPU, GPU, memory, and communication chipset, and / or reductions in display brightness. Sleep state Sleep-2 can advantageously trigger "hibernation," which requires shutting down various parts of the CPU, saving volatile data (if any), shutting down most components, and / or preparing the system for hibernation. Sleep state Sleep-3 can advantageously complete the shutdown of the XR device.

[0042] According to a preferred embodiment of the invention, the registration process can be advantageously facilitated via near field communication (NFC), wherein the user taps the XR device (i.e., XR1 or XR2) against the hub device (XRH). Registration advantageously allows for differences in the identifying characteristics of the hub device (XRH), particularly when multiple devices are registered, and identifies the spatial location of the XR device (i.e., XR1 or XR2).

[0043] Alternatively or additionally, the registration process between the XR device (i.e., XR1 or XR2) and the hub device (XRH) can be accomplished through other communication means, including optical or wired information exchange, WiFi, Bluetooth and / or RF protocols that enable such information exchange.

[0044] According to another embodiment of the invention, an XR device (i.e., XR1 or XR2) that has been initialized and registered with the hub device (XRH) and has entered sleep mode (e.g., as part of an energy-saving feature) advantageously reinitializes itself by sensing physical movement of the XR device (i.e., XR1 or XR2), or by being triggered by an onboard sensor of the XR device (i.e., XR1 or XR2), or by a predetermined signal / message from the hub device (XRH). According to another embodiment of the invention, while re-registration is not mandatory, it can advantageously be triggered by the hub device (XRH), preferably depending on various factors such as the time elapsed since the last activity, activity loss over time, etc.

[0045] According to another preferred embodiment, an XR device (i.e., XR1 or XR2) requesting registration with a hub device (XRH) maps its physical environment and submits it to the hub device (XRH). The hub device (XRH) then constructs a volumetric representation of the physical environment, such as a point cloud, in which it operates and tracks the registered XR devices. A point cloud or point cloud map in the sense of this invention is a discrete collection of data points in space, where each point is located with its set of coordinates, i.e., Cartesian coordinates (x, y, z). These points can represent 3D shapes or objects. Point cloud is a well-defined term in the XR field. Advantageously, each of these points can also have its own set of attributes, which can include parameters such as color, brightness, and transparency. These points can also have associated and / or reactive functionalities, such as behavior based on neighboring points, responses to interactions, etc. Therefore, when devices (especially more than one XR device) exist in the same physical space (e.g., a room) connected to a hub device (XRH), the challenge of XR occlusion can be advantageously addressed, for example, by combining sensor inputs from all devices connected to that hub device (especially XR devices (i.e., XR1 or XR2)) and / or advantageously, features of that hub device (XRH) to form a point cloud and a point cloud map created separately by that hub device (XRH). To this end, the hub device (XRH) advantageously utilizes algorithms (such as RANS) AC (Random Sample Consensus) and ICP (Iterative Closest Point)). With sufficient participation from XR devices, including a third XR device (XR3), a fourth XR device (XR4), and / or more XR devices (XRn), point clouds can be advantageously made into dynamic point clouds, and thus advantageously allow the creation of real-world digital twins for interacting with virtual objects. The more XR devices (i.e., XR1, XR2 to XRn) that join this collaborative process with the hub device (XRH), the better the fidelity and resolution of such dynamic point clouds in the real world.

[0046] Advantageously, the virtual space uses inside-out (unmarked / marked) or outside-in trackers, preferably anchored in the physical space via or using SLAM, to determine the physical coordinates of the XR device. When multiple XR devices are used in the same physical space to access the same virtual space, the following cannot be guaranteed:

[0047] a) Individual XR devices (especially in the case of unmarked, inside-out systems) will advantageously lock onto the same mark as other XR devices;

[0048] b) The XR device has the advantage of knowing the relative position and orientation of other XR devices sharing the same physical space, i.e. the direction in which the XR device is oriented;

[0049] c) The spatial anchor generated by the XR device will not drift over time, as illustrated in the following example:

[0050] 1. A spacecraft rotating around an astronaut who is using XR equipment inside or during a spacewalk, where the spacecraft / ship may not provide sufficient coverage for anchoring;

[0051] 2. Underwater exploration using XR equipment where visual markers may be scarce, such as in underwater oil drilling rig training / maintenance;

[0052] 3. Outdoor situations where visual markers are scattered and sparse, such as military training in a desert environment;

[0053] d) XR devices are advantageous for creating and detecting anchors in low-contrast or low-light spaces / environments;

[0054] e) XR devices are advantageous for creating and detecting anchors in dynamic spaces / environments such as busy conference rooms, trade shows, outdoor gatherings, etc.

[0055] f) XR devices can advantageously create and detect anchors in repetitive / identical spaces / environments, preferably visually or otherwise; for example, most meeting rooms in an office building may follow the same dimensions and decor;

[0056] g) When many other people wearing XR devices are in the same physical space (e.g., a classroom full of students, a concert, a trade show, or any gathering of people), XR devices can advantageously minimize occlusion.

[0057] This is equally advantageous when using multiple XR devices in different physical spaces to access the same virtual space, or when using several XR devices in the same physical space to access different virtual spaces.

[0058] Another embodiment of the invention optionally provides, for example, that if the current mapping information is found to be insufficient in terms of completeness and / or resolution, any additional XR devices that have registered with the hub device (XRH) also perform mapping exercises. This advantageously allows the hub device (XRH) to add information to the point cloud map, such as filling gaps, updating or increasing the density or resolution of existing point cloud maps stored in the hub device (XRH).

[0059] Optionally, another embodiment of the invention provides that point cloud maps are represented and stored as polygons or voxels or a combination thereof. Preferably, the hub device (XRH) optionally provides the user of the XR device with the option to store the point cloud map in a binary / multiple / all-select model. Each of these representations of the surrounding spatial features offers advantages / trades depending on the nature of the surroundings, available storage space, and / or computational requirements. In the binary / multiple / all-select model, the hub device (XRH) advantageously decides where to store the map and / or associated dates, preferably in local storage within the physical environment or in cloud storage. This decision is advantageously pre-configured, user-defined, dynamic, or AI-generated (AI: Artificial Intelligence). The decision regarding where to store the generated map depends on various factors, including but not limited to the nature of the application, latency requirements, security requirements, privacy requirements, asset management, digital licenses, etc.

[0060] Advantageously, the hub device (XRH) acts as a feature transmission medium between XR devices (i.e., XR1, XR2 to XRn) that participate in the collaborative process with the hub device (XRH), for example, acting as the world origin of the environment scanned by a capable XR device (i.e., XR1). A first XR device (XR1) (e.g., representing a high-end device with advanced hardware capabilities) contacts the hub device (XRH), registers itself with the hub device (XRH), and scans the environment. A second XR device (XR2) (e.g., representing a low-end device with advanced hardware capabilities) can also register itself with the hub device (XRH), but is limited to a lower-quality mapping by its hardware (e.g., its camera, LIDAR (light imaging, detection, and ranging), etc.). Since the hub device (XRH) acts as the origin of the point cloud map, it advantageously transfers the higher quality point cloud map obtained from / together with the first XR device (XR1) to the second XR device (XR2), and thus advantageously enhances the feature set of the second XR device (XR2), and it also has lower hardware capabilities, such as regarding its camera, LiDAR, etc.

[0061] In another embodiment of the invention, the hub device (XRH) is advantageously configured to receive location-related information from a first XR device (XR1) to create a point cloud map including the locations of the first XR device (XR1) and the second XR device (XR2). This is advantageous because, for example, a low-end device (i.e., a second XR device (XR2) that may not have built-in location and / or mapping capabilities and may require certain capabilities in the collaborative space) can then communicate with the hub device (XRH) to obtain location-related information from it, particularly including the locations of the first XR device (XR1) and the second XR device (XR2) in the point cloud map. For example, if the second XR device (XR2) wants to present objects(s) at a specific spatial location to its user, it can query the hub device (XRH) using the currently captured image as input. In response to the query, the hub device (XRH) will locate the second XR device (XR2) based on the stored point cloud map and transmit the queried features (image) to the second XR device (XR2). This advantageously provides collaboration and distribution. The stored point cloud map of the hub device (XRH) is precisely a high-quality point cloud map because it is created from sensor input from the first XR device (XR1). Optionally or additionally, the hub device (XRH) may store the point cloud map in a central repository or outside the physical range of the hub device (XRH) itself, but in a location it can access and recall when needed.

[0062] According to another preferred embodiment, the hub device (XRH) is therefore also configured to: receive images or image streams captured by the second XR device (XR2); determine the location of the second XR device (XR2) based on the images or image stream; and provide the determined location to the second device (XR2). Thus, the second XR device (XR2) can now determine its location without having the corresponding functionality itself. In the presence of more than one high-end XR device (i.e., similar to the first XR device (XR1)), all such XR devices can provide the acquired point cloud map data to the hub device (XRH), which uses the acquired point cloud map data to create a global point cloud map. The hub device can use various SLAM algorithms, such as ARKit, ARCore, Orb SLAM 3, etc.

[0063] In another preferred embodiment of the invention, optionally or additionally, LIDAR, SONAR, TOF and / or other mechanisms and / or sensors may be advantageously used as alternatives to and / or in combination with images or image streams for positioning and / or mapping.

[0064] In another preferred embodiment of the invention, where the first XR device (XR1) and the second XR device (XR2) have similar capabilities, features, and / or availability, the mapping data obtained from the second XR device (XR2) is used to update and / or enhance the point cloud map in terms of accuracy, density, and / or filling gaps that the previous mapping may have missed. Advantageously, the data generated by the first XR device (XR1) and the second XR device (XR2) can be enhanced by using implementations such as OctoMap (Hornung, A., Wurm, KM, Bennewitz, M. et al., "OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees," Autonomous Robotics, Vol. 34, pp. 189-206, 2013)).

[0065] Advantageously, if there is a temporal difference between the maps generated by the first XR device (XR1) and the second XR device (XR2), the data difference is used to locate and identify objects that have moved. If the frequency is appropriately increased to match the application, a dynamic volumetric map with dynamic object movement mapping and prediction is achieved. Further advantageously, an implementation of trajectory prediction, well-known in the field of computer vision, preferably kernel-based tracking and / or contour tracking, can be employed.

[0066] In another preferred embodiment of the invention, where the first XR device (XR1) and the second XR device (XR2) have different sensing capabilities, the mapping data obtained from the second XR device (XR2) is used to update and / or enhance the point cloud map in terms of accuracy, density, and / or filling gaps that the previous mapping may have missed. This is advantageously achieved using widely available multi-sensor SLaM (Simultaneous Localization and Mapping) processes and / or algorithms. Multi-sensor SLaM advantageously requires combining the outputs of various sensors such as LiDAR, IR, UV, visible spectral cameras, touch-based sensors, etc., to generate a spatial map with higher fidelity and resolution than that that can be generated using a single sensor. This can be advantageously achieved using existing well-known multi-sensor SLaM algorithms, such as loosely coupled and tightly coupled systems based on LiDAR-IMU.

[0067] In another preferred embodiment of the invention, in the case where the first XR device (XR1) is a high-end device with higher features and capabilities and the second XR device (XR2) is a low-end device with fewer features and capabilities, if the second XR device (XR2) does not have hand tracking features, the hub device (XRH) can still assign the hand tracking feature to the second XR device (XR2) as long as the user's hand is within the field of view of the first XR device (XR1). This is advantageously achieved by the hub device (XRH) by processing sensor data from the first XR device (XR1), locating the user's hand using the second XR device (XR2), and bringing the hand tracking feature to the second XR device (XR2). Therefore, the system according to the invention advantageously realizes a hand tracking feature for the second XR device (XR2) that has so far been impossible due to hardware limitations.

[0068] Another embodiment of the invention is characterized in that the hub device (XRH) is configured to receive sensor data from a first XR device (XR1) to provide a second XR device (XR2) with at least one feature defined in a first feature set.

[0069] Advantageously, the improved quality and / or performance is achieved by combining data from multiple sensors across all connected XR devices. This ensures that users who may not be within the tracking range of one XR device but are within the range of other participating XR devices can be tracked, thus collaboratively compensating for occlusion.

[0070] Another advantageous embodiment of the invention provides that sensor data from only a few XR devices can be selected for processing, depending on capabilities, computing power requirements, energy optimization, range, user configuration, application requirements, etc.

[0071] In another preferred embodiment of the invention, the hub device (XRH) itself may also have sensors or sensor arrays to track state changes in its physical environment and in its vicinity / range of operation and extend various features to all XR devices registered to it. This advantageously includes: the aforementioned user posture, user facial movement, hand tracking, user body movement, XR device movement, changes in the state of light, sound, VOCs, heat, moisture, gas composition and content, and / or RF (radio frequency) in its vicinity / range of operation.

[0072] According to a preferred embodiment, when the second XR device (XR2) registers with the hub device (XRH), it will be notified that it can optionally enable a set of features – which are either not present in the second XR device (XR2) itself or exist together with the second XR device (XR2) itself.

[0073] In another embodiment of the invention, some features that are not present in the two XR devices (the first XR device (XR1) and the second XR device (XR2)) can be advantageously enabled by the system, particularly via a hub device (XRH).

[0074] A preferred application of this type of feature enhancement is facial expression tracking, where the user's face is within the range or field of view of one of the participating XR devices. Facial expression tracking, also known as face tracking, advantageously includes algorithms for detecting and / or tracking faces and their facial features (e.g., eyes, eyebrows, eyelids, pupils, nose, lips, and / or contours) in images and / or videos from cameras, sensors, or color, grayscale, and / or infrared, millimeter-wave (mm: millimeter), particularly near-infrared video files. It advantageously allows for the tracking of one or more faces and / or their facial features, such as 3D head pose, the position of facial landmarks, gaze direction, etc. Thus, features that were previously unavailable in any participating XR device are achieved collaboratively. Therefore, participating XR devices can track all motion-related features. This enables features such as real-time expression tracking to be achieved in individual XR devices without the need for additional sensors.

[0075] Another preferred application of such feature enhancement is body movement, posture, gesture, movement of other people, movement of objects, movement of other XR devices participating in the same physical space and connected to the hub device (XRH), and movement of other XR devices participating in the same physical space but not attached to / connected to the same hub device (XRH). Essentially, any movement and / or change that can be captured by another XR device includes light, sound, VOCs, heat, moisture, gas composition and / or content, and / or radio frequency (WiFi, Bluetooth, etc.).

[0076] According to another advantageous embodiment of the invention, the first XR device can provide the functions and features of a hub device (XRH). Therefore, advantageously, all the features of the hub device XRH are implemented in the first XR device (XR1). Advantageously, the first XR device (XR1) functions as a hub device (XRH) in addition to being an XR device (i.e., XR1) as part of the system according to the invention. Therefore, the solution according to the invention advantageously includes at least two devices, particularly XR devices (XR1, XR2), one of which provides a hub interface or a hub device (XRH). Another embodiment of the invention proposes implementing a simplified set of features of the hub device (XRH) in the first XR device (XR1). This achieves the limited set of features and advantages of the system according to the invention.

[0077] According to a preferred aspect of the invention, the first XR device (XR1) and the second XR device (XR2) are provided to be located in the same environment, such as physical space and virtual space. This is particularly advantageous because, regardless of whether it is VR / MR or AR, the physical projection into the digital world will be the same except for the viewpoint.

[0078] According to another embodiment of the invention, the system is provided to provide the aforementioned features and advantages, regardless of whether the first XR device (XR1) and the second XR device (XR2) are involved in different extended realities (AR / VR / MR), as long as the first XR device (XR1) and the second XR device (XR2) are prohibited from the same physical space, so that a sensor from one XR device can monitor desired movement / changes in the other XR device. These features include endowing one or more of the following: pose tracking, eye tracking, face tracking, hand tracking, motion tracking, etc.

[0079] In another embodiment, when several XR devices are used in the same physical space to access different virtual spaces, it is not guaranteed that each XR device will latch onto the same markers or anchors as other XR devices, or that the XR devices will be aware of the relative positions and orientations of other XR devices sharing the same physical space. In this scenario, the hub device (XRH) facilitates that XR devices sharing the same physical space have a common spatial reference / point cloud map in which they can operate, regardless of the XR devices participating in different virtual spaces.

[0080] According to another preferred embodiment of the invention, the spatial map of the physical space is dynamically refreshed to accommodate changes in the physical space, which are now advantageously available for reflection / projection in the respective virtual spaces of the participating XR devices. As an example, suppose a first XR device (XR1) and a second XR device (XR2) are head-mounted displays (HMDs) operating in the same physical space, but users participate in different virtual spaces, and the user of the first XR device (XR1) moves a new physical object into the physical space in which both the first XR device (XR1) and the second XR device (XR2) operate. This new physical object needs to be reflected / projected, preferably in a form suitable for the experience of the user of the second XR device (XR2), and the user is also immersed in the experience of the second XR device (XR2), so that the second XR device (XR2) is aware of it. This reduces the risk of users of the XR devices unintentionally interacting with physical objects dynamically presented in the physical environment. These physical objects could also be other users of the XR devices.

[0081] Advantageously, the hub device (XRH) is configured to provide a real-time communication channel toward a first XR device (XR1) and / or a second XR device (XR2). For example, an instance of the hub device (when connected to an XR device) may be able to initiate real-time communication and / or control protocols (such as ROS, ExpressLRS, etc.) to enable real-time sensor and stimulator communication in addition to control, telemetry, location, orientation, capability negotiation, announcement, etc.

[0082] According to another preferred embodiment, the first XR device (XR1) and / or the second XR device (XR2) are communicatively connected to the hub device (XRH) via at least one of the following: wireless connectivity including Wi-Fi, Bluetooth, and / or NFC; and wired connectivity including fiber optic, metal, and Ethernet. For example, the real-time protocol can be implemented using a physical cable or a wireless interface such as Wi-Fi. Furthermore, the hub device (XRH) can also be connected to other hub devices (XRH) via physical cables or via wireless interfaces in various topologies, including but not limited to daisy-chain, star, mesh, and circular topologies.

[0083] Variations of the Hub Device (XRH) can also advantageously allow for a flexible platform that can be expanded via hardware or software updates. For example, one version of the Hub Device (XRH) can provide free time slots to support additional XR devices on demand, allowing for future expansion by adding compute cards.

[0084] According to another aspect of the invention, the second XR device (XR2) is configured to provide a second set of features, and the first XR device (XR1) is configured to use at least one of the features defined in the second set of features. This has the advantage that, in cases where both XR devices are high-end devices but have different specific features, the features can be used for both XR devices involved. Accordingly, a hub device (XRH) can be used for the mutual exchange of features between the first XR device (XR1) and the second XR device (XR2).

[0085] Advantageously, the second XR device (XR2) is configured to provide a second set of features to the hub device (XRH) and / or directly to the first XR device (XR1). This allows the first XR device (XR1) and the second XR device (XR2) to subscribe to each other's features.

[0086] Preferably, the first XR device (XR1) and / or the second XR device (XR2) are wearable devices. Wearable devices in the sense of this invention can be a single device worn on the head, such as a head-mounted display (HMD) assisted by a handheld controller, or a collection of multiple devices worn on the body. For example, a wearable device can be a combination of an integrated HMD or XR glasses and sensors, such sensors and such glasses may not coexist in a spatial physical structure / casing. Examples of such non-airborne sensors can be in the form of sensors that monitor finger muscle movement, where the sensor patch / device is worn as a glove or wristband / watch, held in the hand or wristband / watch. Another example of a non-airborne sensor can be a brain-computer interface (BCI) worn on the head, or even a separate depth camera placed at a distance from the body. Thus, such XR devices can be considered as different collections of electronic components intended to collect input and deliver stimuli to the end user, where the components themselves may not be housed in a single physical container.

[0087] According to another embodiment of the present invention, the first XR device (XR1) and / or the second XR device (XR2) are: augmented reality (AR) devices, virtual reality (VR) devices, or mixed reality (MR) devices.

[0088] According to another embodiment, XR devices that can be connected to a hub device (XRH) are not limited to human wearable devices. These devices can be advantageously simple, such as webcams, drones, home or industrial appliances, sensors, monitors, smart structures, medical equipment, etc. In essence, any physical object that needs to be represented in the XR world is considered an XR device.

[0089] Preferably, the hub device (XRH) includes a computing section, a communication section, and a platform section. The computing section advantageously includes integrated computing elements such as GCPU (General Purpose CPU (Central Processing Unit)), GPU, AI (Artificial Intelligence), and ML (Machine Learning) components to provide an XR platform for serving many different pairs of XR devices and other non-wearable devices (typically non-airborne sensors). The communication section of the hub device (XRH) can be broadly divided into a backend and a frontend. The backend communication section attempts to ensure that the hub device (XRH) is connected to the internet as needed via various means (e.g., via 5G cellular base stations, satellite connections, another hub via Ethernet / optical networks, etc.). The backend communication section is also advantageously responsible for connecting to other hub devices (XRH) via various technologies (e.g., 5G, Ethernet, Wi-Fi, etc.). The frontend communication section connects to all XR devices and other related hardware required to deliver the XR experience. The platform section advantageously includes mechanisms for controlling and managing several functions, preferably:

[0090] - Create computational models of virtual or digital equivalents of physical objects juxtaposed in the physical domain; the data used to create the computational models of the physical objects can advantageously come from links connected to sets.

[0091] XR equipment for line-of-sight (XRH) devices;

[0092] - Ensure synchronization between backend and frontend communication based on each use case;

[0093] - Ensure the waiting time requirements for XR devices are met;

[0094] - Provides an abstraction layer for applications (e.g., operating systems) to be developed or deployed.

[0095] Further details, features, and / or advantages of the present invention will be explained with reference to the following detailed description of embodiments of the invention shown in the accompanying drawings.

[0096] in:

[0097] Figure 1 A flowchart illustrating an embodiment of the interconnection in a system including a first XR device (XR1), a second XR device (XR2), and a hub device (XRH);

[0098] Figure 2 A block diagram illustrating an embodiment utilizing global SLAM located on a hub device (XRH);

[0099] Figure 3 A flowchart illustrating an embodiment of the initial phase of initialization and registration of an XR device and hub device (XRH) according to the present invention; and

[0100] Figure 4 A table showing the components included in an XR device, particularly an XR wearable device, according to the present invention.

[0101] Figure 1 An embodiment of the interconnection in a system including a first XR device 100, a second XR device 200, and a hub device 300 is illustrated. In a first step S11, the second XR device 200 (e.g., a low-end device) may subscribe to the hub device 300 to be notified of a first set of features that can be provided to the hub device 300 by the first XR device 100.

[0102] When the first XR device 100 (e.g., a high-end device) has provided its features to the hub device 300 in step S12, the hub device 300 subsequently notifies the second XR device 200 of the features due to the previous subscription. Alternatively, it is foreseeable that the first XR device 100 directly notifies the second XR device 200 of the available features.

[0103] According to embodiments, particularly if there are more than two devices, each participating XR device can advertise its capabilities and margins to other XR devices directly (i.e., via an instance of the hub device interface) or via a physical instance of the hub device it is already connected to. Less capable XR devices (e.g., low-end devices) can now leverage the computing, tracking, and / or other capabilities of the XR devices around them to participate in the XR experience.

[0104] Subsequently, the second XR device 200 may transmit a feature request to the hub device 300 in step S14. The request relates to a feature that has been previously notified to the second XR device 200, namely a feature that is available only on the first XR device 100 but not on the second XR device 200.

[0105] Subsequently, in step S15, hub device 300 notifies first XR device 100 that it has requested the device's features. For example, a user of second XR device 200, which cannot perform hand tracking itself, wants to participate in an XR experience with hand tracking. For this purpose, first XR device 100 lends itself (i.e., its corresponding features) to provide hand tracking for second XR device 200. According to step S16, this is accomplished by hub device 300 by combining sensor data from first XR device 100, which has the user's hand in its field of view. The corresponding sensor stream obtained by first XR device 100 will enable the user of second XR device 200 to see his / her hand, even though the user's own second XR device 200 cannot perform hand tracking.

[0106] The hub device 300 may include a computing section 305, a communication section 310, and a platform section 315.

[0107] The computing unit 305 may include a general-purpose central processing unit (GCPU), a graphics processing unit (GPU), an artificial intelligence (AI) unit, and a machine learning (ML) component integrated together to provide an XR platform for serving many different pairs of XR devices (i.e., not just a pair of first XR devices 100 and second XR devices 200) as well as other non-body wearable devices (such as non-airborne sensors).

[0108] The communication portion 310 of hub device 300 can be broadly divided into a backend and a frontend. The backend ensures that hub device 300 connects to the Internet, for example, using cellular networks (such as 5G), satellite connections, or another hub via Ethernet / optical networks. The backend can also be responsible for connecting to other hub devices via various technologies including 5G, Ethernet, and Wi-Fi. Conversely, the frontend connects to all XR devices, such as the first XR device 100 and the second XR device 200, as well as other related hardware required to deliver the XR experience.

[0109] Finally, platform portion 315 includes software that manages several functions, including the creation of computational models, wherein data for creating such computational models of real-world physical objects may come from XR devices connected to hub device 300. Furthermore, platform portion 315 includes software that ensures synchronization between backend and frontend communications and the latency requirements of XR devices, for example, by providing an abstraction layer for applications to be developed and deployed.

[0110] Figure 2 An embodiment is shown in which global SLAM located on hub device 300 is used to provide location data to a low-end device (i.e., a second XR device 200).

[0111] according to Figure 2 In one embodiment, the Robot Operating System (ROS) can be used as middleware to allow communication between the hub device 300 and the AR devices 100 / 200. This is represented by dashed lines.

[0112] The hub device 300 can act as a ROS master to establish communication between clients (i.e., the first XR device 100 and the second XR device 200). As described above, a publisher / subscriber model can be used to communicate data between clients.

[0113] In addition, from Figure 2 It is evident that a global simultaneous localization and mapping (SLAM) exists on hub device 300, storing maps from all devices capable of providing corresponding data (i.e., the first XR device 100 (which is the high-end device)). Accordingly, in Figure 2 In step S21, the first XR device 100 provides such data to the hub device 300.

[0114] When the low-end device (i.e., the second XR device 200) needs the corresponding capability, it communicates with the hub device 300 to obtain it, as indicated in step S22. For example, if the second XR device 200 wants to render an object in a specific world pose, it can query the global map from the hub device 300, taking the current image (i.e., the image captured by the second XR device 200) as input.

[0115] In response, hub device 300 locates the second XR device 200 based on a map created from data retrieved from other devices such as the first XR device 100 (i.e., high-end devices), and thereby transmits corresponding features to the second XR device 200 through cooperation and distribution. Therefore, although the second XR device 200 does not have its own SLAM capabilities, it can still determine its own location.

[0116] according to Figure 3 The flowchart illustrates an embodiment of the initial phase of initializing and registering an XR device with a hub device (XRH) according to the present invention. On the XR device side, the XR device taps the hub device (XRH) to register itself and scan the environment. On the hub device (XRH) side, the tap serves as an initial alert. The hub device (XRH) determines whether the XR device has previously registered with it. If not, the hub device (XRH) queries the XR device to provide and exchange a first set of features, specifically including the XR device's capabilities and location. If the XR device has previously registered with the hub device (XRH), the hub device (XRH) determines whether the map / point cloud map used for collaboration needs updating. If the software or application hosted on the hub device (XRH) deems a map update necessary, the hub device (XRH) requests the registered XR device to update the map. Typically, this is the message conveyed to the user of the XR1, referred to as the higher-capability device XR1 in the example presented here. The XR device, then with the assistance of its user, will scan its surroundings and exchange the scanned data with the hub device (XRH). If map updates are not required or have already been updated, the hub device (XRH) exchanges session data, parameters, and protocols used within the session, thereby completing the XR device's registration with that hub device (XRH).

[0117] Figure 4 A table showing components of an XR device, particularly an XR wearable device, according to the invention is illustrated. These advantageously include a collection of sensors and stimulators for delivering an XR experience to the end user.

[0118] The embodiments shown in the accompanying drawings and the embodiments explained therein, in particular Figure 2 The examples of messages in the text are only used to explain the solutions presented according to the present invention and are not limited thereto.

[0119] Figure Labels

[0120] 100 First XR Device (XR1)

[0121] 200 Second XR device (XR2)

[0122] 300 Hub Equipment (XRH)

[0123] Calculation section of 305 hub device (XRH)

[0124] Communication section of the 310 hub device (XRH)

[0125] Platform portion of the 315 hub device (XRH)

[0126] S11 Method Steps

[0127] S12 Method Steps

[0128] S13 Method Steps

[0129] S14 Method Steps

[0130] S15 Method Steps

[0131] S16 Method Steps

[0132] S21 Method Steps

[0133] S22 Method Steps

[0134] S23 Method Steps

Claims

1. A system for enabling collaboration among multiple extended reality XR devices (XR1, XR2, 100, 200), the system comprising: A first XR device (XR1, 100) having a first set of features; A second XR device (XR2, 200) having a second set of features; as well as A hub device (XRH, 300) communicatively connected to the first XR device (XR1, 100) and / or the second XR device (XR2, 200). The first XR device (XR1, 100) is configured to provide the first feature set. The second XR device (XR2, 200) is configured to use at least one feature defined in the first feature set and / or a feature of the hub device (XRH, 300).

2. The system of claim 1, wherein the first XR device (XR1, 100) is configured to provide the first feature set to the hub device (XRH, 300) and / or directly to the second XR device (XR2, 200).

3. The system of claim 2, wherein the second XR device (XR2, 200) is configured to subscribe to the hub device (XRH, 300) to be notified of the first feature set when the first feature set is provided to the hub device (XRH, 300).

4. The system of any one of claims 1 to 3, wherein the hub device (XRH, 300) is configured to receive sensor data from the first XR device (XR1, 100) to provide the second XR device (XR2, 200) with the at least one feature defined in the first feature set.

5. The system of claim 4, wherein the first XR device (XR1, 100) and the second XR device (XR2, 200) are located in the same environment.

6. The system according to any one of claims 1 to 5, wherein the hub device (XRH, 300) is configured to receive location-related information from the first XR device (XR1, 100) to create a map, preferably a point cloud map, including the locations of the first XR device (XR1, 100) and the second XR device (XR2, 200).

7. The system of claim 6, wherein the hub device (XRH, 300) is further configured to: Receive images and / or image streams captured by the second XR device (XR2, 200) from the second XR device (XR2, 200). The location of the second XR device (XR2, 200) is determined based on the image and / or image stream, and The determined position is provided to the second XR device (XR2, 200).

8. The system of any one of claims 1 to 7, wherein the hub device (XRH, 300) is configured to provide a real-time communication channel toward the first XR device (XR1, 100) and / or the second XR device (XR2, 200).

9. The system of any one of claims 1 to 8, wherein the first XR device (XR1, 100) and / or the second XR device (XR2, 200) are communicatively connected to the hub device (XRH, 300) via at least one of the following: Wireless connectivity, including Wi-Fi, Bluetooth, and / or NFC; and / or Wired connection, including fiber optic, metal and / or Ethernet.

10. The system of any one of claims 1 to 9, wherein the hub device (XRH, 300) is configured to control the first XR device (XR1, 100) and / or the second XR device (XR2, 200) to operate collaboratively in point cloud generation.

11. The system of any one of claims 1 to 10, wherein the second XR device (XR2, 200) is configured to provide a second feature set, and the first XR device (XR1, 100) is configured to use at least one feature defined in the second feature set.

12. The system of claim 11, wherein the second XR device (XR2, 200) is configured to provide the second feature set to the hub device (XRH, 300) and / or directly to the first XR device (XR1, 100).

13. The system of any one of claims 1 to 12, wherein the first XR device (XR1, 100) and / or the second XR device (XR2, 200) are wearable devices.

14. The system according to any one of claims 1 to 13, wherein the first XR device (XR1, 100) and / or the second XR device (XR2, 200) are: Augmented Reality (AR) devices Virtual reality (VR) devices, or Mixed Reality (MR) devices.

15. The system according to any one of claims 1 to 14, wherein the hub device (XRH, 300) includes a computing section (305), a communication section (310), and a platform section (315).

16. The system of one or more of claims 1 to 15, wherein the hub device (XRH, 300) itself has a sensor or sensor array to track state changes in its physical environment and in the vicinity / range of its operation and extends various features to all XR devices (XR1, XR2, 100, 200) registered to it.

17. The system of one or more of claims 1 to 16, wherein the hub device (XRH, 300) provides facial expression tracking, particularly as long as the face of the user of the XR device is within the range or field of view of one of the participating XR devices (XR1, XR2, 100, 200).

18. The system of claim 17, comprising: User attitude; User facial movement; hand tracking; User body movement; XR device movement; changes in the state of light, sound, VOCs, heat, moisture, gas composition and content; and / or RF (radio frequency) in the vicinity / range of its operation.

19. The system of one or more of claims 1 to 18, wherein the first XR device (XR1, 100) is capable of providing the functions and features of the hub device (XRH, 300).

20. The system of one or more of claims 1 to 19, wherein the features of the hub device (XRH, 300) are implemented in the first XR device (XR1, 100).

21. The system of one or more of claims 1 to 20, wherein the first XR device (XR1, 100) serves as a hub device (XRH, 300) in addition to being part of the system.

22. A method for enabling collaboration between devices, particularly multiple extended reality (XR) devices. A first device having a first set of features, particularly a first XR device (XR1, 100), as well as A second device having a second set of features, particularly a second XR device (XR2, 200), in, In the first device, particularly the first XR device (XR1, 100), or the second device, particularly the second XR device (XR2, 200), and the hub device (XRH) or hub device (XRH, Establish a communication connection between the hub interfaces of 300) The communication connection is preferably initiated by the first device, particularly the first XR device (XR1, 100), or the second device, particularly the second XR device (XR2, 200); The established communication connection is used to provide the first set of features of the first device, particularly the first XR device (XR1, 100), or the second set of features of the second device, particularly the second XR device (XR2, 200), to the hub device (XRH, 300) or the hub interface of the hub device (XRH, 300) and / or to the second device, particularly the second XR device (XR2, 200). The established communication connection is used for receiving, at the second device, particularly the second XR device (XR2), at the hub device (XRH) or the hub interface of the hub device (XRH) or the first device, particularly the first XR device (XR1), at least one feature defined in the first feature set and / or a feature of the hub device (XRH), or for receiving, at the first device, particularly the first XR device (XR1), at the hub device (XRH) or the hub interface of the hub device (XRH), or for receiving, at the second device, particularly the second XR device (XR2), at at least one feature defined in the second feature set and / or a feature of the hub device (XRH, 300), and... The second device, particularly the second XR device (XR2, 200), is configured to utilize at least one feature defined in the first feature set and / or the features of the hub device (XRH, 300), or the first device, particularly the first XR device (XR1, 200), 100) is configured to utilize at least one feature defined in the second feature set and / or the features of the hub device (XRH, 300).

23. The method of claim 22, wherein it is designed and / or adapted to a system according to one or more of claims 1 to 21.

24. A hub device (XRH, 300) for enabling collaboration between devices, particularly multiple extended reality (XR) devices. A first device having a first set of features, particularly a first XR device (XR1, 100), as well as A second device having a second set of features, particularly a second XR device (XR2, 200), The hub device (XRH, 300) Allows the establishment of communication connections to the first device (XR1, 100) and / or the second device (XR2, 200). Designed and / or adapted to receive information, respectively, of at least a portion of the first feature set of the first device (XR1, 100) and the second feature set of the second device (XR2, 200). And / or Information is designed and / or adapted to transmit at least a portion of the first set of features of the first device (XR1, 100) and the second set of features of the second device (XR2, 200), respectively.

25. The hub device (XRH, 300) of claim 24, wherein the hub device (XRH, 300) is configured to receive sensor data from the first device (XR1, 100) to provide the second device (XR2, 200) with the at least one feature defined in the first feature set.

26. The hub device (XRH) according to claim 24 or claim 25, wherein the hub device (XRH, 300) is configured to receive location-related information from the first device (XR1, 100) to create a map, preferably a point cloud map, including the locations of the first device (XR1, 100) and the second device (XR2, 200).

27. The hub device (XRH, 300) according to one or more of claims 14 to 26, wherein the hub device (XRH, 300) is configured to: Receive images and / or image streams captured by the second device (XR2, 200) from the second device (XR2, 200). The location of the second device (XR2, 200) is determined based on the images and / or image stream, and The determined position is provided to the second device (XR2, 200).

28. The hub device (XRH, 300) according to one or more of claims 24 to 27, wherein the hub device (XRH, 300) is configured to provide a real-time communication channel toward the first device (XR1, 100) and / or the second device (XR2, 200).

29. The hub device (XRH, 300) according to one or more of claims 24 to 28, wherein the hub device (XRH, 300) is configured to control the first device (XR1, 100) and / or the second device (XR2, 200) to operate collaboratively in point cloud generation.

30. The hub device (XRH, 300) according to one or more of claims 24 to 29, the hub device (XRH, 300) comprising a computing section (305), a communication section (310) and a platform section (315).

31. The hub device (XRH, 300) according to one or more of claims 24 to 30, wherein the hub device (XRH, 300) itself has a sensor or sensor array to track state changes in its physical environment and in the vicinity / range of its operation and extends various features to all devices registered to it, particularly XR devices (XR1, XR2, 100, 200).

32. The hub device (XRH, 300) according to one or more of claims 24 to 31, wherein the hub device (XRH, 300) provides facial expression tracking, particularly as long as the face of the user of the device, particularly the XR device, is within the range or field of view of one of the participating devices, particularly the XR devices (XR1, XR2, 100, 200).

33. The hub device (XRH, 300) according to claim 32, comprising: User attitude; User facial movement; hand tracking; User body movement; XR device movement; changes in the state of light, sound, VOCs, heat, moisture, gas composition and content; and / or RF (radio frequency) in the vicinity / range of its operation.

34. The hub device (XRH, 300) according to one or more of claims 24 to 34, wherein the hub device (XRH, 300) is designed and / or adapted to perform the steps of the method according to claim 22 or claim 23 performed by the hub device (XRH, 300), and / or is designed and / or adapted to the system according to one or more of claims 1 to 21.

35. A device having a set of features, particularly an XR device (XR1, XR2, 100, 200), said device being designed, configured and / or set up as Establish a communication connection with at least one other device, particularly an XR device (XR2, XR1, 200, 100) and / or a hub device (XRH, 300) or a hub interface of a hub device (XRH, 300). The established communication connection is used to provide the hub device (XRH, 300) or the hub interface of the hub device (XRH, 300) with its feature set (the first feature set of XR1 or the second feature set of XR2). Receive at least one feature defined in the feature set of the other devices, particularly the other XR devices (receiving the first feature set of XR1 at XR2 or the second feature set of XR2 at XR1), and / or advantageously receive features of the hub device (XRH, 300). and / or Utilize at least one feature defined in the feature set of the other devices, particularly the other XR devices (utilizing the first feature set of XR1 at XR2 or the second feature set of XR2 at XR1) and / or advantageously utilize the features of the hub device (XRH, 300).

36. The device according to claim 35, particularly the XR device (XR1, XR2), wherein the device is designed and / or adapted to perform the steps of the method according to claim 22 or claim 23 performed by the device, particularly the XR device (XR1, XR2, 100, 200), and / or is designed and / or adapted to the system according to one or more of claims 1 to 21.

37. The device according to claim 35 or claim 36, particularly the XR device (XR1, XR2), the device providing a hub interface of a hub device (XRH, 300) or hub device (XRH, 300) according to one or more of claims 24 to 34.

38. The device of claim 37, wherein the device provides the functions and features of the hub device (XRH, 300).

39. The device according to one or more of claims 35 to 38, wherein the features of the hub device (XRH, 300) are implemented in the device (XR1, XR2, 100, 200).

40. The device as claimed in one or more of claims 35 to 39, wherein the device (XR1, XR2, 100, 200) functions as a hub device (XRH, 300) in addition to being part of the system as claimed in one or more of claims 1 to 21.

Images