A multi-space real-time object synchronization method based on MR glasses

Abstract

The present disclosure relates to a multi-space real-time object synchronization method based on MR devices, comprising: establishing a P2P communication connection between multiple MR devices; establishing a local SLAM point cloud and extracting a set of ORB feature points; exchanging the extracted set of ORB feature points between the multiple MR devices to determine a coordinate transformation matrix between the multiple MR devices, and performing space alignment and establishing a global space reference system based on the coordinate transformation matrix; establishing a virtual scene including at least one virtual object; continuously updating local state information at each MR device; when the change in the local state information meets a preset condition, determining state update information based on the change in the local state information, and transmitting the state update information between the multiple MR devices; and at each MR device, rendering the at least one virtual object in the virtual scene based on the local state information and the state update information from other MR devices.

Description

Technical Field

[0001] This disclosure relates to Mixed Reality (MR) technology, specifically to a method and system for real-time multi-space object synchronization based on MR devices. Background Technology

[0002] With the rapid development of Mixed Reality (MR) technology, MR has shown enormous application potential in fields such as military command, virtual training, industrial design, and remote collaboration. In these multi-user interactive collaborative scenarios, multiple users distributed across different physical spaces typically need to participate in collaborative work within the same virtual scene through their respective MR terminal devices, such as interacting with the same virtual object within that scene. Therefore, MR systems must possess extremely high spatial consistency, real-time state performance, and consistency in interaction logic.

[0003] Currently, the technical solutions for achieving spatial synchronization of multiple MR terminals mainly rely on centralized architectures, including cloud-based collaborative architectures, local base station synchronization schemes, and edge computing and relay schemes. In cloud-based collaborative architectures, each MR device is typically required to independently build a local SLAM map and upload its features to a cloud server. The server then uniformly calculates the spatial transformation matrix between multiple devices and distributes it to each terminal to establish a unified coordinate system. In local base station synchronization schemes, a host device or central control device is typically introduced into the local area network. This device is responsible for collecting the SLAM maps from each terminal, establishing a unified local coordinate system, and broadcasting object location updates. Furthermore, edge computing and relay schemes attempt to utilize specific edge computing nodes or network relay devices to share the computational burden of the host device and expand coverage.

[0004] However, the aforementioned existing technologies still have significant bottlenecks and shortcomings in practical applications. First, the centralized architecture is susceptible to single-point-of-failure risks; if the master node or cloud server fails or goes offline, the entire collaborative system will experience a synchronization interruption. Given that existing solutions heavily rely on the wide area internet, cloud servers, or specific central coordination nodes, they cannot operate in closed environments without internet coverage or in network-free environments such as underground or in the wild. Furthermore, using public networks poses a risk of data leakage in highly confidential scenarios such as military operations. Second, the traditional map fusion process involves numerous feature extraction, uploading, matching, and feedback steps, resulting in synchronization delays typically ranging from hundreds of milliseconds to several seconds, making it difficult to meet the demands of real-time interaction. In addition, most solutions only transmit position and pose data, neglecting complex attributes such as object animation, material representation, and interactive events, and often employ full-state broadcasting, leading to severe network bandwidth waste and transmission redundancy. Simultaneously, due to the lack of optimization mechanisms for the limited computing resources and rendering load of MR glasses, frame rate drops, pose drift, or animation stuttering are highly likely to occur during multi-user synchronization.

[0005] Therefore, there is an urgent need in this field for a multi-MR device synchronization solution that can achieve self-organized communication in a local network, has efficient synchronization performance, and superior rendering performance. Summary of the Invention

[0006] According to a first aspect of this disclosure, a method for real-time object synchronization in multiple spaces based on MR devices is provided, comprising: establishing a P2P communication connection between multiple MR devices; establishing a local SLAM point cloud and extracting an ORB feature point set from the local SLAM point cloud at each of the multiple MR devices; exchanging the extracted ORB feature point sets among the multiple MR devices to determine a coordinate transformation matrix between the multiple MR devices, and performing spatial alignment and establishing a global spatial reference system among the multiple MR devices based on the coordinate transformation matrix between the multiple MR devices; establishing a virtual scene, the virtual scene including at least one virtual object; continuously updating local state information at each of the multiple MR devices, the local state information being used to represent the rendering state of at least one virtual object in the virtual scene; determining state update information based on the amount of change in the local state information when the change in the local state information meets a preset condition, and transmitting the state update information among the multiple MR devices; and rendering at least one virtual object in the virtual scene at each of the multiple MR devices based on the local state information and the state update information from other MR devices.

[0007] According to a second aspect of this disclosure, a multi-space real-time object synchronization system based on MR devices is provided, comprising a plurality of MR devices, wherein each MR device is configured to perform the method steps according to a first aspect of this disclosure.

[0008] This disclosure aims to address at least one of the following technical problems of existing multi-terminal MR spatial synchronization technologies: high dependence on external central nodes or public network environments, leading to difficulties in deployment in confidential scenarios or environments without networks, and the risk of single point of failure; large map fusion latency, single state synchronization attributes, and serious bandwidth waste caused by full data broadcasting; and poor real-time synchronization issues such as visual drift and rendering stuttering.

[0009] This disclosure establishes a distributed P2P communication connection between multiple MR devices, enabling spatial synchronization and virtual object state synchronization among multiple devices within a local area network without requiring a cloud network. By exchanging ORB feature point sets between devices, a global spatial reference system can be continuously derived and converged from the transformation matrices between each pair of devices, achieving spatial alignment of multiple MR devices. Update synchronization is only performed when the state of the virtual object changes and the change meets preset conditions, avoiding redundant data transmission and reducing bandwidth consumption. The claimed technical solution achieves fully decentralized localized synchronization collaboration, improving deployment flexibility and reliability in complex scenarios and confidential environments, and helping to ensure visual consistency among multiple devices in complex virtual scenes.

[0010] The above and other aspects of this disclosure will be clear from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure, exemplary embodiments are explained below with the help of schematic diagrams and reference numerals.

[0012] Figure 1 A flowchart of a multi-space real-time object synchronization method based on an MR device according to an embodiment of the present disclosure is shown; Figure 2 A schematic diagram of the structure of an MR device for performing a multi-space real-time object synchronization method based on an MR device according to an embodiment of the present disclosure is shown.

[0013] List of reference numerals 10 MR equipment 100 P2P communication network 200 Device Access Layer 210 P2P networking module 220 Data Acquisition Module 300 spatial alignment layers 310 SLAM Mapping Module 320 Feature Extraction Module 330 Alignment Calculation Module 400 State Synchronization Layer 410 Status Update Module 420 Priority Scheduling Module 430 Status Information Encoding Module 500 Delay Compensation Layer 510 State Prediction Module 520 Filtering and Noise Reduction Module 600 rendering adaptation layers 610 Parameter Adaptive Module 620 Real-time rendering module. Detailed Implementation

[0014] The accompanying drawings are included to provide further understanding. The same reference numerals denote elements or parts that have the same function. As long as elements or parts correspond to each other in function in different drawings, their description will not be repeated for each subsequent drawing. For clarity, elements may not have corresponding reference numerals in all drawings.

[0015] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0016] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0017] According to a first aspect of this disclosure, a method for real-time object synchronization in multiple spaces based on MR devices is provided. The method includes: establishing a P2P communication connection between multiple MR devices; at each of the multiple MR devices, establishing a local SLAM point cloud and extracting an ORB feature point set from the local SLAM point cloud; exchanging the extracted ORB feature point sets among the multiple MR devices to determine a coordinate transformation matrix between the multiple MR devices, and based on the coordinate transformation matrix between the multiple MR devices, performing spatial alignment among the multiple MR devices and establishing a global spatial reference system; establishing a virtual scene, the virtual scene including at least one virtual object; continuously updating local state information at each of the multiple MR devices, the local state information representing the rendering state of at least one virtual object in the virtual scene; when the change in the local state information meets a preset condition, determining state update information based on the amount of change in the local state information, and transmitting the state update information among the multiple MR devices; and at each of the multiple MR devices, rendering at least one virtual object in the virtual scene based on the local state information and state update information from other MR devices.

[0018] As an example, after MR devices (such as HoloLens, XVisio, etc.) are started, connections are first established between multiple MR devices based on point-to-point (P2P) communication, thereby building a distributed peer-to-peer network without relying on external servers. Each MR device uses its onboard sensors to perform a simultaneous localization and mapping (SLAM) process in real time, generating a sparse point cloud (SLAM point cloud) reflecting the current physical environment structure. To achieve coordinate unification across space, the system extracts unique ORB feature points from the point cloud. These feature points typically correspond to significant geometric features in the environment, such as corners and object edges. As an example, the number of ORB feature points extracted per frame can be controlled to within 500, which helps balance the computational load. Preferably, the MR device described in this disclosure is MR glasses.

[0019] Furthermore, by leveraging P2P communication connections, devices exchange their extracted ORB feature points to find common visual anchor points. For example, when three or more common feature points are identified between MR device A and MR device B, the spatial transformation matrix T_AB between the devices can be calculated. This matrix T_AB indicates the translation vector and rotation matrix from the coordinate system of device B to the coordinate system of device A. Further, all devices connected via P2P communication iterate through this pairwise calculation method until the local coordinate systems of each device converge to a global spatial reference system, ensuring that the spatial alignment error remains at the centimeter level. For example, the origin and three-axis directions of the local coordinate system of one of the initially connected devices can be specified as the initial values ​​of the global spatial reference system. Once spatial alignment between devices is completed, the real-time collaboration phase begins. This implements a local SLAM map alignment algorithm that ensures consistent spatial alignment of SLAM maps generated by different devices. Even if each device generates its own SLAM map in an independent space, a unified global coordinate system can ultimately be achieved through feature point matching and coordinate transformation matrices. This enables the system to achieve extremely high precision in spatial synchronization between different devices. In complex military demonstrations or virtual training, this technology can avoid visual inconsistencies caused by coordinate drift or alignment errors, improving the accuracy and realism of multi-device collaboration.

[0020] During the real-time collaboration phase, MR devices build a virtual scene containing at least one virtual object based on a real-time 3D engine (such as Unity or Unreal Engine). Continuously updating local state information at each MR device means acquiring the current local state information in real time at a predetermined operating frequency. This local state information represents the current rendering state of the at least one virtual object in the virtual scene. When changes in the local state information meet preset conditions, state update information is determined based on the amount of change in the local state information. This is to share the change information with other MR devices when a virtual object changes, ensuring that the rendering state of the same virtual object remains synchronized across different MR devices. This disclosure describes a state differential synchronization mechanism that only performs synchronization transmission when the rendering state of a virtual object changes, and transmits differential data associated with the amount of change rather than the full rendering state data. This setup avoids redundant data transmission, thereby reducing bandwidth consumption.

[0021] The status update information is distributed to other MR devices within the local area network via a P2P communication network. After receiving the status update information, the receiving MR device overlays it with the current local status information of the virtual object stored locally, and performs smoothing and noise reduction optimization to calculate the final renderable state, which is then provided to the real-time driving 3D engine to complete the real-time rendering output of the virtual object.

[0022] Through the aforementioned state-difference-based synchronization and rendering scheme, the technical solution disclosed herein can achieve high-precision state synchronization of virtual objects between multiple terminals locally without relying on cloud servers. The described mechanism of transmitting only the changes in the virtual object's state can improve synchronization efficiency. For example, it can filter redundant data that remains unchanged for a long time in static scenes, and reduce network bandwidth usage by about 70% during dynamic collaboration, greatly reducing the instantaneous communication pressure and communication overhead during multi-user concurrency.

[0023] In some embodiments, at least one MR device performs device search and identification via a local area network UDP, and if at least one other MR device is identified, an encrypted P2P communication connection is established between the at least one MR device and the at least one other MR device.

[0024] As an example, when an MR device starts up, its internal P2P networking module initiates an automatic scan broadcast to the current subnet using the LAN UDP protocol. This UDP-based discovery mechanism allows the device to quickly identify active peer nodes in the vicinity without manually entering IP addresses or relying on external directory servers, by listening to response heartbeat packets within the LAN. Once at least one other MR device is identified, the initiating device hands off the identified device using a preset security protocol, negotiates a communication key, and establishes an encrypted P2P communication channel, thereby building a distributed peer-to-peer network. During this process, if the networking attempt fails due to environmental interference or physical distance, the MR device will continuously retry according to a preset strategy until the encrypted link is successfully established and local status data collection begins.

[0025] The aforementioned self-discovery and encrypted P2P connection scheme based on LAN UDP achieves a fully decentralized network initialization process, avoiding single points of failure and dependence on external WAN connections found in existing solutions. Since all state calculations and updates are completed within a local P2P peer-to-peer network of MR devices, all sensitive state data related to the virtual scene is not relayed through any cloud server, avoiding the risk of sensitive data being transmitted to the external network. This effectively enhances data security in demanding application scenarios such as military exercises and industrial security demonstrations, while also ensuring that even in a closed environment with zero internet access, multiple MR devices can quickly establish highly reliable collaborative communication connections.

[0026] In some embodiments, the local state information includes at least one of the following object attributes: the position and pose of the virtual object in the virtual scene, the playback progress of the virtual object's appearance change animation, the appearance material parameters of the virtual object, the interaction commands associated with the virtual object, and the semantic information of the virtual object. Each object attribute is encoded according to the same data format.

[0027] As an example, the local state information may also include other object attributes, such as the three-axis velocity and three-axis acceleration of the virtual object in the global spatial reference frame. The object attributes included in the local state information can be configured according to the application of the MR device, and this disclosure is not intended to impose any limitations on this.

[0028] For ease of understanding, this assumes that local state information includes five object attributes: the virtual object's position and orientation in the virtual scene, the playback progress of the virtual object's appearance change animation, the virtual object's appearance material parameters, the interaction commands associated with the virtual object, and the virtual object's semantic information. These five object attributes are formatted using a unified encoding format. For example, the virtual object's position and orientation in the virtual scene represent its three-dimensional coordinates (e.g., xyz orthogonal coordinate system) and rotational orientation (e.g., rpy Euler angles) in a global reference frame. For example, the virtual object is a button in the virtual scene; when the button is pressed, an appearance change animation lasting 0.5 seconds plays. In this case, the playback progress can represent a real-time playback percentage, such as displaying that the animation has played to 65%. For example, appearance material parameters define the object's visual characteristics, such as RGB color values ​​or texture mapping parameters; for example, RGB(255,0,0) represents a red appearance. For example, interaction events are user-triggered actions associated with virtual objects, such as clicking a specific button. For example, semantic information represents high-level business logic data such as tactical symbol types and priority markers.

[0029] Each object attribute is encoded using the same data format. As an example, a formatted data body may include: an identifier to identify the object attribute, the data update frequency, the data generation timestamp, and the data content. For instance, the data body for position and orientation attributes would be {S1;60Hz;20260101090130123;{(x:1.23,y:2.45,z:3.67,roll:15,pitch:-5,yaw:217)}}. Serializing all object attributes using a uniform encoding format helps ensure consistency in the parsing of state data across different hardware terminals in a distributed network.

[0030] In some embodiments, the preset condition includes: the change in any object attribute in the local status information exceeds a preset threshold. The method according to this disclosure further includes: determining object attributes whose changes satisfy the preset condition as object attributes to be updated; determining status update information based on the object attributes to be updated; and transmitting the status update information among multiple MR devices via a P2P communication connection, wherein the status update information includes differential data of the object attributes to be updated before and after the change.

[0031] As an example, on each MR device's local end, rendered object attribute values ​​can be used as reference baselines, and the object attribute values ​​in the latest collected local status information can be continuously compared with the baseline values ​​to monitor the changes in each object attribute. Considering rendering effects, communication costs, and rendering latency, different preset thresholds can be set for different object attributes. As an example, the threshold for the object attribute "position and pose of the virtual object in the virtual scene" can be set to 0.5cm, and the preset threshold for animation progress can be set to 5%. Once the change in an object attribute exceeds the threshold, it is marked as an object attribute to be updated, and the difference data before and after the change is calculated. For example, when the position coordinates of a virtual object change from (x:1.23, y:2.45, z:3.67) to (x:1.23, y:3.14, z:3.67), if the change in displacement of the virtual object in the y-axis direction exceeds the preset threshold of 0.5cm, then the object attribute "position and pose of the virtual object in the virtual scene" is marked as an object attribute to be updated. As an example, the state update information only contains identifiers and differential data. For instance, the data body of the state update information is {S1:{Δx=0,Δy=0.69,Δz=0}}. This can greatly compress the byte size of the parameters representing the rendering state and avoid static properties or object properties with low update frequency from occupying transmission bandwidth.

[0032] In some embodiments, each object attribute of the local state information has a preset event priority; wherein, the state update information includes the event priority of the object attribute to be updated. When the MR device receives multiple state update messages from other MR devices, these multiple state update messages are sorted based on the reception time of each state update message and the event priority in each state update message.

[0033] In some embodiments, when the MR device receives multiple interaction instructions associated with the same virtual object, the unique effective interaction instruction is determined based on the sorting result.

[0034] In some embodiments, event priorities include high priority and low priority; wherein, the step of transmitting status update information among multiple MR devices further includes: comparing the event priorities of the attributes of the object to be updated, transmitting status update information including the attributes of the object to be updated with high priority first, and transmitting the status update information including the attributes of the object to be updated with low priority after multi-frame synthesis.

[0035] By assigning differentiated scheduling weights to different object attributes, the real-time nature of core interactions can be ensured. For example, position and pose, as well as interaction events, can be set to high priority, while material parameters and semantic information can be set to low priority. Thus, when the system determines that the changes in multiple object attributes simultaneously exceed a preset threshold, the MR device will prioritize processing the high-priority object attributes to be updated, calculating their state update information and allocating network bandwidth resources preferentially for the transmission and synchronization of this state update information. For instance, the MR device allocates approximately 60% of the total bandwidth for state update information representing differential data of position and pose, and pushes it to the P2P network in real time at a high frequency of 60Hz. For example, low-priority object attributes to be updated employ a merged transmission strategy; for example, differential update data within three consecutive frames is encapsulated and synthesized before being pushed to the P2P network, thereby reducing transmission overhead.

[0036] As an example, when status update information is received from multiple sources simultaneously, the time of receipt of each status update is recorded as the reception time, and this information is stored in an information cache queue. As an optional step, the timestamp in each message can be extracted as the sending or generation time of that message. The MR device extracts the event priority from each status update and performs a weighted sorting based on the reception time (optionally, also the generation time). This setup ensures that virtual objects are rendered in the correct manner and order. When conflicting interaction commands for the same virtual object are received from multiple MR devices, an event lock can be set: as an example, when status update information indicating "enable control" and "disable control" commands are received simultaneously, the status update information is sorted according to the order of reception time, only the first received command is executed while subsequent conflicting commands are discarded, and the processing result of the interaction command is fed back to all other devices.

[0037] This disclosure describes a multi-device synchronization and conflict resolution scheme based on event priority, which can avoid different rendering results of the same virtual object on different MR devices due to concurrent conflicts, ensuring that all users have a consistent interactive experience in the same virtual scene when multiple devices collaborate. For example, in military exercise simulations or multi-person industrial collaboration applications, participants in different geographical locations can reach a consistent operational understanding of virtual objects, which helps to enhance the logical stability of the system under extreme interactive loads.

[0038] In some embodiments, the step of rendering virtual objects in a virtual scene further includes: determining a transmission delay based on the sending and receiving times of state update information from other MR devices; calculating predicted state information using a linear prediction model or a second-order prediction model based on the transmission delay, the local state information of at least one virtual object in the previous and current frames, and the frame rendering interval; performing noise reduction processing on the predicted state information using a Kalman filter algorithm to determine rendering information; and rendering at least one virtual object in the virtual scene based on the rendering information.

[0039] As an example, for each received state update data frame, it is categorized according to the attribute of the object to be updated (e.g., categorized according to the identifier of the data body of the state update information). For each attribute of the object to be updated, its most recent twenty data frames are recorded and the average transmission delay is calculated. T_avg As a time benchmark for prediction. Based on this average delay T_avg Local state information of virtual objects in the previous frame S_prev and the local state information of the current frame S_current and preset frame rendering interval fi The predicted state information is calculated. As an example, for low-speed motion scenarios, such as when the change in position is less than 10cm, a linear prediction model is used to calculate the predicted state information. S_pred As shown in the following formula: (1).

[0040] For example, in a 60 frames per second operating environment, the frame interval is approximately 16.7 milliseconds. The system uses the displacement vector between the current state and the previous frame state to perform linear compensation according to the ratio of the average delay to the frame interval.

[0041] Alternatively, for high-speed motion scenarios, a second-order prediction model can be used to compute the predicted state message. S_ pred This helps improve prediction accuracy. Subsequently, the system uses a Kalman filter algorithm to process the initially generated prediction state information. S_pred Filtering is performed to eliminate noise (such as transmission jitter and numerical jitter caused by sensor errors) in order to determine the final rendering information. S_final This allows for the rendering of virtual objects within the virtual scene.

[0042] This disclosure provides a two-step delay compensation method. In the first step, a prediction strategy is switched based on the motion characteristics of the virtual object to obtain prediction state information. S_predIn the second step, the preliminary predicted state information is denoised and smoothed using the Kalman filter algorithm. This eliminates instantaneous jumps in the predicted values ​​and effectively corrects displacement errors caused by network transmission. Based on the Kalman-filtered optimized rendering information... S_final To render virtual objects, it can ensure visual continuity, especially in environments with poor network conditions (such as low bandwidth or high latency). It can still maintain low latency and high precision synchronization, avoiding common object jitter or screen lag, and greatly improving adaptability and stability in dynamic scenarios (such as military exercises and tactical simulations).

[0043] In some embodiments, the step of rendering at least one virtual object in a virtual scene further includes: adaptively adjusting the rendering precision based on the hardware performance of the MR device, wherein adjusting the rendering precision includes at least one of the following: adjusting the playback frame rate of the virtual object animation, adjusting the texture precision of the virtual object appearance material parameters, and adjusting the rendering frequency.

[0044] As an example, based on the processor and graphics processing unit (GPU) power of MR devices, they can be categorized into different performance levels. Higher rendering frame rates and / or higher animation playback frame rates and / or higher texture precision can be applied to high-performance devices, and vice versa. Furthermore, the device's rendering frame rate can be monitored in real-time during operation, and rendering precision adjustments can be initiated when the frame rate falls below a preset threshold (e.g., below 55 frames per second). This could involve reducing the update frequency of non-core synchronization states, lowering the playback frame rate of virtual object animations, or reducing the texture precision of virtual object appearance material parameters. In other words, when rendering pressure is excessive, reducing rendering precision can reduce the device's rendering load, or prioritize the rendering precision of object attributes with high event priority while reducing the rendering precision of other object attributes, thus preventing rendering stuttering or frame drops.

[0045] The described technical solution can adaptively adjust the rendering accuracy for MR devices with different hardware specifications, and can also automatically adjust the transmission strategy under different network conditions, thereby ensuring that the best synchronization effect can be obtained on MR devices with different computing power, which helps to improve the user experience.

[0046] In some embodiments, the method disclosed herein further includes: managing a state cache pool in an MR device using an LRU cache eviction algorithm, the state cache pool being used to store historical state information associated with at least one virtual object.

[0047] As an example, historical state information associated with a virtual object includes its position and pose coordinates. As another example, historical state information associated with a virtual object can be local state information collected by the current MR device within a past time period. As a preferred example, the state cache pool also stores the local SLAM point cloud and the extracted ORB feature point set. To optimize memory resource utilization and maintain long-term operational stability, an LRU cache eviction algorithm is used to manage the state cache pool in the MR device. As an example, the state cache pool only retains the position and pose coordinates of each virtual object in the most recent one hundred frames. When new state update information is received and the cache space reaches its limit, the data of the frames that have not been accessed or updated in the state cache pool are cleared. This design reduces redundant data and improves the stability of the system during long-term operation.

[0048] It should be understood that the features and benefits described with respect to the method of the first aspect of this disclosure are also disclosed with respect to the multi-space real-time object synchronization system based on MR devices described below, and vice versa.

[0049] According to a second aspect of this disclosure, a multi-space real-time object synchronization system based on MR devices is provided, comprising a plurality of MR devices, wherein each MR device is configured to perform the method according to any of the above embodiments.

[0050] The following will combine Figure 1 and Figure 2 Embodiments of the method according to this disclosure and embodiments of the system according to this disclosure are described respectively.

[0051] Figure 1 A flowchart illustrating a multi-space real-time object synchronization method based on an MR device according to an embodiment of the present disclosure is shown. (Reference) Figure 1 The method shown includes the following steps: Device Networking: After startup and initialization, the MR device first performs an automatic scan via LAN UDP broadcast to discover other MR devices. If other online devices are detected, it attempts to establish an encrypted P2P communication connection with those devices to build a peer-to-peer network. If the networking process fails, the system continues to retry until the networking is successful.

[0052] Spatial Alignment: After network formation, devices connected via a P2P network synchronously collect local SLAM point cloud data and extract key ORB feature point sets such as wall corners and object edges. Subsequently, multiple MR devices exchange their extracted ORB feature point sets via the P2P network. Shared feature points are matched between each pair of devices, the spatial transformation matrix between devices is calculated, and finally, a global spatial reference frame is formed, thus achieving spatial alignment between multiple devices. After spatial alignment is achieved, the system enters the real-time collaboration phase.

[0053] Status acquisition: Each MR device continuously acquires various object attributes of virtual objects, such as position, posture, animation, material, and interaction events.

[0054] Differential transmission: The status update information of the changed part is determined and transmitted only when the change in the local status information meets the preset conditions. This information is transmitted to other MR devices via a P2P network.

[0055] Delay compensation: At the receiving end, after the MR device receives the state update information, it performs the following steps: First, it calculates the average transmission delay and actively predicts the state of the virtual object in the next frame using a linear prediction model or a second-order prediction model to obtain the initial predicted state information; Second, after Kalman filtering noise reduction optimization, it obtains rendering information that can be used by the 3D engine to perform rendering tasks.

[0056] Real-time rendering: Rendering is performed based on the noise-reduced and optimized rendering information. During the rendering process, the rendering precision is adaptively adjusted according to the real-time computing load and the computing power of the device itself to ensure stable output of synchronized images.

[0057] Throughout the collaboration process, the system continuously loops through the aforementioned state acquisition, differential transmission, latency compensation, and real-time rendering steps. After completing the real-time rendering step, it returns to the state acquisition step by default; if the user actively terminates the collaboration, the device automatically exits the P2P communication connection and clears its local state cache pool.

[0058] Figure 2 A schematic diagram of the structure of an MR device 10 for performing a multi-spatial real-time object synchronization method based on an embodiment of the present disclosure is shown. The MR device 10 includes five functional layers: a device access layer 200, a spatial alignment layer 300, a state synchronization layer 400, a latency compensation layer 500, and a rendering adaptation layer 600. The MR device 10 implements reference through these layers. Figure 1 The described method and steps.

[0059] The device access layer 200 includes a P2P networking module 210 and a data acquisition module 220, used for performing reference... Figure 1 The described method steps include a device networking process. Specifically, the P2P networking module 210 is configured to discover devices and establish encrypted peer-to-peer connections via LAN UDP broadcast, and the data acquisition module 220 is configured to record the network attribute information of each MR device object with an established P2P connection.

[0060] The spatial alignment layer 300 includes a SLAM mapping module 310, a feature extraction module 320, and an alignment calculation module 330, used to perform reference... Figure 1The spatial alignment process described in the method steps involves the SLAM mapping module 310 and the feature extraction module 320 working together. The SLAM mapping module 310 builds a local SLAM point cloud from the raw sensor data, while the feature extraction module 320 extracts key ORB feature point sets, such as corners and edges, from the local SLAM point cloud. The alignment calculation module 330 calculates the spatial transformation matrix between the devices based on the ORB feature point set from another MR device and sends it back to that other MR device. It also receives spatial transformation matrices between other MR devices to align the MR device 10 to the global spatial coordinate system.

[0061] The state synchronization layer 400 includes a state update module 410, a priority scheduling module 420, and a state information encoding module 430, used to execute reference... Figure 1 The described method includes a state acquisition and differential transmission process. Specifically, the state update module 410 continuously acquires various object attributes of the virtual object, such as position, pose, animation, material, and interaction events, and detects in real time whether these object attributes have changed and whether the changes meet preset conditions. The state information encoding module 430 calculates differential data for object attributes that meet the preset conditions and encodes it into state update information. The priority scheduling module 420 allocates transmission bandwidth resources and determines the transmission frequency based on the event priority of each object attribute, and controls the transmission task of the state update information.

[0062] The delay compensation layer 500 includes a state prediction module 510 and a filtering and noise reduction module 520, used to perform reference... Figure 1 The described method steps include a delay compensation process. The state prediction module 510 calculates initial predicted state information, which is then optimized by the filtering and denoising module 520 using a Kalman filter algorithm to obtain rendering information suitable for the 3D engine to perform rendering tasks.

[0063] The rendering adaptation layer 600 includes a parameter adaptation module 610 and a real-time rendering module 620, used to perform reference... Figure 1 The described method steps include a real-time rendering process. The parameter adaptation module 610 monitors the computational resource load and rendering frame rate of the MR device 10 in real time, and dynamically adjusts the rendering precision when performance is limited. The real-time rendering module 620 includes a real-time 3D engine that renders virtual objects based on the rendering information optimized by the filtering and noise reduction module 520.

[0064] The multi-space real-time object synchronization system based on MR devices according to the second aspect of this disclosure preferably includes a plurality of references. Figure 2 The MR device 10 described.

[0065] Various modifications to the embodiments described in this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings, are intended to cover non-exclusive inclusion.

Claims

1. A method for real-time multi-space object synchronization based on MR devices, the method comprising: Establish P2P communication connections between multiple MR devices; At each of the plurality of MR devices, a local SLAM point cloud is established and an ORB feature point set is extracted from the local SLAM point cloud; The extracted ORB feature point sets are exchanged among the multiple MR devices to determine the coordinate transformation matrix between the multiple MR devices, and based on the coordinate transformation matrix between the multiple MR devices, spatial alignment is performed among the multiple MR devices and a global spatial reference system is established. A virtual scene is created, the virtual scene including at least one virtual object; Local state information is continuously updated at each of the plurality of MR devices, the local state information being used to represent the rendering state of the at least one virtual object in the virtual scene; When the change in the local status information meets the preset conditions, status update information is determined based on the amount of change in the local status information, and the status update information is transmitted among the multiple MR devices. as well as At each of the plurality of MR devices, the at least one virtual object is rendered in the virtual scene based on the local state information and state update information from the other MR devices.

2. The method according to claim 1, wherein, The steps to establish a P2P communication connection between multiple MR devices include: At least one MR device performs device search and identification via LAN UDP, and if at least one other MR device is identified, an encrypted P2P communication connection is established between the at least one MR device and the at least one other MR device.

3. The method according to claim 1, wherein, The local state information includes at least one of the following object attributes: the position and / or posture of the virtual object in the virtual scene, the playback progress of the virtual object's appearance change animation, the appearance material parameters of the virtual object, the interaction instructions associated with the virtual object, and the semantic information of the virtual object; wherein, each object attribute is encoded in the same data format.

4. The method according to claim 3, wherein, The preset conditions include: the change in any object attribute in the local status information exceeds a preset threshold. The method further includes: determining object attributes whose changes meet preset conditions as object attributes to be updated, determining status update information based on the object attributes to be updated, and transmitting the status update information between the multiple MR devices via the P2P communication connection, wherein the status update information includes differential data of the object attributes to be updated before and after the change.

5. The method according to claim 4, wherein, Each object attribute of the local state information has a preset event priority; wherein, the state update information includes the event priority of the object attribute to be updated; In this case, when an MR device receives multiple status update messages from other MR devices, the multiple status update messages are sorted based on the reception time of each status update message and the event priority in each status update message.

6. The method according to claim 5, wherein, When an MR device receives multiple interaction commands associated with the same virtual object, the unique and effective interaction command is determined based on the sorting result.

7. The method according to claim 5, wherein, The event priority includes high priority and low priority; wherein, the step of transmitting the status update information among the plurality of MR devices includes: The event priorities of the attributes of the object to be updated are compared. The state update information including the attributes of the object to be updated with high priority is transmitted first, and the state update information including the attributes of the object to be updated with low priority is synthesized into multiple frames and then transmitted.

8. The method according to claim 1, wherein, The step of rendering the at least one virtual object in the virtual scene further includes: The transmission delay is determined based on the sending and receiving times of status update information from other MR devices; Based on the transmission delay, the local state information of the at least one virtual object in the previous and current frames, and the frame rendering interval, a linear prediction model or a second-order prediction model is used to calculate the predicted state information. The predicted state information is denoised using a Kalman filter algorithm to determine the rendering information; and The at least one virtual object is rendered in the virtual scene according to the rendering information.

9. The method according to claim 1, wherein, The step of rendering the at least one virtual object in the virtual scene further includes: Based on the hardware performance of the MR device, the rendering accuracy is adaptively adjusted. Adjusting rendering precision includes at least one of the following: adjusting the playback frame rate of virtual object animation, adjusting the texture precision of virtual object appearance material parameters, and adjusting the rendering frequency.

10. The method according to claim 1, further comprising: The state cache pool in the MR device is managed using an LRU cache eviction algorithm. The state cache pool is used to store historical state information associated with the at least one virtual object.

11. A multi-space real-time object synchronization system based on MR devices, comprising multiple MR devices, wherein, Each MR device is configured to perform the method according to any one of claims 1 to 10.

Images