Mixed reality multi-person interaction large-space positioning, aligning and collaborating system and method

Abstract

The invention relates to the technical field of mixed reality, in particular to a mixed reality multi-user interaction large-space positioning, aligning and collaborating system and method, and aims to solve the problems that in the prior art, the user interaction experience is improved only by collecting multi-aspect data for processing, efficient collaboration work of multiple users in a large space cannot be achieved, and the user experience is poor. Information sharing, task distribution and real-time interaction among users cannot be realized; information sharing, task distribution and real-time interaction among users are realized through the multi-user cooperation module, cross-device operation consistency is ensured through a multi-device sharing coordinate system, configuration is simplified, cooperation efficiency is improved, real-time data transmission is realized through multi-device network synchronization, delay is reduced, cooperation fluency is enhanced, and user experience is improved. The interaction reality sense is enhanced through physical interaction logic synchronization, the working process is optimized, and the cooperation experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality technology, and more specifically, to a mixed reality multi-person interactive large space positioning, alignment, and collaboration system and method. Background Art

[0002] With the rapid development of science and technology, Mixed Reality (MR) technology, as a combination of Virtual Reality (VR) and Augmented Reality (AR), is gradually becoming a hot topic in the field of human-computer interaction. By integrating the physical world and the digital world, MR technology provides users with an unprecedented immersive experience and shows great application potential in many fields such as education, entertainment, and industrial manufacturing. However, in the multi-person interaction scenario of MR, how to achieve precise positioning, alignment, and efficient collaborative work in a large space has become a technical problem that needs to be solved urgently. In this context, the Mixed Reality Multi-Person Interaction Large Space Positioning, Alignment, and Collaboration System came into being. The system aims to achieve precise position perception, spatial alignment, and efficient collaborative operation of multiple users in a shared MR environment by integrating multiple advanced technologies.

[0003] The patent application with reference publication number CN111796670A discloses a large-space multi-person virtual reality interaction system and method, including: a laser motion capture device, used to collect data and perform positioning through an optical camera; a motion capture device, used to capture human motion data through a trackable tool; a hybrid motion capture server, which is respectively connected to the laser motion capture device and the motion capture device through a network device, and is used to receive laser data and motion data, and perform calculation processing; a backpack computer device, which is connected to the hybrid motion capture server through a preset system and a network, and is used to obtain data from the hybrid motion capture server and process it into 3D data information; a head-mounted display device, which is used to receive and display 3D data information transmitted by the backpack computer device; the present invention also provides an interaction method accordingly, whereby the present invention greatly improves the user's interaction experience through multi-faceted data collection;

[0004] However, the existing technology only improves the user interaction experience by collecting and processing various data. It cannot achieve efficient collaborative work of multiple users in a large space, cannot achieve information sharing, task allocation and real-time interaction between users, and cannot achieve precise alignment of the coordinate systems of various devices in MR multi-person interaction scenarios. It cannot achieve accurate and stable multi-terminal collaborative work and cannot provide users with a smooth and consistent experience.

[0005] To this end, we propose a mixed reality multi-person interactive large-space positioning, alignment, and collaboration system and method to address the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a mixed reality multi-person interaction large-space positioning, alignment, and collaboration system and method, which solves the problem that the prior art only improves the user interaction experience by collecting and processing various data, cannot achieve efficient collaboration of multiple users in a large space, cannot achieve information sharing, task allocation and real-time interaction between users, and cannot achieve precise alignment of the coordinate systems of various devices in the MR multi-person interaction scenario, cannot achieve accurate and stable multi-terminal collaboration, and cannot provide users with a smooth and consistent experience.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] Mixed reality multi-person interactive large space positioning, alignment, and collaboration system, including cloud service platform, space construction module, space anchor positioning module, space information analysis module, mixed reality module, multi-person collaboration module, and multi-terminal coordinate alignment and calibration module;

[0009] The space construction module builds a virtual space based on MR glasses, depth sensors and scanning systems;

[0010] The spatial anchor positioning module anchors the center point of the virtual space through spatial anchor positioning technology;

[0011] The spatial information analysis module performs computational simulation and mapping of the real-time status of objects in space;

[0012] The mixed reality module combines virtual objects with real space to form a highly immersive hyper-dimensional space experience of mixed reality;

[0013] Multi-person collaboration module, which uses network communication technology and collaborative algorithms to achieve information sharing, task allocation and real-time interaction among users;

[0014] The multi-terminal coordinate alignment and calibration module calibrates and aligns the coordinate systems of various devices through spatial positioning technology and coordinate conversion algorithms.

[0015] As a preferred embodiment of the present invention, the MR glasses device uses its high-definition color perspective function to ensure that the images penetrating the real space are consistent with the colors of the real space seen by the naked eye; the depth sensor measures the distance from the surface of the object to the sensor by emitting and receiving light, thereby acquiring the depth information of the object; the scanning system can intelligently understand and respond to objects in the physical space, and generate a grid model by scanning the object with a light beam.

[0016] As a preferred embodiment of the present invention, the specific process of the space anchor positioning module anchoring the center point of the virtual space through the space anchor technology is as follows:

[0017] Add the OVRSpatialAnchor component to the actual developed GameObject. Through this layer of code association, after the space is created, the information of the origin of the space (0,0,0) will be calculated. The anchor points created in the space have unique identification symbols and can be obtained and called in the form of logs.

[0018] After obtaining the above basic information, adjust the logical relationship at the application level. The code layer only retains the basic layer of obtaining the origin of the own space (0,0,0) without any processing. Shift the focus to the anchor point. After creating an unbound positioning anchor point, instantiate GameObject+OVRSpatialAnchor, bind the anchor point to the instance, and use the SaceAnchorAsvnc method to persist the anchor point. The return information during the anchor point persistence process includes coordinate information and three-dimensional coordinate axis system information. The anchor point coordinate information (x, y, z) value is reset to the new coordinate (0,0,0), and the coordinate axis system information is assigned to the world coordinate axis system as the overall reference object. After processing these two definitions, write the judgment condition to automatically align the relative coordinates and orientations of the objects in the world added later to the new anchor point, thereby completing the anchoring of all world coordinate centers.

[0019] As a preferred embodiment of the present invention, during the anchoring process, two core processes need to be completed:

[0020] Process 1: In the real space, place a three-dimensional reference object at the planned anchor point, and mark this place as the anchor point (0,0,0);

[0021] Process 2: Define the three-axis orientation in the virtual space. According to the edge lines of the reference object and the rule that the angle between each vector in the three-dimensional coordinate system is 90 degrees, draw vertices and connect them to define the XYZ three-axis orientation.

[0022] As a preferred embodiment of the present invention, the specific process of the spatial information analysis module performing computational simulation and mapping presentation on the real-time state of an object in space is as follows:

[0023] Based on the inside-out spatial monitoring of SLAM technology, the depth and spatial distance information are obtained, and the displacement information and rotation information of the object are processed respectively. For displacement information, the length, width and height information of the space are obtained through the OVRSpatialAnchor component. Since the anchor point (0,0,0) has been defined, the distances of all points in this space relative to the anchor point are mapped to the instance object through code conditions. With the inside-out spatial monitoring of the device, the real-time displacement calculation and mapping of the object can be realized;

[0024] In terms of rotation information, the XYZ axis vector of the anchor point is used as the world axis reference. Through code rotation control, the rotation of all points in space is calculated relative to the coordinate axis of the anchor point and synchronously mapped to the instance object. In combination with the inside-out space monitoring of the device, real-time displacement calculation and mapping of objects can be achieved.

[0025] By binding the displacement and rotation three-dimensional spatial information of the object with the anchor point, the real-time state of the object in space can be simulated and mapped.

[0026] As a preferred embodiment of the present invention, the specific process of obtaining depth and spatial distance information through inside-out space monitoring based on SLAM technology is as follows:

[0027] The Inside-out SLAM system uses RGB-D cameras, LiDAR, and IMU sensors to obtain environmental information. The RGB-D camera obtains color images and depth images, and the depth image directly obtains the distance information from each pixel to the camera. The IMU provides the acceleration and angular velocity information of the device to help estimate the posture change of the device.

[0028] Extract feature points from color images and perform feature matching between consecutive frames, use algorithms to detect key points, and calculate descriptors for each key point;

[0029] Through the matched feature points, the PnP (Perspective-n-Point) algorithm is used to estimate the camera's pose (position and orientation). The PnP formula is:

[0030] Given a set of 3D points Pm = (Xm, Ym, Zm) and the corresponding 2D image points pm = (Um, Vm), the mapping is established through the following relationship:

[0031]

[0032] Where K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation vector, and s is the scale factor;

[0033] Combine the matched feature points and pose information to reconstruct the 3D model of the environment through the depth image and camera pose;

[0034] Use BA algorithm to optimize camera trajectory and map point position to reduce cumulative error;

[0035] BA formula:

[0036] The goal is to minimize the reprojection error:

[0037]

[0038] Where Pmn is the observed 2D point, π is the projection function;

[0039] Calculate the distance between any two points in space through the camera's position and depth information;

[0040] Distance formula:

[0041] For two points A(Xa,Ya,Za) and B(Xb,Yb,Zb) in 3D space, their spatial distance d is calculated by the following formula:

[0042]

[0043] Combined with IMU data, filters are used to update status and achieve real-time environmental monitoring.

[0044] As a preferred embodiment of the present invention, the specific process of the mixed reality module combining virtual objects with real space to form a mixed reality, an immersive hyper-dimensional space experience, is as follows:

[0045] By integrating spatial positioning technology and spatial distance calculation functions, virtual objects are created using 3D modeling technology and real-time lighting post-processing effect rendering.

[0046] At the development level, virtual objects are anchored in the space scene through real-time coordinate calculation, gesture recognition and posture tracking gyroscope external technology are added, specific triggered object feedback information is written, collision objects are added, real physical collision and rebound mechanical phenomena are simulated, real physical perception is created, and finally projected in real time on the device screen;

[0047] Use gestures and posture tracking gyroscopes to trigger object interactions, and highly simulate physical mechanics of the actual physical environment to comprehensively form a highly immersive hyper-dimensional space experience of mixed reality.

[0048] As a preferred embodiment of the present invention, the multi-person collaboration module realizes information sharing, task allocation and real-time interaction among users through network communication technology and collaborative algorithms, which is specifically achieved through the following three aspects:

[0049] Multi-device shared coordinate system: In the state of multiple devices, each device follows the unified original standard process "space creation--anchor point (0,0,0) creation--axis definition" to complete the fixation of the same world coordinate origin and axis. The SaceAnchorAsvnc method is used to persist the anchor point, and a cache mechanism is developed to record the information in a local folder. In actual use, a command will be issued before the code runs, leading all devices to call the cache file in the folder at the same time, thereby establishing a unified object mapping rule;

[0050] Multi-device network synchronization: We independently develop a real-time communication mechanism specifically for LAN environments, and design an efficient server-client information distribution and processing process. All devices communicate in the wireless LAN. After the client role device goes online, it first sends a message to the server role device. The server role device is responsible for centralized detection and judgment of the number and status of the client role devices. When these information meet certain judgment requirements, the server device sends a start command to each client device to start content playback together.

[0051] Synchronization of physical interaction logic: According to the actual interaction process and result goals, the underlying logic of the physical simulation engine is embedded in the spatial world system to realize the underlying functional requirements of physical simulation in 3D space, and then extended based on this, including interactive collaboration and workflow simulation.

[0052] As a preferred embodiment of the present invention, the multi-terminal coordinate alignment and calibration module calibrates and aligns the coordinate systems of various devices through spatial positioning technology and coordinate conversion algorithms, which is specifically achieved through the following three aspects:

[0053] AR image recognition: independently develop positioning alignment client, integrate AR image recognition technology and processing capabilities, design recognition map of complex patterns with specific rules. After the camera captures the image, it will take pictures and detect the feature points of the complex pattern shape, color, structure and light and dark, and construct the anchored three-dimensional information of the image. Through the interface, the object model with specific three-dimensional vector coordinates will be anchored on the coordinates of the reconstructed image center point and mapped to the client screen in real time;

[0054] Axial center point alignment calibration: Based on the anchor point in the MR glasses device, align and calibrate the scene origin (0,0,0) of the third-party perspective client. After the image is successfully recognized, drag the recognition image to the anchor point position in the MR glasses device, and rotate the image to make the position and orientation of the 3D vector coordinate object model anchored on the image consistent with the 3D vector coordinate object model in the MR glasses. After ensuring alignment, the client saves the position information calibration conversion data file locally. All subsequent fusion clients that need to be used will read this data file as the only anchor point aligned with the center point of the MR glasses.

[0055] Screen fusion and synchronization: Aiming at the application scenarios of fusion screens, we independently develop a fusion screen client, whose core functions focus on two points: first, using real-time rendering technology to accurately integrate virtual images into the real scenes captured by real cameras, to achieve synchronized display of virtual and real scenes without boundaries; second, using a mirrored multi-person synchronization mechanism, in which the computer serves as a single instance of the integrated server and local client, and achieves efficient internal communication and scene sharing through direct function calls and message queues, while the remote MR glasses client is connected to the server through a local area network, participates in real-time interaction, and jointly builds an immersive multi-person mixed reality experience.

[0056] As a preferred embodiment of the present invention, a mixed reality multi-person interactive large space positioning, alignment, and collaboration method specifically includes the following steps:

[0057] Step 1: Use the visual input provided by the MR glasses device, combined with the depth sensor and scanning system to scan and model the real environment, and build a virtual space corresponding to the real environment in the virtual environment;

[0058] Step 2: Select one or more center points in the virtual space as spatial anchor points, and use positioning technology to accurately anchor these anchor points to ensure that their positions in the virtual space are fixed;

[0059] Step 3: Perform computational simulation on the collected spatial information to determine the real-time status of virtual objects in the virtual space, map the virtual objects into the real environment, and achieve integration with the real space;

[0060] Step 4: Combine the mapped virtual objects with the real space to form a mixed reality environment. Users can get an immersive hyper-dimensional space experience through MR glasses.

[0061] Step 5: Share information, assign tasks and interact in real time among users to achieve multi-device network synchronization and ensure that all client devices start playing content together;

[0062] Step 6: Calibrate the coordinate system of each user device. Based on the calibration results, align the coordinate systems of each device to ensure that the display positions of virtual objects on different devices are consistent, thus achieving seamless collaboration and interactive experience across devices.

[0063] Compared with the prior art, the advantages of the present invention are:

[0064] (1) In the present invention, information sharing, task allocation and real-time interaction between users are realized through a multi-person collaboration module. The multi-device shared coordinate system ensures cross-device operation consistency, simplifies configuration, and improves collaboration efficiency. Multi-device network synchronization realizes real-time data transmission, reduces delays, and enhances collaboration fluency. Physical interaction logic synchronization enhances interaction realism, optimizes workflow, and improves collaboration experience. The combination of multi-device shared coordinate system, multi-device network synchronization and physical interaction logic synchronization improves collaboration efficiency and fluency on the one hand, and enhances user immersion and participation on the other hand, providing strong support for team collaboration and project management.

[0065] (2) In the present invention, the coordinate systems of various devices are calibrated and aligned through a multi-terminal coordinate alignment and calibration module. AR image recognition is used to identify efficient AR technology, accurately identify and locate, adapt to changing environments, and improve calibration efficiency. The axial center point alignment and calibration are used to accurately calculate the alignment, reduce error accumulation, ensure the coordination and consistency of multiple devices, and enhance system stability. The screen fusion synchronization is used to synchronize the screen in real time to ensure content consistency and seamless switching experience, thereby improving user satisfaction. The combination of AR image recognition, axial center point alignment and calibration, and screen fusion synchronization achieves efficient, accurate, and stable multi-terminal collaborative work, providing users with a smoother and more consistent experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a system block diagram of the present invention;

[0067] Figure 2 This is a flow chart of a method for large-space positioning, alignment, and collaboration of mixed reality multi-person interaction in Embodiment 2 of the present invention;

[0068] Figure 3 It is a schematic diagram of the XYZ three-axis orientation in the first embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will combine the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all the embodiments. All other embodiments obtained by ordinary technicians in this field without creative work based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0070] Embodiment 1: Figure 1 and Figure 3 As shown, the mixed reality multi-person interactive large space positioning, alignment, and collaboration system proposed by the present invention includes a cloud service platform, a space construction module, a space anchor point positioning module, a space information analysis module, a mixed reality module, a multi-person collaboration module, and a multi-terminal coordinate alignment and calibration module;

[0071] The cloud service platform is respectively connected to the space construction module, the space anchor positioning module, the space information analysis module, the mixed reality module, the multi-person collaboration module and the multi-terminal coordinate alignment and calibration module;

[0072] The space construction module builds a virtual space based on MR glasses, depth sensors and scanning systems;

[0073] MR glasses use the high-definition color perspective function to make the images that penetrate the real space consistent with the colors seen by the naked eye. At the same time, thanks to the distortion correction algorithm, the spatial distortion is reduced and the sense of reality is improved. The depth sensor measures the distance from the surface of the object to the sensor by emitting and receiving light, thereby obtaining the depth information of the object. The depth sensor can measure 3D depth more accurately than ordinary RGB cameras. The scanning system can intelligently understand and respond to objects in the physical space, and generate a grid model by scanning the object with a light beam.

[0074] The spatial anchor positioning module anchors the center point of the virtual space through spatial anchor positioning technology;

[0075] The specific process of the spatial anchor positioning module anchoring the center point of the virtual space through the spatial anchor technology is as follows:

[0076] Add the OVRSpatialAnchor component to the actual developed GameObject. Through this layer of code association, after the space is created, the information of the origin of the space (0,0,0) will be calculated. The anchor points created in the space have unique identification symbols and can be obtained and called in the form of logs.

[0077] After obtaining the above basic information, adjust the logical relationship at the application level. The code layer only retains the basic layer of obtaining the self-space origin (0,0,0) without any processing. The focus is shifted to the anchor point. After creating an unbound positioning anchor point, instantiate GameObject+OVRSpatialAnchor and bind the anchor point to the instance, that is, you can instantiate the anchor point at a custom angle, exclude the original logic starting from the self-origin, and use the SaceAnchorAsvnc method to persist the anchor point. The return information during the anchor point persistence process includes coordinate information and three-dimensional coordinate axis system information. The anchor point coordinate information (x, y, z) value is reset to the new coordinate (0,0,0), and the coordinate axis system information is assigned to the world coordinate axis system as the overall reference object. After processing these two definitions, write the judgment condition to automatically align the relative coordinates and orientations of the objects in the world added later to the new anchor point, thereby completing the anchoring of all world coordinate centers;

[0078] When using the SaveSpatialAnchorAsync method to persist an anchor point, if you need to process the returned information, which includes coordinate information and 3D coordinate axis information, you can do so by following these steps:

[0079] In Unity, first create a GameObject and add an OVRSpatialAnchor component to it, setting its position and rotation to match the position you want to persist in physical space;

[0080] Before calling SaveSpatialAnchorAsync, get the position and rotation information from the GameObject's transform component;

[0081] Call the SaveSpatialAnchorAsync method of the OVRSpatialAnchor component to asynchronously save the anchor point and process the result in the callback;

[0082] In the callback function, check whether the operation was successful and get the unique identifier of the anchor;

[0083] An example of a callback function is as follows:

[0084] csharp

[0085] void OnSaveCompleted(OVRSpatialAnchor anchor,bool success)

[0086] {

[0087] if(success)

[0088] {

[0089] string anchorUuid=anchor.GetUuid().ToString();

[0090] / / Save UUID, coordinates and rotation information

[0091] PlayerPrefs.SetString("SpatialAnchorUuid",anchorUuid);

[0092] PlayerPrefs.SetFloat("SpatialAnchorPosX",position.x);

[0093] PlayerPrefs.SetFloat("SpatialAnchorPosY",position.y);

[0094] PlayerPrefs.SetFloat("SpatialAnchorPosZ",position.z);

[0095] PlayerPrefs.SetFloat("SpatialAnchorRotX",rotation.x);

[0096] PlayerPrefs.SetFloat("SpatialAnchorRotY",rotation.y);

[0097] PlayerPrefs.SetFloat("SpatialAnchorRotZ",rotation.z);

[0098] PlayerPrefs.SetFloat("SpatialAnchorRotW",rotation.w);

[0099] PlayerPrefs.Save();

[0100] Debug.Log("Spatial anchor saved successfully with UUID:"+anchorUuid);

[0101] }

[0102] else

[0103] {

[0104] Debug.LogError("Failed to save spatial anchor.");

[0105] }

[0106] }

[0107] If the save fails, log the error message and provide feedback to the user;

[0108] During the anchoring process, two core processes need to be completed:

[0109] Process 1: In the real space, place a three-dimensional reference object at the planned anchor point, and mark this place as the anchor point (0,0,0);

[0110] Process 2: Define the three-axis orientation in the virtual space. According to the edge lines of the reference object and the rule that the angle between each vector in the three-dimensional coordinate system is 90 degrees, draw the vertices and connect them to define the XYZ three-axis orientation;

[0111] The spatial anchor positioning module, relying on advanced spatial anchor positioning technology, accurately anchors the center point of the virtual space, showing multiple significant advantages. First, this technology achieves high-precision position locking, ensuring the seamless integration of virtual objects and the physical environment, bringing users an unprecedented immersive experience; secondly, its cross-platform compatibility enables augmented reality applications on different devices and operating systems to be seamlessly connected, expanding the breadth of application scenarios; thirdly, the persistent storage characteristics of spatial anchors ensure that users can enjoy a consistent and continuous augmented reality experience at different times and on different devices.

[0112] The spatial information analysis module performs computational simulation and mapping of the real-time status of objects in space;

[0113] The specific process of the spatial information analysis module to perform computational simulation and mapping of the real-time status of objects in space is as follows:

[0114] The Inside-out SLAM system uses RGB-D cameras, LiDAR, and IMU sensors to obtain environmental information. The RGB-D camera obtains color images and depth images, and the depth image directly obtains the distance information from each pixel to the camera. The IMU provides the acceleration and angular velocity information of the device to help estimate the posture change of the device.

[0115] Extract feature points from color images and perform feature matching between consecutive frames, use algorithms to detect key points, and calculate descriptors for each key point;

[0116] Through the matched feature points, the PnP (Perspective-n-Point) algorithm is used to estimate the camera's pose (position and orientation). The PnP formula is:

[0117] Given a set of 3D points Pm = (Xm, Ym, Zm) and the corresponding 2D image points pm = (Um, Vm), the mapping is established through the following relationship:

[0118]

[0119] Where K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation vector, and s is the scale factor;

[0120] Combine the matched feature points and pose information to reconstruct the 3D model of the environment through the depth image and camera pose;

[0121] Use BA algorithm to optimize camera trajectory and map point position to reduce cumulative error;

[0122] BA formula:

[0123] The goal is to minimize the reprojection error:

[0124]

[0125] Where P mn is the observed 2D point, π is the projection function;

[0126] Calculate the distance between any two points in space through the camera's position and depth information;

[0127] Distance formula:

[0128] For two points A(Xa,Ya,Za) and B(Xb,Yb,Zb) in 3D space, their spatial distance d is calculated by the following formula:

[0129]

[0130] Combined with IMU data, the filter is used to update the status and achieve real-time environmental monitoring;

[0131] Then, the displacement information and rotation information of the object are processed separately. For displacement information, the length, width and height information of the space are obtained through the OVRSpatialAnchor component. Since the anchor point (0,0,0) has been defined, the distances of all points in this space relative to the anchor point are mapped to the instance object through code conditions. With the inside-out space monitoring of the device, the real-time displacement calculation and mapping of the object is realized.

[0132] In terms of rotation information, the XYZ axis vector of the anchor point is used as the world axis reference. Through code rotation control, the rotation of all points in space is calculated relative to the coordinate axis of the anchor point and synchronously mapped to the instance object. In combination with the inside-out space monitoring of the device, real-time displacement calculation and mapping of objects can be achieved.

[0133] By binding the displacement and rotation 3D space information of the object with the anchor point, the computational simulation and mapping presentation of the real-time state of the object in space can be realized;

[0134] The spatial information analysis module has many advantages in the computational simulation and mapping presentation of the real-time status of objects. First, the module can efficiently process complex spatial data and accurately capture the real-time position, posture and motion trajectory of objects in three-dimensional space, providing a solid data foundation for decision-making; secondly, through high-precision computational simulation, it can predict the future changes in the spatial state of objects and provide forward-looking support for planning, scheduling and risk management; thirdly, the module's powerful mapping presentation capabilities can present complex spatial information to users in an intuitive and easy-to-understand way, improving information understanding and decision-making efficiency.

[0135] The mixed reality module combines virtual objects with real space to form a highly immersive hyper-dimensional space experience of mixed reality;

[0136] The specific process of the mixed reality module combining virtual objects with real space to form a highly immersive hyper-dimensional space experience of mixed reality is as follows:

[0137] By integrating spatial positioning technology and spatial distance calculation functions, virtual objects are created using 3D modeling technology and real-time lighting post-processing effect rendering.

[0138] At the development level, virtual objects are anchored in the space scene through real-time coordinate calculation (they can be fixed relative to the center point or can move relative to the center point in real time), gesture recognition and posture tracking gyroscope external technology are added, specific trigger object feedback information is written, collision objects are added, real physical collision and rebound mechanical phenomena are simulated, real physical perception is created, and finally projected in real time on the device screen;

[0139] Use gestures (clenched fist, open hand) and posture tracking gyroscope (raised hand, waved hand) to trigger object interaction, highly consistent with the physical mechanics simulation of the actual physical environment, and comprehensively form a highly immersive hyper-dimensional space experience of mixed reality;

[0140] Multi-person collaboration module, which uses network communication technology and collaborative algorithms to achieve information sharing, task allocation and real-time interaction among users;

[0141] The mixed reality module seamlessly integrates virtual objects into the real space to create a hyper-dimensional mixed reality experience. It has significant advantages. First, it breaks the boundaries between physical and digital, allowing users to immerse themselves in a highly interactive and lifelike environment and experience an unprecedented sense of immersion. Secondly, mixed reality technology promotes the intuitive display and deep interaction of information, making complex concepts easy to understand, and improving the efficiency and fun of learning, training and entertainment. Thirdly, the module supports a variety of application scenarios, from education, medical care to industrial design, which can achieve a deep integration of virtual and reality and promote industry innovation and development.

[0142] The multi-person collaboration module uses network communication technology and collaborative algorithms to achieve information sharing, task allocation, and real-time interaction among users. This is achieved through the following three aspects:

[0143] Multi-device shared coordinate system: In the state of multiple devices, each device follows the unified original standard process "space creation--anchor point (0,0,0) creation--axis definition" to complete the fixation of the same world coordinate origin and axis. The SaceAnchorAsvnc method is used to persist the anchor point, and a cache mechanism is developed to record the information in a local folder. In actual use, a command will be issued before the code runs, leading all devices to call the cache file in the folder at the same time, thereby establishing a unified object mapping rule;

[0144] Multi-device network synchronization: We independently develop a real-time communication mechanism specifically for LAN environments, and design an efficient server-client information distribution and processing process. All devices communicate in the wireless LAN. After the client role device goes online, it first sends a message to the server role device. The server role device is responsible for centralized detection and judgment of the number and status of the client role devices. After these information meet certain judgment requirements, the server device sends a start command to each client device to start content playback together. The entire process can reach millisecond level through the optimization of object resources and the optimization of underlying code logic, with almost no delay, ensuring the smoothness of synchronization. At the same time, this set of information processing mechanisms has high flexibility. One eyeglass end can be used as a server and other eyes can be deployed as clients; it can also be deployed on a computer as a server and all eyeglass devices as clients, which can achieve multi-device network synchronization effects;

[0145] Synchronization of physical interaction logic: According to the actual interaction process and result goals, the underlying logic of the physical simulation engine is embedded in the spatial world system to realize the underlying functional requirements of physical simulation in 3D space, and then extended based on this, including interactive collaboration and workflow simulation;

[0146] Interactive collaboration is a solution for multi-person custom editing of 3D content in MR mixed reality. Every time the client performs a step, it will exchange information with the server device. The server device will update the 3D object instance data and determine the next frame, and synchronize it to the client to achieve a joint viewing effect.

[0147] Workflow simulation is a complete process system created by combining multiple production industrial processes, including 3D modeling, keyframe animation and game development. Each interactive event is marked as a key node. The judgment of the key node forms multiple event branches. Multiple event branches correspond to multiple final results, triggering different animation presentation methods. This logic is calculated and judged in real time and synchronized to each client through the server to realize workflow simulation of multi-person joint operation.

[0148] The multi-terminal coordinate alignment and calibration module calibrates and aligns the coordinate systems of various devices through spatial positioning technology and coordinate conversion algorithms;

[0149] The multi-terminal coordinate alignment and calibration module calibrates and aligns the coordinate systems of each device through spatial positioning technology and coordinate conversion algorithms, which is achieved through the following three aspects:

[0150] AR image recognition: independently develop positioning alignment client, integrate AR image recognition technology and processing capabilities, design recognition map of complex patterns with specific rules. After the camera captures the image, it will take pictures and detect the feature points of the complex pattern shape, color, structure and light and dark, and construct the anchored three-dimensional information of the image. Through the interface, the object model with specific three-dimensional vector coordinates will be anchored on the coordinates of the reconstructed image center point and mapped to the client screen in real time;

[0151] Axial center point alignment calibration: Based on the anchor point in the MR glasses device, align and calibrate the scene origin (0,0,0) of the third-party perspective client. After the image is successfully recognized, drag the recognition image to the anchor point position in the MR glasses device, and rotate the image to make the position and orientation of the 3D vector coordinate object model anchored on the image consistent with the 3D vector coordinate object model in the MR glasses. After ensuring alignment, the client saves the position information calibration conversion data file locally. All subsequent fusion clients that need to be used will read this data file as the only anchor point aligned with the center point of the MR glasses.

[0152] Screen fusion and synchronization: Aiming at the application scenarios of fusion screens, we independently develop a fusion screen client, whose core functions focus on two points: first, using real-time rendering technology to accurately integrate virtual images into the real scenes captured by real cameras, to achieve synchronized display of virtual and real scenes without boundaries; second, using a mirrored multi-person synchronization mechanism, in which the computer serves as a single instance of the integrated server and local client, and achieves efficient internal communication and scene sharing through direct function calls and message queues, while the remote MR glasses client is connected to the server through a local area network, participates in real-time interaction, and jointly builds an immersive multi-person mixed reality experience.

[0153] Embodiment 2: The technical solution of the embodiment of the present invention is different from that of embodiment 1 in that:

[0154] like Figure 2 As shown, the mixed reality multi-person interactive large space positioning, alignment, and collaboration method specifically includes the following steps:

[0155] Step 1: Use the visual input provided by the MR glasses device, combined with the depth sensor and scanning system to scan and model the real environment, and build a virtual space corresponding to the real environment in the virtual environment;

[0156] Step 2: Select one or more center points in the virtual space as spatial anchor points, and use positioning technology to accurately anchor these anchor points to ensure that their positions in the virtual space are fixed;

[0157] Step 3: Perform computational simulation on the collected spatial information to determine the real-time status of virtual objects in the virtual space, map the virtual objects into the real environment, and achieve integration with the real space;

[0158] Step 4: Combine the mapped virtual objects with the real space to form a mixed reality environment. Users can get an immersive hyper-dimensional space experience through MR glasses.

[0159] Step 5: Share information, assign tasks and interact in real time among users to achieve multi-device network synchronization and ensure that all client devices start playing content together;

[0160] Step 6: Calibrate the coordinate system of each user device. Based on the calibration results, align the coordinate systems of each device to ensure that the display positions of virtual objects on different devices are consistent, thus achieving seamless collaboration and interactive experience across devices.

[0161] The working principle of the present invention: when in use, in the present invention, information sharing, task allocation and real-time interaction between users are realized through a multi-person collaboration module, a multi-device shared coordinate system is used to ensure cross-device operation consistency, simplify configuration, and improve collaboration efficiency, and real-time data transmission is realized through multi-device network synchronization, delays are reduced, and collaboration fluency is enhanced. The physical interaction logic synchronization enhances the interactive realism, optimizes the workflow, and improves the collaboration experience. Through the combination of the multi-device shared coordinate system, the multi-device network synchronization, and the physical interaction logic synchronization, on the one hand, the efficiency and fluency of collaboration are improved, and on the other hand, the user's immersion and sense of participation are enhanced, which provides team collaboration and project management with a better experience. It provides strong support, and calibrates and aligns the coordinate systems of various devices through the multi-terminal coordinate alignment and calibration module. Through AR image recognition and efficient AR technology, it accurately identifies and locates, adapts to changing environments, and improves calibration efficiency. Through axial center point alignment and calibration, it accurately calculates and aligns, reduces error accumulation, ensures multi-device coordination and consistency, and enhances system stability. Through screen fusion synchronization, real-time synchronization of images ensures content consistency, seamless switching experience, and improves user satisfaction. Through the combination of AR image recognition, axial center point alignment and calibration, and screen fusion synchronization, efficient, accurate, and stable multi-terminal collaborative work is achieved, providing users with a smoother and more consistent experience.

[0162] The above are only preferred specific implementation modes of the present invention; however, the protection scope of the present invention is not limited thereto; any technician familiar with the technical field within the technical scope disclosed by the present invention; any equivalent replacement or change based on the technical solution and improved concept of the present invention shall be covered within the protection scope of the present invention.

Claims

1. Mixed reality multi-person interactive large space positioning, alignment, and collaboration system, characterized by: It includes cloud service platform, space construction module, space anchor positioning module, space information analysis module, mixed reality module, multi-person collaboration module and multi-terminal coordinate alignment and calibration module; The space construction module builds a virtual space based on MR glasses, depth sensors and scanning systems; The spatial anchor point positioning module anchors the center point of the virtual space through the spatial anchor point positioning technology; The spatial information analysis module performs computational simulation and mapping of the real-time status of objects in space; The mixed reality module combines virtual objects with real space to form a highly immersive hyper-dimensional space experience of mixed reality; Multi-person collaboration module, which uses network communication technology and collaborative algorithms to achieve information sharing, task allocation and real-time interaction among users; The multi-terminal coordinate alignment and calibration module calibrates and aligns the coordinate systems of various devices through spatial positioning technology and coordinate conversion algorithms.

2. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: MR glasses use their high-definition color perspective function to make the images that penetrate the real space consistent with the colors seen by the naked eye in the real space; the depth sensor measures the distance from the surface of the object to the sensor by emitting and receiving light, thereby obtaining the depth information of the object; The scanning system can intelligently understand and respond to objects in the physical space, and generate a mesh model by scanning the object with a light beam.

3. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: The specific process of the spatial anchor positioning module anchoring the center point of the virtual space through the spatial anchor technology is as follows: Add the OVRSpatialAnchor component to the actual developed GameObject. Through this layer of code association, after the space is created, the information of the origin of the space (0,0,0) will be calculated. The anchor points created in the space have unique identification symbols and can be obtained and called in the form of logs. After obtaining the above basic information, adjust the logical relationship at the application level. The code layer only retains the basic layer of obtaining the origin of the own space (0,0,0) without any processing. Shift the focus to the anchor point. After creating an unbound positioning anchor point, instantiate GameObject+OVRSpatialAnchor, bind the anchor point to the instance, and use the SaceAnchorAsvnc method to persist the anchor point. The return information during the anchor point persistence process includes coordinate information and three-dimensional coordinate axis system information. The anchor point coordinate information (x, y, z) value is reset to the new coordinate (0,0,0), and the coordinate axis system information is assigned to the world coordinate axis system as the overall reference object. After processing these two definitions, write the judgment condition to automatically align the relative coordinates and orientations of the objects in the world added later to the new anchor point, thereby completing the anchoring of all world coordinate centers.

4. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 3 is characterized in that: During the anchoring process, two core processes need to be completed: Process 1: In the real space, place a three-dimensional reference object at the planned anchor point, and mark this place as the anchor point (0,0,0); Process 2: Define the three-axis orientation in the virtual space. According to the edge lines of the reference object and the rule that the angle between each vector in the three-dimensional coordinate system is 90 degrees, draw vertices and connect them to define the XYZ three-axis orientation.

5. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: The specific process of the spatial information analysis module performing computational simulation and mapping presentation on the real-time state of an object in space is as follows: Based on the inside-out spatial monitoring of SLAM technology, the depth and spatial distance information are obtained, and the displacement information and rotation information of the object are processed respectively. For displacement information, the length, width and height information of the space are obtained through the OVRSpatialAnchor component. Since the anchor point (0,0,0) has been defined, the distances of all points in this space relative to the anchor point are mapped to the instance object through code conditions. With the inside-out spatial monitoring of the device, the real-time displacement calculation and mapping of the object can be realized; In terms of rotation information, the XYZ axis vector of the anchor point is used as the world axis reference. Through code rotation control, the rotation of all points in space is calculated relative to the coordinate axis of the anchor point and synchronously mapped to the instance object. In combination with the inside-out space monitoring of the device, real-time displacement calculation and mapping of objects can be achieved. By binding the displacement and rotation three-dimensional spatial information of the object with the anchor point, the real-time state of the object in space can be simulated and mapped.

6. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 5, characterized in that: Inside-out space monitoring based on SLAM technology, the specific process of obtaining depth and space distance information is as follows: The Inside-out SLAM system uses RGB-D cameras, LiDAR, and IMU sensors to obtain environmental information. The RGB-D camera obtains color images and depth images, and the depth image directly obtains the distance information from each pixel to the camera. The IMU provides the acceleration and angular velocity information of the device to help estimate the posture change of the device. Extract feature points from color images and perform feature matching between consecutive frames, use algorithms to detect key points, and calculate descriptors for each key point; The PnP algorithm is used to estimate the camera's pose through the matched feature points. The PnP algorithm specifically includes the following steps: Given a set of 3D points Pm = (Xm, Ym, Zm) and the corresponding 2D image points pm = (Um, Vm), the mapping is established through the following relationship: Where K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation vector, and s is the scale factor; Combine the matched feature points and pose information to reconstruct the 3D model of the environment through the depth image and camera pose; Use BA algorithm to optimize camera trajectory and map point position to reduce cumulative error; The BA algorithm is as follows: The goal is to minimize the reprojection error: Where P mn is the observed 2D point, π is the projection function; Calculate the distance between any two points in space through the camera's position and depth information; The distance calculation steps are as follows: For two points A(Xa,Ya,Za) and B(Xb,Yb,Zb) in 3D space, their spatial distance d is calculated by the following formula: Combined with IMU data, filters are used to update status and achieve real-time environmental monitoring.

7. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: The specific process of the mixed reality module combining virtual objects with real space to form a mixed reality, an immersive hyper-dimensional space experience, is as follows: By integrating spatial positioning technology and spatial distance calculation functions, virtual objects are created using 3D modeling technology and real-time lighting post-processing effect rendering. At the development level, virtual objects are anchored in the space scene through real-time coordinate calculation, gesture recognition and posture tracking gyroscope external technology are added, specific triggered object feedback information is written, collision objects are added, real physical collision and rebound mechanical phenomena are simulated, real physical perception is created, and finally projected in real time on the device screen; Use gestures and posture tracking gyroscopes to trigger object interactions, and highly simulate physical mechanics of the actual physical environment to comprehensively form a highly immersive hyper-dimensional space experience of mixed reality.

8. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: The multi-person collaboration module uses network communication technology and collaborative algorithms to achieve information sharing, task allocation and real-time interaction among users, which is specifically achieved through the following three aspects: Multi-device shared coordinate system: In the state of multiple devices, each device follows the unified original standard process "space creation--anchor point (0,0,0) creation--axis definition" to complete the fixation of the same world coordinate origin and axis. The SaceAnchorAsvnc method is used to persist the anchor point, and a cache mechanism is developed to record the information in a local folder. In actual use, a command will be issued before the code runs, leading all devices to call the cache file in the folder at the same time, thereby establishing a unified object mapping rule; Multi-device network synchronization: We independently develop a real-time communication mechanism specifically for LAN environments, and design an efficient server-client information distribution and processing process. All devices communicate in the wireless LAN. After the client role device goes online, it first sends a message to the server role device. The server role device is responsible for centralized detection and judgment of the number and status of the client role devices. When these information meet certain judgment requirements, the server device sends a start command to each client device to start content playback together. Synchronization of physical interaction logic: According to the actual interaction process and result goals, the underlying logic of the physical simulation engine is embedded in the spatial world system to realize the underlying functional requirements of physical simulation in 3D space, and then extended based on this, including interactive collaboration and workflow simulation.

9. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: The multi-terminal coordinate alignment and calibration module calibrates and aligns the coordinate systems of each device through spatial positioning technology and coordinate conversion algorithms, which is specifically achieved through the following three aspects: AR image recognition: independently develop positioning alignment client, integrate AR image recognition technology and processing capabilities, design recognition map of complex patterns with specific rules. After the camera captures the image, it will take pictures and detect the feature points of the complex pattern shape, color, structure and light and dark, and construct the anchored three-dimensional information of the image. Through the interface, the object model with specific three-dimensional vector coordinates will be anchored on the coordinates of the reconstructed image center point and mapped to the client screen in real time; Axial center point alignment calibration: Based on the anchor point in the MR glasses device, align and calibrate the scene origin (0,0,0) of the third-party perspective client. After the image is successfully recognized, drag the recognition image to the anchor point position in the MR glasses device, and rotate the image to make the position and orientation of the 3D vector coordinate object model anchored on the image consistent with the 3D vector coordinate object model in the MR glasses. After ensuring alignment, the client saves the position information calibration conversion data file locally. All subsequent fusion clients that need to be used will read this data file as the only anchor point aligned with the center point of the MR glasses. Screen fusion and synchronization: Aiming at the application scenarios of fusion screens, we independently develop a fusion screen client, whose core functions focus on two points: first, using real-time rendering technology to accurately integrate virtual images into the real scenes captured by real cameras, to achieve synchronized display of virtual and real scenes without boundaries; second, using a mirrored multi-person synchronization mechanism, in which the computer serves as a single instance of the integrated server and local client, and achieves efficient internal communication and scene sharing through direct function calls and message queues, while the remote MR glasses client is connected to the server through a local area network, participates in real-time interaction, and jointly builds an immersive multi-person mixed reality experience.

10. A mixed reality multi-person interactive large space positioning, alignment, and collaboration method, characterized in that: The specific steps include: Step 1: Use the visual input provided by the MR glasses device, combined with the depth sensor and scanning system to scan and model the real environment, and build a virtual space corresponding to the real environment in the virtual environment; Step 2: Select one or more center points in the virtual space as spatial anchor points, and use positioning technology to accurately anchor these anchor points to ensure that their positions in the virtual space are fixed; Step 3: Perform computational simulation on the collected spatial information to determine the real-time status of virtual objects in the virtual space, map the virtual objects into the real environment, and achieve integration with the real space; Step 4: Combine the mapped virtual objects with the real space to form a mixed reality environment. Users can get an immersive hyper-dimensional space experience through MR glasses. Step 5: Share information, assign tasks and interact in real time among users to achieve multi-device network synchronization and ensure that all client devices start playing content together; Step 6: Calibrate the coordinate system of each user device. Based on the calibration results, align the coordinate systems of each device to ensure that the display positions of virtual objects on different devices are consistent, thus achieving seamless collaboration and interactive experience across devices.

Images