Mixed reality multi-person interaction large-space positioning, aligning and collaborating system and method

Abstract

The invention relates to the technical field of mixed reality, and discloses a mixed reality multi-user interaction large-space positioning, aligning and collaborating system and method. Comprising the following steps: constructing a gridded virtual space, carrying out central point positioning on a client in the virtual space, carrying out simulation mapping on a real object and the virtual space, carrying out virtual-real combination on the virtual object and real motion, generating a mixed reality scene, and constructing a collaborative interaction network with sub-channels and data enclosing walls. Carrying out fusion alignment on the real-time picture and the mixed reality scene; according to the invention, a data basis for carrying out simulation mapping on the virtual space can be provided for the real object, a consistent mapping association relationship between the real object and the world coordinate system of the virtual space is ensured, and a synchronous transmission and display effect of virtual and real pictures of a plurality of clients and servers in a mixed reality scene is ensured; the effects of accurate calculation simulation and efficient mapping presentation of the real-time state of the real object in the virtual space are achieved.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality technology, and more specifically, to a mixed reality multi-person interaction large-space positioning, alignment, and collaboration system and method. Background Art

[0002] Mixed reality technology, as a combination of virtual reality and augmented reality, is gradually providing users with an unprecedented immersive experience and showing great application potential in many fields such as education and entertainment. In multi-person interaction in mixed reality, it is necessary to ensure that multiple clients can be accurately positioned in a large space and achieve coordinated data transmission and screen alignment.

[0003] Patent application referenced with publication number CN115712351A discloses a hierarchical rendering and interaction method and system for multi-person remote mixed reality sharing scenes. By clustering participants' perspectives, the system significantly reduces repeated rendering of similar perspectives in cloud services. This effectively increases the number of online users in a shared scene using the same cloud computing resources. Rapid processing and fusion of background and foreground images by side services allows for adaptive adjustment of the viewing screen while waiting for cloud rendering data to be delivered, improving the real-time viewing experience on the lower side of the cloud rendering architecture. The system can also support different types of mixed reality applications, such as virtual reality and augmented reality. By separating and processing interactive masks, the side service filters out invalid interaction information, improving the efficiency of cloud service interaction processing.

[0004] Existing mixed reality multi-person interaction systems usually rely on single-dimensional sensor data as a reference for virtual-reality simulation mapping, such as IMU data or visual data, to achieve the mapping association effect between real objects and virtual space. However, when real objects are in high-speed motion or complex occlusion, using single-dimensional mapping data as a mapping reference is prone to mapping errors in the process of virtual-reality simulation mapping, resulting in a large offset between the object state simulation in the virtual space and the real physical state, thereby destroying the consistency of the simulation mapping between real objects and virtual space.

[0005] In view of this, the present invention proposes a mixed reality multi-person interaction large space positioning, alignment, and collaboration system and method to solve the above problems. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned objectives, the present invention provides the following technical solution: a mixed reality multi-person interactive large-space positioning, alignment, and collaboration system, applied to a mixed reality server, comprising:

[0007] The coordinate system positioning module is used to scan the real space clockwise at the scanning point, construct a gridded virtual space, draw a world coordinate system in the virtual space, and locate the center point of the client in the virtual space;

[0008] The mapping binding module is used to identify the spatial area of real objects in the real world, calculate the mapping features of the spatial area, and simulate the mapping between the real objects and the virtual space based on the spatial mapping rules. The mapping features include displacement mapping values and rotation mapping values.

[0009] The spatial mapping rule is: map the spatial points of the spatial area point by point from the outside to the inside;

[0010] The mixed reality module is used to render virtual objects in the virtual space and combine virtual objects with real movements to generate mixed reality scenes;

[0011] The network collaboration module is used to convert discrete clients and servers into network nodes based on interactive collaboration principles, and build a collaborative interaction network with sub-channels and data fences;

[0012] The interactive cooperation principle is: a sub-channel is established between each subordinate node and the superior node;

[0013] The synchronization and alignment module is used to obtain the real-time image of the client, integrate and align the real-time image with the mixed reality scene, and control the collaborative interactive network to synchronously display the real-time image and the virtual image.

[0014] Furthermore, the method for constructing the virtual space is:

[0015] Mark the two adjacent side walls of the room in real space one by one, and record the intersection of the two side walls as the scanning point, obtaining A scanning points;

[0016] Taking A scanning points as reference points and the preset scanning speed as standard, the laser scanner is used to scan the point cloud data of all points inside the room clockwise, and the point cloud data of all points of A scanning points are aggregated to generate A point cloud sets;

[0017] Eliminate duplicate point cloud data in A point cloud sets, and align the remaining point cloud data in A point cloud sets;

[0018] The remaining point cloud data is converted into a mesh model through triangulation technology, and the point cloud data in the missing areas of the mesh model are automatically filled to construct a virtual space.

[0019] Furthermore, the method for center point positioning is:

[0020] Connect any two non-adjacent scanning points in sequence to obtain B scanning lines, and record the area enclosed by the B scanning lines as the central area;

[0021] Identify the midpoint of the central area through computer vision technology and mark the model point where the midpoint of the area is located in the virtual space;

[0022] Use the model point as the coordinate origin, and the right direction, upward direction, and rear direction of the coordinate origin as the X-axis direction, Y-axis direction, and Z-axis direction respectively to draw the world coordinate system;

[0023] Based on the world coordinate system, the three-dimensional coordinates of all point cloud data in the virtual space are redistributed, driving C clients to uniformly locate the center point and axis in the virtual space.

[0024] Furthermore, the calculation method of the displacement mapping value is:

[0025] Mark the point cloud data of E spatial points in D spatial areas one by one, and generate point cloud coordinates after converting the point cloud data into the world coordinate system;

[0026] Using the Euclidean distance calculation formula, calculate the Euclidean distances between the E spatial points in the D spatial regions and the coordinate origin of the world coordinate system one by one, and record the calculated Euclidean distances as displacement mapping values;

[0027] The rotation mapping value is calculated as:

[0028] The spatial point corresponding to the maximum value of the Euclidean distance in the D spatial regions is recorded as the sub-marker point, and the sub-coordinate system with the same axis as the world coordinate system is drawn with the sub-marker point as the coordinate origin to obtain D sub-coordinate systems;

[0029] Mark the coordinates of the spatial points of the D spatial regions on the corresponding sub-coordinate systems one by one, obtain E sub-marking information, identify the virtual points corresponding to the sub-marking information one by one in the virtual space, and mark the coordinates of the virtual points one by one in the world coordinate system, which are recorded as rotation coordinates;

[0030] Using the X-axis, Y-axis, and Z-axis of the world coordinate system as the rotation axes and the origin of the world coordinate system as the rotation base point, the E spatial points in the D spatial regions are rotated one by one to the rotation coordinates of the world coordinate system;

[0031] The rotation Euler angles of E spatial points in D spatial regions between the sub-marker information and the rotation coordinates are measured respectively and recorded as rotation mapping values.

[0032] Furthermore, the control method of the simulation mapping is:

[0033] A1: Number the D spatial regions in ascending order according to the order in which the spatial regions are identified, starting with 1;

[0034] A2: Match all spatial points on the facade in the spatial region with the corresponding displacement mapping values and rotation mapping values one by one in ascending order to generate mapping units, obtaining F mapping units.

[0035] A3: Using the displacement mapping value and the rotation mapping value as the mapping standard, simulate and map the spatial points in the F mapping units one by one to the virtual space;

[0036] A4: Repeat steps A2-A3 until all E spatial points in the D spatial regions are simulated and mapped to the virtual space.

[0037] Furthermore, realistic motion includes gesture motion and collision motion;

[0038] The method for generating mixed reality scenes is:

[0039] In the virtual space, pair the virtual objects with the coordinates of D clients one by one to obtain D virtual sub-scenes;

[0040] In D virtual sub-scenes, the client's real-time palm position is collected through the depth camera, and the coordinates corresponding to the real-time palm position are unified with the coordinates of the virtual object in the world coordinate system to construct D virtual sub-scenes with gesture movement;

[0041] The real-time arm position of the client is collected one by one through the posture tracking gyroscope, the coordinates corresponding to the real-time arm position and the coordinates of the virtual object are unified in the world coordinate system, and the coordinates where the virtual object's shape changes are recorded in real time as trigger coordinates;

[0042] The trigger coordinates and the corresponding real-time arm position coordinates are unified in the world coordinate system to construct D virtual sub-scenes with collision motion. The D virtual sub-scenes with gesture motion and collision motion are then fused to generate a mixed reality scene.

[0043] Furthermore, the network nodes include upper-level nodes and lower-level nodes;

[0044] The method for constructing a collaborative interaction network is:

[0045] Construct a local area network with two upper and lower network layers, and establish an upper node and C lower nodes in the upper network layer and the lower network layer respectively;

[0046] Import the server into the upper node, causing the upper node to be converted into an upper-level node, and import the C clients into the C lower nodes one by one, causing the C clients to be converted into C lower-level nodes;

[0047] C spaced upper ports are configured on the upper node, one lower port is configured on each of the C lower nodes, and sub-channels with annular cross-sections are constructed between the C upper ports and the corresponding C lower ports to obtain C sub-channels;

[0048] Identify the cross-sectional center of the sub-channel, and build a data fence through the cross-sectional center to divide the sub-channel into an upload channel and a download channel, thereby transforming the local area network into a collaborative interactive network.

[0049] Furthermore, the fusion alignment method is:

[0050] The depth camera collects the real-time images of C subordinate nodes at each moment in real time to obtain C real-time images;

[0051] The lower left corner of the C real-time images is used as the coordinate origin, and the right direction, the upper direction, and the front direction of the real-time images are used as the X-axis direction, the Y-axis direction, and the Z-axis direction respectively. The X-axis, Y-axis, and Z-axis are drawn to construct the image coordinate system of the C real-time images;

[0052] Import C real-time images into the mixed reality scene one by one, and based on the coordinate origin of the world coordinate system, align the center points and axial directions of the C image coordinate systems with the world coordinate system to obtain C aligned real-time images.

[0053] Furthermore, the method for synchronously displaying the real-time image and the virtual image is as follows:

[0054] Collect the real-world motions of C subordinate nodes at each moment to obtain C real-world motions. Bind and aggregate the C aligned real-time images with the corresponding C real-world motions to generate C real-world data sets.

[0055] Transmit C real-world data sets to the upper-level node through C upload channels, and aggregate and simulate the real-world motion and real-time images in the C real-world data sets within the collaborative interactive network to simulate a complete virtual image.

[0056] The virtual image is synchronously transmitted to C lower-level nodes through C download channels to control the synchronous display of the real image and the virtual image.

[0057] A mixed reality multi-person interactive large-space positioning, alignment, and collaboration method is applied to a mixed reality server and is implemented based on a mixed reality multi-person interactive large-space positioning, alignment, and collaboration system, including:

[0058] S1: Scan the real space clockwise at the scanning point to construct a gridded virtual space, draw a world coordinate system in the virtual space, and locate the center point of the client in the virtual space;

[0059] S2: Identify the spatial regions of real objects in the real world, calculate the mapping features of the spatial regions, and simulate the mapping between real objects and virtual space based on spatial mapping rules;

[0060] S3: Render virtual objects in the virtual space and combine them with real-world motion to generate a mixed reality scene.

[0061] S4: Based on the interactive collaboration principle, discrete clients and servers are converted into network nodes to build a collaborative interactive network with sub-channels and data walls;

[0062] S5: Obtain the real-time image of the client, integrate and align the real-time image with the mixed reality scene, and control the collaborative interactive network to synchronously display the real-time image and the virtual image.

[0063] The technical effects and advantages of the mixed reality multi-person interactive large space positioning, alignment, and collaboration system and method of the present invention are as follows:

[0064] The present invention introduces a world coordinate system into the virtual space to provide a unified coordinate reference for the center point and axis. This avoids the positioning uncertainty caused by the automatic calibration of the axis and center point by the coordinate system in three-dimensional space, improves the positioning accuracy of multiple clients in a large space, and provides a data basis for simulating the mapping of real objects to the virtual space by calculating the displacement mapping values and rotation mapping values of real objects in the virtual space. This ensures that the real objects maintain a consistent mapping relationship with the world coordinate system of the virtual space, and achieves accurate calculation and simulation of the real state of real objects in the virtual space and efficient mapping presentation. By combining virtual objects with real motion, a mixed reality scene is generated, which can achieve a one-to-one restoration of real objects in the virtual space, ensuring that virtual objects in the virtual space maintain visual consistency with real objects. Finally, by constructing a collaborative interactive network with sub-channels and data fences, it is possible to ensure that multiple clients and servers can independently, securely, and efficiently synchronize the transmission and display of virtual and real images in the mixed reality scene, ensuring that multiple clients can perform mixed reality simulation operations in the same large space, thereby achieving the integrated purpose of positioning, alignment, and collaboration in large-scale multi-person interactive spaces in mixed reality applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of the modules of the mixed reality multi-person interactive large space positioning, alignment, and collaboration system provided in the first embodiment of the present invention;

[0066] Figure 2This is a flow chart of the mixed reality multi-person interactive large space positioning, alignment, and collaboration system provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0068] Example 1: Please refer to Figure 1 As shown, the mixed reality multi-person interactive large space positioning, alignment, and collaboration system described in this embodiment is applied to a mixed reality server and includes:

[0069] The coordinate system positioning module scans the real space clockwise at the scanning point, constructs a gridded virtual space, and draws a world coordinate system in the virtual space to locate the center point of the client in the virtual space;

[0070] Before engaging in mixed reality multi-person large-space interaction, each user will be treated as a client, allowing multiple clients to perform subsequent positioning, alignment, and collaborative processing in a larger room;

[0071] In order to achieve the positioning, alignment and collaborative operation of multiple clients, each client needs to be equipped with the required hardware equipment, including MR glasses, depth cameras and laser scanners. MR glasses can ensure that the images penetrating the real space are consistent with the colors seen by the naked eye in the real space. The depth camera can measure 3D depth more accurately than ordinary RGB cameras. By emitting and receiving light, the depth camera can measure the distance from the surface of an object in the real space to the depth camera, thereby obtaining the depth information of the object. The laser scanner can intelligently understand and respond to objects in the real space, scan objects through light beams and generate mesh models.

[0072] The real space is the physical space that the client can see directly with the naked eye, which can be regarded as a room where multiple clients are located.

[0073] After scanning objects in the real space, the scanning results can be used to construct a virtual space corresponding to the real space, so that the virtual space can maintain consistency with the layout, object placement, room shape and structure of the real space, thereby achieving a one-to-one conversion effect between the real space and the virtual space.

[0074] The method of constructing the virtual space is:

[0075] Mark two adjacent side walls in the room in real space one by one, and record the intersection of the two side walls as a scanning point to obtain A scanning points. The scanning points are used to provide laser scanning points for the laser scanner. They correspond to different positions in the room, and any two scanning points are independent of each other.

[0076] Taking A scanning point as the reference point and the preset scanning speed as the standard, the laser scanner scans the point cloud data of all points inside the room clockwise, and then aggregates the point cloud data of all points at A scanning points to generate A point cloud sets; the preset scanning speed refers to the swing speed of the laser scanner when scanning clockwise, which can be used to numerically limit the movement speed of the laser scanner;

[0077] Remove duplicate point cloud data from the A point cloud sets, and perform registration and alignment on the remaining point cloud data in the A point cloud sets. Registration and alignment is the process of aligning point cloud data acquired from different perspectives or at different times to the same coordinate system. Registration and alignment is common knowledge among those skilled in the art and is not considered an innovation in this application, so it will not be elaborated on in detail.

[0078] The remaining point cloud data is converted into a mesh model through triangulation technology, and the point cloud data in the missing areas of the mesh model is automatically filled in to construct a virtual space. Triangulation technology is the process of converting point cloud data or other geometric data into a mesh surface composed of triangles. Triangulation technology is common knowledge to those skilled in the art and is not considered an innovation in this application, so it will not be elaborated on here.

[0079] After the virtual space is constructed, it is necessary to integrate multiple clients into the virtual space at the same time and make them have the same coordinate reference system in the virtual space, so that multiple clients can achieve a unified center point positioning effect in the virtual space;

[0080] When achieving unified center point positioning for multiple clients, it is necessary to draw a world coordinate system in the virtual space so that the world coordinate system can provide a reference coordinate system for center point positioning for multiple clients in the virtual space. Moreover, within the constraints of the world coordinate system, the center point of the client can be positioned so that multiple clients can have the same origin and axis in the world coordinate system.

[0081] The method of center point positioning is:

[0082] Connect any two non-adjacent scanning points in sequence to obtain B scanning lines, and record the area enclosed by the B scanning lines as the central area;

[0083] The midpoint of the central area is identified using computer vision technology, and the model point where the midpoint is located is marked in virtual space. The model point refers to the specific location in the model corresponding to the midpoint of the area in virtual space, and serves as the basis for the subsequent construction of the world coordinate system.

[0084] Use the model point as the coordinate origin, and the right direction, upward direction, and rear direction of the coordinate origin as the X-axis direction, Y-axis direction, and Z-axis direction respectively to draw the world coordinate system;

[0085] Based on the world coordinate system, the three-dimensional coordinates of all point cloud data in the virtual space are redistributed, driving C clients to uniformly locate the center point and axis in the virtual space.

[0086] It should be noted that after the C clients are uniformly positioned at the center point and along the axis in the virtual space, multiple clients will use the same coordinate system for the same object in the virtual space, allowing the information seen by the C clients in the virtual space to remain consistent, laying the foundation for subsequent alignment and collaborative interaction.

[0087] The mapping binding module identifies the spatial area of real objects in the real world, calculates the mapping features of the spatial area, and simulates the mapping between the real objects and the virtual space based on the spatial mapping rules. The mapping features include displacement mapping values and rotation mapping values.

[0088] Real objects are the various types of specific items and equipment within a room in real space. They serve as the direct targets for subsequent client-side mixed reality large-space interaction. Since real objects have a specific shape and volume in real space, in order to spatially correspond real objects with virtual space, it is necessary to determine the spatial region of the real objects so that the spatial region can represent the specific morphological structure of the real objects.

[0089] The method for identifying spatial regions is:

[0090] Mark the facades of D real objects inside the room in real space, and query the point cloud data corresponding to the points on the facades one by one to obtain the regional point cloud;

[0091] Based on the regional point cloud, a regional outline is constructed in virtual space. The point cloud data within the regional outline and the area corresponding to the regional point cloud are recorded as spatial regions. D spatial regions are obtained, and the points contained in the spatial regions are recorded as spatial points. The regional outline is the basic outline for constructing the spatial region and is used to provide the overall external contour constraints for the spatial region.

[0092] When the spatial regions are identified, the D spatial regions can respectively accurately represent the positions of the D real objects, and enable the spatial regions to be used as objects for subsequent displacement and rotation adjustments with the virtual space.

[0093] When performing a mapping process of displacement and rotation between a spatial region and a virtual space, it is necessary to calculate the mapping features of the spatial region in the mapping process so that the mapping features can serve as a basis for mapping the spatial region in the virtual space;

[0094] Specifically, the mapping features include displacement mapping values and rotation mapping values;

[0095] The displacement mapping value is used to represent the displacement amplitude between the coordinate origin of the world coordinate system when mapping the spatial area into the virtual space;

[0096] The displacement map value is calculated as:

[0097] Mark the point cloud data of E spatial points in D spatial areas one by one, and generate point cloud coordinates after converting the point cloud data into the world coordinate system;

[0098] Using the Euclidean distance calculation formula, calculate the Euclidean distances between the E spatial points in the D spatial regions and the coordinate origin of the world coordinate system one by one, and record the calculated Euclidean distances as displacement mapping values;

[0099] The calculation formula of Euclidean distance is:

[0100]

[0101] Where, OS jlde is the Euclidean distance of the e-th spatial point in the d-th spatial region, d = 1, 2...D, e = 1, 2...E, X 1de is the X-axis element of the e-th spatial point in the d-th spatial region, and Y 1de is the Y-axis element of the e-th spatial point in the d-th spatial region, and Z 1de is the Z-axis element of the e-th spatial point in the d-th spatial region, and X0, Y0, and Z0 are the X-axis element, Y-axis element, and Z-axis element of the coordinate origin of the world coordinate system, respectively.

[0102] The rotation mapping value is used to represent the rotation amplitude between the axis of the world coordinate system when mapping the spatial area into the virtual space;

[0103] The rotation mapping value is calculated as:

[0104] The spatial point corresponding to the maximum value of the Euclidean distance in the D spatial regions is recorded as the sub-marker point, and the sub-coordinate system with the same axis as the world coordinate system is drawn with the sub-marker point as the coordinate origin to obtain D sub-coordinate systems;

[0105] Mark the coordinates of the spatial points in the D spatial regions on the corresponding sub-coordinate system one by one to obtain E sub-marking information;

[0106] The virtual points corresponding to the sub-marker information are identified one by one in the virtual space, and the coordinates of the virtual points are marked one by one on the world coordinate system, which are recorded as rotation coordinates. The method of marking the rotation coordinates on the world coordinate system can accurately and uniquely identify the specific coordinate position corresponding to the sub-marker information on the world coordinate system, thereby providing a precise point positioning effect for subsequent rotation mapping of the spatial area to the virtual space;

[0107] Using the X-axis, Y-axis, and Z-axis of the world coordinate system as the rotation axes and the origin of the world coordinate system as the rotation base point, the E spatial points in the D spatial regions are rotated one by one to the rotation coordinates of the world coordinate system;

[0108] The rotation Euler angles of E spatial points in D spatial regions between the sub-marker information and the rotation coordinates are measured respectively and recorded as rotation mapping values.

[0109] After calculating the displacement mapping value and the rotation mapping value, the real object can be simulated and mapped according to the displacement mapping value and the rotation mapping value, so that the real object can be accurately and effectively mapped to the virtual space and maintain a mapping relationship with the world coordinate system, thereby realizing the computational simulation and mapping presentation effect of the real-time state of the object in space.

[0110] When simulating the mapping of real objects, it is necessary to perform the mapping under the constraints of spatial mapping rules to ensure that the mapping process of real objects has high accuracy and orderliness;

[0111] The spatial mapping rule is: map the spatial points of the spatial area point by point from the outside to the inside; this ensures that each spatial area can maintain the order of the overall simulation mapping from the outside to the inside, and can also perform independent and orderly simulation mapping on each spatial point.

[0112] The control method of simulation mapping is:

[0113] A1: Number the D spatial regions in ascending order according to the order in which the spatial regions are identified, starting with 1;

[0114] A2: Match all spatial points on the facade in the spatial region with the corresponding displacement mapping values and rotation mapping values one by one in ascending order to generate mapping units, obtaining F mapping units.

[0115] A3: Using the displacement mapping value and the rotation mapping value as the mapping standard, simulate and map the spatial points in the F mapping units one by one to the virtual space;

[0116] A4: Repeat steps A2-A3 until all E spatial points in the D spatial regions are simulated and mapped to the virtual space.

[0117] It should be noted that, in the simulation mapping process, each mapping unit corresponds to different mapping displacement parameters and mapping rotation parameters, which can ensure that all real objects in the real space can be accurately and orderly integrated into the virtual space.

[0118] The mixed reality module renders virtual objects in a virtual space and combines virtual objects with real-world motion to generate a mixed reality scene.

[0119] Virtual objects are virtual objects that are created by mapping real objects into virtual space and rendering their colors. This allows real objects to be restored one-to-one in virtual space, ensuring that virtual objects in virtual space maintain visual consistency with real objects.

[0120] Since virtual objects have colors, rendering operations are required when constructing virtual objects in the virtual space to ensure that the virtual objects can be realistically projected onto the client's visual screen;

[0121] Specifically, when rendering virtual objects in a virtual space, a simulated mapping spatial area is first marked in the virtual space. The original colors, lighting data under different colors, and lighting data under different brightness of D real objects are collected one by one through the MR glasses. Then, the collected original colors, lighting data under different colors, and lighting data under different brightness of the real objects are matched one by one with the world coordinate system of the virtual space. Finally, through 3D modeling technology, the collected original colors, lighting data under different colors, and lighting data under different brightness of the real objects are rendered in real time onto the spatial area to render the virtual object.

[0122] After rendering the virtual object, it is necessary to effectively combine the virtual object with the real movement, so that the impact of the client's physical movement in the real space on the virtual object can be mixed into virtual and real, so that the virtual object can make corresponding changes based on the real movement to achieve the effect of mixed reality. The state of the combination of real movement and virtual object is recorded as a mixed reality scene.

[0123] Realistic motion includes gesture motion and collision motion. Gesture motion is used to represent changes in the client's palm, including but not limited to clenching a fist, spreading fingers, and bending fingers. Collision motion is used to represent changes in the client's arm position, including but not limited to waving, swinging, and raising hands.

[0124] The method for generating mixed reality scenes is:

[0125] In the virtual space, virtual objects are paired with the coordinates of D clients one by one to obtain D virtual sub-scenes. Virtual sub-scenes are the smallest unit for building mixed reality scenes, providing a unique and independent virtual reality mixing foundation for each client.

[0126] In D virtual sub-scenes, the client's real-time palm position is collected through the depth camera, and the coordinates corresponding to the real-time palm position are unified with the coordinates of the virtual object in the world coordinate system to construct D virtual sub-scenes with gesture movement;

[0127] The real-time arm position of the client is collected one by one through the posture tracking gyroscope, the coordinates corresponding to the real-time arm position and the coordinates of the virtual object are unified in the world coordinate system, and the coordinates where the virtual object's shape changes are recorded in real time as trigger coordinates;

[0128] The trigger coordinates and the corresponding real-time arm position coordinates are unified in the world coordinate system to construct D virtual sub-scenes with collision motion. The D virtual sub-scenes with gesture motion and collision motion are then fused to generate a mixed reality scene.

[0129] The network collaboration module converts discrete clients and servers into network nodes based on interactive collaboration principles, building a collaborative interaction network with sub-channels and data fences.

[0130] After the mixed reality scene is constructed, multiple discrete clients and servers can be integrated into the mixed reality scene one by one, allowing the client and server to interact in the same virtual environment.

[0131] The server is a port device that can provide virtual video images to the client and is used to communicate with the client's MR glasses via a wireless network. Specifically, the server includes but is not limited to computers, VR devices, etc.

[0132] Since multiple clients are usually in a discrete state, in order to improve the network synchronization and collaboration between multiple clients and the server, it is necessary to convert the discrete clients and servers into network nodes respectively, and then build a collaborative interaction network based on the converted network nodes;

[0133] The collaborative interactive network is a local area network that ensures that the client and server can exchange data and instructions within the same network environment. This provides a stable and closed network environment for the exchange of data and instructions between the client and server. The network node is the smallest component of the collaborative interactive network and can independently represent different port devices.

[0134] Network nodes include upper-level nodes and lower-level nodes; upper-level nodes are nodes in the collaborative interactive network used to receive data and generate and send instructions, and upper-level nodes correspond to clients; lower-level nodes are nodes in the collaborative interactive network used to generate data and receive instructions, and lower-level nodes correspond to clients.

[0135] In a collaborative interaction network, it is necessary to ensure that the upper node and each lower node can maintain a relatively independent data transmission state. Therefore, when building a collaborative interaction network, it is necessary to combine the interactive cooperation principles to build it;

[0136] The interactive collaboration principle is: a sub-channel is established between each lower-level node and the upper-level node; thus ensuring that each client can maintain an independent and stable transmission interaction state with the server.

[0137] The method for constructing a collaborative interaction network is:

[0138] Construct a local area network with two upper and lower network layers, and establish an upper node and C lower nodes in the upper network layer and the lower network layer respectively;

[0139] Import the server into the upper node, causing the upper node to be converted into an upper-level node, and import the C clients into the C lower nodes one by one, causing the C clients to be converted into C lower-level nodes;

[0140] C spaced upper-level ports are configured on the upper-level node, and one lower-level port is configured on each of the C lower-level nodes. Sub-channels with annular cross-sections are constructed between the C upper-level ports and the corresponding C lower-level ports to obtain C sub-channels. The sub-channels with annular cross-sections can ensure the closed transmission of data within the sub-channels, preventing data leakage, and also facilitate the subsequent division of sub-channels.

[0141] Identify the cross-sectional center of the sub-channel, and build a data fence through the cross-sectional center to divide the sub-channel into an upload channel and a download channel, thereby transforming the local area network into a collaborative interactive network.

[0142] It should be noted that the data wall is a virtual network fence used to divide the sub-channel into two. It can not only achieve the effect of dividing the channel cavity of the sub-channel into two, but also ensure the independence of the divided upload channel and download channel, and prevent the data in the upload channel and download channel from interfering with each other.

[0143] The synchronization and alignment module obtains the real-time image of the client, integrates and aligns the real-time image with the mixed reality scene, and controls the collaborative interactive network to display the real-time image and the virtual image synchronously;

[0144] The real-time image refers to the real image captured in real time by the depth camera worn by the client, so that the real-time image can be used as the image for the client to input data into the mixed reality scene, ensuring that the mixed reality scene can effectively combine the real scene and the virtual scene;

[0145] After obtaining the real-time image, since the coordinate system of each client in the mixed reality scene is inconsistent, the coordinates corresponding to the real-time image obtained by the client are also inconsistent. In order to ensure that all real-time images can maintain consistent coordinates with the mixed reality scene, it is necessary to perform a fusion alignment operation on the real-time image and the mixed reality to ensure that the coordinate system of the real-time image is consistent with the coordinate system of the mixed reality scene.

[0146] The fusion alignment method is:

[0147] The depth camera collects the real-time images of C subordinate nodes at each moment in real time to obtain C real-time images;

[0148] The lower left corner of the C real-time images is used as the coordinate origin, and the right direction, the upper direction, and the front direction of the real-time images are used as the X-axis direction, the Y-axis direction, and the Z-axis direction respectively. The X-axis, Y-axis, and Z-axis are drawn to construct the image coordinate system of the C real-time images;

[0149] Import C real-time images into the mixed reality scene one by one, and based on the coordinate origin of the world coordinate system, align the center points and axial directions of the C image coordinate systems with the world coordinate system to obtain C aligned real-time images.

[0150] After the real-time image is integrated and aligned with the mixed reality scene, the C clients can now achieve large-space positioning, alignment, and collaboration with the server. This allows for the subsequent interactive transmission effect of the collaborative interactive network, allowing for the synchronous display of the real-time image and the virtual image in the mixed reality scene, ensuring the continuity of the image in the mixed reality scene and preventing image lag.

[0151] The method for synchronously displaying real-time images and virtual images is as follows:

[0152] Collect the real-world motions of C subordinate nodes at each moment to obtain C real-world motions. Bind and aggregate the C aligned real-time images with the corresponding C real-world motions to generate C real-world data sets.

[0153] C real-world data sets are transmitted to the upper-level node through C upload channels. The real-world motion and real-time images in the C real-world data sets are aggregated and simulated within the collaborative interactive network to simulate a complete virtual image. Once all the real-world motion and real-time images are aggregated on the server side, the corresponding virtual images can be simulated according to the pre-set mixed reality simulation rules, so that the virtual images can provide feedback and interact with all the real-world motion and real-time images.

[0154] The virtual image is synchronously transmitted to C lower-level nodes through C download channels to control the synchronous display of the real image and the virtual image.

[0155] In this embodiment, by introducing a world coordinate system in the virtual space to provide a unified coordinate reference basis for the center point and the axis, the positioning uncertainty caused by the automatic calibration of the axis and the center point by the coordinate system in the three-dimensional space can be avoided, and the positioning accuracy of multiple clients in a large space can be improved. By calculating the displacement mapping value and rotation mapping value of the real object in the virtual space, a data basis for simulating the mapping of the real object to the virtual space can be provided, which can ensure that the real object and the world coordinate system of the virtual space maintain a consistent mapping relationship, and realize the accurate calculation simulation and efficient mapping presentation of the real object's real-time state in the virtual space. By combining virtual objects with real-world movements to generate mixed reality scenes, real objects can be restored one-to-one in the virtual space, ensuring that virtual objects in the virtual space can maintain visual consistency with real objects. Finally, by building a collaborative interactive network with sub-channels and data walls, it can be ensured that multiple clients and servers can independently, securely and efficiently transmit and display virtual and real images synchronously in the mixed reality scene, ensuring that multiple clients can perform mixed reality simulation operations in the same large space, thereby achieving the integrated purpose of positioning, alignment and collaboration of large-scale interactive spaces for multiple people in mixed reality applications.

[0156] Example 2: Please refer to Figure 2 As shown, for parts not described in detail in this embodiment, please refer to the description of Example 1. A method for large-space positioning, alignment, and collaboration in mixed reality multi-person interaction is provided, which is applied to a mixed reality server and is implemented based on a large-space positioning, alignment, and collaboration system for mixed reality multi-person interaction, including:

[0157] S1: Scan the real space clockwise at the scanning point to construct a gridded virtual space, draw a world coordinate system in the virtual space, and locate the center point of the client in the virtual space;

[0158] S2: Identify the spatial regions of real objects in the real world, calculate the mapping features of the spatial regions, and simulate the mapping between real objects and virtual space based on spatial mapping rules;

[0159] S3: Render virtual objects in the virtual space and combine them with real-world motion to generate a mixed reality scene.

[0160] S4: Based on the interactive collaboration principle, discrete clients and servers are converted into network nodes to build a collaborative interaction network with sub-channels and data walls;

[0161] S5: Obtain the real-time image of the client, integrate and align the real-time image with the mixed reality scene, and control the collaborative interactive network to synchronously display the real-time image and the virtual image.

[0162] The mapping features include displacement mapping values and rotation mapping values. The spatial mapping rule is: mapping the spatial points of the spatial area point by point from the outside to the inside.

[0163] The network nodes include upper and lower nodes, and the interactive cooperation principle is: a sub-channel is established between each lower node and the upper node;

[0164] Realistic motion includes gesture motion and collision motion;

[0165] The method for identifying spatial regions is:

[0166] Mark the facades of D real objects inside the room in real space, and query the point cloud data corresponding to the points on the facades one by one to obtain the regional point cloud;

[0167] Based on the regional point cloud, a regional outline is constructed in the virtual space, and the point cloud data located within the regional outline and the area corresponding to the regional point cloud are recorded as spatial regions. D spatial regions are obtained, and the points contained in the spatial regions are recorded as spatial points.

[0168] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A mixed reality multi-person interactive large space positioning, alignment, and collaboration system, applied to a mixed reality server, characterized by: include: The coordinate system positioning module is used to scan the real space clockwise at the scanning point, construct a gridded virtual space, draw a world coordinate system in the virtual space, and locate the center point of the client in the virtual space; The mapping binding module is used to identify the spatial area of real objects in the real world, calculate the mapping features of the spatial area, and simulate the mapping between the real objects and the virtual space based on the spatial mapping rules. The mapping features include displacement mapping values and rotation mapping values. The spatial mapping rule is: map the spatial points of the spatial area point by point from the outside to the inside; The mixed reality module is used to render virtual objects in the virtual space and combine virtual objects with real movements to generate mixed reality scenes; The network collaboration module is used to convert discrete clients and servers into network nodes based on interactive collaboration principles, and build a collaborative interaction network with sub-channels and data fences; The interactive cooperation principle is: a sub-channel is established between each subordinate node and the superior node; The synchronization and alignment module is used to obtain the real-time image of the client, integrate and align the real-time image with the mixed reality scene, and control the collaborative interactive network to synchronously display the real-time image and the virtual image.

2. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 1, characterized in that: The method of constructing the virtual space is: Mark the two adjacent side walls of the room in real space one by one, and record the intersection of the two side walls as the scanning point, obtaining A scanning points; Taking A scanning points as reference points and the preset scanning speed as standard, the laser scanner is used to scan the point cloud data of all points inside the room clockwise, and the point cloud data of all points of A scanning points are aggregated to generate A point cloud sets; Eliminate duplicate point cloud data in A point cloud sets, and align the remaining point cloud data in A point cloud sets; The remaining point cloud data is converted into a mesh model through triangulation technology, and the point cloud data in the missing areas of the mesh model are automatically filled to construct a virtual space.

3. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 2, characterized in that: The method of center point positioning is: Connect any two non-adjacent scanning points in sequence to obtain B scanning lines, and record the area enclosed by the B scanning lines as the central area; Identify the midpoint of the central area through computer vision technology and mark the model point where the midpoint of the area is located in the virtual space; Use the model point as the coordinate origin, and the right direction, upward direction, and rear direction of the coordinate origin as the X-axis direction, Y-axis direction, and Z-axis direction respectively to draw the world coordinate system; Based on the world coordinate system, the three-dimensional coordinates of all point cloud data in the virtual space are redistributed, driving C clients to uniformly locate the center point and axis in the virtual space.

4. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 3 is characterized in that: The displacement map value is calculated as: Mark the point cloud data of E spatial points in D spatial areas one by one, and generate point cloud coordinates after converting the point cloud data into the world coordinate system; Using the Euclidean distance calculation formula, calculate the Euclidean distances between the E spatial points in the D spatial regions and the coordinate origin of the world coordinate system one by one, and record the calculated Euclidean distances as displacement mapping values; The rotation mapping value is calculated as: The spatial point corresponding to the maximum value of the Euclidean distance in the D spatial regions is recorded as the sub-marker point, and the sub-coordinate system with the same axis as the world coordinate system is drawn with the sub-marker point as the coordinate origin to obtain D sub-coordinate systems; Mark the coordinates of the spatial points of the D spatial regions on the corresponding sub-coordinate systems one by one, obtain E sub-marking information, identify the virtual points corresponding to the sub-marking information one by one in the virtual space, and mark the coordinates of the virtual points one by one in the world coordinate system, which are recorded as rotation coordinates; Using the X-axis, Y-axis, and Z-axis of the world coordinate system as the rotation axes and the origin of the world coordinate system as the rotation base point, the E spatial points in the D spatial regions are rotated one by one to the rotation coordinates of the world coordinate system; The rotation Euler angles of E spatial points in D spatial regions between the sub-marker information and the rotation coordinates are measured respectively and recorded as rotation mapping values.

5. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 4 is characterized in that: The control method of analog mapping is: A1: Number the D spatial regions in ascending order according to the order in which the spatial regions are identified, starting with 1; A2: Match all spatial points on the facade in the spatial region with the corresponding displacement mapping values and rotation mapping values one by one in ascending order to generate mapping units, obtaining F mapping units. A3: Using the displacement mapping value and the rotation mapping value as the mapping standard, simulate and map the spatial points in the F mapping units one by one to the virtual space; A4: Repeat steps A2-A3 until all E spatial points in the D spatial regions are simulated and mapped to the virtual space.

6. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 5, characterized in that: Realistic motion includes gesture motion and collision motion; The method for generating mixed reality scenes is: In the virtual space, pair the virtual objects with the coordinates of D clients one by one to obtain D virtual sub-scenes; In D virtual sub-scenes, the client's real-time palm position is collected through the depth camera, and the coordinates corresponding to the real-time palm position are unified with the coordinates of the virtual object in the world coordinate system to construct D virtual sub-scenes with gesture movement; The real-time arm position of the client is collected one by one through the posture tracking gyroscope, the coordinates corresponding to the real-time arm position and the coordinates of the virtual object are unified in the world coordinate system, and the coordinates where the virtual object's shape changes are recorded in real time as trigger coordinates; The trigger coordinates and the corresponding real-time arm position coordinates are unified in the world coordinate system to construct D virtual sub-scenes with collision motion. The D virtual sub-scenes with gesture motion and collision motion are then fused to generate a mixed reality scene.

7. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 6, characterized in that: Network nodes include upper-level nodes and lower-level nodes; The method for constructing a collaborative interaction network is: Construct a local area network with two upper and lower network layers, and establish an upper node and C lower nodes in the upper network layer and the lower network layer respectively; Import the server into the upper node, causing the upper node to be converted into an upper-level node, and import the C clients into the C lower nodes one by one, causing the C clients to be converted into C lower-level nodes; C spaced upper ports are configured on the upper node, one lower port is configured on each of the C lower nodes, and sub-channels with annular cross-sections are constructed between the C upper ports and the corresponding C lower ports to obtain C sub-channels; Identify the cross-sectional center of the sub-channel, and build a data fence through the cross-sectional center to divide the sub-channel into an upload channel and a download channel, thereby transforming the local area network into a collaborative interactive network.

8. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 7, characterized in that: The fusion alignment method is: The depth camera collects the real-time images of C subordinate nodes at each moment in real time to obtain C real-time images; The lower left corner of the C real-time images is used as the coordinate origin, and the right direction, the upper direction, and the front direction of the real-time images are used as the X-axis direction, the Y-axis direction, and the Z-axis direction respectively. The X-axis, Y-axis, and Z-axis are drawn to construct the image coordinate system of the C real-time images; Import C real-time images into the mixed reality scene one by one, and based on the coordinate origin of the world coordinate system, align the center points and axial directions of the C image coordinate systems with the world coordinate system to obtain C aligned real-time images.

9. The mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to claim 8, characterized in that: The method for synchronously displaying real-time images and virtual images is as follows: Collect the real-world motions of C subordinate nodes at each moment to obtain C real-world motions. Bind and aggregate the C aligned real-time images with the corresponding C real-world motions to generate C real-world data sets. Transmit C real-world data sets to the upper-level node through C upload channels, and aggregate and simulate the real-world motion and real-time images in the C real-world data sets within the collaborative interactive network to simulate a complete virtual image. The virtual image is synchronously transmitted to C lower-level nodes through C download channels to control the synchronous display of the real image and the virtual image.

10. A mixed reality multi-person interactive large space positioning, alignment, and collaboration method, applied to a mixed reality server, based on a mixed reality multi-person interactive large space positioning, alignment, and collaboration system according to any one of claims 1 to 9, characterized in that: include: S1: Scan the real space clockwise at the scanning point to construct a gridded virtual space, draw a world coordinate system in the virtual space, and locate the center point of the client in the virtual space; S2: Identify the spatial regions of real objects in the real world, calculate the mapping features of the spatial regions, and simulate the mapping between real objects and virtual space based on spatial mapping rules; S3: Render virtual objects in the virtual space and combine them with real-world motion to generate a mixed reality scene. S4: Based on the interactive collaboration principle, discrete clients and servers are converted into network nodes to build a collaborative interactive network with sub-channels and data walls; S5: Obtain the real-time image of the client, integrate and align the real-time image with the mixed reality scene, and control the collaborative interactive network to synchronously display the real-time image and the virtual image.

Images