Multi-AR-device collaborative interaction method and system based on dynamic identification method

The spatial conversion matrix is ​​obtained through dynamic identification method and three-dimensional imaging sensors, and the problem of information barriers between AR devices is solved, real-time synchronous interaction of virtual objects in the same physical space of multiple AR devices is realized, ensuring the consistency of user experience and resource efficiency. .

CN118963551APending Publication Date: 2024-11-15GUANGDONG UNIV OF TECH
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411001162.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The information barrier between different AR devices makes it difficult to maintain consistency in the coordinated interaction effect and spatial location of the same virtual object in the same physical space, and existing remote collaboration technologies cannot accurately synchronize environmental information.

Method used

Using a multi-AR device collaborative interaction method based on dynamic identification method, by selecting an AR device as the server and other devices as the client, a spatial conversion matrix is ​​obtained based on dynamic identification diagrams and three-dimensional imaging sensors, realizing coordinate conversion and synchronization of virtual objects , ensure real-time interaction of multiple AR devices in the same physical space.

Benefits of technology

Real-time synchronous interaction between virtual objects between multiple AR devices is realized, breaking through information barriers, avoiding complex resource synchronization, reducing the waste of transmission bandwidth resources, and ensuring a consistent interactive experience for users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963551A_ABST
    Figure CN118963551A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-AR (Augmented Reality) equipment collaborative interaction method and system based on a dynamic identification method, and the method comprises the steps: selecting server-side AR equipment and client-side AR equipment, and synchronizing the pose of the client-side AR equipment relative to a dynamic identification graph based on the dynamic identification method, thereby obtaining a spatial conversion matrix; calculating a coordinate conversion relation of static virtual objects in the client AR equipment and the server AR equipment based on the spatial conversion matrix to obtain synchronous coordinates of the static virtual objects; and calculating coordinates of the client AR device and the server AR device after interaction on the dynamic virtual object based on the coordinate conversion relationship to obtain synchronous coordinates of the dynamic virtual object. The system comprises an initial pose synchronization module, a static virtual object synchronization module, a dynamic virtual object synchronization module and an AR resource library synchronization module. According to the invention, real-time synchronous interaction of the multiple AR devices on the virtual object in the same physical space can be realized. The method can be widely applied to the technical field of augmented reality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a multi-AR device collaborative interaction method and system based on a dynamic identification method. Background Art

[0002] Augmented Reality (AR) is an emerging human-computer interaction technology. It generates virtual objects in a virtual three-dimensional world through computers, and uses algorithms and sensor technologies to achieve precise alignment and real-time interaction between virtual objects and the real environment, providing users with an immersive and seamless sensory experience.

[0003] With the popularity of AR devices including smartphones, tablets and AR glasses, users can interact with virtual objects in a variety of ways, and the collaborative interaction of multiple AR devices has a wide range of application value. However, there is a gap in information between different AR devices, and it is difficult for different users to maintain consistent interaction effects and spatial positions when they interact collaboratively with the same virtual object in the same physical space. Because the existing remote collaboration technology cannot accurately synchronize environmental information to different AR devices, the virtual objects in the server cannot match the environment of the AR device because the environment of the server is different from that of the AR device. Different AR devices can only synchronize virtual objects but cannot bind them to the same physical space. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a multi-AR device collaborative interaction method and system based on a dynamic identification method, which can realize real-time synchronous interaction of multiple AR devices with virtual objects in the same physical space.

[0005] The first technical solution adopted by the present invention is: a multi-AR device collaborative interaction method based on a dynamic identification method, comprising the following steps:

[0006] Select at least one AR device as the server and other AR devices as the client, and synchronize the position of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain the space transformation matrix;

[0007] Calculate the coordinate transformation relationship of the static virtual object in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronization coordinate of the static virtual object;

[0008] The coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other are calculated based on the coordinate conversion relationship to obtain the synchronous coordinates of the dynamic virtual object.

[0009] Further, the step of selecting at least one AR device as a server and other AR devices as clients, and synchronizing the pose of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain the space transformation matrix specifically includes:

[0010] Select at least one AR device as the server and other AR devices as the client, and generate a dynamic logo on the server AR device based on the physical space position;

[0011] The client AR device uses its own three-dimensional imaging sensor to obtain a 3D point cloud image of the dynamic logo image, and obtains the virtual position of the feature point of the dynamic logo image;

[0012] Constructing a relationship between the physical space position and the virtual position based on a rotation matrix and a translation vector;

[0013] The rotation matrix and the translation vector between the physical space position and the virtual position are solved to obtain a space transformation matrix.

[0014] Furthermore, the step of solving the rotation matrix and the translation vector between the physical space position and the virtual position to obtain the space transformation matrix specifically includes:

[0015] Convert the problem of solving the rotation matrix and translation vector into the problem of minimizing the root mean square error of two point sets;

[0016] With the goal of minimizing the error between the source point cloud and the target point cloud, the ICP algorithm is used to solve the rotation matrix and translation vector to obtain the final rotation matrix and translation vector;

[0017] Construct a spatial transformation matrix based on the final rotation matrix and translation vector.

[0018] Furthermore, the step of calculating the coordinate transformation relationship of the static virtual objects in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronous coordinates of the static virtual objects specifically includes:

[0019] The client AR device generates a static virtual object and uses the centroid of the dynamic logo as a spatial root anchor point;

[0020] Construct the coordinate system transformation matrix of all client AR devices relative to the server AR device based on the space transformation matrix and imaging center offset;

[0021] Based on the coordinate system conversion matrix, the static virtual object generated by the client AR device is transformed into the server AR device to obtain the first static virtual object coordinates;

[0022] The first static virtual object coordinates are transformed to the other client AR devices based on the coordinate system conversion matrix of the other client AR devices to obtain the second static virtual object coordinates.

[0023] Furthermore, the coordinate system transformation matrix is ​​expressed as follows:

[0024]

[0025] Among them, v mark Represents the centroid coordinates of the dynamic logo graph; represents the final rotation matrix; Represents the final translation vector; ΔX, ΔY and ΔZ represent offset constants; Trans(ΔX, ΔY, ΔZ) represents the translation transformation matrix; I represents the 3×3 identity matrix; Rot(Z, θ3) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the Z axis with an angle of θ3; Rot(X, θ2) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the X axis with an angle of θ2; Rot(Y, θ1) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the Y axis with an angle of θ1; n represents the number of coordinates in the coordinate point set; Represents the coordinates of the center of mass in the client AR device's own coordinate system; v i Represents the i-th coordinate of the dynamic identification graph coordinate point set; Represents the client AR device C i The spatial transformation matrix compared to the centroid of the dynamic logo; Represents the client AR device C i To the coordinate system transformation matrix of the server AR device S

[0026] Further, the step of calculating the coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other based on the coordinate transformation relationship to obtain the synchronous coordinates of the dynamic virtual object specifically includes:

[0027] If the client AR device interacts with the dynamic virtual object in situ, the coordinates of the dynamic virtual object in the server AR device are updated based on the position of the dynamic virtual object after the interaction and the coordinate conversion matrix to obtain the first dynamic virtual object coordinates;

[0028] If the client AR device is displaced and interacts with the dynamic virtual object, the coordinate transformation matrix of the device is updated based on the position of the client AR device after the displacement to obtain an updated coordinate transformation matrix;

[0029] If the server-side AR device is displaced and interacts with the dynamic virtual object, the coordinate transformation matrix of the device is updated based on the position of the server-side AR device after the displacement to obtain an updated coordinate transformation matrix;

[0030] Based on the position of the dynamic virtual object after the interaction and the updated coordinate transformation matrix, the coordinates of the dynamic virtual object in the server AR device are updated to obtain the first dynamic virtual object coordinates;

[0031] The first dynamic virtual object coordinates are transformed to other client AR devices based on the coordinate system conversion matrix of the other client AR devices to obtain the second dynamic virtual object coordinates.

[0032] The second technical solution adopted by the present invention is: a multi-AR device collaborative interaction system based on a dynamic identification method, comprising:

[0033] An initial posture synchronization module is used to select at least one AR device as a server and other AR devices as clients, and synchronize the posture of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain a space transformation matrix;

[0034] A static virtual object synchronization module calculates the coordinate transformation relationship of the static virtual object in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronization coordinate of the static virtual object;

[0035] A dynamic virtual object synchronization module calculates the coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other based on the coordinate conversion relationship to obtain the synchronization coordinates of the dynamic virtual object;

[0036] The AR resource library synchronization module is used to determine whether the AR resources of the required virtual objects exist in the local warehouse, and load the missing AR resources from the cloud object repository and save them to the local warehouse.

[0037] The beneficial effects of the method and system of the present invention are as follows: the method of the present invention breaks through the information barriers between different AR devices through spatial coordinate transformation based on the dynamic identification method, accurately synchronizes the initial postures of multiple AR devices, avoids secondary synchronization calibration operations, and realizes real-time synchronous interaction of multiple AR devices with virtual objects in the same physical space; the system of the present invention synchronizes the AR device resource library by dynamically updating the local warehouse and the cloud object repository, so that the AR device can adjust the state of the virtual object according to the zoom factor and the AR special effect instruction number, without synchronizing redundant resource information, resulting in a waste of transmission bandwidth resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of the steps of a multi-AR device collaborative interaction method based on a dynamic identification method of the present invention;

[0039] Figure 2 It is a structural block diagram of a multi-AR device collaborative interaction system based on a dynamic identification method of the present invention;

[0040] Figure 3 It is a schematic diagram of posture synchronization of a multi-AR device collaborative interaction method based on a dynamic identification method of the present invention;

[0041] Figure 4 It is a schematic diagram of coordinate transformation after movement of a client AR device and a dynamic virtual object in a multi-AR device collaborative interaction method based on a dynamic identification method of the present invention;

[0042] Figure 5 It is a schematic diagram of coordinate transformation of a server-side AR device and a dynamic virtual object after movement in a multi-AR device collaborative interaction method based on a dynamic identification method of the present invention;

[0043] Figure 6 It is a schematic diagram of an AR resource library synchronization module of a multi-AR device collaborative interaction system based on a dynamic identification method of the present invention;

[0044] Figure numerals: 1. virtual object before moving and scaling; 2. virtual object after moving and scaling; 3. client AR device before moving; 4. client AR device after moving; 5. dynamic logo; 6. server AR device before moving; 7. server AR device after moving; 8. AR device; 9. local warehouse; 10. cloud object repository. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0046] Reference Figure 1 The present invention provides a multi-AR device collaborative interaction method based on a dynamic identification method, the method comprising the following steps:

[0047] S1. Select at least one AR device as the server and other AR devices as the client, and synchronize the position of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain the space transformation matrix;

[0048] S1.1. Select at least one AR device as the server and other AR devices as the client, and generate a dynamic logo on the server AR device based on the physical space position;

[0049] Specifically, refer to Figure 3, a dynamic logo image of size b×b is generated on the server AR device screen, logo Figure 4 The coordinates of the vertices are [0,0,0], [0,b,0], [b,0,0] and [b,b,0] respectively. The feature points of the identification graph constitute the positions of the server-side AR device in the physical space. All feature points on the identification graph form a point set G.

[0050] S1.2, the client AR device uses its own three-dimensional imaging sensor to obtain a 3D point cloud image of the dynamic logo image, and obtains the virtual position of the feature point of the dynamic logo image;

[0051] Specifically, the three-dimensional imaging sensor includes but is not limited to a binocular camera, a three-dimensional scanner and an RGB-D camera; the virtual positions of the feature points of the dynamic identification graph are represented by a point set H.

[0052] S1.3, constructing the relationship between the physical space position and the virtual position based on the rotation matrix and translation vector;

[0053] Specifically, the relationship between the physical space position and the virtual position is expressed as follows:

[0054] g=RH+t (1)

[0055] Among them, g represents the point set of all feature points of the dynamic identification map (including physical space position information); H represents the virtual position point set of the feature points of the dynamic identification map (including virtual space position information); R represents the rotation matrix; t represents the translation vector.

[0056] S1.4. Solve the rotation matrix and translation vector between the physical space position and the virtual position to obtain a space transformation matrix.

[0057] Specifically, the problem of solving the rotation matrix and translation vector is transformed into the problem of minimizing the root mean square error of two point sets, which is expressed as follows:

[0058]

[0059] Where E represents the mean square error; Represents the set of points that the rotation matrix R and translation vector t can take.

[0060] The present invention adopts the ICP algorithm to solve the rotation matrix R and translation vector t in the formula. The algorithm starts from the initial transformation matrix and continuously iteratively adjusts the position and posture of the source point cloud to minimize the error with the target point cloud. When the error is less than the set threshold, the algorithm stops iterating and outputs the final rotation matrix and translation vector. Then, the final rotation matrix and translation vector and formula 1 are used to synchronize the initial posture of the client AR device and the server AR device to obtain the space conversion matrix. After all the client AR devices and the server AR devices complete the initial posture synchronization, the virtual space and the physical space identification are aligned.

[0061] The spatial transformation matrix is ​​expressed as follows:

[0062]

[0063] in, Represents the client AR device C i The spatial transformation matrix compared to the centroid of the dynamic logo; represents the final rotation matrix; Represents the final translation vector.

[0064] S2. Calculate the coordinate transformation relationship of the static virtual objects in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronization coordinates of the static virtual objects;

[0065] S2.1. The client AR device generates a static virtual object and uses the centroid of the dynamic logo as a spatial root anchor point;

[0066] Specifically, the key to the synchronous interaction of static virtual objects is to keep the virtual objects in all AR devices synchronized with the root anchor point in the physical space. The client AR device generates a static virtual object and uses the centroid of the dynamic logo map as the spatial root anchor point. The centroid of the dynamic logo map is calculated as follows:

[0067]

[0068] Among them, v mark represents the centroid coordinates of the dynamic identification graph; n represents the number of coordinates in the coordinate point set; v i Represents the i-th coordinate of the dynamic identification graph coordinate point set.

[0069] S2.2. Construct the coordinate system transformation matrix of all client AR devices relative to the server AR device based on the space transformation matrix and the imaging center offset;

[0070] Specifically, the coordinate system transformation matrix is ​​expressed as follows:

[0071]

[0072] Among them, v mark Represents the centroid coordinates of the dynamic logo graph; represents the final rotation matrix; Represents the final translation vector; ΔX, ΔY and ΔZ represent offset constants; Trans(ΔX, ΔY, ΔZ) represents the translation transformation matrix; I represents the 3×3 identity matrix; Rot(Z, θ3) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the Z axis with an angle of θ3; Rot(X, θ2) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the X axis with an angle of θ2; Rot(Y, θ1) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the Y axis with an angle of θ1; Represents the coordinates of the center of mass in the client AR device's own coordinate system; Represents the client AR device C i The spatial transformation matrix compared to the centroid of the dynamic logo; Represents the client AR device C i To the coordinate system transformation matrix of the server AR device S.

[0073] The coordinate system transformation matrix P of all client AR devices relative to the server AR device can be obtained through the above equations 5, 6 and 7. Although the coordinates of the same virtual object in the virtual space are different for each device, the position of the virtual object can be synchronized to all AR devices based on the coordinate system transformation matrix.

[0074] S2.3. transforming the static virtual object generated by the client AR device to the server AR device based on the coordinate system conversion matrix to obtain the first static virtual object coordinates;

[0075] Specifically, the spatial coordinates of the static virtual object generated by the client AR device C1 are After the coordinate system transformation matrix It can be transformed to the server AR device S to obtain the spatial coordinates of the static virtual object relative to the server AR device S. That is, the coordinates of the first static virtual object, and its expression is as follows:

[0076]

[0077] in, Represents the spatial coordinates of the static virtual object relative to the server AR device S; Represents the coordinate system transformation matrix of the client AR device C1 relative to the server AR device S; Represents the spatial coordinates of a static virtual object.

[0078] S2.4. Based on the coordinate system conversion matrix of the other client AR device, transform the first static virtual object coordinates to the other client AR device to obtain the second static virtual object coordinates.

[0079] Specifically, the client AR device C2 is calculated by formula 8 The spatial coordinates in the local coordinate system:

[0080]

[0081] in, Represents the coordinates of the virtual object in the client AR device C2; Represents the coordinate system transformation matrix from the server AR device S to the client AR device C2.

[0082] The specific embodiment of the present invention uses the characteristics of the rotation matrix and the translation vector in the coordinate system transformation matrix to solve the coordinate system transformation matrix of the server AR device S relative to the client AR device C2 with a smaller amount of calculation. It can be decomposed into the form of rotation matrix and translation vector:

[0083]

[0084] in, Represents the rotation matrix from the server AR device S to the client AR device C2; Indicates the translation vector from the server AR device S to the client AR device C2.

[0085] Since the rotation matrix has an orthogonal property, then:

[0086] Right now

[0087] in, Represents the rotation matrix from the client AR device C2 to the server AR device S.

[0088] Then The transformation from S to C2 is expressed as follows:

[0089]

[0090] in, Indicates the translation vector from the client AR device C2 to the server AR device S.

[0091] because is the translation vector of C2 relative to C2, which is the 0 vector. It can be deduced that the translation vector of the server AR device S relative to the client AR device C2 is therefore, It can be calculated by the following expression:

[0092]

[0093] The client AR device C2 receives It can complete the conversion of the virtual object space coordinates relative to its own coordinate system. Without relying on the client to re-request the server's spatial pose, the client AR device can quickly complete the calculation of the virtual object's spatial coordinates. The calculation formula is as follows:

[0094]

[0095] in, Represents the coordinates of the virtual object in the client AR device C2's own coordinate system.

[0096] S3. Calculate the coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other based on the coordinate conversion relationship to obtain the synchronous coordinates of the dynamic virtual object.

[0097] Specifically, after completing the synchronization of static virtual object coordinates of multiple AR devices, it is also necessary to synchronize the interactive operations of any AR device on dynamic virtual objects to other AR devices. Interactive operations include removing, moving, rotating, and scaling dynamic virtual objects; interactive synchronization of dynamic virtual objects requires real-time updates of the status of dynamic virtual objects to ensure that users of AR devices have a consistent interactive experience.

[0098] S3.1. If the client AR device interacts with the dynamic virtual object in situ, the coordinates of the dynamic virtual object in the server AR device are updated based on the position of the dynamic virtual object after the interaction and the coordinate conversion matrix to obtain the first dynamic virtual object coordinates;

[0099] Specifically, if the dynamic virtual object of the client AR device changes from the coordinate Move to coordinates Calculate first and The difference is updated based on the difference and the coordinate transformation matrix Coordinates relative to the server AR device That is, the coordinates of the first dynamic virtual object, and its calculation expression is as follows:

[0100]

[0101] in, and Respectively represent the coordinate differences before and after moving on the X-axis, Y-axis, and Z-axis; Represents the coordinate system transformation matrix from the client AR device C1 to the server AR device S.

[0102] S3.2. If the client AR device is displaced and interacts with a dynamic virtual object, a coordinate transformation matrix of the device is updated based on the position of the client AR device after the displacement to obtain an updated coordinate transformation matrix;

[0103] Specifically, refer to Figure 4 , if the position of the client AR device moves from C1 to C'1, the dynamic virtual object moves from the coordinate The coordinates relative to C'1 after movement and scaling are The client AR device needs to obtain the sensor parameters to obtain the coordinate transformation matrix of the moved AR device relative to the original position.

[0104] S3.3. If the server-side AR device is displaced and interacts with a dynamic virtual object, the coordinate transformation matrix of the device is updated based on the position of the server-side AR device after the displacement to obtain an updated coordinate transformation matrix;

[0105] Specifically, refer to Figure 5 , if the server-side AR device and the dynamic virtual object move and rotate at the same time, the position of the server-side AR device moves from S to S', and the position of the dynamic virtual object changes from coordinate Move to coordinates Since the AR camera position of the server-side AR device that generates the dynamic logo is the spatial anchor point coordinate of all client AR devices, when the server-side AR device moves and interacts with virtual objects, it is also necessary to obtain sensor parameters to update the coordinate transformation matrix.

[0106] S3.4. Update the coordinates of the dynamic virtual object in the server AR device based on the position of the dynamic virtual object after the interaction and the updated coordinate transformation matrix to obtain the first dynamic virtual object coordinates.

[0107] Specifically, when the client AR device moves and interacts with the dynamic virtual object, the coordinate transformation matrix renew Coordinates relative to the server AR device Its calculation expression is as follows:

[0108]

[0109] When the server-side AR device moves and interacts with dynamic virtual objects, the coordinate transformation matrix is ​​used. renew Coordinates relative to the server AR device Its calculation expression is as follows:

[0110]

[0111] S3.5. Based on the coordinate system conversion matrix of the other client AR device, transform the first dynamic virtual object coordinates to the other client AR device to obtain the second dynamic virtual object coordinates.

[0112] Specifically, the coordinates of the first dynamic virtual object Synchronize to all AR devices to complete global synchronization of virtual objects.

[0113] Reference Figure 2 The present invention provides a multi-AR device collaborative interaction system based on a dynamic identification method, comprising:

[0114] An initial posture synchronization module is used to select at least one AR device as a server and other AR devices as clients, and synchronize the posture of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain a space transformation matrix;

[0115] A static virtual object synchronization module calculates the coordinate transformation relationship of the static virtual object in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronization coordinate of the static virtual object;

[0116] A dynamic virtual object synchronization module calculates the coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other based on the coordinate conversion relationship to obtain the synchronization coordinates of the dynamic virtual object;

[0117] The AR resource library synchronization module is used to determine whether the AR resources of the required virtual objects exist in the local warehouse, and load the missing AR resources from the cloud object repository and save them to the local warehouse.

[0118] Reference Figure 6 When the client AR device receives a virtual object generation instruction from other client AR devices, it first determines whether there is an AR resource for the virtual object in the local warehouse. If the local warehouse does not have the corresponding AR resource, the corresponding AR resource is loaded from the cloud object repository and saved to the local warehouse. Finally, the AR device generates a virtual object according to the user's interaction instruction. The advantages of the AR resource library synchronization module also include that the scaling operations and AR special effects operations of virtual objects by the client AR device can be simplified into scaling multiples and AR special effects instruction numbers. Other client AR devices can adjust the status of virtual objects according to the scaling multiples and AR special effects instruction numbers without having to synchronize redundant resource information every time, resulting in a waste of transmission bandwidth resources.

[0119] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0120] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for collaborative interaction of multiple AR devices based on dynamic identification method, characterized in that: The following steps are involved: Select at least one AR device as the server and other AR devices as the client, and synchronize the position of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain the space transformation matrix; Calculate the coordinate transformation relationship of the static virtual object in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronization coordinate of the static virtual object; The coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other are calculated based on the coordinate conversion relationship to obtain the synchronous coordinates of the dynamic virtual object.

2. According to claim 1, a method for collaborative interaction of multiple AR devices based on dynamic identification method, characterized in that: The step of selecting at least one AR device as a server and other AR devices as clients, and synchronizing the pose of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain the space transformation matrix specifically includes: Select at least one AR device as the server and other AR devices as the client, and generate a dynamic logo on the server AR device based on the physical space position; The client AR device uses its own three-dimensional imaging sensor to obtain a 3D point cloud image of the dynamic logo image, and obtains the virtual position of the feature point of the dynamic logo image; Constructing a relationship between the physical space position and the virtual position based on a rotation matrix and a translation vector; The rotation matrix and the translation vector between the physical space position and the virtual position are solved to obtain a space transformation matrix.

3. According to claim 1, a method for collaborative interaction of multiple AR devices based on dynamic identification method, characterized in that: The step of solving the rotation matrix and the translation vector between the physical space position and the virtual position to obtain the space transformation matrix specifically includes: Convert the problem of solving the rotation matrix and translation vector into the problem of minimizing the root mean square error of two point sets; With the goal of minimizing the error between the source point cloud and the target point cloud, the ICP algorithm is used to solve the rotation matrix and translation vector to obtain the final rotation matrix and translation vector; Construct a spatial transformation matrix based on the final rotation matrix and translation vector.

4. According to claim 1, a method for collaborative interaction of multiple AR devices based on dynamic identification method, characterized in that: The step of calculating the coordinate transformation relationship of the static virtual objects in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronous coordinates of the static virtual objects specifically includes: The client AR device generates a static virtual object and uses the centroid of the dynamic logo as a spatial root anchor point; Construct the coordinate system transformation matrix of all client AR devices relative to the server AR device based on the space transformation matrix and imaging center offset; Based on the coordinate system conversion matrix, the static virtual object generated by the client AR device is transformed into the server AR device to obtain the first static virtual object coordinates; The first static virtual object coordinates are transformed to the other client AR devices based on the coordinate system conversion matrix of the other client AR devices to obtain the second static virtual object coordinates.

5. According to claim 4, a method for collaborative interaction of multiple AR devices based on dynamic identification method is characterized in that: The coordinate system transformation matrix is ​​expressed as follows: Among them, v mark Represents the centroid coordinates of the dynamic logo graph; represents the final rotation matrix; Represents the final translation vector; ΔX, ΔY and ΔZ represent offset constants; Trans(ΔX, ΔY, ΔZ) represents the translation transformation matrix; I represents the 3×3 identity matrix; Rot(Z, θ3) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the Z axis with an angle of θ3; Rot(X, θ2) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the X axis with an angle of θ2; Rot(Y, θ1) represents the rotation matrix transformation of any vector in three-dimensional space with respect to the Y axis with an angle of θ1; n represents the number of coordinates in the coordinate point set; Represents the coordinates of the center of mass in the client AR device's own coordinate system; v i Represents the i-th coordinate of the dynamic identification graph coordinate point set; Represents the client AR device C i The spatial transformation matrix compared to the centroid of the dynamic logo; Represents the client AR device C i To the coordinate system transformation matrix of the server AR device S.

6. According to claim 1, a method for collaborative interaction of multiple AR devices based on dynamic identification method, characterized in that: The step of calculating the coordinates of the dynamic virtual object after the client AR device and the server AR device interact with each other based on the coordinate conversion relationship to obtain the synchronous coordinates of the dynamic virtual object specifically includes: If the client AR device interacts with the dynamic virtual object in situ, the coordinates of the dynamic virtual object in the server AR device are updated based on the position of the dynamic virtual object after the interaction and the coordinate conversion matrix to obtain the first dynamic virtual object coordinates; If the client AR device is displaced and interacts with the dynamic virtual object, the coordinate transformation matrix of the device is updated based on the position of the client AR device after the displacement to obtain an updated coordinate transformation matrix; If the server-side AR device is displaced and interacts with the dynamic virtual object, the coordinate transformation matrix of the device is updated based on the position of the server-side AR device after the displacement to obtain an updated coordinate transformation matrix; Based on the position of the dynamic virtual object after the interaction and the updated coordinate transformation matrix, the coordinates of the dynamic virtual object in the server AR device are updated to obtain the first dynamic virtual object coordinates; The first dynamic virtual object coordinates are transformed to other client AR devices based on the coordinate system conversion matrix of the other client AR devices to obtain the second dynamic virtual object coordinates.

7. A multi-AR device collaborative interaction system based on dynamic identification method, characterized in that: include: An initial posture synchronization module is used to select at least one AR device as a server and other AR devices as clients, and synchronize the posture of the client AR device relative to the dynamic identification map based on the dynamic identification method to obtain a space transformation matrix; A static virtual object synchronization module calculates the coordinate transformation relationship of the static virtual object in the client AR device and the server AR device based on the space transformation matrix to obtain the synchronization coordinate of the static virtual object; The dynamic virtual object synchronization module calculates the client AR device and the server based on the coordinate transformation relationship. The coordinates of the dynamic virtual object after the AR device interacts with it, and obtains the synchronized coordinates of the dynamic virtual object; The AR resource library synchronization module is used to determine whether the AR resources of the required virtual objects exist in the local warehouse, and load the missing AR resources from the cloud object repository and save them to the local warehouse.

Citation Information

Cited By

  • Interaction method and system for mixed reality

    CN121033342A

  • A mixed reality interaction method and system

    CN121033342B

  • Virtual object attitude determination method and system based on dynamic reference object

    CN121170232A

  • A dynamic reference-based virtual object pose determination method and system

    CN121170232B