A mixed reality interaction method and system

Abstract

The application provides a mixed reality interaction method and system, and relates to the technical field of computers.The method comprises the following steps: a server acquires anchor observation data sent by at least one user equipment, the anchor observation data being used for describing coordinate data of a plurality of anchors in a local coordinate system; the server determines an alignment transformation matrix of each user equipment based on anchor position data and the anchor observation data, the anchor position data being used for indicating coordinate data of the anchors in a global coordinate system; the server performs fusion processing on the alignment transformation matrix of each user equipment, generates a correction transformation matrix corresponding to each user equipment, and sends the correction transformation matrix; and the user equipment maps local coordinates of a virtual object in the local coordinate system to the global coordinate system based on the correction transformation matrix, and controls the virtual object according to the global coordinate system.The application solves the problem that the positions of virtual objects presented in different user terminals are inconsistent, and improves operation reliability.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a mixed reality interaction method and system. Background Technology

[0002] Mixed Reality (MR) is the fusion and upgrade of Virtual Reality (VR) and Augmented Reality (AR) technologies. It uses digital technology to seamlessly combine virtual content with the real environment, creating an interactive space that blends the virtual and the real.

[0003] Existing MR systems typically rely on each user terminal (such as AR glasses or mobile devices) to run Simultaneous Localization and Mapping (SLAM) algorithms locally, establishing their own spatial awareness models to overlay virtual content onto the real environment. Users interact with virtual objects within their own independent coordinate systems, and the system perceives the user's physical space through local mapping. However, the lack of a unified spatial reference leads to inconsistent placement of virtual objects across different user terminals. Therefore, in multi-device interactive tasks, this spatial error can cause collaboration failures or misoperations, particularly in tasks requiring high precision (such as engineering training or puzzle games). Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides a mixed reality interaction method and system to solve the problem of inconsistent virtual object positions on different user terminals, thereby improving operational efficiency and reliability.

[0005] Firstly, this application provides a mixed reality interaction method, comprising:

[0006] The server obtains anchor point observation data sent by at least one user device. The anchor point observation data is used to describe the coordinate data of multiple preset anchor points in the local coordinate system constructed by the user device itself.

[0007] The server determines the alignment transformation matrix for each user device based on the anchor point location data and the anchor point observation data. The anchor point location data is used to indicate the coordinate data of the anchor point in the global coordinate system, and the alignment transformation matrix is ​​used to indicate the mapping relationship between the local coordinate system and the global coordinate system.

[0008] The server performs a fusion process on the alignment transformation matrices of each user device, generates a corrected transformation matrix corresponding to each user device, and then sends it.

[0009] The user equipment maps the local coordinates of the virtual object in the local coordinate system to the global coordinate system based on the correction transformation matrix, and controls the virtual object according to the global coordinate system.

[0010] In one possible implementation, the user equipment obtains anchor point observation data through the following steps:

[0011] Acquire environmental data of the surrounding environment, including two-dimensional images and depth data;

[0012] Feature points are extracted from each image frame in the two-dimensional image, and the three-dimensional coordinates of the feature points in any image frame are determined based on the environmental data. The three-dimensional coordinates are determined by the pixel coordinates of each image frame in the two-dimensional image, the depth data, and the camera intrinsic parameter matrix corresponding to the user device.

[0013] Based on the three-dimensional coordinates, predict the observed pixel position of the feature point in the next image frame, and form a feature point pair by the three-dimensional point coordinates and the observed pixel position in the adjacent image frames;

[0014] Based on the three-dimensional coordinates of the feature point pair and the observed pixel position, the feature matching error is determined, and the first rotation matrix and the first translation vector corresponding to minimizing the feature matching error are extracted.

[0015] Based on the first rotation matrix and the first translation vector, the pose transformation matrix between adjacent frames is determined, and the pose transformation matrices of consecutive frames are superimposed to form the camera motion trajectory of the user device.

[0016] Based on the feature point pairs and the acquired inertial measurement data, the camera motion trajectory is updated;

[0017] Record the key frame poses in the pre-selected key frames, whereby the key frame poses include spatial feature points, depth data, timestamps, and initial poses;

[0018] Based on the camera motion trajectory and the keyframe pose, a local spatial map in the local coordinate system is formed, so that the anchor point observation data of the anchor point can be determined according to the local spatial map.

[0019] In one possible implementation, the server determines the alignment transformation matrix for each user equipment based on the anchor point location data and the anchor point observation data, including:

[0020] Map the anchor point observation data sent by any user equipment to the anchor point location data, and construct a corresponding first mapping error model;

[0021] Calculate the first centroid coordinates corresponding to the anchor point observation data of the user equipment for any anchor point, and calculate the second centroid coordinates corresponding to the anchor point position data of the anchor point;

[0022] The anchor point observation data is decentralized based on the first centroid coordinates to obtain the anchor point observation point set, and the anchor point position data is decentralized based on the second centroid coordinates to obtain the anchor point position point set.

[0023] Based on the anchor point observation point set and anchor point location point set corresponding to each anchor point, construct the first covariance matrix;

[0024] Singular value decomposition is performed on the first covariance matrix to obtain a first orthogonal matrix and a second orthogonal matrix;

[0025] Based on the first orthogonal matrix and the second orthogonal matrix, a second rotation matrix is ​​constructed, and based on the first centroid coordinates, the second centroid coordinates, and the second rotation matrix, a second translation vector is constructed.

[0026] Using the second rotation matrix and the second translation vector as the solution that minimizes the first mapping error model, the alignment transformation matrix of the user equipment is determined based on the second rotation matrix and the second translation vector.

[0027] In one possible implementation, the server determines the alignment transformation matrix for each user device based on anchor point location data and the anchor point observation data, and further includes:

[0028] The weight of the anchor point is obtained, and the weight is determined by the observation features of the anchor point, including one of anchor point recognition confidence, image sharpness, and depth sensor noise;

[0029] Based on the weights, the anchor point observation data, and the anchor point position data, a corresponding second mapping error model is constructed.

[0030] Based on the weights, the set of anchor observation points and the set of anchor location points corresponding to each anchor point, a second covariance matrix is ​​constructed.

[0031] By performing singular value decomposition on the second covariance matrix, a solution that minimizes the second mapping error model is obtained.

[0032] In one possible implementation, the user equipment controls the virtual object based on the global coordinate system, including:

[0033] The user equipment responds to the control operation request for the target virtual object, generates a status event packet, and sends it to the server. The status event packet includes the operation timestamp of the control operation request, the object identifier of the target virtual object, the device identifier of the user equipment that initiated the control operation request, and the object state vector of the target virtual object. The object state vector includes object position data and object pose quaternion. The object position data is used to indicate the coordinates of the target virtual object in the global coordinate system.

[0034] When the server detects that the operation timestamp carried by the received status event packet falls within a preset receiving time window, it stores the status event packet within the receiving time window into a buffer.

[0035] The server performs state interpolation and / or rotational state interpolation on the state data corresponding to the transmission delay based on the state event packets in the buffer, obtains the interpolation result, and obtains the object state stream of the target virtual object based on the object state vector in the state event packets and the interpolation result. The transmission delay is the duration between the operation timestamp and the time of receiving the state event packets.

[0036] In one possible implementation, the server performs state interpolation and / or rotational state interpolation on the state data corresponding to the transmission delay based on the state event packets in the buffer, to obtain the interpolation result, including:

[0037] Upon receiving the state event packet, the state event packet is added to the state queue of the target virtual object, wherein the state queue contains multiple timestamps arranged in chronological order, and each of the multiple timestamps corresponds to an object state vector;

[0038] Obtain the target time point, wherein the target time point is within the time interval formed by the two timestamps located at the end of the state queue;

[0039] Based on the target time point, the latest timestamp, and the previous timestamp adjacent to the latest timestamp, a weighting factor is determined, which is used to indicate the relative position of the target time point between two adjacent timestamps;

[0040] Based on the weighting factor, the object state vector corresponding to the latest timestamp, and the object state vector corresponding to the previous timestamp, state interpolation is performed on the target time point to obtain the object state interpolation corresponding to the target time point.

[0041] Based on the obtained rotation state quaternions of two adjacent specified frames, the target interpolation angle weight determined by the rotation quaternions, and the weight factor, rotation state interpolation is performed on the target time point to determine the rotation state quaternion corresponding to the target time point.

[0042] The transmission delay is compensated based on the state interpolation and / or rotation state interpolation corresponding to the target time point to obtain the interpolation result.

[0043] In one possible implementation, the user equipment controls the virtual object based on the global coordinate system, further comprising:

[0044] The server obtains a preset logical frame and its corresponding logical timestamp, wherein the logical timestamp is incremented by a preset initial timestamp in a fixed manner.

[0045] When the server reaches the logical timestamp and detects that the buffer stores a state event packet corresponding to the logical timestamp, it obtains the object state stream of the target virtual object based on the stored state event packet.

[0046] When the server reaches the logical timestamp but does not detect that the corresponding state event packet is stored in the buffer, it extracts the previous logical timestamp and the corresponding state event packet from the buffer. Based on the object state vector of the previous logical timestamp, the interval time, and the state change rate of the previous logical timestamp, it determines the object state prediction vector of the logical timestamp, so that the object state flow of the target virtual object is obtained based on the object state prediction vector of the logical timestamp.

[0047] In one possible implementation, the control operation request includes user location data, a request timestamp initiating the control operation request, and a behavior identifier corresponding to the operation to be controlled, wherein the user location data is used to indicate the user's coordinates in the global coordinate system; the user equipment controls the virtual object according to the global coordinate system, further comprising:

[0048] When the server receives multiple user devices' requests to control the same target virtual object in the current logical frame, it determines the interaction distance between the user and the target virtual object for any user device based on the corresponding user location data and the object location data in the status event packet.

[0049] The server determines the control score value of the user device based on the system time corresponding to the current logical frame, the request timestamp, the preset time decay factor, the interaction distance, the preset maximum effective interaction distance, the user device's user permission factor, operation time weight, spatial weight, and user permission weight.

[0050] The server selects the user device that maximizes the control score value and uses it as the controller of the target virtual object in the current logical frame. After generating the corresponding control permissions, the server issues them to the controller.

[0051] The user device corresponding to the controller controls the target virtual object to perform the operation behavior according to the control permissions.

[0052] In one possible implementation, the server further includes the following steps:

[0053] Construct a task graph for multi-person collaborative tasks, wherein a node in the task graph represents a task unit, and an edge in the task graph represents the task dependency relationship between two connected task units. The task dependency relationship includes one of task temporal relationship and causal relationship. The task unit is associated with a set of virtual objects and a set of user devices that execute the task unit.

[0054] When the node state of the predecessor node corresponding to any node in the task graph is detected to be the first task state, the node is triggered to update the corresponding node state to the second task state and broadcast it, and operation permissions are sent to the user equipment associated with the node. The first task state is used to indicate that the task has been completed, and the second task state is used to indicate that the task is in progress.

[0055] Secondly, this application provides a mixed reality interactive system, including: a server and at least one user device;

[0056] The user equipment is used to send anchor point observation data, which describes the coordinate data of multiple preset anchor points in the local coordinate system constructed by the user equipment itself.

[0057] The server is configured to acquire the anchor point observation data; determine the alignment transformation matrix for each user device based on the anchor point position data and the anchor point observation data, wherein the anchor point position data is used to indicate the coordinate data of the anchor point in the global coordinate system, and the alignment transformation matrix is ​​used to indicate the mapping relationship between the local coordinate system and the global coordinate system; perform fusion processing on the alignment transformation matrices of each user device to generate the correction transformation matrix corresponding to each user device and then send it.

[0058] The user equipment is used to map the local coordinates of the virtual object in the local coordinate system to the global coordinate system based on the correction transformation matrix, and to control the virtual object according to the global coordinate system.

[0059] The mixed reality interaction method and system provided in this application, by considering a unified anchor point reference benchmark, allows each user device to identify anchor point observation data of multiple anchor points in its local coordinate system. The server, based on the anchor point observation data and the anchor point position data of multiple anchor points in the global coordinate system, obtains an alignment transformation matrix to convert the local coordinate system to the global coordinate system. Then, by fusing the alignment transformation matrices of each user device, a consistent global coordinate mapping benchmark is generated, achieving high-precision alignment of spatial coordinates and eliminating spatial error problems. Thus, the user device maps the local coordinates of virtual objects in its local coordinate system to the global coordinate system based on the corrected transformation matrix, and controls the virtual objects according to the global coordinate system, ensuring the consistency of virtual object coordinates. This solves the problem of inconsistent virtual object positions across different user terminals, improving operational efficiency and reliability. Attached Figure Description

[0060] Figure 1 A flowchart illustrating a mixed reality interaction method provided in an embodiment of this application;

[0061] Figure 2 This is a schematic diagram of the structure of a mixed reality interactive system provided in an embodiment of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Figure 1 This is a flowchart illustrating a mixed reality interaction method provided in an embodiment of this application. The method is used in a system that includes at least one user device and a server. The mixed reality interaction method includes steps S101 to S104.

[0064] S101. The server obtains anchor point observation data sent by at least one user device. The anchor point observation data is used to describe the coordinate data of multiple preset anchor points in the local coordinate system constructed by the user device itself.

[0065] In this application, the user equipment can be an electronic device with environmental perception, spatial positioning, and data transmission capabilities, such as a smartphone, AR glasses, or tablet computer. It should be noted that the user equipment is equipped with an RGB-D camera or depth sensor, capable of acquiring real-time two-dimensional image information and depth data of the user's environment. By running a simultaneous localization and mapping (SLAM) algorithm based on visual-inertial joint estimation, continuous tracking of the user's current position and modeling of the three-dimensional structure of the environment are achieved, thereby obtaining a local spatial map in the local coordinate system of the user equipment.

[0066] Therefore, this application establishes multiple anchor points (these anchor points can be natural anchor points, such as wall corners or objects with rich textures, or artificial anchor points, such as QR codes or infrared markers). The user equipment identifies these multiple anchor points based on the local spatial map to obtain anchor point observation data in the user equipment's local coordinate system. This anchor point observation data includes local location coordinates and pose quaternions. Subsequently, this anchor point observation data (including local location coordinates, timestamps, confidence scores, and device IDs) is packaged and sent to the server as input for global fusion.

[0067] Furthermore, the server receives anchor point observation data from multiple user devices. Optionally, the server can perform spatiotemporal synchronization based on timestamps, sorting different observations of the same anchor point by timestamp to ensure spatiotemporal synchronization of observation results for the same anchor point. The server can determine whether anchor points observed by different user devices belong to the same physical entity based on the anchor point's feature description, avoiding misclassifying anchor points at different locations as the same object.

[0068] S102. The server determines the alignment transformation matrix for each user device based on the anchor point location data and the anchor point observation data. The anchor point location data is used to indicate the coordinate data of the anchor point in the global coordinate system, and the alignment transformation matrix is ​​used to indicate the mapping relationship between the local coordinate system and the global coordinate system.

[0069] In this application, for the same anchor point, a rigid transformation determined by the corresponding anchor point position data and the anchor point observation data maps the local coordinate system of each user device to a unified global coordinate system, thereby obtaining an alignment transformation matrix. This alignment transformation matrix indicates the mapping relationship between the local coordinate system and the global coordinate system. Therefore, this application considers a unified anchor point reference datum to enable multiple user devices to achieve spatial alignment based on anchor point observation data under different positions and perspectives, eliminating differences in the local coordinate systems of different users (such as starting point, direction, and scaling ratio), and ensuring spatial consistency for multi-user collaboration.

[0070] S103. The server performs fusion processing on the alignment transformation matrices of each user device, generates the corrected transformation matrix corresponding to each user device, and then sends it.

[0071] In this application, due to the observation errors of user equipment (such as sensor noise and viewing angle limitations), the alignment transformation matrix may have deviations. To address this, the alignment transformation matrices of multiple user equipment are fused to achieve high-precision alignment of spatial coordinates, thereby eliminating observation errors between equipment, improving the consistency of the global coordinate system, and enhancing robustness to dynamic environments.

[0072] Optionally, the fusion processing of the alignment transformation matrix can be a weighted average fusion, that is, calculating a weighted average matrix based on the device confidence of each user device for the alignment transformation matrices uploaded by multiple user devices. The device confidence can be determined by the following factors: the number of feature points matched when the user device observes the anchor point, the device motion stability (e.g., assessed through IMU noise level (e.g., accelerometer variance), and the accuracy of historical observations (e.g., the deviation between the transformation matrix previously uploaded by the device and the global fusion result). Optionally, this application can also obtain a relative transformation matrix, that is, the transformation matrix between the local coordinate systems of any two user devices, and perform fusion processing based on the alignment transformation matrix of each user device (local coordinate system mapped to global coordinate system) and the relative transformation matrix between user devices (local coordinate system mapped to the local coordinate system of another user device), taking into account the interaction between user devices. It should be noted that the modified transformation matrix of each user device may be consistent or different.

[0073] Therefore, this application achieves the correction of the alignment transformation matrix by fusing the alignment transformation matrices of each user device, eliminating the impact of single device observation errors on global alignment, improving the consistency of the global coordinate system, and realizing spatial alignment between devices.

[0074] S104. The user equipment maps the local coordinates of the virtual object in the local coordinate system to the global coordinate system based on the correction transformation matrix, and controls the virtual object according to the global coordinate system.

[0075] In this application, the user device receives a correction transformation matrix to map the local coordinates of the virtual object to the global coordinate system, thereby controlling the rendering position of the virtual object according to the global coordinate system and ensuring a consistent spatial layout for multiple users. For example, in a multiplayer AR game, all players see the same position of the virtual props; in collaborative design, when users modify the same 3D model on different devices, the model positions are aligned.

[0076] In some embodiments, the user equipment obtains anchor point observation data through the following steps:

[0077] Acquire environmental data of the surrounding environment, including two-dimensional images and depth data;

[0078] Feature points are extracted from each image frame in the two-dimensional image, and the three-dimensional coordinates of the feature points in any image frame are determined based on the environmental data. The three-dimensional coordinates are determined by the pixel coordinates of each image frame in the two-dimensional image, the depth data, and the camera intrinsic parameter matrix corresponding to the user device.

[0079] Based on the three-dimensional coordinates, predict the observed pixel position of the feature point in the next image frame, and form a feature point pair by the three-dimensional point coordinates and the observed pixel position in the adjacent image frames;

[0080] Based on the three-dimensional coordinates of the feature point pair and the observed pixel position, the feature matching error is determined, and the first rotation matrix and the first translation vector corresponding to minimizing the feature matching error are extracted.

[0081] Based on the first rotation matrix and the first translation vector, the pose transformation matrix between adjacent frames is determined, and the pose transformation matrices of consecutive frames are superimposed to form the camera motion trajectory of the user device.

[0082] Based on the feature point pairs and the acquired inertial measurement data, the camera motion trajectory is updated;

[0083] Record the key frame poses in the pre-selected key frames, whereby the key frame poses include spatial feature points, depth data, timestamps, and initial poses;

[0084] Based on the camera motion trajectory and the keyframe pose, a local spatial map in the local coordinate system is formed, so that the anchor point observation data of the anchor point can be determined according to the local spatial map.

[0085] In this embodiment, each user device (i.e., a mixed reality terminal device) relies on its built-in RGB-D camera or depth sensor to acquire two-dimensional images and depth data of the user's environment in real time. Optionally, the user device converts the two-dimensional pixel coordinates in the two-dimensional image into three-dimensional coordinates in the user device's local coordinate system using its camera intrinsic parameter matrix. Specifically, the three-dimensional point coordinates are calculated using the depth data, the homogeneous representation of the pixel coordinates of each image frame in the two-dimensional image, and the corresponding camera intrinsic parameter matrix of the user device.

[0086] Furthermore, feature points are extracted, such as ORB features (Oriented FAST and Rotated BRIEF) or FAST corner points, which are rotation and scale invariant. By matching feature points in adjacent frames, the observed pixel position of the feature points in the next image frame is predicted, resulting in feature point pairs. Subsequently, the feature matching error is determined based on the 3D coordinates of the feature point pairs and the observed pixel positions (i.e., feature point pairs). By solving for the first rotation matrix and the first translation vector, the feature matching error is minimized, achieving pose estimation. Thus, based on the first rotation matrix and the first translation vector, the pose transformation matrix between adjacent frames is determined, thereby forming the camera motion trajectory of the user equipment's camera at different times.

[0087] It should be noted that inertial measurement data (such as acceleration and angular velocity) is acquired through inertial measurement units (such as accelerometers and gyroscopes) on the user equipment. Subsequently, based on image feature matching results (such as feature point pairs) and inertial measurement data, extended Kalman filters (EKF) or sliding window optimization techniques are used to jointly model the image with accelerometer and gyroscope information, thereby improving the robustness of camera trajectory, especially maintaining high-precision tracking even in environments with weak textures or short-term occlusion.

[0088] Subsequently, during the mapping process, based on the local sparse point cloud map, the spatial feature points, depth data, timestamps, and initial poses extracted from the keyframes are recorded in the local cache. Each keyframe includes a timestamp and initial pose, facilitating spatiotemporal alignment and relative transformation calculations of multi-terminal spatial maps in the subsequent fusion stage. Thus, based on the camera motion trajectory and keyframe poses, a local spatial map in the local coordinate system is formed, and anchor point observation data is determined based on this local spatial map.

[0089] Therefore, the local spatial map constructed in this embodiment not only provides a basic layer for the identification and positioning of anchor points, but also provides a reliable spatial reference for the initial positioning of virtual objects, effectively improving the system's spatial understanding ability when the user initially enters the scene, and laying a high-precision data foundation for subsequent multi-terminal spatial fusion and synchronous operation.

[0090] In some embodiments, the server determines the alignment transformation matrix for each user equipment based on anchor point location data and the anchor point observation data, including:

[0091] Map the anchor point observation data sent by any user equipment to the anchor point location data, and construct a corresponding first mapping error model;

[0092] Calculate the first centroid coordinates corresponding to the anchor point observation data of the user equipment for any anchor point, and calculate the second centroid coordinates corresponding to the anchor point position data of the anchor point;

[0093] The anchor point observation data is decentralized based on the first centroid coordinates to obtain the anchor point observation point set, and the anchor point position data is decentralized based on the second centroid coordinates to obtain the anchor point position point set.

[0094] Based on the anchor point observation point set and anchor point location point set corresponding to each anchor point, construct the first covariance matrix;

[0095] Singular value decomposition is performed on the first covariance matrix to obtain a first orthogonal matrix and a second orthogonal matrix;

[0096] Based on the first orthogonal matrix and the second orthogonal matrix, a second rotation matrix is ​​constructed, and based on the first centroid coordinates, the second centroid coordinates, and the second rotation matrix, a second translation vector is constructed.

[0097] Using the second rotation matrix and the second translation vector as the solution that minimizes the first mapping error model, the alignment transformation matrix of the user equipment is determined based on the second rotation matrix and the second translation vector.

[0098] In this embodiment, the local coordinate systems independently constructed by each user device are mapped to a unified global spatial coordinate system, achieving spatial consistency in the multi-user collaborative mixed reality interactive experience. Since each user device's local coordinate system (i.e., local spatial map) established using a local SLAM algorithm is independent and lacks a common reference benchmark, this embodiment uses multiple anchor points for unified alignment to ensure that all virtual objects seen by users have a consistent position and orientation. Specifically, this embodiment pre-sets multiple anchor points in the physical space. Each user device identifies the anchor point's observation data in its local coordinate system and transforms the local coordinate system to a unified global coordinate system through a rigid transformation.

[0099] More specifically, by using anchor point observation data and anchor point position data, the local coordinate system is transformed to a unified global coordinate system, and a first mapping error model is constructed, as shown below:

[0100] ;

[0101] in, This represents the first mapping error model. Denotes the second rotation matrix. Denotes the second translation vector. Indicates the first Anchor point position data in the global coordinate system. Indicates user equipment i to the first The anchor point observation data in the local coordinate system, where n represents the number of anchor points.

[0102] To solve the first mapping error model, the anchor point observation data and anchor point position data are decentered, i.e., the coordinates of the first centroid and the second centroid are calculated separately:

[0103] , ;

[0104] in, Let the first centroid coordinates be... The coordinates of the second centroid are then used. Subsequently, the coordinates of each point are subtracted from the corresponding centroid coordinates to obtain the decentralized point set:

[0105] ;

[0106] in, This represents the set of anchor observation points for the k-th anchor point. This represents the set of anchor points for the k-th anchor point.

[0107] Then, construct the first covariance matrix. ,Right now: .

[0108] More specifically, this embodiment performs Singular Value Decomposition (SVD) on the first covariance matrix. Based on the first and second orthogonal matrices obtained from the SVD, a second rotation matrix is ​​constructed, and a second translation vector is constructed based on the first centroid coordinates, the second centroid coordinates, and the second rotation matrix. Subsequently, the obtained second rotation matrix and second translation vector are used as the solution to the aforementioned first mapping error model.

[0109] Therefore, based on the second rotation matrix and the second translation vector, the homogeneous transformation matrix used to map the local coordinate system to the global coordinate system is obtained, as shown below:

[0110] ;

[0111] in, Denotes the homogeneous transformation matrix. A group structure representing three-dimensional rigid motion, including rotation and translation.

[0112] Therefore, this embodiment maps all virtual objects and spatial features of user devices to the global coordinate system after transformation using a homogeneous transformation matrix, achieving consistent alignment of spatial coordinates among multiple users. This allows the multi-user collaborative system to unify dispersed local coordinate systems with a shared global reference system, ensuring that the position, orientation, and trajectory of virtual objects are strictly consistent across all user devices, thereby constructing a reliable mixed reality environment.

[0113] In some embodiments, the server determines the alignment transformation matrix for each user equipment based on anchor point location data and the anchor point observation data, and further includes:

[0114] The weight of the anchor point is obtained, and the weight is determined by the observation features of the anchor point, including one of anchor point recognition confidence, image sharpness, and depth sensor noise;

[0115] Based on the weights, the anchor point observation data, and the anchor point position data, a corresponding second mapping error model is constructed.

[0116] Based on the weights, the set of anchor observation points and the set of anchor location points corresponding to each anchor point, a second covariance matrix is ​​constructed.

[0117] By performing singular value decomposition on the second covariance matrix, a solution that minimizes the second mapping error model is obtained.

[0118] It should be noted that anchor point recognition can be affected by ambient light, obstructions, and recognition angle, which can easily lead to spatial mapping errors and cause inconsistent positions of virtual objects on different user devices. To address this, this embodiment introduces weights into the observation error of each anchor point when constructing the first mapping error model, forming a second mapping error model. Solving this second mapping error model effectively reduces anchor point drift caused by image occlusion, unclear recognition, or sensor errors.

[0119] Specifically, to enhance the robustness of the alignment, a second mapping error model is constructed based on the weights, anchor point observation data, and anchor point position data, as shown below:

[0120] ;

[0121] in, This represents the second mapping error model. Let be the rotation matrix of the second mapping error model. Let be the translation vector of the second mapping error model. The weight represents the k-th anchor point, which can be determined by factors such as recognition confidence (e.g., feature point matching score), image sharpness (e.g., Laplacian variance detection blur), and depth sensor noise estimation (e.g., ToF camera ranging error model). Optionally, this weight can be dynamically adjusted to improve alignment stability under low light, occlusion, or motion blur conditions.

[0122] In this embodiment, a second covariance matrix is ​​constructed using weights, the set of anchor observation points corresponding to each anchor point, and the set of anchor position points. This allows the solution to minimize the second mapping error model by solving the second covariance matrix, thus obtaining the corresponding rotation matrix and translation vector.

[0123] Therefore, this embodiment takes into account the weight of anchor points, which can effectively reduce anchor point drift caused by image occlusion, unclear recognition or sensor error, improve alignment stability under low light, occlusion or motion blur conditions, and ensure that all users have a unified position, posture and interaction benchmark for virtual objects in different spaces, thereby significantly improving collaborative reliability.

[0124] In some embodiments, the user equipment controls the virtual object based on the global coordinate system, including:

[0125] The user equipment responds to the control operation request for the target virtual object, generates a status event packet, and sends it to the server. The status event packet includes the operation timestamp of the control operation request, the object identifier of the target virtual object, the device identifier of the user equipment that initiated the control operation request, and the object state vector of the target virtual object. The object state vector includes object position data and object pose quaternion. The object position data is used to indicate the coordinates of the target virtual object in the global coordinate system.

[0126] When the server detects that the operation timestamp carried by the received status event packet falls within a preset receiving time window, it stores the status event packet within the receiving time window into a buffer.

[0127] The server performs state interpolation and / or rotational state interpolation on the state data corresponding to the transmission delay based on the state event packets in the buffer, obtains the interpolation result, and obtains the object state stream of the target virtual object based on the object state vector in the state event packets and the interpolation result. The transmission delay is the duration between the operation timestamp and the time of receiving the state event packets.

[0128] It should be noted that under conditions of unstable network environment and inconsistent terminal frame rate, existing mixed reality systems cannot guarantee the continuous synchronization of virtual object states, causing interactive actions to jitter, lag, or jump, especially when multiple people operate objects simultaneously, resulting in state conflicts or behavioral chaos. To address this, this embodiment uses timestamp-driven state event packets for synchronization. To address the issue of inconsistency between reception time and operation time, state interpolation is performed for position and velocity states, and rotation state interpolation is performed for rotation states, ensuring a smooth transition of states in the physically continuous space.

[0129] In this embodiment, whenever a user performs an operation on a virtual object (such as moving, rotating, scaling, or interactive triggering), i.e., the corresponding user device initiates a control operation request for the target virtual object, the operation is encapsulated into a state event packet, which includes the following fields: operation timestamp, object identifier of the target virtual object (such as the object's global ID number), device identifier of the user device initiating the control operation request, and object state vector. The object state vector may include the target virtual object's object position data, object pose quaternion, velocity, or other additional attributes. Optionally, the state event packet is sent to other user devices via a P2P connection or server relay. All user devices sort the state event packets according to the operation timestamp and update their local state, achieving multi-user operation synchronization.

[0130] Furthermore, to reduce contention caused by multi-user state writes, this embodiment introduces a time window buffer mechanism. Each user equipment has a fixed-length receiving time window, and all state event packets received within this window will participate in interpolation estimation. The window length can be dynamically adjusted based on network latency statistics to balance real-time performance and stability. Specifically, after receiving a state event packet, the server checks whether its operation timestamp falls within the receiving time window. If so, the state event packet is stored in a temporary buffer for subsequent interpolation. Otherwise, the state event packet is temporarily stored in a backup queue (which can be reactivated if the window moves later and the packet is included in the range).

[0131] Furthermore, considering the uncertainty of network transmission, the actual time when the terminal receives the state event packet is usually later than the original operation time, i.e., there is a non-zero delay. If the local object state is updated directly at the time of reception, the object will "jump" or the action will be discontinuous due to system clock drift, network jitter, and other issues, which will damage the user experience. Therefore, this embodiment uses a timestamp-based linear interpolation compensation strategy to smoothly transition the object state while maintaining synchronization accuracy.

[0132] Specifically, for additive variables such as position and velocity, linear interpolation (i.e., state interpolation) is used, which is computationally simple and suitable for continuous changes. For rotational states (represented by quaternions), spherical linear interpolation (i.e., rotational state interpolation) is used to avoid the uneven rotation rate or non-unit quaternion problems caused by linear interpolation. Thus, based on the obtained interpolation compensation and the received object state vector, the state flow of the virtual object is obtained, achieving a smooth transition of object states.

[0133] Therefore, this embodiment uses interpolation to dynamically compensate for state data, ensuring the consistency and continuity of object states during multi-user collaboration even in the presence of network transmission delays or inconsistent frame rates. Simultaneously, it reduces synchronization errors and timing issues caused by network latency, achieving smooth and consistent presentation of virtual objects across different terminals, thus guaranteeing the continuity and predictability of operational responses in collaborative tasks.

[0134] Based on the above embodiments, in some embodiments, the server performs state interpolation and / or rotational state interpolation on the state data corresponding to the transmission delay based on the state event packets in the buffer, to obtain the interpolation result, including:

[0135] Upon receiving the state event packet, the state event packet is added to the state queue of the target virtual object, wherein the state queue contains multiple timestamps arranged in chronological order, and each of the multiple timestamps corresponds to an object state vector;

[0136] Obtain the target time point, wherein the target time point is within the time interval formed by the two timestamps located at the end of the state queue;

[0137] Based on the target time point, the latest timestamp, and the previous timestamp adjacent to the latest timestamp, a weighting factor is determined, which is used to indicate the relative position of the target time point between two adjacent timestamps;

[0138] Based on the weighting factor, the object state vector corresponding to the latest timestamp, and the object state vector corresponding to the previous timestamp, state interpolation is performed on the target time point to obtain the object state interpolation corresponding to the target time point.

[0139] Based on the obtained rotation state quaternions of two adjacent specified frames, the target interpolation angle weight determined by the rotation quaternions, and the weight factor, rotation state interpolation is performed on the target time point to determine the rotation state quaternion corresponding to the target time point.

[0140] The transmission delay is compensated based on the state interpolation and / or rotation state interpolation corresponding to the target time point to obtain the interpolation result.

[0141] In this embodiment, after receiving the status event packet, the user equipment sorts the status event packet according to the operation timestamp and updates the status queue of the virtual object.

[0142] For interpolation of additivity variables such as position and velocity, state interpolation uses the following formula:

[0143] ;

[0144] in, This represents the object state interpolation corresponding to the target time point. Indicates the target time point. As a weighting factor, This represents the object's state vector at the latest timestamp (time m). Indicates the previous timestamp (the... The object's state vector at time ( ). Indicates the previous timestamp, This indicates the latest timestamp.

[0145] For the rotation state of an object, quaternions are used to represent it. Using spherical linear interpolation (Slerp), the interpolated rotation state is:

[0146] ;

[0147] in, This represents the quaternion representing the rotation state at the target time point. The quaternion represents the rotation state of the first specified frame. The quaternion representing the rotation state of the second specified frame. This represents the target interpolation angle weight determined by the rotation quaternion.

[0148] Therefore, this embodiment reduces the computational difficulty by performing linear interpolation on additive variables such as position and velocity, making it suitable for continuously changing situations. By performing spherical linear interpolation on the rotation state, it avoids the problem of uneven rotation rate or non-unit quaternion caused by linear interpolation. By adding a weighting factor, it dynamically reflects the relative position of the target time in the two frames before and after, ensuring a smooth transition.

[0149] In some embodiments, the user equipment controlling the virtual object based on the global coordinate system further includes:

[0150] The server obtains a preset logical frame and its corresponding logical timestamp, wherein the logical timestamp is incremented by a preset initial timestamp in a fixed manner.

[0151] When the server reaches the logical timestamp and detects that the buffer stores a state event packet corresponding to the logical timestamp, it obtains the object state stream of the target virtual object based on the stored state event packet.

[0152] When the server reaches the logical timestamp but does not detect that the corresponding state event packet is stored in the buffer, it extracts the previous logical timestamp and the corresponding state event packet from the buffer. Based on the object state vector of the previous logical timestamp, the interval time, and the state change rate of the previous logical timestamp, it determines the object state prediction vector of the logical timestamp, so that the object state flow of the target virtual object is obtained based on the object state prediction vector of the logical timestamp.

[0153] In this embodiment, to ensure inter-frame synchronization, a time-driven approach is used to ensure that each user equipment maintains a uniform step size as it advances along the logical timeline, thereby guaranteeing temporal consistency of state updates across spatially aligned terminals. Specifically, the logical timestamp of each frame increments by a fixed step size to ensure strict alignment of the logical timelines of all terminals. Then, logical frames are advanced at the standard frame rate. At each specified logical timestamp, a state event packet is extracted from the buffer, and linear or spherical interpolation is performed to generate an object state vector for the specified logical timestamp. Furthermore, when the terminal fails to generate logical frames on time due to system jitter (such as excessive CPU load), resulting in dropped frames (such as failure to generate the corresponding frame's state data at a specified time) or incorrect frames (such as the deviation between the current frame number and the global logical frame number exceeding a threshold), the future state is predicted by the rate of change of historical states to avoid users seeing "lag" or "jumps". In this regard, if the user device does not receive a new event (i.e., a new state event packet) or misses a frame update at a specified logical timestamp, the state change rate of the previous logical timestamp, the object state vector of the previous logical timestamp, and the interval time are calculated to predict the future state vector (i.e., the object state prediction vector of the specified logical timestamp).

[0154] Therefore, this embodiment fills in missing state data based on historical state change trends, ensuring the accuracy of predictions and compatibility with the global timeline, ultimately achieving a consistent and synchronized user experience. It ensures that every user operation has perceptible real-time performance and behavioral consistency across the entire system, effectively avoiding issues such as information delays, state jumps, and data conflicts during multi-user collaborative interactions.

[0155] In some embodiments, the control operation request includes user location data, a request timestamp initiating the control operation request, and a behavior identifier corresponding to the operation to be controlled, wherein the user location data is used to indicate the user's coordinates in the global coordinate system; the user equipment controls the virtual object according to the global coordinate system, further comprising:

[0156] When the server receives multiple user devices' requests to control the same target virtual object in the current logical frame, it determines the interaction distance between the user and the target virtual object for any user device based on the corresponding user location data and the object location data in the status event packet.

[0157] The server determines the control score value of the user device based on the system time corresponding to the current logical frame, the request timestamp, the preset time decay factor, the interaction distance, the preset maximum effective interaction distance, the user device's user permission factor, operation time weight, spatial weight, and user permission weight.

[0158] The server selects the user device that maximizes the control score value and uses it as the controller of the target virtual object in the current logical frame. After generating the corresponding control permissions, the server issues them to the controller.

[0159] The user device corresponding to the controller controls the target virtual object to perform the operation behavior according to the control permissions.

[0160] It should be noted that in multi-user mixed reality collaborative scenarios, virtual objects may simultaneously face interactive behaviors from multiple user devices, such as simultaneous clicking, dragging, and rotating. This can easily lead to control conflicts, command overriding, and state jitter, severely impacting interaction logic and collaboration efficiency. To address this, this embodiment dynamically calculates a control priority score based on the current object state and all operation requests, and determines the object's sole controller based on this score.

[0161] Specifically, the control score value of the user equipment is determined using the following formula:

[0162] ;

[0163] in, This represents the control score value of the j-th user equipment. This indicates the system time corresponding to the current logical frame. This represents the request timestamp of the j-th user device. This represents the time decay factor, used to decay the weight of earlier operations; Represents object location data. Represents user location data. Indicates the interaction distance. To maximize the effective interaction distance, This represents the user permission factor for the j-th user device, which indicates the system's trust weight for user priority. For example, higher values ​​can be set for master users and administrators. Indicates the weight of operation time. Indicates spatial weights, This indicates the user's permission weight.

[0164] Furthermore, the user device with the highest score is selected as the sole controller of the target virtual object in the current frame, and control permissions are assigned to it. Other requests are ignored or placed in a candidate queue.

[0165] Therefore, this embodiment determines the control score of the user equipment that initiates the control operation request for the same target virtual object, and selects the user equipment with the highest score as the controller of the target virtual object in the current logic frame, so as to ensure consistency and uniqueness in a collaborative environment with multiple users operating simultaneously and complex role division.

[0166] In some embodiments, the server further includes the following steps:

[0167] Construct a task graph for multi-person collaborative tasks, wherein a node in the task graph represents a task unit, and an edge in the task graph represents the task dependency relationship between two connected task units. The task dependency relationship includes one of task temporal relationship and causal relationship. The task unit is associated with a set of virtual objects and a set of user devices that execute the task unit.

[0168] When the node state of the predecessor node corresponding to any node in the task graph is detected to be the first task state, the node is triggered to update the corresponding node state to the second task state and broadcast it, and operation permissions are sent to the user equipment associated with the node. The first task state is used to indicate that the task has been completed, and the second task state is used to indicate that the task is in progress.

[0169] In this embodiment, a task graph for multi-person collaborative tasks is constructed using a directed graph structure. Nodes in the task graph represent task units (such as disassembling mechanisms, assembling parts, and deducing processes), and edges in the task graph represent task dependencies (such as task B can only be executed after task A is completed).

[0170] Subsequently, the node's state includes a first task state (completed), a second task state (in progress), and a third task state (not started). For example, if any node is in the first task state across all its predecessor nodes, then that node is activated and its state is updated to the second task state. If not all predecessor nodes of that node are in the first task state, then that node is in the third task state. After updating the node state to the second task state, permissions are distributed to the user devices associated with that node. For each operation event processed, the corresponding task state is updated and broadcast to all terminals.

[0171] Therefore, this embodiment can adaptively allocate control rights by constructing a task graph, avoid disorderly preemption, achieve clear interaction logic and consistent behavior, and still ensure the uniqueness and fairness of control behavior in high-frequency interaction or task-intensive scenarios.

[0172] Figure 2 This is a schematic diagram of the structure of a mixed reality interactive system provided in an embodiment of this application. The mixed reality interactive system 200 includes: a server 201 and at least one user device 202;

[0173] The user equipment 202 is used to send anchor point observation data, which is used to describe the coordinate data of multiple preset anchor points in the local coordinate system constructed by the user equipment itself.

[0174] The server 201 is used to acquire the anchor point observation data; based on the anchor point position data and the anchor point observation data, determine the alignment transformation matrix of each user device, wherein the anchor point position data is used to indicate the coordinate data of the anchor point in the global coordinate system, and the alignment transformation matrix is ​​used to indicate the mapping relationship between the local coordinate system and the global coordinate system; perform fusion processing on the alignment transformation matrices of each user device to generate the correction transformation matrix corresponding to each user device and then send it.

[0175] The user equipment 202 is used to map the local coordinates of the virtual object in the local coordinate system to the global coordinate system based on the correction transformation matrix, and to control the virtual object according to the global coordinate system.

[0176] The system in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each device in the system of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each device in the system, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0177] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A mixed reality interaction method, characterized by, The method comprises the following steps: The server acquires anchor observation data sent by at least one user equipment, wherein the anchor observation data is used to describe coordinate data of a plurality of preset anchors in a local coordinate system constructed by the user equipment itself; The server determines an alignment transformation matrix of each user equipment based on anchor position data and the anchor observation data, wherein the anchor position data is used to indicate coordinate data of the anchors in a global coordinate system, and the alignment transformation matrix is used to indicate a mapping relationship between the local coordinate system and the global coordinate system; The server performs fusion processing on the alignment transformation matrix of each user equipment to generate a correction transformation matrix corresponding to each user equipment and then sends the correction transformation matrix; The user equipment maps local coordinates of a virtual object in the local coordinate system to the global coordinate system based on the correction transformation matrix, and controls the virtual object according to the global coordinate system, comprising: the user equipment generates a state event package in response to a control operation request for a target virtual object, and then sends the state event package to the server, wherein the state event package comprises an operation timestamp of the control operation request, an object identifier of the target virtual object, a device identifier of the user equipment initiating the control operation request, and an object state vector of the target virtual object, the object state vector comprises object position data and object attitude quaternion, and the object position data is used to indicate coordinates of the target virtual object in the global coordinate system; the server stores state event packages in a buffer area when it is detected that an operation timestamp carried in a received state event package falls within a preset receiving time window; the server performs state interpolation and / or rotational state interpolation on state data corresponding to a transmission delay based on state event packages in the buffer area to obtain an interpolation result, and obtains an object state stream of the target virtual object based on the object state vector in the state event package and the interpolation result, wherein the transmission delay is a time length between the operation timestamp and a time when the state event package is received.

2. The mixed-reality interaction method of claim 1, wherein, The user equipment obtains anchor observation data by the following steps, comprising: Acquiring environment data of an environment, wherein the environment data comprises a two-dimensional image and depth data; Extracting feature points of each image frame in the two-dimensional image, and determining three-dimensional point coordinates of the feature points in any image frame based on the environment data, wherein the three-dimensional point coordinates are determined by pixel point coordinates of each image frame in the two-dimensional image, the depth data, and a camera intrinsic matrix corresponding to the user equipment; Predicting observed pixel positions of the feature points in a next image frame based on the three-dimensional point coordinates, and constructing a feature point pair from three-dimensional point coordinates and observed pixel positions in adjacent image frames; Determining a feature matching error based on the three-dimensional coordinates and observed pixel positions in the feature point pair, and extracting a first rotation matrix and a first translation vector corresponding to a minimum feature matching error; Determining a pose transformation matrix between adjacent frames based on the first rotation matrix and the first translation vector, and superimposing pose transformation matrices of consecutive frames to form a camera motion trajectory of the user equipment; updating the camera motion trajectory based on the feature point pairs and the obtained inertial measurement data; recording a key frame pose in a pre-selected key frame, the key frame pose including a spatial feature point, depth data, a timestamp, and an initial pose; forming a local spatial map in a local coordinate system based on the camera motion trajectory and the key frame pose, so that anchor point observation data of the anchor points is determined according to the local spatial map.

3. The mixed-reality interaction method of claim 2, wherein, The server determines the alignment transformation matrix of each user equipment based on the anchor point position data and the anchor point observation data, including: mapping the anchor point observation data sent by any user equipment to the anchor point position data, and constructing a corresponding first mapping error model; calculating a first centroid coordinate corresponding to the anchor point observation data of any anchor point of the user equipment, and calculating a second centroid coordinate corresponding to the anchor point position data of the anchor point; performing a decentralization operation on the anchor point observation data based on the first centroid coordinate to obtain an anchor point observation point set, and performing a decentralization operation on the anchor point position data based on the second centroid coordinate to obtain an anchor point position point set; constructing a first covariance matrix based on the anchor point observation point set and the anchor point position point set corresponding to each anchor point; performing singular value decomposition on the first covariance matrix to obtain a first orthogonal matrix and a second orthogonal matrix; constructing a second rotation matrix based on the first orthogonal matrix and the second orthogonal matrix, and constructing a second translation vector based on the first centroid coordinate, the second centroid coordinate, and the second rotation matrix; taking the second rotation matrix and the second translation vector as a solution for minimizing the first mapping error model, so that the alignment transformation matrix of the user equipment is determined based on the second rotation matrix and the second translation vector.

4. The mixed-reality interaction method of claim 3, wherein, The server determines the alignment transformation matrix of each user equipment based on the anchor point position data and the anchor point observation data, and further includes: obtaining a weight of the anchor point, the weight being determined by an observation feature of the anchor point, the observation feature including one of anchor point recognition confidence, image clarity, and depth sensor noise; constructing a corresponding second mapping error model based on the weight, the anchor point observation data, and the anchor point position data; constructing a second covariance matrix based on the weight, the anchor point observation point set, and the anchor point position point set corresponding to each anchor point; obtaining a solution for minimizing the second mapping error model by performing singular value decomposition on the second covariance matrix.

5. The mixed-reality interaction method of claim 4, wherein, The server performs state interpolation and / or rotational state interpolation on state data corresponding to transmission delay based on the state event package in the buffer, to obtain an interpolation result, including: when the state event package is received, adding the state event package to a state queue of the target virtual object, wherein the state queue contains a plurality of time stamps arranged in chronological order, and each time stamp in the plurality of time stamps corresponds to an object state vector; obtaining a target time point, wherein the target time point is in a time interval formed by two time stamps located at the end of the state queue; determine a weight factor based on the target time point, the latest time stamp and a previous time stamp adjacent to the latest time stamp, the weight factor being used to indicate a relative position of the target time point between the two adjacent time stamps; perform state interpolation on the target time point based on the weight factor, an object state vector corresponding to the latest time stamp and an object state vector corresponding to the previous time stamp, to obtain an object state interpolation corresponding to the target time point; perform rotation state interpolation on the target time point based on the weight factor, the rotation state quaternions of the two adjacent specified frames, a target interpolation angle weight determined based on the rotation quaternions, to determine a rotation state quaternion corresponding to the target time point; compensate for transmission delay according to the state interpolation and / or the rotation state interpolation corresponding to the target time point, to obtain an interpolation result.

6. The mixed-reality interaction method of claim 5, wherein, The user equipment controls the virtual object according to the global coordinate system, and further includes: The server obtains a preset logical frame and a corresponding logical time stamp, wherein the logical time stamp is incremented by a preset initial time stamp by a fixed step; When the logical time stamp is reached and it is detected that the state event package corresponding to the logical time stamp is stored in the buffer, the server obtains the object state stream of the target virtual object based on the stored state event package; When the logical time stamp is reached but it is not detected that the corresponding state event package is stored in the buffer, the server extracts the previous logical time stamp and the corresponding state event package before the logical time stamp from the buffer, and determines an object state prediction vector of the logical time stamp based on the object state vector of the previous logical time stamp, the interval time and the state change rate of the previous logical time stamp, so that the object state stream of the target virtual object is obtained based on the object state prediction vector of the logical time stamp.

7. The mixed-reality interaction method of claim 6, wherein, The control operation request includes user position data, a request time stamp of initiating the control operation request, and a behavior identifier corresponding to the operation behavior to be controlled, and the user position data is used to indicate the coordinates of the user in the global coordinate system; The user equipment controls the virtual object according to the global coordinate system, and further includes: When the server receives control operation requests of multiple user equipment for the same target virtual object in the current logical frame, for any user equipment, the server determines an interaction distance between the user and the target virtual object based on the corresponding user position data and the object position data in the state event package; The server determines a control score value of the user equipment based on the system time corresponding to the current logical frame, the request time stamp, a preset time decay factor, the interaction distance, a preset maximum interaction effective distance, a user permission factor of the user equipment, an operation time weight, a space weight and a user permission weight; The server filters out the user equipment corresponding to the maximum control score value as the controller of the target virtual object in the current logical frame, generates a corresponding control permission and then sends it to the controller. The user equipment controls the virtual object according to the global coordinate system, and further includes: The server obtains a preset logical frame and a corresponding logical time stamp, wherein the logical time stamp is incremented by a preset initial time stamp by a fixed step; When the logical time stamp is reached and it is detected that the state event package corresponding to the logical time stamp is stored in the buffer, the server obtains the object state stream of the target virtual object based on the stored state event package; When the logical time stamp is reached but it is not detected that the corresponding state event package is stored in the buffer, the server extracts the previous logical time stamp and the corresponding state event package before the logical time stamp from the buffer, and determines an object state prediction vector of the logical time stamp based on the object state vector of the previous logical time stamp, the interval time and the state change rate of the previous logical time stamp, so that the object state stream of the target virtual object is obtained based on the object state prediction vector of the logical time stamp. The control operation request includes user position data, a request time stamp of initiating the control operation request, and a behavior identifier corresponding to the operation behavior to be controlled, and the user position data is used to indicate the coordinates of the user in the global coordinate system; The user equipment controls the virtual object according to the global coordinate system, and further includes: When the server receives control operation requests of multiple user equipment for the same target virtual object in the current logical frame, for any user equipment, the server determines an interaction distance between the user and the target virtual object based on the corresponding user position data and the object position data in the state event package; The server determines a control score value of the user equipment based on the system time corresponding to the current logical frame, the request time stamp, a preset time decay factor, the interaction distance, a preset maximum interaction effective distance, a user permission factor of the user equipment, an operation time weight, a space weight and a user permission weight; The server filters out the user equipment corresponding to the maximum control score value as the controller of the target virtual object in the current logical frame, generates a corresponding control permission and then sends it to the controller. A user equipment corresponding to the controller controls the target virtual object to perform the operation behavior according to the control authority.

8. The mixed-reality interaction method of claim 7, wherein, The server further comprises the following steps: constructing a task graph of a multi-person collaborative task, wherein a node in the task graph represents a task unit, an edge in the task graph represents a task dependency relationship between two connected task units, the task dependency relationship includes one of a task timing relationship and a causal relationship, and the task unit is associated with a virtual object set and a set of user equipment sets performing the task unit; when detecting that a node state of a predecessor node corresponding to any node in the task graph is a first task state, triggering the node to broadcast after updating the corresponding node state to a second task state, and sending an operation authority to a user equipment associated with the node, wherein the first task state is used to indicate that a task is completed, and the second task state is used to indicate that a task is in progress.

9. A mixed reality interactive system, characterized by Comprise: a server and at least one user equipment; The user equipment is configured to send anchor point observation data, wherein the anchor point observation data is used to describe coordinate data of a plurality of preset anchor points in a local coordinate system constructed by the user equipment itself. The server is configured to obtain the anchor point observation data. Based on anchor point position data and the anchor point observation data, determine an alignment transformation matrix of each user equipment, wherein the anchor point position data is used to indicate coordinate data of the anchor points in a global coordinate system, and the alignment transformation matrix is used to indicate a mapping relationship between the local coordinate system and the global coordinate system; fuse the alignment transformation matrices of the user equipments to generate a correction transformation matrix corresponding to each user equipment and then send the correction transformation matrix; The user equipment is configured to map a local coordinate of a virtual object in a local coordinate system to the global coordinate system based on the correction transformation matrix, and control the virtual object according to the global coordinate system, including: the user equipment generates a state event package in response to a control operation request for a target virtual object and sends the state event package to the server, wherein the state event package includes an operation timestamp of the control operation request, an object identifier of the target virtual object, a device identifier of a user equipment initiating the control operation request, and an object state vector of the target virtual object, the object state vector includes object position data and object attitude quaternion, and the object position data is used to indicate the coordinates of the target virtual object in the global coordinate system; the server stores the state event packages in a buffer area when detecting that the operation timestamp carried by the received state event package falls within a preset receiving time window; the server performs state interpolation and / or rotational state interpolation on state data corresponding to a transmission delay based on the state event packages in the buffer area to obtain an interpolation result, and obtains an object state stream of the target virtual object based on the object state vector in the state event package and the interpolation result, wherein the transmission delay is the time length between the operation timestamp and the time when the state event package is received.

Images