Virtual-real interaction multi-prop matching pursuit method and system based on multiple sensors

Abstract

The invention discloses a virtual-real interaction multi-prop matching tracking method and system based on multiple sensors, and the method comprises the steps: S1, receiving the image data, obtained by a multi-view camera module, of infrared light-emitting points on a plurality of prop bodies, calculating the coordinates of each infrared light-emitting point in a three-dimensional space, and calculating the coordinates of each infrared light-emitting point; obtaining a first movement track set; s2, receiving IMU attitude data sent by an inertial measurement unit on each prop body, and estimating an IMU track of an infrared light-emitting point on each prop body through pre-integration and coordinate transformation to obtain a second motion track set; s3, aligning and normalizing the tracks, and calculating the similarity score of each track through a dynamic time warping algorithm to obtain a similarity score matrix; s4, inputting the similarity score matrix into a Hungary algorithm for optimal matching to obtain an identity binding result; and S5, motion prediction is carried out on the prop body whose identity is successfully bound according to the first motion track, and continuous pose tracking is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a virtual-real interaction multi-prop matching tracking method and system based on multiple sensors. BACKGROUND

[0002] With the rapid development of virtual reality (VR), augmented reality (AR) and human-computer interaction technology, virtual interaction devices are increasingly widely used in commercial display, industrial simulation, education and training, and entertainment games. In these interaction scenarios, physical props serve as an important bridge connecting users and virtual environments, and the design of the physical props directly affects the naturalness of the interaction and the immersion of the users. In particular, in complex scenarios where multiple people operate together or multiple props are used simultaneously, how to accurately and stably bind and track the unique identity (ID) of each physical prop with its real-time pose (position and attitude) in three-dimensional space is a key technical problem for achieving high-quality virtual-real interaction.

[0003] At present, there are various technical solutions in the industry for three-dimensional space positioning and tracking. One traditional solution is an optical motion capture system based on multiple retroreflective markers. This type of system arranges multiple retroreflective markers on the surface of the prop, uses multiple high-precision infrared cameras deployed around the scene for irradiation and capture, and calculates the three-dimensional coordinates of the center of the retroreflective markers by calculating the image coordinates of the center. Although this type of system can achieve high positioning accuracy, it has significant drawbacks. First, the system structure is complex and costly, requiring a precise multi-camera array and dedicated retroreflective markers for calibration, which makes it difficult to popularize on a large scale in commercial landing scenarios. Second, the surface of the retroreflective markers is easily worn, contaminated or obstructed, resulting in poor system robustness. In addition, in a multi-person multi-prop scenario, it is difficult to stably and correctly distinguish and match the IDs among multiple props based on visual information alone, which can easily lead to the ID swapping problem.

[0004] To overcome the limitations of pure visual systems, existing technologies have also introduced solutions that combine inertial measurement units (IMUs). For example, some virtual reality handles are equipped with IMU sensors to track the attitude changes of the handles. However, single IMU data has inherent cumulative error (Drift) and cannot provide absolute spatial position information, and the accuracy will decrease significantly after long-term use. Although a combination of vision and IMU can be used, in a dynamic scenario with multiple people and multiple props, how to correctly and efficiently match multiple visual positioning points and multiple IMU data streams from different physical carriers collected asynchronously remains a difficult problem. Simple spatio-temporal nearest neighbor matching methods are prone to matching errors when the props move dramatically, intersect with each other or are temporarily obstructed, leading to tracking failure and interaction interruption.

[0005] Therefore, in view of the above prior art, the present application provides a multi-prop matching tracking method and system for virtual-real interaction based on multiple sensors. SUMMARY

[0006] The present application aims at the defects of the prior art, and provides a multi-prop matching tracking method and system for virtual-real interaction based on multiple sensors, which has a simple appearance and good structural strength, a low cost of a single prop, and can be used for cooperation with a motion capture system when multiple people hold the prop, binding the IDs of the user and the prop, and ensuring the robustness of the interaction.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] A multi-prop matching tracking method for virtual-real interaction based on multiple sensors, comprising:

[0009] Step S1. Receiving image data of infrared light emitting points on multiple prop bodies obtained by a multi-view camera module, and calculating coordinates of each infrared light emitting point in a three-dimensional space to obtain a first motion trajectory set;

[0010] Step S2. Receiving IMU attitude data sent by an inertial measurement unit on each prop body, and estimating an IMU trajectory of each infrared light emitting point on the prop body through pre-integration and coordinate transformation to obtain a second motion trajectory set;

[0011] Step S3. Aligning and normalizing the first motion trajectory set and the second motion trajectory set, and calculating a similarity score of each first motion trajectory and second motion trajectory through a dynamic time warping algorithm to obtain a similarity score matrix;

[0012] Step S4. Inputting the similarity score matrix into a Hungarian algorithm for optimal matching to obtain an identity binding result of the infrared light emitting points and the prop bodies;

[0013] Step S5. For the prop bodies with successful identity binding, performing motion prediction according to the first motion trajectory to realize continuous pose tracking of the prop bodies.

[0014] Further, the step S1 is specifically:

[0015] S11. Receiving image data of infrared light emitting points on multiple prop bodies obtained by each camera in a multi-view camera module;

[0016] S12. Forming a camera pair with any two cameras in the multi-view camera module, and extracting feature pixel coordinates of the infrared light emitting point images obtained by the two cameras, and internal and external parameter matrices of the cameras, and constructing a space ray from the optical centers of the two cameras to the infrared light emitting point;

[0017] S13. Calculate the common perpendicular of the two spatial rays and determine whether the length of the calculated common perpendicular is less than or equal to the preset threshold. If so, determine that the two spatial rays intersect and set the midpoint of the common perpendicular as the initial estimated position of the current infrared emitting point in the three-dimensional space below the camera.

[0018] S14. Traverse all camera pairs and repeat steps S12-S13 to obtain multiple initial estimated positions in three-dimensional space for the current infrared emitting point;

[0019] S15. Perform cluster analysis on multiple initial estimated positions in three-dimensional space, and use nonlinear optimization to solve for the coordinates of the three-dimensional space point with the smallest reprojection error for the spatial rays belonging to the same cluster, which is then used as the first motion trajectory of the current infrared emitting point.

[0020] S16. Traverse the image data of all infrared emitting points and repeat steps S12-S15 to obtain the first set of motion trajectories of all infrared emitting points.

[0021] Furthermore, step S2 specifically includes:

[0022] S21. Receive IMU attitude data sent by the inertial measurement units on all props;

[0023] S22. Pre-integrate the IMU attitude data of the current prop body to obtain the relative pose change of the current prop body;

[0024] S23. Obtain the fixed positional relationship between the infrared emitting point and the inertial measurement unit in the current prop body coordinate system, and perform coordinate transformation on the fixed positional relationship and the relative pose change of the current prop body to estimate the motion trajectory of the infrared emitting point of the current prop body and obtain the second motion trajectory.

[0025] S24. Iterate through the IMU attitude data sent by the inertial measurement units on all prop bodies, and repeat steps S22-S13 to obtain the second motion trajectory set of all infrared light-emitting points on the prop bodies.

[0026] Furthermore, the alignment of the first set of motion trajectories and the second set of motion trajectories in step S3 specifically involves:

[0027] Calculate the covariance matrix of the first set of motion trajectories and the second set of motion trajectories respectively;

[0028] The calculated covariance matrix is ​​decomposed into eigenvalues ​​to obtain the principal directions corresponding to the first set of motion trajectories and the second set of motion trajectories.

[0029] Based on the principal directions corresponding to the first and second sets of motion trajectories, the rotation matrix is ​​calculated using the Rodrigues formula, and one of the motion trajectories is rotated around itself according to the rotation matrix until the principal direction of the currently rotated motion trajectory is aligned with the principal direction of the other motion trajectory.

[0030] Furthermore, in step S3, the similarity score of each first motion trajectory and second motion trajectory is calculated using the dynamic time warping algorithm as follows:

[0031] The dynamic time warping algorithm is used to perform point matching on the first and second motion trajectories, and to establish the correspondence between multiple points on the first and second motion trajectories to obtain matching point pairs.

[0032] Calculate the average Euclidean distance between matching point pairs and use the average Euclidean distance as the similarity score between the first and second motion trajectories.

[0033] Furthermore, step S5 specifically includes:

[0034] S51. For each successfully bound item, predict the spatial position of the current item in the next moment based on the corresponding first motion trajectory;

[0035] S52. Receive the next moment image data of all infrared emitting points on the prop body acquired by the multi-view camera module, and search for infrared emitting points that match the current prop body within the neighborhood of the predicted spatial location;

[0036] S53. Determine whether a matching infrared emitting point is found within the domain. If so, update the coordinates of the found infrared emitting point to the latest pose of the current prop body and maintain the identity binding state to achieve continuous pose tracking.

[0037] Furthermore, in step S53, it is determined whether a matching infrared emitting point is found within the field. If not, it indicates that the infrared emitting point tracking has failed. The historical first motion trajectory of the current infrared emitting point in the last preset time period before the tracking failure is obtained, and steps S1-S5 are re-executed.

[0038] Correspondingly, a multi-sensor-based virtual-real interaction multi-prop system is also provided, including:

[0039] At least one prop body, each prop body includes an infrared light-emitting point and an inertial measurement unit, used to emit light through the infrared light-emitting point and acquire the IMU attitude data of the current prop body through the inertial measurement unit, and send the acquired IMU attitude data to the data acquisition and processing module.

[0040] The multi-view camera module is used to acquire image data of infrared emitting points on multiple props and send the acquired image data of infrared emitting points to the data acquisition and processing module.

[0041] The data acquisition and processing module is communicatively connected to the prop body and the multi-view camera module, respectively, and is used to receive the IMU attitude data of the current prop body and the image data of infrared emitting points on multiple prop bodies, and execute the aforementioned multi-sensor-based virtual-real interaction multi-prop matching and tracking method.

[0042] Furthermore, each camera in the multi-camera module is equipped with an infrared filter.

[0043] Furthermore, each prop body sends IMU attitude data to the data acquisition and processing module through the corresponding UDP port number.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. This invention uses low-cost infrared LEDs to replace high-precision, fragile reflective spheres, and employs a general-purpose IMU module and microcontroller, significantly reducing the hardware cost of a single prop. It abandons the scheme that relies on special arrangement (such as multiple rigid reflective spheres) or specific optical features (such as specific flashing codes) for identification, reducing the requirements for camera resolution and accuracy, and simplifying the prop manufacturing process.

[0046] 2. Redundant information is created through multi-sensor data fusion. When the vision system loses the target due to occlusion or rapid movement, IMU data can provide short-term dead reckoning to maintain tracking; conversely, the vision system can provide absolute position to correct the cumulative drift of the IMU.

[0047] 3. The continuous tracking mechanism of "prediction-verification" and the dynamic loss recovery and new item registration mechanism together constitute a stable system that can maintain itself and can easily cope with dynamic changes in real-world scenarios such as items being temporarily obscured, leaving the field of vision, and the addition of new items.

[0048] 4. The core of this invention, the trajectory similarity-based matching algorithm, is an identity binding scheme independent of the number of props. The system does not need to preset a unique visual identifier for each prop. Regardless of the number of props, the matching process is logically consistent, enabling the system to easily support flexible expansion from single-person single-prop to multi-person multi-prop. Attached Figure Description

[0049] Figure 1 This is a structural diagram of the prop body provided in Example 1;

[0050] Figure 2This is a flowchart of a multi-sensor-based virtual-real interaction multi-prop matching and tracking method provided in Embodiment 2. Detailed Implementation

[0051] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0052] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-sensor-based method and system for matching and tracking virtual and real interactive props.

[0053] Example 1

[0054] This embodiment provides a virtual-real interaction multi-prop system based on multiple sensors, including at least one prop body, a multi-view camera module, and a data acquisition and processing module.

[0055] like Figure 1 As shown, each prop's casing contains an LED infrared point light source 1, an inertial measurement unit 2, a microcontroller 3, a battery, a vibration motor, etc.

[0056] The LED infrared point light source 4 emits light continuously, and is wrapped by a soft light cover 11 to form a near-spherical light source; the inertial measurement unit 2 (hereinafter referred to as the IMU unit) integrates a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer; the microcontroller 3 has a built-in Wi-Fi unit and a microcontroller, which is responsible for sending the attitude data (including three-axis acceleration, three-axis angular acceleration and three-axis magnetic field) collected by the IMU unit to the data acquisition and processing module (hereinafter referred to as DAS) through the UDP port.

[0057] The multi-camera unit consists of multiple (e.g., four) high-frame-rate industrial cameras deployed above the virtual interactive scene, with overlapping fields of view to ensure coverage of the interactive space. Each camera is precisely synchronized and equipped with an infrared filter to filter out ambient light interference, capturing only the infrared light of a specific wavelength emitted by the prop itself. All cameras are pre-calibrated with intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters (position and orientation in the world coordinate system).

[0058] DAS is a high-performance computer that connects to a multi-camera module via network cable, USB signal cable, etc. It synchronously acquires and processes image data of infrared emitting points obtained by all cameras, and then obtains the coordinates of multiple infrared emitting points in three-dimensional space.

[0059] DAS also communicates with all the props via a wireless network, and then receives and processes IMU data packets sent by the props through UDP port numbers. Each IMU data packet contains its source UDP port number, timestamp, and data such as three-axis acceleration and three-axis angular velocity. This allows DAS to determine the identity ID information of the connected prop through the port number.

[0060] This embodiment uses low-cost infrared LEDs instead of high-precision, fragile reflective spheres, and employs a general-purpose IMU module and microcontroller, significantly reducing the hardware cost of a single prop. It abandons the scheme that relies on special arrangement patterns (such as multiple rigid reflective spheres) or specific optical characteristics (such as specific flashing codes) for identification, reducing the requirements for camera resolution and accuracy, and simplifying the prop manufacturing process.

[0061] Example 2

[0062] This embodiment provides a multi-sensor-based virtual-real interaction multi-prop matching and tracking method. This method is based on a multi-sensor-based virtual-real interaction multi-prop system from Embodiment 1, such as... Figure 2 As shown, the matching tracking method includes:

[0063] Step S1. Receive image data of multiple infrared emitting points on the prop body acquired by the multi-view camera module, and calculate the coordinates of each infrared emitting point in three-dimensional space to obtain the first set of motion trajectories;

[0064] Step S2. Receive the IMU attitude data sent by the inertial measurement unit on each prop body, and estimate the IMU trajectory of the infrared emitting point on each prop body through pre-integration and coordinate transformation to obtain the second motion trajectory set;

[0065] Step S3. Align and normalize the first set of motion trajectories and the second set of motion trajectories, and calculate the similarity score of each first motion trajectory and the second motion trajectory using the dynamic time warping algorithm to obtain the similarity score matrix;

[0066] Step S4. Input the similarity score matrix into the Hungarian algorithm for optimal matching to obtain the identity binding results between the infrared emitting points and the prop body;

[0067] Step S5. For the prop body that has been successfully bound to the identity, perform motion prediction based on the first motion trajectory to achieve continuous pose tracking of the prop body.

[0068] It should be noted that steps S1 and S2 in this embodiment can be executed simultaneously and are not limited by this order.

[0069] In step S1, image data of multiple infrared emitting points on the prop body acquired by the multi-view camera module are received, and the coordinates of each infrared emitting point in three-dimensional space are calculated to obtain the first set of motion trajectories; specifically:

[0070] S11. Receive image data of multiple infrared light-emitting points on the prop body acquired by each camera in the multi-camera module;

[0071] When multiple props are held or moved into the scene by multiple users, the DAS synchronously receives image data from each camera in the multi-camera module. To ensure timing consistency, all cameras are synchronously exposed via hardware trigger signals from the DAS. Because the cameras are equipped with infrared filters, the infrared emitting points on the props appear as bright white spots in the acquired images, while the environmental background is almost black, greatly simplifying subsequent feature extraction.

[0072] S12. Form a camera pair from any two cameras in the multi-camera module, and extract the feature pixel coordinates of the infrared emitting point images obtained by the two cameras as well as the intrinsic and extrinsic parameter matrices of the cameras, and construct a spatial ray from the optical center of the two cameras pointing to the infrared emitting point.

[0073] For each infrared emitting point image acquired by the camera, DAS first performs Gaussian filtering to suppress noise. Subsequently, using an algorithm based on gray-scale centroid method or ellipse fitting, the center feature pixel coordinates [u, v] of each spot are extracted with sub-pixel accuracy, where u represents the pixel column coordinate and v represents the pixel row coordinate.

[0074] For each extracted pixel coordinate [u, v], it is transformed to the camera normalized coordinate system using the pre-calibrated camera intrinsic parameter matrix K to eliminate the influence of lens distortion, as shown below:

[0075] p norm = K - ¹ ∙ [u, v, 1] T

[0076] Among them, K - ¹ denotes the inverse of the camera intrinsic parameter matrix K; [u, v, 1] T p represents a column vector of homogeneous pixel coordinates; norm Represents the calculated normalized coordinate point [x] norm , y norm , 1] T It is a three-dimensional vector with a z-component of 1.

[0077] In the camera coordinate system, starting from the camera optical center O(0,0,0), passing through point p normA ray whose unit direction vector v can be obtained through v=p norm / z p norm We obtain, where z p norm p norm The norm of the z-component, due to p norm The z-component is 1.

[0078] Finally, using the camera's extrinsic parameter matrix T = [R|t], the ray is transformed to a unified world coordinate system. The starting point of the ray in the world coordinate system is the camera's optical center A. w = t, direction vector is v w =Rv.

[0079] S13. Calculate the common perpendicular of the two spatial rays and determine whether the length of the calculated common perpendicular is less than or equal to the preset threshold. If so, determine that the two spatial rays intersect and set the midpoint of the common perpendicular as the initial estimated position of the current infrared emitting point in the three-dimensional space below the camera.

[0080] For a camera pair (Cam i Cam j There are two rays Ray in the world coordinate system. i (A wi , v wi ) and Ray j (A wj , v wj Due to the presence of errors, these are usually skew lines that do not want to intersect. Therefore, it is necessary to calculate the common perpendicular of these two skew rays, whose direction vector n can be obtained from v. wi With v wj The cross product is obtained, i.e., n = v wi ×v wj .

[0081] The foot of the perpendicular P of the common perpendicular to the two rays i (in Ray) i (above) and P j (in Ray) j The above can be obtained by solving the following system of linear equations:

[0082] (P i - P j ) ·v wi = 0, indicating vector (P) i - P j ) and v wi vertical;

[0083] (P i - P j ) ·vwj = 0, indicating vector (P) i - P j ) and v wj vertical;

[0084] P i = A wi + λ * v wi , indicating P i Ray i Above, λ represents the scale parameter;

[0085] P j = A wj +μ * v wj , indicating P j Ray j Above, μ represents the scale parameter.

[0086] The parameters λ and μ can be solved by simultaneous equations, and then P can be obtained. i and P j .

[0087] Calculate P i and P j The Euclidean distance L between them. A preset distance threshold d (e.g., 5mm). If L≤d, the two rays are considered to intersect within the allowable error range, and the midpoint M of the common perpendicular is determined. ij = (P i + P j ) / 2 serves as the initial estimated three-dimensional spatial position of the current infrared emitting point, calculated by the camera.

[0088] S14. Traverse all camera pairs and repeat steps S12-S13 to obtain multiple initial estimated positions in three-dimensional space for the current infrared emitting point;

[0089] DAS iterates through all possible camera pair combinations. For a module with N cameras, theoretically, C(N,2) camera pairs can be generated for the same physical luminous point, resulting in C(N,2) initial estimated positions in 3D space. These initial estimated points constitute the point cloud estimate of the true position of the luminous point.

[0090] S15. Perform cluster analysis on multiple initial estimated positions in three-dimensional space, and use nonlinear optimization to solve for the coordinates of the three-dimensional space point with the smallest reprojection error for the spatial rays belonging to the same cluster, which is then used as the first motion trajectory of the current infrared emitting point.

[0091] Due to noise and errors, the initial estimated point cloud obtained in step S14 will be closely clustered around the actual physical location, but there may be individual outliers that deviate significantly. DAS uses a clustering algorithm (such as DBSCAN) to cluster these points. The cluster that belongs to the same group and contains the most points is considered the effective estimated point set, while discrete points are discarded.

[0092] For all initial estimated points belonging to the same valid cluster, the associated spatial rays from different cameras (usually far more than two) are pooled together. DAS uses these rays, along with an initial 3D point (such as the centroid of the cluster), as input and performs nonlinear optimization using bundle adjustment. The objective function of the optimization is to minimize the reprojection error, i.e., to adjust the coordinates of the 3D point P such that the sum of the squares of the differences between P's reprojected position on the 2D image of each participating camera and the initially detected feature pixel coordinates [u, v] is minimized.

[0093] S16. Traverse the image data of all infrared emitting points and repeat steps S12-S15 to obtain the first set of motion trajectories of all infrared emitting points.

[0094] The DAS performs steps S12-S15 repeatedly, either in parallel or cyclically, for each detected infrared emitting point in the scene. Finally, it outputs a first set of motion trajectories {Traj1, Traj2, ..., Traj...}. m}, where M is the number of currently tracked glowing points. At this stage, these trajectories are anonymous; the system knows their spatial paths, but they are not yet bound to any item entity with a specific ID.

[0095] In step S2, the IMU attitude data sent by the inertial measurement unit on each prop body is received, and the IMU trajectory of the infrared emitting point on each prop body is estimated through pre-integration and coordinate transformation to obtain the second set of motion trajectories, specifically:

[0096] S21. Receive IMU attitude data sent by the inertial measurement units on all props;

[0097] The DAS continuously listens on a specific UDP port via a wireless network (such as Wi-Fi). Each prop's microcontroller sends data packets to the DAS at a fixed frequency (e.g., 100Hz) using its unique UDP port number. Each data packet contains:

[0098] Data header: contains item identification information (implicit in the source UDP port number) and timestamp.

[0099] IMU attitude data: including three-axis acceleration [a]x , a y , a z ], triaxial angular velocity [ω x , ω y , ω z ], and optional triaxial magnetic field data [m x , m y , m z ].

[0100] After receiving the data packet, DAS parses it according to its source port number and distributes it to the corresponding item data processing thread.

[0101] S22. Pre-integrate the IMU attitude data of the current prop body to obtain the relative pose change of the current prop body;

[0102] DAS maintains a data buffer for the current item (e.g., item A corresponding to UDP port 5001). When matching is required, it captures all IMU data of the item within a recent specific time period Δt (e.g., 0.5 seconds from T0 to T1).

[0103] Pre-integration is performed on all continuous IMU data within this time period. The pre-integration outputs the relative pose change of the prop body, including displacement increment Δp, velocity increment Δv, and rotation increment ΔR. Simultaneously, this process also outputs the initial angular velocity ω0 and initial velocity v0 at the initial time T0. The displacement increment Δp is obtained by double integration of the rotationally compensated acceleration, representing the IMU's displacement in the prop coordinate system from T0 to T1. The velocity increment Δv is obtained by single integration of the rotationally compensated acceleration. The rotation increment ΔR is obtained by integrating the angular velocity, typically represented as a rotation matrix or quaternion, representing the IMU's attitude change from T0 to T1.

[0104] The pre-integration method used in this embodiment is an IMU processing technique that performs integration in the local coordinate system (i.e., the IMU body coordinate system) to directly calculate the relative pose change from time T0 to T1, rather than the absolute pose.

[0105] S23. Obtain the fixed positional relationship between the infrared emitting point and the inertial measurement unit in the current prop body coordinate system, and perform coordinate transformation on the fixed positional relationship and the relative pose change of the current prop body to estimate the motion trajectory of the infrared emitting point of the current prop body and obtain the second motion trajectory.

[0106] The DAS reads the structural parameters of the current prop body from a pre-calibrated configuration database. These parameters record the precise relative positional relationship between the infrared emitter and the inertial measurement unit (IMU) in the prop body's coordinate system. This positional relationship is a three-dimensional spatial vector representing a fixed offset from the IMU center to the infrared emitter, which is determined through precise measurement during prop manufacturing and stored in the DAS system.

[0107] The DAS receives the pre-integration processing results, including the relative rotation and relative displacement changes of the prop body over a time period. These data describe the IMU's own motion trajectory in three-dimensional space, but the positional offset between the emitting point and the IMU has not yet been considered.

[0108] DAS uses the principles of rigid body kinematics to convert the relative pose changes of the IMU into the motion trajectory of the infrared emitter. The specific calculation process is as follows: for the IMU pose at each time point, firstly, the position and attitude of the IMU in the world coordinate system are calculated, and then, based on a fixed position offset, the position of the infrared emitter in the world coordinate system at the same moment is calculated using the coordinate transformation formula.

[0109] By calculating a continuous time series (e.g., T0-T1), a complete motion trajectory of the infrared emitter is generated, resulting in a second motion trajectory. This trajectory describes how the infrared emitter should move in three-dimensional space if only IMU data is used for estimation.

[0110] In this embodiment, in order to ensure a fair comparison between the second motion trajectory and the first motion trajectory, the initial second motion trajectory needs to be compensated, specifically as follows:

[0111] DAS uses the initial angular velocity ω0 and initial velocity v0 obtained through pre-integration to mathematically compensate for the entire initial second motion trajectory. This operation is mathematically equivalent to subtracting the positional offset caused by the initial motion state from each point on the trajectory. After compensation, an equivalent second motion trajectory (IMU trajectory) T is obtained, which is equivalent to motion starting from zero initial velocity. imu .

[0112] S24. Iterate through the IMU attitude data sent by the inertial measurement units on all prop bodies, and repeat steps S22-S13 to obtain the second motion trajectory set of all infrared light-emitting points on the prop bodies.

[0113] Steps S22-S23 are repeated in parallel or cyclically for each prop that has established a connection with DAS (i.e., each valid UDP data stream). Finally, DAS outputs a second set of motion trajectories {T}. imu1 , T imu2 , ..., T imun}, where n represents the number of currently moving item bodies. During this phase, each second movement trajectory is associated with a unique item body ID via its UDP port number.

[0114] In this embodiment, the initial velocity of the first motion trajectory is compensated, transforming it to a zero initial velocity state.

[0115] The first motion trajectory is a continuous three-dimensional coordinate sequence directly observed by the multi-view camera module. When the DAS begins tracking an infrared emitting point (i.e., the starting time T of the trajectory),... start The infrared emitting point may already be in motion, meaning it has an unknown, non-zero initial velocity V. v0 Therefore, compensation is needed for this trajectory, specifically:

[0116] DAS selects a segment of data from the beginning of the first motion trajectory (e.g., the first N data points, corresponding to time T). start To T start +△t).

[0117] By performing numerical differentiation on the position data of these points (e.g., by using the least squares method to perform linear or quadratic fitting and then taking the derivative), the starting time T of the trajectory of the infrared emitting point can be estimated. start The initial velocity vector V v0 = [v x , v y ,v z ]ᵀ.

[0118] For each data point P on the first motion trajectory i Its displacement relative to the starting point P0 can be considered as being determined by the initial velocity V. v0 This is provided in conjunction with its subsequent acceleration / deceleration motion. The goal of compensation is to mathematically subtract the amount caused by the initial velocity V. v0 The displacement provided separately.

[0119] After the above processing, both the first and second motion trajectories are unified to the same baseline of zero initial velocity. At this point, the difference between the two trajectories mainly stems from the essential difference in the motion patterns they describe, rather than the difference in their initial states. This allows the DTW algorithm to focus on comparing the similarity of the two trajectories in dynamic characteristics such as acceleration and turning, greatly improving the accuracy and robustness of the identity binding process.

[0120] In step S3, the first set of motion trajectories and the second set of motion trajectories are aligned and normalized, and the similarity score of each first motion trajectory and the second motion trajectory is calculated by the dynamic time warping algorithm to obtain the similarity score matrix.

[0121] S31. Calculate the covariance matrix of the first set of motion trajectories and the second set of motion trajectories respectively;

[0122] For any trajectory V containing n three-dimensional points: V = {p1, p2, ..., p...} n}, where p k =[x k , y k ,z k ] T .

[0123] Calculate the centroid p of the trajectory c = [x c , y c , z c ]ᵀ, where x c = (x1 + x2 + ... + x n ) / n, y c = (y1 + y2 + ... + y n ) / n, z c = (z1 + z2 + ... + z n ) / n.

[0124] The decentralized coordinate matrix is ​​calculated as follows:

[0125] P = [p1 - p c , p2 - p c , ..., p n - pc ] T

[0126] The covariance matrix C is then expressed as:

[0127]

[0128] Where i,j = 1,2,3 correspond to the x, y, and z coordinate axes, respectively.

[0129] S32. Perform eigenvalue decomposition on the calculated covariance matrix to obtain the principal directions corresponding to the first set of motion trajectories and the second set of motion trajectories;

[0130] The eigenvalue decomposition of the covariance matrix C of any trajectory is expressed as:

[0131]

[0132] Where Λ represents the diagonal matrix formed by the eigenvalues, and Λ = diag(λ1, λ2, λ3), λ1 ≥ λ2 ≥ λ3; V represents the eigenvector matrix, and V = [v1, v2, v3], where each column vector v i For the corresponding eigenvalue λ i The unit eigenvector.

[0133] The first eigenvector v1 (corresponding to the largest eigenvalue λ1) is taken as the principal direction of the trajectory. This principal direction represents the main extension direction of the trajectory in three-dimensional space.

[0134] S33. Based on the principal directions corresponding to the first set of motion trajectories and the second set of motion trajectories, calculate the rotation matrix using the Rodrigues formula, and rotate one of the motion trajectories around itself according to the rotation matrix until the principal direction of the currently rotated motion trajectory is aligned with the principal direction of the other motion trajectory;

[0135] Let the principal direction of the first trajectory be v1. v The principal direction of the second motion trajectory is v1 i ;

[0136] The axis of rotation is represented as follows:

[0137] k = v1 v × v1 i

[0138] The rotation angle is calculated as follows:

[0139] θ = arccos(v1) v v1 i )

[0140] The rotation matrix R is calculated using Rodrigues' formula and is expressed as follows:

[0141] R = I + sinθ × K + (1-cosθ) × K 2

[0142] Where K represents the cross matrix of vector k;

[0143] Applying rotational transformation: p′= R(p- p) c ) + p c Where p' represents the coordinates of the rotated point; p represents the original coordinates of the point. c R represents the centroid of the trajectory; R represents the rotation matrix calculated using Rodrigues' formula.

[0144] Based on the rotational changes, one of the motion trajectories is rotated around itself until the main direction of the currently rotating motion trajectory is aligned with the main direction of the other motion trajectory.

[0145] S34. Normalize the aligned trajectory;

[0146] The two aligned trajectories are then normalized. First, the scale factor for each trajectory is calculated, determined by the distribution range of the trajectory point set in 3D space. Specifically, the difference between the maximum and minimum values ​​of all points on each coordinate axis is found, and its Euclidean norm is used as the scale representation of that trajectory. Then, the scale factors of the two trajectories are unified. Typically, the scale of one trajectory is used as a benchmark, and the coordinates of all points on the other trajectory are scaled proportionally to ensure that both trajectories have the same scale range.

[0147] Building upon scale normalization, the amplitude variations of the trajectories are further normalized. By calculating the mean and standard deviation of the amplitude for each trajectory, a standard deviation normalization method is used to adjust the amplitude distribution of the two trajectories to the same numerical range. Specifically, the mean of each trajectory is subtracted from the coordinate value of that trajectory, and then divided by the standard deviation, ensuring that both processed trajectories have zero mean and unit variance.

[0148] S35. Use the dynamic time warping algorithm to perform point matching on the first motion trajectory and the second motion trajectory, and establish the correspondence between multiple points on the first motion trajectory and the second motion trajectory to obtain matching point pairs;

[0149] First, a distance matrix is ​​constructed between the two trajectories. The rows of this matrix correspond to points on the first trajectory, and the columns correspond to points on the second trajectory. Each element in the matrix stores the Euclidean distance between corresponding point pairs. This calculation forms a complete two-dimensional distance matrix, visually displaying the spatial distance relationships between all possible point pairs on the two trajectories.

[0150] Based on the constructed distance matrix, a dynamic time warping algorithm is used to find the optimal matching path between two trajectories. The algorithm starts from the top left corner of the matrix and gradually moves towards the bottom right corner, finding the path with the minimum cumulative distance while satisfying the constraints of path continuity and monotonicity. This process is implemented through dynamic programming, maintaining a cumulative distance matrix to record the minimum cumulative distance to each position.

[0151] By backtracking along the optimal path, a detailed point correspondence is established between the two trajectories. This correspondence is a many-to-many mapping, allowing one point in one trajectory to correspond to multiple points in another trajectory, thus effectively handling potential local time scaling and rate variations between trajectories. The system records the correspondence of all these matching point pairs, forming a complete matching result.

[0152] The DTW algorithm compares two sequences of potentially different lengths (in this case, trajectory point sequences A and B), finds the optimal non-linear alignment between them, and calculates the minimum cumulative distance required to achieve this alignment. The core of the algorithm is to use dynamic programming to find a path (i.e., the alignment) that minimizes the cumulative distance. Specifically:

[0153] Step 1: Initialize the dynamic programming matrix:

[0154] Create a matrix D of size (m+1) x (n+1), where m and n are the lengths of trajectories A and B, respectively.

[0155] The top-left element D[0][0] of the matrix is ​​initialized to 0, which means that the cumulative distance between the two empty sequences is 0.

[0156] The first column (D[1..m][0]) and the first row (D[0][1..n]) of the matrix are initialized to an infinite (∞) value. This is a boundary condition, the physical meaning of which is that the cost of aligning a non-empty sequence with an empty sequence is infinite, thus ensuring that the backtracking path does not exceed the effective range of the matrix.

[0157] Step 2: Fill in the distance matrix (forward calculation):

[0158] This is a double loop process that iterates through each cell D[i][j] of the matrix from i = 1 to m and j = 1 to n.

[0159] For each cell (i, j):

[0160] Calculate local cost: Calculate the Euclidean distance cost between the (i-1)th point in trajectory A (because the sequence index starts from 0) and the (j-1)th point in trajectory B.

[0161] Determine the minimum cumulative path: The value of cell D[i][j] is equal to the local cost mentioned above, plus the minimum value among the three adjacent cells from the left (i, j-1), above (i-1, j), and upper left (i-1, j-1).

[0162] From the top left (i-1, j-1): This represents a "match" operation, where a point in A is aligned with a point in B.

[0163] From above (i-1, j): represents an "insertion" of sequence A or a "compression" of sequence B, where a point of A is aligned with multiple points of B.

[0164] From the left (i, j-1): This represents an "insertion" of sequence B or a "compression" of sequence A, where a point of B is aligned with multiple points of A.

[0165] In this way, D[i][j] records the minimum cumulative distance among all possible aligned paths from the beginning of the sequence (0, 0) to the current point (i, j).

[0166] Step 3: Backtracking to find the optimal alignment path:

[0167] After matrix D is filled, backtracking is performed from the endpoint (m, n) to the starting point (0, 0) to construct a specific point pair matching sequence alignment.

[0168] Initialization: Create an empty list `alignment` to store the matched point pairs and set the current coordinates to (m, n).

[0169] Backtracking loop: When i > 0 and j > 0 (i.e., before backtracking to the matrix boundary), the following operations are performed in the loop:

[0170] Record the current match: Add the original sequence index (i-1, j-1) corresponding to the current point to the head of the alignment list.

[0171] Determine the backtracking direction: Check the values ​​of the three cells to the left (i, j-1), above (i-1, j), and to the upper left (i-1, j-1) of the current position (i, j).

[0172] Move to the direction of the minimum value: Select the smallest of the three values ​​and move the current coordinate (i, j) to the position corresponding to the minimum value.

[0173] If the minimum value is in the upper left corner, then move to (i-1, j-1) (one-to-one matching).

[0174] If the minimum value is above, then move to (i-1, j) (one point of A matches multiple points of B).

[0175] If the minimum value is on the left, then move to (i, j-1) (one point of B matches multiple points of A).

[0176] This backtracking process will eventually find a path from (m, n) to (1, 1), which determines the least costly alignment between sequences A and B.

[0177] Step 4: Return the results:

[0178] Final distance: The value of the element D[m][n] in the lower right corner of the matrix is ​​the minimum cumulative distance between the two trajectories after optimal alignment.

[0179] Alignment path: The list alignment stores all point pairs that make up the optimal alignment.

[0180] The algorithm returns two results: (alignment, distance).

[0181] S36. Calculate the average Euclidean distance between matching point pairs and use the average Euclidean distance as the similarity score between the first and second motion trajectories.

[0182] For each pair of matching points found through dynamic time warping, calculate its Euclidean distance in the original space. This distance directly reflects the actual degree of deviation between the two corresponding points in three-dimensional space and is an important indicator for measuring local similarity.

[0183] The Euclidean distances of all matching point pairs are statistically averaged to obtain the overall average distance. During the calculation process, the system considers the weighting factors of different matching point pairs to ensure that the calculation results accurately reflect the overall similarity between the two trajectories.

[0184] The calculated average Euclidean distance is used as the similarity score ε between the two trajectories. This score has a clear physical meaning: the smaller the score, the more similar the two trajectories are in spatial shape and motion pattern; the larger the score, the greater the difference between the two trajectories.

[0185] This embodiment calculates the overall similarity score of two trajectories A and B after alignment using the DTW algorithm. The calculation steps are as follows:

[0186] Obtaining the alignment path: First, obtain the alignment path from the results of the DTW algorithm. This path is a sequence where each element is a coordinate pair (i, j), indicating that the i-th point in trajectory A is aligned (matched) with the j-th point in trajectory B.

[0187] Initialization: Initialize the total distance accumulator `score` to 0. Also, obtain the number `c` of all matching pairs in the alignment path.

[0188] Cumulative distance: Iterate through each matching pair in the alignment path. For each matching pair (i,j):

[0189] Calculate the Euclidean distance (or other predefined distance function dist) between the i-th point in trajectory A and the j-th point in trajectory B.

[0190] This distance value is added to the total distance score.

[0191] Calculate the average: After traversing all matching pairs, divide the accumulated total distance score by the total number of matching pairs c to obtain an average distance.

[0192] Output: The final score is the similarity score between the two trajectories. The smaller the score, the more similar the shapes of the two trajectories; the larger the score, the lower the similarity.

[0193] In step S4, the similarity score matrix is ​​input into the Hungarian algorithm for optimal matching to obtain the identity binding result between the infrared emitting point and the prop body.

[0194] S41. Preprocess the similarity score matrix.

[0195] Let the similarity score matrix be E m×n , where m is the number of first motion trajectories (number of light-emitting points) and n is the number of second motion trajectories (number of IMU data streams).

[0196] When m ≠ n, the matrix can be expanded into a square matrix by adding virtual rows or columns:

[0197] If m < n, add nm rows, and set the virtual row element value to the maximum penalty value ε. max .

[0198] If m > n, add a column mn, and set the element values ​​of the virtual column to the maximum penalty value ε. max .

[0199] Convert the similarity score matrix into a cost matrix C: C(i,j) = ε(i,j);

[0200] Where ε(i,j) represents the similarity score between the i-th infrared emission point and the j-th IMU; i = 1,...,m, j =1,...,n.

[0201] S42. Use the Hungarian algorithm to analyze E. m×n The scores are used for optimal matching (by adding virtual nodes with scores of 0 to complete the m x n matrix to a square matrix) to obtain the correspondence between the emitting points and the IMUs. Since the connection between the IMUs and the DAS is usually stable and continuous, n is usually greater than or equal to m. When the two are not equal, some IMUs will be in a state of no matching results; their data will still be recorded and participate in future matching calculations, specifically:

[0202] Preprocessing extension:

[0203] Input: An m×n distance (or cost) matrix D.

[0204] Objective: To expand a non-square input matrix into a square matrix so that the standard Hungarian algorithm can be applied.

[0205] Operation: Calculate p = max(m, n). Expand the original m×n matrix D into a p×p square matrix. Fill the new rows or columns with 0. For example, if m < n, add (nm) rows of zeros to the bottom of the matrix; if n < m, add (mn) columns of zeros to the right of the matrix. This transforms the problem into finding a perfect match on a square matrix.

[0206] Core solution:

[0207] The standard Hungarian algorithm is run on the expanded p×p matrix. This algorithm reveals an optimal matching scheme step by step through matrix transformations. Its core steps are as follows:

[0208] Detailed steps of the standard Hungarian algorithm

[0209] Step 1: Line Reduction

[0210] Objective: To ensure that each row contains at least one 0.

[0211] Operation: Iterate through each row of the matrix. For each row, find the minimum value in that row. Then subtract this minimum value from all elements in that row.

[0212] Step 2: Column Reduction

[0213] Objective: To ensure that each column contains at least one 0.

[0214] Operation: Iterate through each column of the matrix. For each column, find the minimum value in that column. Then subtract this minimum value from all elements in that column.

[0215] At this point, there are enough zeros in the matrix. The goal is to find p independent zeros located in different rows and columns to form a perfect match.

[0216] Step 3: Cover all zero elements

[0217] Objective: To cover all zeros in a matrix using the fewest possible rows and columns.

[0218] Operation: Use a sub-algorithm (such as the line drawing method) to find the smallest set of rows and columns that can cover all 0s, denoted as cover_rows (the covered rows) and cover_cols (the covered columns).

[0219] Step 4: Optimality Test

[0220] Verification: If the sum of the number of covered rows and columns equals the dimension p of the matrix (i.e., len(cover_rows) + len(cover_cols) == p), it means that p independent zeros have been successfully found, and the algorithm ends.

[0221] Otherwise: If the number of covering lines is less than p, it means that the current distribution of zero elements cannot form a perfect match, and we need to proceed to step 5 to adjust the matrix.

[0222] Step 5: Matrix Adjustment

[0223] Objective: To create new zero elements without changing the optimal solution to the problem, in order to help find more independent zeros.

[0224] Operation: Call the adjust function:

[0225] Find the minimum uncovered value: Find the minimum value min_val among the cells where all uncovered rows and columns intersect.

[0226] Adjustment matrix:

[0227] For all uncovered cells (rows and columns not covered), execute D[i][j] -= min_val.

[0228] For all cells that are double-covered (both rows and columns are covered), execute D[i][j] += min_val.

[0229] (For cells that are only covered by rows or only covered by columns, their values ​​remain unchanged.)

[0230] After adjustment, return to step 3 and continue trying to cover all zero elements.

[0231] Final result extraction.

[0232] When the loop exits (the condition in step 4 is met), an optimal match is extracted based on the position of 0 in the final matrix. The length of this match is min(m, n) because it ignores the matches resulting from virtual rows or columns added during the expansion process.

[0233] This extended Hungarian algorithm cleverly transforms a non-square matrix into a standard assignment problem by padding it with zeros. It then creates zero elements through row and column reduction and systematically transforms the cost matrix using a strategy of cyclic covering and adjustment, ultimately finding an optimal matching scheme that minimizes the total cost. The adjust function is crucial in the iterative process; by selectively adding or subtracting minimum values, it gradually guides the solution space towards the optimal solution while creating new zeros.

[0234] In step S5, for the prop body whose identity has been successfully bound, motion prediction is performed based on the first motion trajectory to achieve continuous pose tracking of the prop body, specifically as follows:

[0235] S51. For each successfully bound item, predict the spatial position of the current item in the next moment based on the corresponding first motion trajectory;

[0236] DAS extracts the spatial coordinates of the item's main body over a recent period (e.g., the last three consecutive timestamps). Based on this historical position data, it calculates the item's instantaneous velocity. The velocity is calculated by dividing the position difference between adjacent timestamps by the time interval. Next, DAS further calculates the instantaneous acceleration, which is obtained by dividing the velocity difference between adjacent timestamps by the time interval.

[0237] DAS uses a uniformly accelerated motion model for prediction; this embodiment uses the three most recent data points. , , A simple prediction of the next moment can be expressed as:

[0238]

[0239] in

[0240]

[0241]

[0242] S52. Receive the next moment image data of all infrared emitting points on the prop body acquired by the multi-view camera module, and search for infrared emitting points that match the current prop body within the neighborhood of the predicted spatial location;

[0243] After the DAS receives the image data of the next moment acquired by the multi-view camera module, it performs the search according to the following process:

[0244] A spherical search area is established centered on the predicted location. The radius of this search area is dynamically adjusted based on the movement of the item, primarily considering factors such as the item's current speed and the system's prediction error. The search radius calculation includes a safety margin to ensure that the target is not lost due to prediction errors.

[0245] The search is performed within the search area according to distance priority. First, a fine-grained search is conducted in the core region immediately adjacent to the predicted location; this region is small but offers high accuracy. If no matching point is found within the core region, the search area is gradually expanded to an extended region. During the search, information on all luminous points detected within this region is recorded.

[0246] S53. Determine whether a matching infrared emitting point is found within the domain. If so, update the coordinates of the found infrared emitting point to the latest pose of the current prop body and maintain the identity binding state to achieve continuous pose tracking.

[0247] First, eliminate glowing points that clearly do not meet the matching conditions, including points that are obviously inconsistent with the historical movement trajectory of the prop, and points that are too different from the movement characteristics of the prop.

[0248] For the remaining candidate luminous points, their motion characteristics will be verified to ensure consistency with the target prop's historical motion patterns. This includes checking characteristic parameters such as the continuity of motion direction and the consistency of speed changes.

[0249] When multiple candidate points exist within the search area, the most likely point will be selected. The selection criteria include proximity to the predicted location and consistency with historical movement trajectories, among other indicators. If multiple candidate points with equal probability exist, the system will be temporarily unable to confirm the point and will initiate a special processing procedure.

[0250] When a matching luminous point is successfully found within the neighborhood, the 3D spatial coordinates of that luminous point are updated to the latest pose of the current item. Simultaneously, relevant data from this match is recorded to optimize subsequent prediction models. The identity binding status remains valid, and the consecutive success tracking counter is updated.

[0251] DAS dynamically optimizes motion prediction parameters based on actual tracking performance. When the prop's motion is stable and tracking is consistently successful, the search range is gradually tightened to improve tracking efficiency. When the prop's motion is complex or there is significant environmental interference, the system appropriately relaxes the search conditions to ensure tracking robustness.

[0252] If no matching luminous point is found within the neighborhood, the identity binding will not be immediately lifted. Instead, a short-term predictive tracking mode will be activated, continuing to use the motion model for prediction over the next few time frames and expanding the search range. Only if a match cannot be found again within the set number of consecutive frames will the item be considered lost in tracking, triggering the corresponding recovery mechanism.

[0253] In this embodiment, if no matching infrared emitting point is found within the range, it indicates that the infrared emitting point tracking has failed or is lost. At this point, it is not immediately determined as tracking failure, but rather a multi-frame verification mechanism is initiated. The search and verification will be repeated over the next 3-5 time frames (approximately 50-100 milliseconds). Only when no matching emitting point is found in multiple consecutive time frames is the tracking failure status officially confirmed, and the corresponding recovery mechanism is triggered, such as re-executing steps S1-S5. This delayed determination mechanism effectively avoids misjudgments caused by temporary occlusion or detection noise.

[0254] In this embodiment, new prop registration is also included: when an unmatched IMU data stream is detected, and a new, untracked luminous point is simultaneously identified in the scene, the new prop registration process is initiated. The new luminous point is first tracked for a short period to obtain its visual trajectory, and then the aforementioned matching process is performed between it and the unmatched IMU trajectory. Upon successful matching, the new prop is included in the system for continuous tracking.

[0255] Example 3

[0256] The difference between the virtual-real interaction multi-prop matching and tracking method provided in this embodiment and Embodiment 2 is that:

[0257] The identity binding in this embodiment can adopt a simplified scheme. For example, after establishing communication with all props, DAS can sequentially send instructions to the microcontroller of each prop to control its infrared LED to flash at a specific frequency. By detecting the flashing frequency of the light-emitting points in the image, visual features can be directly associated with the prop ID without the need for complex trajectory matching algorithms. This scheme is suitable for cost-sensitive scenarios with a small number of props.

[0258] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A multi-sensor-based method for matching and tracking multiple props in virtual-real interaction, characterized in that, include: Step S1. Receive image data of multiple infrared emitting points on the prop body acquired by the multi-view camera module, and calculate the coordinates of each infrared emitting point in three-dimensional space to obtain the first set of motion trajectories; Step S2. Receive the IMU attitude data sent by the inertial measurement unit on each prop body, and estimate the IMU trajectory of the infrared emitting point on each prop body through pre-integration and coordinate transformation to obtain the second motion trajectory set; Step S3. Align and normalize the first set of motion trajectories and the second set of motion trajectories, and calculate the similarity score of each first motion trajectory and the second motion trajectory using the dynamic time warping algorithm to obtain the similarity score matrix; Step S4. Input the similarity score matrix into the Hungarian algorithm for optimal matching to obtain the identity binding results between the infrared emitting points and the prop body; Step S5. For the prop body that has been successfully bound to the identity, perform motion prediction based on the first motion trajectory to achieve continuous pose tracking of the prop body.

2. The method for matching and tracking multiple props in virtual-real interaction based on multiple sensors according to claim 1, characterized in that, Step S1 specifically involves: S11. Receive image data of multiple infrared light-emitting points on the prop body acquired by each camera in the multi-camera module; S12. Form a camera pair from any two cameras in the multi-camera module, and extract the feature pixel coordinates of the infrared emitting point images obtained by the two cameras as well as the intrinsic and extrinsic parameter matrices of the cameras, and construct a spatial ray from the optical center of the two cameras pointing to the infrared emitting point. S13. Calculate the common perpendicular of the two spatial rays and determine whether the length of the calculated common perpendicular is less than or equal to the preset threshold. If so, determine that the two spatial rays intersect and set the midpoint of the common perpendicular as the initial estimated position of the current infrared emitting point in the three-dimensional space below the camera. S14. Traverse all camera pairs and repeat steps S12-S13 to obtain multiple initial estimated positions in three-dimensional space for the current infrared emitting point; S15. Perform cluster analysis on multiple initial estimated positions in three-dimensional space, and use nonlinear optimization to solve for the coordinates of the three-dimensional space point with the smallest reprojection error for the spatial rays belonging to the same cluster, which is then used as the first motion trajectory of the current infrared emitting point. S16. Traverse the image data of all infrared emitting points and repeat steps S12-S15 to obtain the first set of motion trajectories of all infrared emitting points.

3. The method for matching and tracking multiple props in virtual-real interaction based on multiple sensors according to claim 2, characterized in that, Step S2 specifically involves: S21. Receive IMU attitude data sent by the inertial measurement units on all props; S22. Pre-integrate the IMU attitude data of the current prop body to obtain the relative pose change of the current prop body; S23. Obtain the fixed positional relationship between the infrared emitting point and the inertial measurement unit in the current prop body coordinate system, and perform coordinate transformation on the fixed positional relationship and the relative pose change of the current prop body to estimate the motion trajectory of the infrared emitting point of the current prop body and obtain the second motion trajectory. S24. Iterate through the IMU attitude data sent by the inertial measurement units on all prop bodies, and repeat steps S22-S13 to obtain the second motion trajectory set of all infrared light-emitting points on the prop bodies.

4. The method for matching and tracking multiple props in virtual-real interaction based on multiple sensors according to claim 3, characterized in that, The specific steps in step S3 for aligning the first set of motion trajectories and the second set of motion trajectories are as follows: Calculate the covariance matrix of the first set of motion trajectories and the second set of motion trajectories respectively; The calculated covariance matrix is ​​decomposed into eigenvalues ​​to obtain the principal directions corresponding to the first set of motion trajectories and the second set of motion trajectories. Based on the principal directions corresponding to the first and second sets of motion trajectories, the rotation matrix is ​​calculated using the Rodrigues formula, and one of the motion trajectories is rotated around itself according to the rotation matrix until the principal direction of the currently rotated motion trajectory is aligned with the principal direction of the other motion trajectory.

5. The method for matching and tracking multiple props in virtual-real interaction based on multiple sensors according to claim 4, characterized in that, In step S3, the similarity score of each first motion trajectory and second motion trajectory is calculated using the dynamic time warping algorithm as follows: The dynamic time warping algorithm is used to perform point matching on the first and second motion trajectories, and to establish the correspondence between multiple points on the first and second motion trajectories to obtain matching point pairs. Calculate the average Euclidean distance between matching point pairs and use the average Euclidean distance as the similarity score between the first and second motion trajectories.

6. The method for matching and tracking multiple props in virtual-real interaction based on multiple sensors according to claim 1, characterized in that, Step S5 specifically involves: S51. For each successfully bound item, predict the spatial position of the current item in the next moment based on the corresponding first motion trajectory; S52. Receive the next moment image data of all infrared emitting points on the prop body acquired by the multi-view camera module, and search for infrared emitting points that match the current prop body within the neighborhood of the predicted spatial location; S53. Determine whether a matching infrared emitting point is found within the domain. If so, update the coordinates of the found infrared emitting point to the latest pose of the current prop body and maintain the identity binding state to achieve continuous pose tracking.

7. In step S53 of claim 6, if a matching infrared emitting point is found within the field, and if not, it indicates that the infrared emitting point tracking has failed. The historical first motion trajectory of the current infrared emitting point in the last preset time period before the tracking failure is obtained, and steps S1-S5 are re-executed.

8. A virtual-real interaction multi-prop system based on multiple sensors, characterized in that, include: At least one prop body, each prop body includes an infrared light-emitting point and an inertial measurement unit, used to emit light through the infrared light-emitting point and acquire the IMU attitude data of the current prop body through the inertial measurement unit, and send the acquired IMU attitude data to the data acquisition and processing module. The multi-view camera module is used to acquire image data of infrared emitting points on multiple props and send the acquired image data of infrared emitting points to the data acquisition and processing module. The data acquisition and processing module is communicatively connected to the prop body and the multi-view camera module, respectively, and is used to receive the IMU attitude data of the current prop body and the image data of the infrared light-emitting points on multiple prop bodies, and execute the virtual-real interaction multi-prop matching and tracking method based on multiple sensors as described in any one of claims 1-7.

9. A virtual-real interaction multi-prop system based on multiple sensors according to claim 8, characterized in that, Each camera in the multi-camera module is equipped with an infrared filter.

10. A multi-sensor-based virtual-real interaction multi-prop system according to claim 9, characterized in that, Each prop body sends IMU attitude data to the data acquisition and processing module through the corresponding UDP port number.

Images