A dynamic alignment method and system based on double reference under weak timing synchronization condition

Abstract

The application discloses a kind of weak time sequence synchronous conditions based on double reference dynamic alignment method and motion capture system.The method comprises: under weak time sequence synchronous conditions, synchronously start inertial measurement unit and reference motion capture system, independently collect motion data;Original inertial measurement unit data is executed combination preprocessing;Fusion synthetic reference signal is constructed dual-stage time alignment mechanism, generates synthetic reference signal and with inertial measurement unit measured signal constitutes double reference, through signal energy evaluation in sliding window, dynamically select left and right hand optimal signal as alignment reference, determine initial time offset using cross-correlation analysis and correct dynamic drift;Based on the T-Pose reference frame of automatic positioning, the integration calibration and gravity compensation of sensor coordinate system to skeletal coordinate system are completed.The application does not need to rely on special time synchronization server, realizes weak time sequence synchronous conditions inertial motion capture data and human kinematics signal high-precision, adaptive dynamic alignment.

Description

Technical Field

[0001] This invention relates to the fields of data processing and human motion analysis technology, and more specifically, to a dynamic alignment method and motion capture system based on dual references under weak temporal synchronization conditions. Background Technology

[0002] Data-driven human motion analysis and synthesis have broad application prospects in fields such as medical rehabilitation, augmented reality, virtual reality, and film and television production. High-quality simulation effects rely on a large amount of accurate and reliable motion data. With the rise of deep learning technology, researchers widely use neural networks to mine correlation patterns in motion data, which places stringent requirements on the scale of the data, consistency of sources, and time synchronization accuracy. To obtain richer motion information, multiple types of sensors (such as inertial measurement units, optical motion capture devices, pressure sensors, etc.) are typically used for multimodal data acquisition. However, different sensors have different data formats, sampling frequencies, and accuracy characteristics, with the most critical challenge being time synchronization between multiple devices.

[0003] The current mainstream solution involves building a dedicated time synchronization server (e.g., configuring a high-performance computer as an NTP or PTP server) and connecting all acquisition devices to this server to obtain a unified clock reference. This method has extremely high requirements for software configuration, network environment, and site conditions, making it difficult to deploy flexibly in complex scenarios such as outdoor, mobile, or single-person wearable devices, severely limiting the versatility of motion capture systems. Existing technologies also include some time alignment methods for multi-sensor fusion. For example, patent CN121856260A discloses an IMU sensor spatiotemporal fusion industrial defect detection system based on the SORT algorithm, which achieves synchronization through hardware timestamps and alignment instructions, mainly applicable to industrial production lines such as conveyor belts; patent CN118832583A discloses a distributed multi-sensor time synchronization control system for space robots, which relies on synchronization board hardware and GNSS signals for hard-triggered synchronization; and patent CN120800346A discloses an attitude calculation method for a vehicle-mounted mobile communication system, which uses time synchronization error as a state variable for Kalman filtering estimation. However, the above methods either rely on dedicated hardware or external signals, or focus on specific problems in different technical fields (such as industrial inspection, robotics, and vehicle navigation). None of them can provide a robust and accurate dynamic alignment solution that does not require additional hardware for the weak temporal synchronization that is common in human motion capture (i.e., the device can only collect data independently under its own clock and cannot or has difficulty obtaining a unified clock source).

[0004] Furthermore, existing methods mostly rely on a single reference signal for signal alignment (such as relying solely on IMU hand acceleration or visual contour features). When human motion patterns are complex (such as high-speed rotation or rapid start-stop) or the single signal is interfered with (such as the hand being occluded), the alignment accuracy drops sharply. Therefore, there is an urgent need for an inertial data and kinematic signal alignment method that can adapt to complex motions, has strong anti-interference capabilities, and does not require a dedicated synchronization server. Summary of the Invention

[0005] The present invention aims to solve at least one of the above-mentioned problems in the prior art, and provides a method and system for dynamically aligning inertial motion capture data and human kinematic signals under weak time synchronization conditions, without relying on a dedicated time synchronization server, and with strong anti-interference ability, which can adapt to high and low speed motion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a dynamic alignment method based on dual reference under weak timing synchronization conditions, comprising the following steps: S1. Under weak timing synchronization conditions, the inertial measurement unit sensor and the reference motion capture system are started synchronously to independently collect motion data. All data are timestamped with their own hardware clock and stored in a standardized format, with the standard T-Pose posture as the starting reference for the data.

[0007] S2. Perform combined preprocessing on the acquired raw inertial measurement unit data. The combined preprocessing includes at least duplicate frame removal, interpolation repair, and smoothing filtering.

[0008] S3. Construct a two-stage time alignment mechanism for the fused and synthesized reference signal, and perform dynamic alignment between the preprocessed inertial measurement unit measured signal and the reference motion capture system signal, specifically including: S31. Based on the posture parameters in the parametric human body model and reference motion capture data, standard joint positions are calculated through forward kinematics, and differential operations are performed on the position signals of specified joints to generate synthetic reference signals. S32. Within the current sliding window, evaluate the energy of the measured signals from similar inertial measurement units on the left and right sides in real time, and adaptively select the signal with higher energy as the dynamic alignment reference. S33. Based on the synthetic reference signal and the dynamic alignment benchmark, the initial time offset is determined by cross-correlation analysis, and the time drift is continuously tracked and corrected within the sliding window.

[0009] S4. Using the standard T-Pose as a reference, automatically detect the standard T-Pose frame in the collected data, calculate the transformation matrix from the sensor coordinate system to the human skeleton coordinate system based on the raw data of the inertial measurement unit of the frame, and perform coordinate system transformation and gravity acceleration removal on all inertial measurement unit data accordingly to obtain calibrated absolute acceleration and attitude data.

[0010] S5. Visualize and compare the human posture driven by the calibrated and synchronized inertial measurement unit data with the standard posture of the reference motion capture system, and support users to manually fine-tune the alignment parameters.

[0011] Furthermore, in step S2, the duplicate frame removal is based on the 2-norm determination rule of the acceleration data:

[0012] in, To determine the threshold, during interpolation repair, linear interpolation is used to fill in missing acceleration data, and spherical linear interpolation is used to fill in missing rotation data. The formula for spherical linear interpolation is:

[0013] in, These are time-based interpolation weights.

[0014] Further, in step S31, the synthesized reference signal is generated by performing a second-order difference operation on the three-dimensional position sequence of the specified joint, and its calculation formula is as follows:

[0015] Where P(t) is the joint position at time t, and Δt is the sampling time interval.

[0016] Furthermore, in step S32, the formula for calculating the signal energy is:

[0017] Where W is the current sliding window, and a(t) is the measured acceleration signal of the inertial measurement unit within the window; the side with the higher energy value is selected as the dynamic alignment reference.

[0018] Further, in step S33, the initial time offset is determined by maximizing the cross-correlation function between the synthesized reference signal and the dynamically aligned reference signal, as shown in the formula:

[0019] in, To synthesize a reference signal, The reference signal is dynamically aligned, and τ is the candidate time offset.

[0020] Furthermore, in step S4, the automatic positioning of the T-Pose standard frame is determined by minimizing the sum of the rotation angles of multiple joints of the human body and the standard T-Pose angle, as shown in the formula:

[0021] Where N is the number of joints involved in the calculation. Let be the rotation matrix of the j-th joint at time t. Let be the rotation matrix of the j-th joint in the standard T-Pose.

[0022] Furthermore, in step S4, the formula for calculating the removal of gravitational acceleration is:

[0023] in, The acceleration data is after coordinate system transformation. This is the purely linear acceleration after removing gravity.

[0024] Secondly, the present invention provides a dynamic alignment system based on dual references under weak timing synchronization conditions, comprising: The data acquisition module is configured to synchronously start the inertial measurement unit sensor and the reference motion capture system under weak temporal synchronization conditions, independently acquire motion data, and use the standard T-Pose posture as the data starting reference. The preprocessing module is configured to perform a combination of preprocessing on the acquired raw inertial measurement unit data, including duplicate frame removal, interpolation repair, and smoothing filtering. A two-stage time alignment module is configured to construct a two-stage time alignment mechanism for the fused synthetic reference signal, performing dynamic alignment between the preprocessed inertial measurement unit measured signal and the reference motion capture system signal; the two-stage time alignment module includes: The synthetic reference signal construction unit is configured to generate a synthetic reference signal based on the attitude parameters in the parameterized human model and reference motion capture data. The dynamic signal selection unit is configured to evaluate the energy of the measured signals from similar inertial measurement units on the left and right sides in real time within the current sliding window, and adaptively select the signal with higher energy as the dynamic alignment reference. The drift correction unit is configured to determine the initial time offset based on the synthetic reference signal and the dynamic alignment reference by using cross-correlation analysis, and to continuously track and correct the time drift within a sliding window. The integrated calibration module is configured to automatically detect the T-Pose standard frame in the acquired data based on the standard T-Pose, calculate the transformation matrix from the sensor coordinate system to the human skeleton coordinate system, and perform coordinate system transformation and gravitational acceleration removal on all inertial measurement unit data accordingly. The visualization output module is configured to visually compare and output the human posture driven by the calibrated and synchronized inertial measurement unit data with the standard posture of the reference motion capture system, and supports users to manually fine-tune the alignment parameters.

[0025] In another aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a dynamic alignment method based on dual reference under any of the weak timing synchronization conditions described above.

[0026] On the other hand, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a dynamic alignment method based on dual reference under weak timing synchronization conditions as described above.

[0027] The above technical solution has the following technical effects: This invention constructs a dual-reference alignment mechanism that integrates a synthetic reference signal and a dynamic signal selection. Under weak temporal synchronization conditions, it generates a theoretically synthesized signal using a parameterized human body model and adaptively switches between left and right hand measured references based on signal energy evaluation within a sliding window. This eliminates the reliance on a dedicated time synchronization server and solves the problem of insufficient accuracy of a single reference signal in complex movements. Through a standardized acquisition system and combined preprocessing workflow, data quality is guaranteed from the source. Based on an automatic positioning T-Pose integrated calibration system, precise mapping of sensor data to the skeletal coordinate system is achieved. This invention significantly improves the alignment accuracy, robustness, and system versatility of inertial motion capture data and human kinematic signals under weak temporal synchronization conditions. Attached Figure Description

[0028] Figure 1 A flowchart illustrating a dynamic alignment method based on dual references under weak timing synchronization conditions, provided in an embodiment of the present invention; Figure 2 This is a flowchart of step S3 in the above embodiment; Figure 3 A schematic diagram of a dynamic alignment system based on dual references under weak timing synchronization conditions, provided for another embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0029] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0030] In the field of human motion capture and analysis, multimodal data fusion technology faces a long-standing core contradiction: on the one hand, inertial measurement units (IMUs) have advantages such as ease of wear, no site restrictions, and high sampling frequency, making them the mainstream choice for motion capture in outdoor and mobile scenarios; on the other hand, IMU data lacks an absolute spatial reference and must be calibrated and verified using ground truth provided by high-precision reference motion capture systems (such as optical motion capture systems). However, these two types of systems are inherently separated in terms of clock sources—optical systems are usually synchronized by a high-performance workstation, while wireless IMU sensors each rely on independent crystal oscillators. Forcing both to connect to the same time synchronization server requires deploying dedicated network equipment and configuring a Precision Time Protocol (PTP), which is not only costly but also severely limits the flexibility of data acquisition, making it difficult to implement in real-world scenarios such as outdoor sports fields, rehabilitation clinics, and home environments. Therefore, how to achieve high-precision dynamic alignment between IMU measured signals and reference kinematic signals under conditions of weak timing synchronization (i.e., relying solely on the device's own clock without a unified time reference) has become a key bottleneck restricting the widespread application of motion capture technology.

[0031] Furthermore, existing alignment methods generally rely on a single reference signal source. For example, using only IMU hand acceleration as a reference can lead to a sharp drop in signal quality when the user performs complex movements such as rapid arm swings or turns, as the hand movements are large but prone to occlusion or electromagnetic interference. Similarly, using only visual features or model-synthesized signals as a reference lacks the ability to adapt to real sensor noise. When the motion pattern changes abruptly or the signal briefly fails, a single reference is highly susceptible to incorrect time offset estimations, potentially causing complete alignment failure. Faced with complex and varied human motion, a robust method that can integrate multi-source reference information, dynamically evaluate signal quality, and adaptively switch alignment references is urgently needed. Based on these technical challenges, this invention proposes a dynamic alignment method and motion capture system based on dual references under weak temporal synchronization conditions.

[0032] The following is combined Figure 1 and Figure 2 The flowchart below provides a detailed explanation of the method of the present invention.

[0033] A first aspect of the present invention provides a dynamic alignment method based on dual references under weak timing synchronization conditions, comprising the following steps: S1. Under weak timing synchronization conditions, the inertial measurement unit sensor and the reference motion capture system are started synchronously to independently collect motion data. All data are timestamped with their own hardware clock and stored in a standardized format, with the standard T-Pose posture as the starting reference for the data.

[0034] S2. Perform combined preprocessing on the acquired raw inertial measurement unit data. The combined preprocessing includes at least duplicate frame removal, interpolation repair, and smoothing filtering.

[0035] S3. Construct a two-stage time alignment mechanism for the fused and synthesized reference signal, and perform dynamic alignment between the preprocessed inertial measurement unit measured signal and the reference motion capture system signal, specifically including: S31. Based on the posture parameters in the parametric human body model and reference motion capture data, standard joint positions are calculated through forward kinematics, and differential operations are performed on the position signals of specified joints to generate synthetic reference signals. S32. Within the current sliding window, evaluate the energy of the measured signals from similar inertial measurement units on the left and right sides in real time, and adaptively select the signal with higher energy as the dynamic alignment reference. S33. Based on the synthetic reference signal and the dynamic alignment benchmark, the initial time offset is determined by cross-correlation analysis, and the time drift is continuously tracked and corrected within the sliding window.

[0036] S4. Using the standard T-Pose as a reference, automatically detect the standard T-Pose frame in the collected data, calculate the transformation matrix from the sensor coordinate system to the human skeleton coordinate system based on the raw data of the inertial measurement unit of the frame, and perform coordinate system transformation and gravity acceleration removal on all inertial measurement unit data accordingly to obtain calibrated absolute acceleration and attitude data.

[0037] S5. Visualize and compare the human posture driven by the calibrated and synchronized inertial measurement unit data with the standard posture of the reference motion capture system, and support users to manually fine-tune the alignment parameters.

[0038] In a specific embodiment of the invention, the user selects the corresponding device parameters according to their wearing configuration (tight-fitting or loose-fitting). After the sensor is activated, it automatically completes a self-test, verifying the relative position of the sensor and the human joints through signal strength detection. Data storage adopts a standardized txt format, recording the timestamps of independent data collection by the device and the acceleration and rotation data collected by the sensor in real time. All data collection processes use the T-Pose as a unified starting posture. The software automatically prompts to start data collection after detecting that the user has maintained a stable T-Pose for 3 seconds.

[0039] In one specific implementation, when performing a dynamic alignment method based on dual references under weak temporal synchronization conditions, 17 wireless IMU sensors (worn on the head, torso, limbs, and hands and feet) and an OptiTrack optical motion capture system are first deployed in an outdoor sports field. There are no hardware synchronization cables or network time servers connecting the two; they operate independently: the IMU sensors collect triaxial acceleration and rotation angle data at a sampling rate of 200Hz and add timestamps using their own crystal oscillator clock; the OptiTrack system collects three-dimensional position data of human body markers at a sampling rate of 120Hz and adds timestamps using its workstation clock. The system software uses the standard T-Pose as the starting posture: the user stands with arms outstretched, and the software automatically starts recording after detecting that the posture is stable for 3 seconds. After acquisition, all data is uniformly stored in txt format, including a separate timestamp column and a sensor data column for each device. Subsequently, combined preprocessing such as duplicate frame removal, linear interpolation and spherical linear interpolation, and sliding window smoothing are performed. During the time alignment phase, synthetic acceleration signals of hand joints are generated from OptiTrack data based on the SMPL model. Simultaneously, the energy values ​​of the left and right hand IMU acceleration signals are calculated within a sliding window (window length 2 seconds, step size 0.5 seconds). The hand with higher energy is selected as the alignment reference. A coarse time offset is determined using a cross-correlation function, and minor drifts are continuously corrected as the sliding window progresses. Finally, the frame with the smallest deviation from the standard T-Pose among the first 1000 frames is automatically searched as the reference frame. The transformation matrix from the IMU coordinate system to the skeletal coordinate system is calculated, completing the coordinate system transformation and gravity acceleration removal. A visual comparison of the synchronized IMU-driven pose and the OptiTrack standard pose is output, and users can fine-tune the alignment parameters using the interface slider.

[0040] In the above technical solution, steps S1 to S5 constitute a complete data processing pipeline in the order of "independent acquisition—preprocessing—dual-stage alignment—integrated calibration—visualization output". The inherent logic of this sequence is as follows: first, standardized acquisition ensures the basic usability of the raw data (S1); then, preprocessing eliminates the interference of sensor noise and data gaps on the subsequent alignment accuracy (S2); then, in S3, a dual reference mechanism of "synthetic reference signal + dynamic measured benchmark" is used to solve the fundamental problem of the unreliability of a single benchmark under weak temporal synchronization; in S4, coordinate system transformation is completed to give the data physical meaning; and finally, in S5, an interactive verification method is provided. There is a clear causal dependency between each step: without the preprocessing in S2, the cross-correlation analysis in S3 will produce erroneous offsets due to noise and duplicate frames; without the time alignment in S3, the T-Pose calibration in S4 will fail to correctly solve the transformation matrix due to time misalignment. This interconnected set of steps allows the time error between IMU data and reference data to be compressed to the sub-millisecond level without any dedicated synchronization hardware. At the same time, the dynamic left-hand switching mechanism automatically maintains the reliability of the alignment reference when the movements change drastically, thus solving the problems of low synchronization accuracy and poor robustness in weak timing synchronization scenarios in existing technologies.

[0041] In some embodiments, in step S2, the duplicate frame removal is based on the 2-norm determination rule of the acceleration data:

[0042] in, To determine the threshold, during interpolation repair, linear interpolation is used to fill in missing acceleration data, and spherical linear interpolation is used to fill in missing rotation data. The formula for spherical linear interpolation is:

[0043] in, These are time-based interpolation weights.

[0044] For example, in some specific preferred implementations, the detection of repeated frames of acceleration data uses a 2-norm decision threshold. =0.01m / s². When the Euclidean distance between the acceleration vectors of two consecutive frames is less than this threshold, the subsequent frame is determined to be a duplicate frame. If a continuous duplicate interval is detected... , If ], then perform linear interpolation on this interval: Let The effective frame time before the interval. For the effective frame time after the interval, the interpolation weights Then the interpolation point acceleration at point For rotated data, such as a continuous motion rotating from 0° to 90° around the Y-axis, if several frames are missing, spherical linear interpolation is used: Let R1 be the initial rotation matrix, R2 be the final rotation matrix, and the interpolation weights w∈[0,1], then the interpolation result R(w) = This method ensures that the interpolation path is the shortest geodesic on the rotating manifold, and that the rotation matrix of the intermediate frame always maintains orthogonal normalization properties.

[0045] The essence of the above combination of technical features lies in the fact that: acceleration data belongs to vectors in linear space, and its missing values ​​can be recovered by linear interpolation with minimal computational cost; while rotation data belongs to elements on a special orthogonal group SO(3), a non-Euclidean manifold. Directly performing linear interpolation on the rotation matrix will destroy its orthogonality and normalization, resulting in distortions such as "collapse" or "distortion" in the interpolated attitude. Therefore, this application uses spherical linear interpolation (Slerp) to fill in the missing rotation data. Slerp directly interpolates along the geodesics on the SO(3) manifold, ensuring that the interpolation path is the shortest path in the rotation space, thereby maintaining the rotational continuity of the interpolation sequence. This interpolation method is equivalent to performing linear interpolation of angle increments in the Lie algebra SO(3) (i.e., the tangent space) of SO(3), and then mapping back to the manifold through exponential mapping. Therefore, it can ensure the orthogonality and determinant constraints of the rotation matrix in each frame of the interpolation sequence, and maintain the continuous physical rationality of the attitude. Meanwhile, the 2-norm determination rule directly corresponds to the magnitude change of the acceleration vector in three-dimensional space, and can more accurately reflect whether the motion has truly stopped than component-by-component threshold determination. The combination of the three (2-norm deduplication, linear interpolation to compensate for acceleration, and Slerp to compensate for rotation) constitutes a differentiated preprocessing strategy for the characteristics of IMU multimodal data, which ensures the quality of the data input to the subsequent alignment algorithm and avoids false time offset estimation caused by missing data or rotation interpolation distortion.

[0046] In some embodiments, in step S31, the synthesized reference signal is generated by performing a second-order difference operation on the three-dimensional position sequence of the specified joint, and its calculation formula is as follows:

[0047] Where P(t) is the joint position at time t, and Δt is the sampling time interval.

[0048] For example, the joints are designated as the left wrist joint and the right wrist joint. Referring to the three-dimensional position sequence P(t) (in meters) of the wrist joints provided by the motion capture system, the sampling interval Δt = 1 / 120 seconds (corresponding to a 120Hz optical system). The formula for calculating the synthesized acceleration signal is as follows: The second-order central difference is equivalent to performing two discrete differentials on the position sequence. The resulting composite acceleration of the wrist joint is physically homogeneous with the hand acceleration measured by the IMU sensor (both have units of meters per second²), and the two should show a high correlation in time waveforms—when a person waves their hand, the peak position of the composite acceleration roughly corresponds to that of the measured acceleration. The only difference is that the measured value of the IMU includes sensor noise and gravity components, while the composite value comes from a noise-free ideal model.

[0049] In the above embodiments, the joint position sequence obtained from the reference data through forward kinematics calculation is smooth and without delay. Its second-order difference operation is actually a high-pass filter, which can highlight the dynamic change components (i.e., acceleration) in motion and filter out position offset information in static or uniform motion. Using the synthetic acceleration signal as the first reference benchmark, its physical meaning completely corresponds to the IMU measured acceleration. This allows cross-correlation analysis to be performed in a homogeneous signal space, avoiding mismatches caused by differences in dimensions and phases between different physical quantities (such as position and acceleration). At the same time, the synthetic signal is not affected by practical problems such as sensor obstruction and electromagnetic interference, providing a "theoretical gold standard" for alignment. This feature is directly related to step S31 in the aforementioned embodiments and is the specific implementation of the "synthetic reference signal" in constructing a dual reference system.

[0050] In some embodiments, the formula for calculating the signal energy in step S32 is:

[0051] Where W is the current sliding window, and a(t) is the measured acceleration signal of the inertial measurement unit within the window; the side with the higher energy value is selected as the dynamic alignment reference.

[0052] For example, the length of the sliding window W is set to 2 seconds (i.e., 200 IMU sampling points, assuming an IMU frequency of 100Hz). Within each window, the energy of the left-hand and right-hand acceleration signals is calculated separately: , .like If the left-hand signal is selected as the dynamic alignment reference Simu for the current window, then the right-hand signal is selected; otherwise, the right-hand signal is selected. The window slides forward in 0.5-second increments, with adjacent windows having a 75% overlap rate to ensure smooth reference switching. For example, when the user performs a left-hand swing, the left-hand acceleration energy is significantly higher than the right-hand, and the system automatically uses the left hand as the reference; when the action switches to a right-hand toss, the right-hand energy exceeds the left-hand energy, and the reference switches accordingly. The switching process achieves a seamless transition through the weighted average of overlapping windows.

[0053] In the above embodiments, signal energy Mathematically equivalent to the time integral of the square of the magnitude of the acceleration vector, physically representing the magnitude of motion intensity within that time period. Under weak temporal synchronization conditions, the signal-to-noise ratio (SNR) is positively correlated with motion intensity—the greater the motion intensity, the higher the acceleration amplitude, and the lower the relative proportion of measurement noise. Therefore, channels with higher energy are more suitable as alignment references. Deploying IMUs on both the left and right sides creates naturally redundant measurement channels. By evaluating and selecting high-energy channels in real time, the system achieves an adaptive mechanism of "selecting the best" in complex motions. This mechanism, together with the synthetic reference signal in the aforementioned embodiments, constitutes a dual reference system: the synthetic signal provides a theoretical reference from the ideal model, while the dynamically selected high-energy measured signal provides a robust reference from the real sensor; the two complement each other. Compared to using only one side or a synthetic signal, this combination can maintain at least one high-confidence reference signal even when the motion changes drastically.

[0054] In some embodiments, in step S33, the initial time offset is determined by maximizing the cross-correlation function between the synthesized reference signal and the dynamically aligned reference signal, as shown in the formula:

[0055] in, To synthesize a reference signal, The reference signal is dynamically aligned, and τ is the candidate time offset.

[0056] For example, let To synthesize a reference signal (length is ), For dynamic alignment reference signal (length is ),and ≈ Calculate the cross-correlation function R( )= ,in The range of values ​​is [- , ], The corresponding maximum expected time offset (e.g., ±2 seconds, i.e.) The step size is equal to 1 / 100 of the sampling interval, with a total of 400 candidate offsets. R( The maximum value corresponding to ) This is the initial time offset Δt. For example, if... The peak occurred at frame 1000. If the peak occurs in frame 1005, then the cross-correlation analysis will output... =+5 frames, corresponding to Δt=5 / 100=0.05 seconds, indicating that the IMU data lags behind the reference data by 50 milliseconds.

[0057] In the above embodiments, cross-correlation analysis is a standard tool in signal processing for detecting the delay between two waveforms. Its theoretical basis is the Wiener-Khinchin theorem—under the assumption of additive white noise, the delay corresponding to the peak value of the cross-correlation is an unbiased estimate of the true delay. This invention applies this classical method between a "synthetic reference signal" and a "dynamically measured signal," rather than between two traditionally measured signals. This combination is chosen because the synthetic signal originates from a noise-free ideal model, providing higher peak discrimination than measured-to-measured cross-correlation, especially under low signal-to-noise ratio conditions. The energy assessment of the measured signals has already been optimized in the preceding steps; therefore, the two input signals used in the cross-correlation analysis are both optimal in their respective categories. This step, occurring before dynamic drift correction, provides a coarse initial alignment value, ensuring that subsequent drift correction within the sliding window searches only within a small range, reducing computational complexity and avoiding local optima traps.

[0058] In some embodiments, in step S4, the automatic positioning of the T-Pose standard frame is determined by minimizing the sum of the rotation angles of multiple joints of the human body and the standard T-Pose angle, as shown in the formula:

[0059] Where N is the number of joints involved in the calculation. Let be the rotation matrix of the j-th joint at time t. Let be the rotation matrix of the j-th joint in the standard T-Pose.

[0060] In this embodiment, for example, N is taken as 24 (corresponding to the 24 joints of the human body defined in the SMPL model). The standard T-Pose joint angle Angle (… Predefined: Arms extended horizontally and perpendicular to the torso, palms down, legs straight and together, feet pointing forward. Within the first 1000 frames of data acquisition (approximately 10 seconds, assuming a 100Hz sampling rate), for each frame t, calculate the sum of the absolute deviations of the rotation angles of all 24 joints from the standard angles. The frame with the smallest sum of deviations is the [missing information - likely a specific frame or unit]. For example, if the sum of the 24 joint angle deviations in frame 235 is 15.3 degrees, in frame 236 it is 14.8 degrees, and in frame 237 it is 16.1 degrees, then frame 236 is selected as the T-Pose reference frame. The system automatically sets this frame as the zero point of the data time and displays the skeleton posture in the 3D visualization window. If the user finds that the automatic positioning is incorrect (for example, the user's actual T-Pose appears in frame 240), they can manually correct it by dragging the timeline with the mouse.

[0061] In the above embodiments, the T-Pose, as a common calibration posture in human motion capture, has clearly defined joint angles. By minimizing the sum of all joint angle deviations, the system can automatically locate the moment closest to a stationary T-Pose from a data set containing arbitrary motion, avoiding the tedious manual marking of the starting frame. The key to this positioning algorithm's computation is that the deviation sum is robust to motion within the first 1000 frames—even if the first few frames are erratic, as long as the user briefly maintains a T-Pose at some point, the algorithm can capture that frame. The first 1000 frames, rather than the entire sequence, are chosen to balance computational efficiency and positioning reliability, as the T-Pose typically appears at the beginning of the acquisition phase. This feature directly corresponds to step S4 in the aforementioned embodiments and is a prerequisite for achieving a fully automated calibration process. Without this automatic positioning function, the user must manually mark the frame during acquisition or search frame by frame afterward, reducing the system's usability.

[0062] In some embodiments, after the orientation is corrected as described above, the core IMU calibration stage begins. The goal of calibration is to calculate the transformation matrix from the sensor coordinate system to the skeletal coordinate system, thereby achieving accurate conversion of the raw data. (The calibrated rotation data is then presented.) With acceleration data The calculation is shown in the following formula.

[0063]

[0064]

[0065] in, The rotation matrix is ​​the original output of the sensor. This is the mapping matrix from the SMPL model space to the IMU space. The offset correction matrix from the device installation position to the human skeleton is used to complete the coordinate system transformation of the rotation data through two matrix multiplications.

[0066] In some embodiments, the formula for calculating gravitational acceleration elimination in step S4 is:

[0067] in, The acceleration data is after coordinate system transformation. This is the purely linear acceleration after removing gravity.

[0068] For example, acceleration after coordinate system transformation Represented as a three-dimensional vector In the northeast-northeast coordinate system, the direction of gravitational acceleration is vertically downwards, that is... However, since the Z-axis of the IMU is usually pointed above the human body (away from the direction of gravity) when it is installed, the actual culling operation is as follows: = - That is, subtracting an upward gravitational vector. For example, when a person is standing still, The theoretical Z-axis value should be +9.80665 m / s² (because the accelerometer measures the supporting force, which is upward), after subtracting gravity. The Z component is close to 0, which is consistent with the physical fact that the human body has no linear acceleration. When a person jumps upwards, The Z component is greater than 9.80665, and after subtracting gravity, a positive upward acceleration is obtained; when the person falls, The Z component is less than 9.80665, and after subtracting gravity, we get a negative (downward) acceleration.

[0069] In the above embodiments, the accelerometer measures "support force" rather than actual linear motion acceleration, and its readings include the contribution of gravitational acceleration. In actual motion, the absolute linear acceleration in the human skeletal coordinate system is the physical quantity describing motion. By subtracting the known gravity vector (in standard T-Pose, the direction of gravity is aligned with the vertical axis of the skeletal coordinate system), the sensor readings can be converted into actual motion acceleration. This step must be placed after coordinate system transformation because the gravity vector rotates with posture changes in the sensor coordinate system, while its direction is constant (vertically downward) in the skeletal coordinate system. The coordinate system transformation in the aforementioned embodiments first rotates the sensor data to the skeletal coordinate system, fixing the gravity component on the Z-axis, so that it can be accurately eliminated by scalar subtraction. This feature, together with other sub-operations in step S4 (T-Pose positioning, transformation matrix calculation), constitutes a complete integrated calibration system, ensuring that the subsequent output acceleration data correctly reflects the actual motion of the human body in a physical sense.

[0070] refer to Figure 3 To facilitate understanding of the second aspect of the invention, a dynamic alignment system based on dual references under weak timing synchronization conditions is provided, comprising: The data acquisition module is configured to synchronously start the inertial measurement unit sensor and the reference motion capture system under weak temporal synchronization conditions, independently acquire motion data, and use the standard T-Pose posture as the data starting reference. The preprocessing module is configured to perform a combination of preprocessing on the acquired raw inertial measurement unit data, including duplicate frame removal, interpolation repair, and smoothing filtering. A two-stage time alignment module is configured to construct a two-stage time alignment mechanism for the fused synthetic reference signal, performing dynamic alignment between the preprocessed inertial measurement unit measured signal and the reference motion capture system signal; the two-stage time alignment module includes: The synthetic reference signal construction unit is configured to generate a synthetic reference signal based on the attitude parameters in the parameterized human model and reference motion capture data. The dynamic signal selection unit is configured to evaluate the energy of the measured signals from similar inertial measurement units on the left and right sides in real time within the current sliding window, and adaptively select the signal with higher energy as the dynamic alignment reference. The drift correction unit is configured to determine the initial time offset based on the synthetic reference signal and the dynamic alignment reference by using cross-correlation analysis, and to continuously track and correct the time drift within a sliding window. The integrated calibration module is configured to automatically detect the T-Pose standard frame in the acquired data based on the standard T-Pose, calculate the transformation matrix from the sensor coordinate system to the human skeleton coordinate system, and perform coordinate system transformation and gravitational acceleration removal on all inertial measurement unit data accordingly. The visualization output module is configured to visually compare and output the human posture driven by the calibrated and synchronized inertial measurement unit data with the standard posture of the reference motion capture system, and supports users to manually fine-tune the alignment parameters.

[0071] For example, the data acquisition module can specifically consist of 17 MTi-630 series IMU sensors and an OptiTrackPrime 41 optical capture system, without any hardware synchronization cables between them. The preprocessing module is deployed on a portable laptop, running repeating frame detection, linear / Slerp interpolation, and Kalman filtering algorithms implemented in Python. The two-stage time alignment module further comprises three sub-units: a synthetic reference signal construction unit that calls the forward kinematics solution function of the SMPL model to generate synthetic hand acceleration; a dynamic signal selection unit that calculates the left and right hand energies in a sliding window (2 seconds window, 0.5-second step) and outputs the signal index of the high-energy side; and a drift correction unit that performs cross-correlation calculation and phase tracking within the sliding window. The integrated calibration module implements the T-Pose automatic search algorithm and the least-squares solution of the coordinate transformation matrix. The visualization output module uses PyQt5 to build a 3D rendering window, displaying a red-green skeleton overlay, and provides a timeline slider and a correlation coefficient display area. Data is transferred between modules via a shared memory queue, forming a real-time processing pipeline.

[0072] In the above embodiments, each stage of the data processing pipeline is encapsulated as an independent module, and the modules interact only through standardized data structures (such as raw data packets, preprocessed data, aligned data, and calibrated data). This decoupled design allows each module to be independently optimized or replaced without affecting the operation of other modules. For example, if a new type of IMU sensor has different noise characteristics, only the filtering parameters in the preprocessing module need to be modified, while the two-stage alignment module and calibration module remain completely unchanged; if the reference system needs to be replaced from optical to electromagnetic, only the input data format of the forward kinematics in the synthetic reference signal construction unit needs to be adjusted. Meanwhile, the three-level sub-unit structure (construction-selection-correction) within the two-stage time alignment module strictly follows the algorithm execution order—first the synthetic reference, then dynamic selection, and finally drift correction, ensuring the timing correctness of the data processing flow. Compared to traditional systems that couple all functions into a black box, this modular system offers substantial differences in development and debugging, functional upgrades, and fault isolation.

[0073] Based on the foregoing embodiments, there are also some embodiments with essentially the same or similar technical concepts, as shown in the following examples: In an alternative implementation, the joints upon which the synthetic reference signal is based are not limited to hand joints, but rather include positional data from other joints whose motion characteristics are more pronounced in human movement. For example, in running motion analysis, the three-dimensional position sequence of the knee or ankle joint can be used as the data source for the synthetic signal; in trunk torsion motion analysis, positional data from the hip or shoulder joint can be used. The specific implementation method remains unchanged: based on the SMPL model and the posture parameters in the reference motion capture data, the three-dimensional position sequence of the specified joint is calculated through forward kinematics, and then a second-order difference operation is performed on the position sequence to generate a synthetic acceleration signal. This synthetic signal and the measured acceleration signal from the IMU worn at the corresponding joint form a dual reference system, used for subsequent cross-correlation alignment and dynamic drift correction. Taking the knee joint as an example, when the human body squats or jumps, the knee joint has a large range of motion and rapid velocity changes, and the waveform characteristics of its synthetic acceleration signal are more obvious than those of the hand, resulting in a higher correlation with the measured IMU signal of the knee joint. This allows for better alignment accuracy than the hand reference in certain lower limb-dominated motion scenarios.

[0074] In the aforementioned alternative, the term "designated joint" is not limited to hand joints. This embodiment specifies it as lower limb joints such as the knee and ankle, demonstrating that the method for constructing the synthetic reference signal is joint position independent—any three-dimensional joint position solvable through forward kinematics can be used as input for differential operations. This extension's feasibility is based on the hierarchical structure of human kinematics: the SMPL model defines the kinematic chains of 24 joints throughout the body, and the position of any joint can be recursively obtained from the root node through rigid body transformation. Therefore, the generation logic of the synthetic signal is consistent for all joints. When the application scenario switches from upper limb motion analysis to lower limb rehabilitation assessment, this embodiment switches the reference joint from the hand to the knee / ankle joint, ensuring that the physical meaning of the synthetic signal matches the actual IMU installation location, avoiding spurious phase differences introduced by inconsistent physical locations of the signal source. Compared to solutions that adhere to the hand reference, this embodiment enables the system to dynamically select the most suitable joint as the synthetic signal source under different motion modes, further expanding the method's scenario adaptability.

[0075] Another embodiment is that the input data used by the world model or alignment algorithm in the foregoing embodiments is not limited to acceleration signals, but can also be extended to a combination of IMU attitude measurements and other measurable data. For example, using triaxial accelerometer readings and triaxial gyroscope readings simultaneously as input features, a six-dimensional signal vector can be constructed. , , , , , ] T The acceleration signal is used to characterize linear motion, while the angular velocity signal is used to characterize rotational motion. During the cross-correlation alignment stage, the cross-correlation functions of the acceleration and angular velocity channels are calculated separately, and the combined time offset is obtained through weighted fusion (e.g., the weights are proportional to the signal energy of each channel). Taking rapid rotation as an example, the peak value of the acceleration signal may not be obvious, but the angular velocity signal has a significant single-peak characteristic. In this case, the angular velocity channel has a dominant weight in the fusion, and it can still accurately capture the alignment reference. Furthermore, the input data can be further extended to magnetometer readings (providing an absolute heading reference), barometer altitude data, etc., to improve the robustness of the alignment algorithm when single sensor signals degrade through multi-channel fusion.

[0076] The technical essence of this alternative lies in the fact that the IMU, as a multi-axis sensor, outputs acceleration, angular velocity, and magnetometer data that physically describe different aspects of motion—acceleration is sensitive to linear motion, angular velocity is sensitive to angular motion, and the magnetometer provides a geomagnetic reference. Under weak temporal synchronization conditions, the signal-to-noise ratio of a single signal dimension may decrease due to a specific motion mode (e.g., acceleration approaches zero during uniform linear motion), but other dimensions maintain a high signal-to-noise ratio (angular velocity during uniform linear motion may also be zero, but the magnetometer remains stable). By combining signals from multiple dimensions, the algorithm can utilize the complementarity between different dimensions—when the cross-correlation peak of one dimension is blurred, the peaks of other dimensions can still provide reliable time delay estimates. This multimodal input fusion strategy essentially constructs a redundant observation system, which is more reliable than schemes relying on a single physical quantity. This embodiment does not change the core steps of the dynamic alignment method based on dual references under any of the above weak temporal synchronization conditions; it merely concretizes the higher-level concept of "measured signal from the inertial measurement unit," demonstrating that the method of this invention can be compatible with all output channels of the IMU without loss of generality.

[0077] In some embodiments, such as Figure 4 This application also provides an electronic device, including a processor 401, a memory 402 and a bus 403 and a computer program stored in the memory, wherein when the processor executes the program, it implements a dynamic alignment method based on dual reference under any of the weak timing synchronization conditions described above.

[0078] Furthermore, as an executable solution, any dynamic alignment system based on dual references under weak timing synchronization conditions can be a computer unit, which can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit; it may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., which are not limited in this respect in the embodiments of the present invention.

[0079] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0080] In some embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic alignment method based on dual reference under any of the weak timing synchronization conditions described in the embodiments of the present invention.

[0081] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0082] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A dynamic alignment method based on dual references under weak temporal synchronization conditions, characterized in that, Includes the following steps: S1. Under weak timing synchronization conditions, the inertial measurement unit sensor and the reference motion capture system are started synchronously to independently collect motion data; all data are timestamped with their own hardware clock and stored in a standardized format, with the standard T-Pose posture as the data starting reference. S2. Perform combined preprocessing on the acquired raw inertial measurement unit data. The combined preprocessing includes at least duplicate frame removal, interpolation repair, and smoothing filtering. S3. Construct a two-stage time alignment mechanism for the fused and synthesized reference signal, and perform dynamic alignment between the preprocessed inertial measurement unit measured signal and the reference motion capture system signal, specifically including: S31. Based on the posture parameters in the parametric human body model and reference motion capture data, standard joint positions are calculated through forward kinematics, and differential operations are performed on the position signals of specified joints to generate synthetic reference signals. S32. Within the current sliding window, evaluate the energy of the measured signals from similar inertial measurement units on the left and right sides in real time, and adaptively select the signal with higher energy as the dynamic alignment reference. S33. Based on the synthesized reference signal and the dynamic alignment benchmark, the initial time offset is determined by cross-correlation analysis, and the time drift is continuously tracked and corrected within the sliding window; S4. Using the standard T-Pose as a reference, automatically detect the standard T-Pose frame in the collected data, calculate the transformation matrix from the sensor coordinate system to the human skeleton coordinate system based on the raw data of the inertial measurement unit of the frame, and perform coordinate system transformation and gravity acceleration removal on all inertial measurement unit data accordingly to obtain calibrated absolute acceleration and attitude data. S5. Visualize and compare the human posture driven by the calibrated and synchronized inertial measurement unit data with the standard posture of the reference motion capture system, and support users to manually fine-tune the alignment parameters.

2. The method according to claim 1, characterized in that, In step S2, the removal of duplicate frames is based on the 2-norm determination rule of acceleration data: in, To determine the threshold; in the interpolation repair, missing acceleration data is filled using linear interpolation, and missing rotation data is filled using spherical linear interpolation. The spherical linear interpolation formula is: in, These are time-based interpolation weights.

3. The method according to claim 1, characterized in that, In step S31, the synthesized reference signal is generated by performing a second-order difference operation on the three-dimensional position sequence of the specified joint, and its calculation formula is as follows: Where P(t) is the joint position at time t, and Δt is the sampling time interval.

4. The method according to claim 1, characterized in that, In step S32, the formula for calculating the signal energy is: Where W is the current sliding window, and a(t) is the measured acceleration signal of the inertial measurement unit within the window; the side with the higher energy value is selected as the dynamic alignment reference.

5. The method according to claim 1, characterized in that, In step S33, the initial time offset is determined by maximizing the cross-correlation function between the synthesized reference signal and the dynamically aligned reference signal, as shown in the formula: in, To synthesize a reference signal, For dynamic alignment reference signal, This represents the candidate time offset.

6. The method according to claim 1, characterized in that, In step S4, the automatic positioning of the T-Pose standard frame is determined by minimizing the sum of the rotation angles of multiple joints of the human body and the standard T-Pose angle, as shown in the formula: Where N is the number of joints involved in the calculation. Let be the rotation matrix of the j-th joint at time t. Let be the rotation matrix of the j-th joint in the standard T-Pose.

7. The method according to claim 1, characterized in that, In step S4, the calculation formula for the removal of gravitational acceleration is: in, The acceleration data is after coordinate system transformation. This is the purely linear acceleration after removing gravity.

8. A dynamic alignment system based on dual references under weak timing synchronization conditions, characterized in that, The system employs the dynamic alignment method based on dual references under weak timing synchronization conditions as described in any one of claims 1 to 7, the system comprising: The data acquisition module is configured to synchronously start the inertial measurement unit sensor and the reference motion capture system under weak temporal synchronization conditions, independently acquire motion data, and use the standard T-Pose posture as the data starting reference. The preprocessing module is configured to perform a combination of preprocessing on the acquired raw inertial measurement unit data, including duplicate frame removal, interpolation repair, and smoothing filtering. A two-stage time alignment module is configured to construct a two-stage time alignment mechanism for the fused synthetic reference signal, performing dynamic alignment between the preprocessed inertial measurement unit measured signal and the reference motion capture system signal; the two-stage time alignment module includes: The synthetic reference signal construction unit is configured to generate a synthetic reference signal based on the attitude parameters in the parameterized human model and reference motion capture data. The dynamic signal selection unit is configured to evaluate the energy of the measured signals from similar inertial measurement units on the left and right sides in real time within the current sliding window, and adaptively select the signal with higher energy as the dynamic alignment reference. The drift correction unit is configured to determine the initial time offset based on the synthesized reference signal and the dynamic alignment reference by using cross-correlation analysis, and to continuously track and correct the time drift within a sliding window; The integrated calibration module is configured to automatically detect the T-Pose standard frame in the acquired data based on the standard T-Pose, calculate the transformation matrix from the sensor coordinate system to the human skeleton coordinate system, and perform coordinate system transformation and gravitational acceleration removal on all inertial measurement unit data accordingly. The visualization output module is configured to visually compare and output the human posture driven by the calibrated and synchronized inertial measurement unit data with the standard posture of the reference motion capture system, and supports users to manually fine-tune the alignment parameters.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images