High-precision motion capture real-time calibration method based on multi-sensor fusion

Abstract

The invention provides a high-precision motion capture real-time calibration method based on multi-sensor fusion, and the method comprises the steps: obtaining motion data and environmental parameters collected by multiple sensors, the multiple sensors comprising an inertial measurement unit and an optical sensor; constructing a dynamic calibration model based on the action data and the environmental parameters, and performing real-time error compensation and fusion processing on the sensor data; and establishing a time-varying error dynamic model, and performing real-time compensation on zero offset drift, vibration noise and electromagnetic interference of the inertial measurement unit based on the environmental parameters. According to the invention, the IMU and the optical sensor are fused, respective advantages are exerted, and motion data are comprehensively collected. And meanwhile, the time-varying error dynamic model compensates errors generated by environmental interference in real time, so that real actions can be accurately restored through action capture.

Description

Technical Field

[0001] The present invention relates to the field of motion capture technology, and in particular to a high-precision motion capture real-time calibration method based on multi-sensor fusion. Background Art

[0002] In today's era of rapid digital advancement, high-precision motion capture technology based on multi-sensor fusion plays a vital role in numerous fields. In virtual reality (VR) and augmented reality (AR), precise motion capture enables users to experience immersive interactions, enhancing the realism and immersion of virtual environments. In film and television production and game development, it enables realistic character movement simulation, enhancing the visual impact and appeal of productions. In sports training and medical rehabilitation, motion capture technology provides athletes with precise movement analysis, assisting in the development of personalized training plans, while helping doctors assess patients' recovery progress and providing data support for treatment.

[0003] However, existing motion capture real-time calibration technology based on multi-sensor fusion still has many shortcomings, which limit its further development and widespread application. In terms of data processing, the data collected by multiple sensors is not only massive in volume but also diverse in type. The measurement principles, accuracy, and sampling frequency of different sensors vary greatly, making it difficult to ensure data consistency. For example, the sampling frequency of an inertial measurement unit (IMU) is high, but its accuracy is greatly affected by environmental factors; optical sensors have high accuracy but are easily affected by occlusion and changes in illumination. When fusing this data, even after preprocessing, it is difficult to eliminate the impact of data differences, which greatly reduces the accuracy of the fusion algorithm. At the same time, the complex algorithms required for multi-sensor fusion and real-time calibration are extremely computationally intensive and place extremely high performance demands on the computing equipment. This not only increases hardware costs but can also lead to a decrease in real-time performance, making it impossible to meet the real-time requirements of demanding application scenarios such as real-time action interactive games.

[0004] In terms of fusion algorithms, existing algorithms are often based on the assumption of a linear relationship between sensor data, while practical applications involve a large number of nonlinear features. Taking human joint movement as an example, the relationship between joint rotation angle and muscle force is not a simple linear relationship, but a complex nonlinear mapping. Fusion algorithms based on linear models cannot accurately restore the true state of the joints, thus affecting the accuracy of motion capture. In addition, there is cross-sensitivity between sensors. For example, changes in air pressure may affect the readings of both the barometer and the accelerometer. Existing calibration models usually treat these interferences as independent factors and cannot accurately describe the true output characteristics of the sensors. When processing multimodal features such as optical image edge features and inertial measurement unit angular velocity data, due to their essential differences in semantic space, although deep learning fusion models can extract high-level features, they have difficulty capturing the nonlinear correlations of the underlying data, resulting in large feature alignment errors.

[0005] Real-time calibration technology also faces challenges. In dynamic environments, the operating conditions of sensors are complex and changeable. Existing technologies generally assume that the sensor operating environment is static or slowly changing, and lack adaptability to fast dynamic environments. For example, when an inertial measurement unit moves rapidly from a low-temperature environment to a high-temperature environment, the thermal expansion of the sensor material will cause the zero-bias drift rate to increase sharply. Existing calibration models usually only consider steady-state temperature compensation and cannot track the impact of the temperature change rate on the sensor in real time. In high-frequency vibration scenarios such as industrial robotic arms or moving human bodies, the output of the accelerometer will be superimposed with vibration noise. Traditional Kalman filtering has difficulty distinguishing between real motion signals and vibration interference, resulting in the accumulation of attitude estimation errors over time. In strong electromagnetic environments, such as those near medical equipment or motors, sensors may be interfered with by transient electromagnetic pulses. Existing anti-interference algorithms are mostly designed based on statistical characteristics and have limited ability to suppress non-periodic pulses.

[0006] In terms of computing resource allocation, existing real-time calibration algorithms suffer from energy efficiency imbalances. Multi-sensor fusion algorithms typically process all data in full. However, in some cases, some sensor data contributes little to the final result, resulting in a large amount of redundant calculations. For example, in a stable attitude, the contribution of the accelerometer is far less than that of the gyroscope. If computing resources are not dynamically adjusted, computing power will be wasted. In distributed sensor networks, data transmission delays can cause time synchronization errors, affecting the accuracy of attitude fusion results. For example, due to communication delays between nodes in a drone cluster system, the attitude fusion results lag by 50ms, seriously affecting flight stability. When deploying complex algorithms on edge devices (such as wearable sensors), due to computing power and power consumption limitations, it is often necessary to reduce the sampling rate or simplify the model, which inevitably leads to a decrease in accuracy.

[0007] Therefore, a high-precision motion capture real-time calibration method based on multi-sensor fusion is proposed. Summary of the Invention

[0008] In view of this, the embodiments of the present invention hope to provide a high-precision motion capture real-time calibration method based on multi-sensor fusion to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.

[0009] To solve the above technical problems, a technical solution adopted in this application is: to provide a high-precision motion capture real-time calibration method based on multi-sensor fusion, comprising the following steps: obtaining motion data and environmental parameters collected by multiple sensors, wherein the multiple sensors include an inertial measurement unit and an optical sensor; constructing a dynamic calibration model based on the motion data and environmental parameters, and performing real-time error compensation and fusion processing on the sensor data; establishing a time-varying error dynamic model, and performing real-time compensation for the zero bias drift, vibration noise and electromagnetic interference of the inertial measurement unit based on the environmental parameters; and outputting the calibrated high-precision motion capture results.

[0010] As a further preferred embodiment of the present technical solution: the construction of the dynamic calibration model includes: deploying an environmental sensor array rigidly connected to the multi-sensor, synchronously collecting the temperature change rate, vibration signal and electromagnetic pulse intensity, and realizing the timestamp alignment of environmental parameters and motion data through a nanosecond precision clock.

[0011] As a further preferred embodiment of the present technical solution, the time-varying error dynamic model includes: a temperature and zero-bias drift rate real-time correction model:

[0012]

[0013] Among them, (f temp ) is a polynomial fitting function containing the first-order derivative of temperature, which is used to compensate for the zero-bias mutation caused by thermal expansion; the variable step-size LMS vibration noise separation filter dynamically adjusts the bandpass filter cutoff frequency according to the real-time estimated frequency distribution of the vibration signal to separate high-frequency vibration noise from low-frequency motion signals; the electromagnetic pulse interference compensation mechanism, when a transient electromagnetic pulse is detected, uses the stable data of the previous 100ms through cubic spline interpolation to generate a state estimate during the interference period.

[0014] As a further preferred embodiment of the present technical solution, the fusion process adopts a nonlinear multimodal fusion architecture, including: constructing a nonlinear kinematic model of human joints based on the DH parameter method and the B-spline interpolation function to describe the nonlinear mapping relationship between muscle force and joint angle:

[0015] [θ j (t) = g j (F muscle (t),θ j (t-1))]

[0016] Among them, (gj ) is the nonlinear dynamic function of the (j)th joint; an augmented state vector containing cross-sensitive physical quantities is constructed, and the unscented Kalman filter is used to jointly estimate the multivariable state space model to decouple the influence of non-target physical quantities such as pressure and temperature on the sensor output; the deep features of visual and inertial data are extracted through multi-stream convolution and LSTM network, and the cross-modal feature cross-correlation matrix is calculated using the attention mechanism to generate a dynamic weight matrix to achieve feature space alignment.

[0017] As a further preferred embodiment of the present technical solution, the visual stream of the multi-stream convolution and LSTM network extracts optical image edge features through a residual network, the inertial stream processes inertial measurement unit time series data through LSTM, and the alignment layer generates fusion features through an attention mechanism:

[0018] [f fusion =W a ·[f vis ;f imu ]]

[0019] Among them, (W a ) is the dynamic weight matrix, (f vis ) and (f imu ) are the visual and inertial eigenvectors respectively.

[0020] As a further preferred embodiment of the present technical solution, it further includes adopting a computing resource optimization strategy to allocate computing resources in the process of executing the dynamic calibration model, and the computing resource optimization strategy includes: defining a data validity index (C i (t)):

[0021]

[0022] Dynamically adjust the sensor data processing frequency according to the above indicators to reduce redundant calculations; build an edge-cloud collaborative architecture, deploy lightweight UKF filters at the edge layer to process high-frequency data, run the full-precision model on the cloud and periodically send parameter update packages; use knowledge distillation and dynamic pruning technology to lightweight the edge device model, and cut redundant connections when the computing power load exceeds the threshold to reduce accuracy loss and computational complexity.

[0023] To solve the above technical problems, another technical solution adopted in this application is: a high-precision motion capture real-time calibration device, including: a data acquisition module, configured to obtain motion data and environmental parameters of multiple sensors; a dynamic calibration module, configured to construct a dynamic calibration model based on the motion data and environmental parameters, and perform real-time error compensation and fusion processing on the sensor data; a result output module, configured to output the calibrated high-precision motion capture results.

[0024] As a further preferred embodiment of the present technical solution, the dynamic calibration module includes: an environmental perception unit, configured to deploy an environmental sensor array, synchronously collect temperature change rate, vibration signal and electromagnetic pulse intensity, and perform timestamp alignment; an error compensation unit, configured to establish a time-varying error dynamic model, and perform real-time compensation for the zero bias drift, vibration noise and electromagnetic interference of the inertial measurement unit; a nonlinear fusion unit, configured to adopt a nonlinear multimodal fusion architecture to achieve nonlinear modeling of joint kinematics, sensor cross-sensitivity decoupling and cross-modal feature alignment.

[0025] To solve the above technical problems, another technical solution adopted in this application is: a computer device, which includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the steps of the high-precision motion capture real-time calibration method based on multi-sensor fusion as described above.

[0026] In order to solve the above technical problems, another technical solution adopted in this application is: a storage medium storing program instructions that can implement the high-precision motion capture real-time calibration method based on multi-sensor fusion as described above.

[0027] The embodiment of the present invention adopts the above technical solution, which has the following advantages:

[0028] First, this invention integrates IMU and optical sensors, leveraging their respective strengths to comprehensively capture motion data. Simultaneously, a time-varying error dynamic model compensates for errors caused by environmental interference in real time, enabling motion capture to accurately reproduce real movements.

[0029] 2. The nonlinear multimodal fusion architecture of the present invention can accurately describe the complex nonlinear relationship between muscle force and joint angle, effectively decouple the problem of sensor cross-sensitivity, and align multimodal features. Even in scenarios with complex human motion postures and a lot of environmental interference, it can capture movements stably and accurately.

[0030] Third, this invention dynamically adjusts the sensor data processing frequency based on data validity indicators to avoid redundant computation. A collaborative architecture between the edge and cloud rationally allocates computing tasks, with edge devices processing high-frequency data and the cloud performing in-depth analysis. Combined with lightweight edge device model technology, this significantly reduces computing resource consumption while ensuring motion capture accuracy, making it applicable to a wide range of hardware conditions.

[0031] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2 Schematic diagram of the module of the calibration device of the present invention

[0035] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0037] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0038] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0039] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0040] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0041] Figure 1 This is a flow chart of a high-precision motion capture real-time calibration method based on multi-sensor fusion according to an embodiment of the present invention. It should be noted that if there are substantially the same results, the method of this application is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown: A high-precision motion capture real-time calibration method based on multi-sensor fusion includes the following steps:

[0042] S100, acquiring motion data and environmental parameters collected by multiple sensors, where the multiple sensors include an inertial measurement unit and an optical sensor;

[0043] Specifically, in actual application scenarios, a virtual reality interactive system is used as an example. The user wears a device that integrates an IMU and an optical marker, which can collect motion data in real time. Among them, the IMU collects acceleration, angular velocity and magnetic field data at a relatively high frequency (for example, 1000Hz), and the optical sensor collects the spatial coordinate information of the marker at a certain frequency (such as 200Hz), thereby obtaining the user's motion posture. At the same time, the environmental sensor array rigidly connected to the multi-sensor starts working, the temperature gradient sensor collects the temperature change rate with an accuracy of ±0.1℃ / ms, the vibration accelerometer collects vibration signals with a bandwidth of 20kHz, and the electromagnetic pulse detector monitors the electromagnetic pulse intensity at the μT level to fully obtain environmental parameters.

[0044] To ensure that the collected data accurately reflects the user's movements and can be effectively integrated in subsequent processing, multi-sensor temporal and spatial synchronization calibration is required. Using a nanosecond-precision clock, each sensor's data is precisely timestamped, achieving precise temporal alignment of motion data with environmental parameters. For example, at a given moment, the IMU collects a set of acceleration data, while the optical sensor collects the coordinates of a marker point, and the environmental sensor array collects the corresponding temperature, vibration, and electromagnetic data. These data are all precisely time-stamped, ensuring temporal consistency. Furthermore, the IMU's accelerometer bias and gyroscope scale factor are calibrated, as are the intrinsic and extrinsic parameters of the optical sensor. For example, the Zhang Zhengyou calibration method is used to determine the optical sensor's intrinsic parameters, such as focal length and distortion coefficient, with a calibration accuracy of <0.1 pixel. A target array is used to establish a coordinate system transformation relationship, and extrinsic parameters, such as the rotation matrix and translation vector, between the IMU and optical sensor are determined. The calibration error is controlled to <0.5mm / 0.1°, thus ensuring the accuracy and consistency of sensor measurements.

[0045] S200, constructing a dynamic calibration model based on the motion data and environmental parameters, and performing real-time error compensation and fusion processing on the sensor data;

[0046] Specifically, the construction of the dynamic calibration model includes: deploying an environmental sensor array rigidly connected to the multi-sensor, synchronously collecting temperature change rate, vibration signal and electromagnetic pulse intensity, and realizing timestamp alignment of environmental parameters and motion data through a nanosecond precision clock.

[0047] The fusion process adopts a nonlinear multimodal fusion architecture, including: constructing a nonlinear kinematic model of human joints based on the DH parameter method and B-spline interpolation function to describe the nonlinear mapping relationship between muscle force and joint angle:

[0048] [θ j (t) = g j (F muscle (t),θ j (t-1))]

[0049] Among them, (g j ) is the nonlinear dynamic function of the (j)th joint; an augmented state vector containing cross-sensitive physical quantities is constructed, and the unscented Kalman filter is used to jointly estimate the multivariable state space model to decouple the influence of non-target physical quantities such as pressure and temperature on the sensor output; the deep features of visual and inertial data are extracted through multi-stream convolution and LSTM network, and the cross-modal feature cross-correlation matrix is calculated using the attention mechanism to generate a dynamic weight matrix to achieve feature space alignment.

[0050] The visual stream of the multi-stream convolution and LSTM network extracts optical image edge features through the residual network, the inertial stream processes the inertial measurement unit time series data through the LSTM, and the alignment layer generates fusion features through the attention mechanism:

[0051] [f fusion =W a ·[f vis ;f imu ]]

[0052] Among them, (W a ) is the dynamic weight matrix, (f vis ) and (f imu ) are the visual and inertial eigenvectors respectively.

[0053] Time-varying error compensation algorithm Temperature compensation: In actual applications, when the device worn by the user moves quickly from a low-temperature indoor environment to a high-temperature outdoor environment, the IMU will be significantly affected by temperature changes. The temperature and zero-bias drift rate dynamic model established according to the present invention is:

[0054]

[0055] Among them, (f temp ) is a polynomial fitting function containing the first-order derivative of temperature, which is used to compensate for the zero-bias mutation caused by thermal expansion. Assume that at a certain moment, the temperature rises rapidly from 20℃ to 35℃, and the temperature change rate The model calculates the bias drift compensation value, which is 0.5°C / s. The IMU measurement data is corrected in real time, effectively compensating for the bias drift caused by thermal expansion and ensuring the accuracy of the IMU measurement data. Vibration noise separation: When the user performs intense movements, such as running, jumping, or waving their arms quickly, the device will be affected by high-frequency vibrations, causing vibration noise to be superimposed on the accelerometer output. At this point, the variable-step-size LMS vibration noise separation filter designed in the present invention comes into play. It dynamically adjusts the bandpass filter cutoff frequency based on the real-time estimated frequency distribution of the vibration signal. For example, when it is detected that the vibration frequency is mainly concentrated in the range of 100-500Hz, the filter automatically adjusts the cutoff frequency to effectively separate the high-frequency vibration noise from the low-frequency motion signal, preventing the vibration noise from interfering with the motion signal and improving the accuracy of motion capture. Electromagnetic interference processing: In some special environments, such as near strong electromagnetic interference sources such as motors and transformers, the sensor may be interfered with by transient electromagnetic pulses. When the electromagnetic pulse detector detects an electromagnetic pulse with an amplitude exceeding a certain threshold (such as 3σ) and a short duration (<10μs), the electromagnetic pulse interference compensation mechanism of the present invention is activated. The stable data of the first 100ms is used to generate the state estimation value during the interference period through the cubic spline interpolation algorithm to avoid the influence of abnormal data on the calibration results and ensure the reliability of the motion capture data.

[0056] To further compensate for the IMU's bias drift due to thermal expansion, the temperature and bias drift rate dynamic model can also be expressed using the following formula:

[0057]

[0058] (k1) and (j2) are temperature sensitivity coefficients, which are obtained by fitting historical data and are used to compensate for the zero bias drift of the IMU caused by thermal expansion.

[0059] S300, establishing a time-varying error dynamic model, and performing real-time compensation for the zero bias drift, vibration noise, and electromagnetic interference of the inertial measurement unit based on the environmental parameters;

[0060] Specifically, the time-varying error dynamic model includes: a temperature and zero-bias drift rate real-time correction model:

[0061]

[0062] Among them, (f temp ) is a polynomial fitting function containing the first-order derivative of temperature, which is used to compensate for the zero-bias mutation caused by thermal expansion; the variable step-size LMS vibration noise separation filter dynamically adjusts the bandpass filter cutoff frequency according to the real-time estimated frequency distribution of the vibration signal to separate high-frequency vibration noise from low-frequency motion signals; the electromagnetic pulse interference compensation mechanism, when a transient electromagnetic pulse is detected, uses the stable data of the previous 100ms through cubic spline interpolation to generate a state estimate during the interference period.

[0063] Among them, joint kinematic modeling: Taking human motion analysis as an example, a nonlinear kinematic model of human joints is constructed based on the DH parameter method and B-spline interpolation function to describe the nonlinear mapping relationship between muscle force and joint angle:

[0064] (θ j (t) = g j (F muscle (t),θ j (t-1)))

[0065] Among them, (g j ) is the nonlinear dynamic function of the jth joint. When analyzing the motion of the human arm, considering the muscle force (F muscle (t)), and the historical angle of the joint (θ j(t-1)), this model can more accurately describe the motion state of the joint. For example, when the arm muscles exert force, based on the magnitude and direction of the muscle force, combined with the current angle and historical angle of the joint, the model can more accurately predict the bending angle of the arm joint, thereby more realistically reflecting the movement of the human arm. Sensor cross-sensitivity decoupling: In response to the cross-sensitivity phenomenon between sensors, such as air pressure changes may affect the readings of the barometer and accelerometer at the same time, the present invention constructs an augmented state vector containing cross-sensitive physical quantities, and uses unscented Kalman filtering (UKF) to jointly estimate the multivariable state space model. Non-target physical quantities such as pressure and temperature are included in the state vector, and these factors are comprehensively considered through UKF to accurately estimate the true state of the sensor, effectively decouple cross-sensitivity errors, and improve the accuracy and reliability of sensor data. Cross-modal feature alignment: Multi-stream convolution and LSTM networks are used to extract deep features of visual and inertial data, and the cross-modal feature correlation matrix is calculated through the attention mechanism to generate a dynamic weight matrix to achieve feature space alignment. In VR scenarios, the visual stream extracts edge features from optical images through a residual network, while the inertial stream processes IMU time series data through an LSTM. For example, when a user performs complex gestures, the visual stream extracts edge features such as the gesture's outline and joints through a residual network, while the inertial stream captures the dynamic changes in angular velocity and acceleration in the IMU data through an LSTM. The inertial stream then uses an attention mechanism to calculate the cross-correlation matrix between the visual and inertial features, generating a dynamic weight matrix that aligns the two features, enabling more accurate gesture recognition and improving motion capture accuracy.

[0066] More specifically, a computing resource optimization strategy is used to allocate computing resources in the process of executing the dynamic calibration model. The computing resource optimization strategy includes: defining a data validity index (C i (t)):

[0067]

[0068] Dynamically adjust the sensor data processing frequency according to the above indicators to reduce redundant calculations; build an edge-cloud collaborative architecture, deploy lightweight UKF filters at the edge layer to process high-frequency data, run the full-precision model on the cloud and periodically send parameter update packages; use knowledge distillation and dynamic pruning technology to lightweight the edge device model, and cut redundant connections when the computing power load exceeds the threshold to reduce accuracy loss and computational complexity.

[0069] Among them, sensor data validity evaluation: the present invention defines the data validity index (C i (t)):

[0070]

[0071] According to this indicator, the sensor data processing frequency is dynamically adjusted to reduce redundant calculations. When the virtual reality device is in a relatively static state, the accelerometer data validity index (C accel If the value of (t) is less than a pre-set threshold (e.g., 0.3), the accelerometer processing frequency is automatically reduced from 1000Hz to 500Hz, reducing unnecessary computing resource consumption without compromising overall motion capture accuracy. Edge-cloud collaborative architecture: In virtual reality applications, edge devices (such as VR headsets worn by users) deploy lightweight UKF filters to perform preliminary processing of high-frequency inertial data, reducing the parameter size by 70% compared to the full model. The cloud runs a full-precision model, receiving low-bandwidth features such as joint angle residuals uploaded by the edge devices, performing in-depth analysis, and periodically sending parameter update packages to the edge devices (e.g., every 10 seconds). This approach meets real-time requirements while fully leveraging the powerful computing power of the cloud. For example, in multi-person VR interaction scenarios, the cloud can analyze the motion data of multiple users in real time to optimize motion capture and calibration, while reducing the computational burden on edge devices. Edge device model lightweighting: Knowledge distillation and dynamic pruning techniques are used to lightweight edge device models. When the computing load of an edge device exceeds a certain threshold (e.g., 80%), the model automatically initiates a dynamic pruning process to trim redundant connections in the convolutional layer. For example, when processing complex gesture motion capture tasks, after pruning, the model inference latency is controlled to less than 15ms while ensuring that the motion capture accuracy loss is less than 3%, effectively balancing computing resources and accuracy.

[0072] S400 outputs calibrated, high-precision motion capture results. After the aforementioned data collection, dynamic calibration model processing, and computing resource optimization, calibrated, high-precision motion capture results are output. In a virtual reality interactive system, calibrated motion data, such as joint angles and position coordinates, is fed back to the VR application in real time, enabling more accurate virtual character motion simulation and enhancing the user's immersive and interactive experience. In sports training, coaches can develop more scientific training plans based on this precise data, helping athletes improve their training outcomes. In medical rehabilitation, doctors analyze patients' motion data to assess their recovery progress, providing a strong basis for adjusting personalized treatment plans.

[0073] Figure 2 : is a functional module diagram of a high-precision motion capture real-time calibration method system based on multi-sensor fusion according to an embodiment of the present application. Figure 2As shown, a high-precision motion capture real-time calibration device includes: a data acquisition module, configured to obtain motion data and environmental parameters of multiple sensors; a dynamic calibration module, configured to build a dynamic calibration model based on the motion data and environmental parameters, and perform real-time error compensation and fusion processing on the sensor data; and a result output module, configured to output the calibrated high-precision motion capture results.

[0074] The dynamic calibration module includes: an environmental perception unit, configured to deploy an environmental sensor array, synchronously collect temperature change rate, vibration signal and electromagnetic pulse intensity, and perform timestamp alignment; an error compensation unit, configured to establish a time-varying error dynamic model, and perform real-time compensation for the zero bias drift, vibration noise and electromagnetic interference of the inertial measurement unit; a nonlinear fusion unit, configured to adopt a nonlinear multimodal fusion architecture to achieve nonlinear modeling of joint kinematics, sensor cross-sensitivity decoupling and cross-modal feature alignment.

[0075] The present invention also provides an example of a VR fitness company using the method of the present invention to perform motion capture and calibration:

[0076] A VR fitness company has launched a new VR fitness game in which players must perform various fitness exercises in a virtual environment, such as boxing, squats, and sword swinging. To provide players with the most realistic and accurate experience, the company adopted the method of the present invention for motion capture and calibration.

[0077] 1. Data collection and synchronization:

[0078] 1.1. Equipment: Players wear a VR headset and a full-body motion capture suit with integrated IMU and optical markers. The suit is equipped with multiple IMU sensors, while multiple optical sensors are placed in the room. In addition, the suit is equipped with an array of micro-environmental sensors, including temperature gradient sensors, vibration accelerometers, and electromagnetic pulse detectors.

[0079] 1.2. Collection parameters:

[0080] IMU: collects acceleration, angular velocity, and magnetic field data at a sampling frequency of 1000 Hz. Assume that at a certain time \(t_1\), the acceleration data collected by the IMU is Angular velocity data is

[0081] Optical sensor: This sensor collects the spatial coordinates of a marker at a frequency of 200 Hz. At time t1, the optical sensor records the spatial coordinates of a marker on the helmet as (x = 2.5 m, y = 1.8 m, z = 1.2 m).

[0082] Environmental sensor array: The temperature gradient sensor collects the temperature change rate with an accuracy of ±0.1°C / ms. At time (t1), the measured temperature is (25°C) and the temperature change rate is (0.02°C / s).

[0083] The vibration accelerometer collects vibration signals with a bandwidth of 20kHz. At time (t1), the amplitude of the vibration signal is (0.05m / S 2 ). The electromagnetic pulse intensity detected by the electromagnetic pulse detector is (0.01\muT).

[0084] Time and space synchronization calibration: Through the nanosecond precision clock, the data collected by each sensor is accurately timestamped to ensure the alignment of motion data and environmental parameters in the time dimension. At the same time, the IMU is calibrated for the accelerometer zero bias and gyroscope scale factor, as well as the internal and external parameters of the optical sensor. Assume that the zero bias calibration value of the IMU accelerometer is The gyroscope scale factor calibration value is (k ω =1.002). After calibration of the optical sensor internal parameters, the focal length (f=50mm) and the distortion coefficient (k1=-0.01) are obtained.

[0085] 2. Dynamic calibration model construction:

[0086] 2.1 Time-varying error compensation algorithm:

[0087] Temperature compensation: Based on the temperature-bias drift rate dynamic model Known (δ bias =(0.02,-0.01,0.04)m / s 2 ), (k1=(0.001,0.002,0.0015)m / s 2 / ℃), (k2=(0.0005,0.0003,0.0004)m / s 2 / (℃ / s)), (T0=20℃), (T(t)=25℃),

[0088] Calculate \(\delta_{\text{bias}}(t)\): (δ bias,x =0.02+0.001×(25-20)+0.0005×0.02=0.02+0.005+0.00001=0.02501m / s 2 )

[0089] (δ bias,y =-0.01+0.002×(25-20)+0.0003×0.02=-0.01+0.01+0.000006=0.000006m / s2 )

[0090] (δ bias,Z =0.04+0.0015×(25-20)+0.0004×0.02=0.04+0.0075+0.000008=0.047508m / s 2 )

[0091] Compensate the acceleration data collected by IMU:

[0092] Vibration noise separation: A variable step size LMS adaptive filter is used to separate vibration noise. Assuming that the main frequency range of the vibration signal is determined to be 100-500Hz through time-frequency analysis, the filter dynamically adjusts the cutoff frequency based on this. After the filter is processed, the influence of the vibration signal on the acceleration data is removed, and purer motion acceleration data is obtained. Assume that the acceleration data becomes

[0093] Electromagnetic interference processing: Since the electromagnetic pulse intensity detected by the electromagnetic pulse detector is (0.01\muT) and does not exceed the set threshold (assuming the threshold is (0.05\muT)), no interpolation compensation processing is required.

[0094] 2.2 Nonlinear Fusion Architecture:

[0095] Joint kinematics modeling: Taking the player's arm joint as an example, the joint kinematics equation is constructed based on the DH parameter method and the muscle force-joint angle B-spline interpolation model. Assume that the muscle force of the arm joint (F muscle =50N), the joint angle at the previous moment (θ j (t-1)=30°), according to the equation (Assume (g j ) is a specific nonlinear function that can be obtained by fitting experimental data), and the current joint angle (θ j (t) = 35°).

[0096] Sensor cross-sensitivity decoupling: Construct an augmented state vector (x(t) = [s(t), P(t), T(t)] T ), where (s(t)) is the main sensor state (such as acceleration, angular velocity, etc.), (P(t)) is the pressure (assuming the pressure is 101 kPa), and (T(t)) is the temperature of 25°C. The unscented Kalman filter (UKF) is used to jointly estimate the multivariate state space model. It is assumed that after UKF processing, a more accurate sensor state estimate is obtained.

[0097] Cross-modal feature alignment: The visual stream extracts the edge features of the joint points in the optical image through the residual network (ResNet-18) and generates a joint point probability map; the inertial stream processes the time series data of the IMU through the long short-term memory network (LSTM) to capture the dynamic relationship between angular velocity and acceleration. The cross-modal feature weight matrix is calculated through the attention mechanism:

[0098] Assume that the visual feature vector (f vis =(0.2,0.3,0.5)), inertial characteristic quantity (f imu =(0.4,0.1,0.5)).

[0099] First calculate

[0100] After processing by the multi-layer perceptron (MLP) and the Softmax function, the cross-modal feature weight matrix (W a ), which is used to perform weighted fusion of visual and inertial features to obtain more accurate motion feature representation.

[0101] 3. Computing resource optimization strategy:

[0102] 3.1 Sensor Data Validity Evaluation: Defining Sensor Data Validity Indicators Assume that at a certain moment, the accelerometer data validity index (c accel (T) = 0.2 < 0.3), the processing frequency of the accelerometer is automatically reduced from 1000 Hz to 500 Hz.

[0103] 3.2. Edge-Cloud Collaborative Computing: The edge device (VR helmet) deploys a lightweight UKF filter to perform preliminary processing on the high-frequency inertial data. Assume that the acceleration data processed by the lightweight UKF filter is Low-bandwidth features such as joint angle residuals are uploaded to the cloud. The cloud runs a full-precision nonlinear model for in-depth analysis and sends parameter update packages to edge devices every 10 seconds.

[0104] 3.3 Edge Device Model Lightweighting: When the computing power load of edge devices exceeds 80%, knowledge distillation and dynamic pruning techniques are used to lightweight the model. Assume that after pruning, the model inference latency is reduced from 20ms to 12ms, while the loss of motion capture accuracy is controlled within 2%.

[0105] 4. Result output and application:

[0106] After the above processing, calibrated, high-precision motion capture results are output. In VR fitness games, this calibrated motion data, such as joint angles and position coordinates, is fed back to the game program in real time, enabling precise mapping of the player's movements in the virtual environment. For example, if a player performs a boxing move in real life, the virtual character in the game will accurately perform the same boxing move, providing a highly immersive fitness experience. The game can also evaluate the player's fitness results based on the motion data, such as calculating the player's exercise intensity and calories burned. Suppose, based on the motion data, the player burns 300 kcal during this fitness session.

[0107] For other details about the technical solutions for implementing each module in the device of the above embodiment, please refer to the description of the high-precision motion capture real-time calibration method based on multi-sensor fusion in the above embodiment, which will not be repeated here.

[0108] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For system-related embodiments, since they are generally similar to method-related embodiments, their description is relatively simple. For relevant details, refer to the description of the method-related embodiments.

[0109] An electronic device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0110] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the high-precision motion capture real-time calibration method based on multi-sensor fusion described in each embodiment of the present disclosure.

[0111] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0112] like Figure 3 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device according to an embodiment of the present invention. Figure 3 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0113] like Figure 3 As shown, the electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0114] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 3 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0115] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by the processor, all or part of the steps of the high-precision motion capture real-time calibration method based on multi-sensor fusion of the embodiment of the present disclosure are performed.

[0116] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0117] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions. When executed by a processor, the non-transitory computer-readable instructions execute all or part of the steps of the high-precision motion capture real-time calibration method based on multi-sensor fusion described in each embodiment of the present disclosure.

[0118] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0119] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0120] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0121] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0122] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0123] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0124] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0125] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0126] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A high-precision motion capture real-time calibration method based on multi-sensor fusion, characterized by: The following steps are involved: Acquiring motion data and environmental parameters collected by multiple sensors, wherein the multiple sensors include an inertial measurement unit and an optical sensor; Building a dynamic calibration model based on the motion data and environmental parameters to perform real-time error compensation and fusion processing on sensor data; Establishing a time-varying error dynamic model to compensate for the zero bias drift, vibration noise and electromagnetic interference of the inertial measurement unit in real time based on the environmental parameters; Output calibrated high-precision motion capture results.

2. The high-precision motion capture real-time calibration method based on multi-sensor fusion according to claim 1, characterized in that: The dynamic calibration model construction includes: An environmental sensor array rigidly connected to the multi-sensor is deployed to synchronously collect temperature change rate, vibration signal and electromagnetic pulse intensity, and realize the timestamp alignment of environmental parameters and motion data through a nanosecond precision clock.

3. The high-precision motion capture real-time calibration method based on multi-sensor fusion according to claim 1, characterized in that: The time-varying error dynamic model includes: Real-time correction model for temperature and bias drift rate: Among them, (f temp ) is a polynomial fitting function containing the first-order temperature derivative term, which is used to compensate for the zero-bias mutation caused by thermal expansion; Variable step-size LMS vibration noise separation filter, which dynamically adjusts the bandpass filter cutoff frequency based on the real-time estimated vibration signal frequency distribution to separate high-frequency vibration noise from low-frequency motion signals; The electromagnetic pulse interference compensation mechanism, when a transient electromagnetic pulse is detected, uses the previous 100ms stable data to generate a state estimate during the interference through cubic spline interpolation.

4. The high-precision motion capture real-time calibration method based on multi-sensor fusion according to claim 1, characterized in that: The fusion process adopts a nonlinear multimodal fusion architecture, including: A nonlinear kinematic model of human joints is constructed based on the DH parameter method and B-spline interpolation function to describe the nonlinear mapping relationship between muscle force and joint angle: [i j (t)=g j (F muscle (t),θ j (t-1))] Among them, (g j ) is the nonlinear dynamic function of the (j)th joint; Construct an augmented state vector containing cross-sensitive physical quantities, use unscented Kalman filtering to jointly estimate the multivariable state space model, and decouple the effects of non-target physical quantities such as pressure and temperature on the sensor output; The deep features of visual and inertial data are extracted through multi-stream convolution and LSTM network, the cross-modal feature correlation matrix is calculated using the attention mechanism, and a dynamic weight matrix is generated to achieve feature space alignment.

5. The high-precision motion capture real-time calibration method based on multi-sensor fusion according to claim 4, characterized in that: The visual stream of the multi-stream convolution and LSTM network extracts optical image edge features through the residual network, the inertial stream processes the inertial measurement unit time series data through the LSTM, and the alignment layer generates fusion features through the attention mechanism: [ f fusion =W a ·[f vis ;f imu ]] Among them, (W a ) is the dynamic weight matrix, (f vis ) and (f imu ) are the visual and inertial eigenvectors respectively.

6. The high-precision motion capture real-time calibration method based on multi-sensor fusion according to claim 1, characterized in that: The method further includes allocating computing resources during the execution of the dynamic calibration model using a computing resource optimization strategy, wherein the computing resource optimization strategy includes: Define the data validity index (C i (t)): Dynamically adjust the sensor data processing frequency according to the indicators to reduce redundant calculations; Build an edge-cloud collaborative architecture, deploy lightweight UKF filters at the edge layer to process high-frequency data, run full-precision models on the cloud, and periodically send parameter update packages; Knowledge distillation and dynamic pruning techniques are used to lightweight edge device models, and redundant connections are pruned when the computing load exceeds the threshold to reduce accuracy loss and computational complexity.

7. A high-precision motion capture real-time calibration device, characterized in that: include: a data acquisition module configured to acquire motion data and environmental parameters from multiple sensors; A dynamic calibration module is configured to construct a dynamic calibration model based on the motion data and environmental parameters, and perform real-time error compensation and fusion processing on the sensor data; The result output module is configured to output the calibrated high-precision motion capture results.

8. The high-precision motion capture real-time calibration device according to claim 7, characterized in that: The dynamic calibration module includes: An environmental sensing unit is configured to deploy an environmental sensor array to synchronously collect temperature change rate, vibration signal and electromagnetic pulse intensity, and align timestamps; an error compensation unit configured to establish a time-varying error dynamic model and compensate for the zero bias drift, vibration noise, and electromagnetic interference of the inertial measurement unit in real time; The nonlinear fusion unit is configured to adopt a nonlinear multimodal fusion architecture to achieve nonlinear modeling of joint kinematics, sensor cross-sensitivity decoupling and cross-modal feature alignment.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the high-precision motion capture real-time calibration method based on multi-sensor fusion described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the high-precision motion capture real-time calibration method based on multi-sensor fusion described in any one of claims 1-6.

Images