A real-time inertial motion capture method and system based on location anchor augmentation

Abstract

The application discloses a real-time inertial motion capture method and system based on position anchor enhancement, collects inertial measurement unit acceleration, rotation signals, limb end position anchors and visibility tags provided by a head-mounted display device, encodes into a multi-modal observation feature vector, acquires human shape parameters and an initial motion state summary to construct a global structured condition vector, generates a frame-by-frame validity mask according to the visibility tag to perform gating processing, inputs the multi-modal feature after the gating processing into a transformer network to output human posture parameters, joint position parameters and motion speed parameters through adaptive layer normalization by using the condition vector, takes visual inertial poses output by the head-mounted display device as a stable head direction reference to perform online dynamic calibration, and reconstructs whole body three-dimensional joint positions and a human mesh model through forward kinematics. The application utilizes spatial position anchors to suppress end error amplification and heading drift, and realizes high-precision, long-time stable wearable motion capture.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a real-time inertial motion capture method and system based on enhanced position anchor points. Background Technology

[0002] With the rapid development of applications such as virtual reality (VR), robotic teleoperation, and large-scale real motion data acquisition, there is an urgent need for full-body motion capture systems that can be deployed long-term, possess high precision, and are not limited by location. Motion capture systems based on sparse inertial measurement units (IMUs) have become a core research direction for motion capture in open-world environments due to their advantages such as requiring no external infrastructure, high portability, and resistance to visual occlusion. However, with a sparse configuration using only a small number of sensors (such as 6 or 10 IMUs), pure inertial systems face severe challenges of accuracy degradation in long-term motion reconstruction. Specifically, this manifests in three major physical bottlenecks: First, there is the sensitivity to integral drift. Acceleration requires double integration to be converted into positional constraints, a process that is extremely sensitive to sensor noise, zero bias, and drift accumulated over time. Second, errors in the long power chain are amplified by cascading. The positions of the limb extremities are calculated from forward kinematics. Due to the lack of direct rotational observation of the intermediate joints, the system's internal degrees of freedom are weakly constrained; even a small angular error in the upstream joints will affect the position along the "shoulder" axis. elbow The "wrist" chain amplifies step by step, eventually manifesting as a huge positional distortion at the wrist. Third, there is cumulative global heading drift. Traditional solutions typically assume that the root IMU at the pelvis is a drift-free reference, but consumer-grade IMUs inevitably produce yaw drift. This deviation is systematically propagated upwards along the kinematic tree, causing the long-term motion trajectory to lose directional consistency in the global reference frame.

[0003] To alleviate the above problems, researchers have attempted to introduce visual inertial odometry (VIO) from head-mounted displays (HMDs) as an external spatial reference. For example, HMDs... The Poser system simply splices the rough hand position provided by the headset with the IMU signal, attempting to constrain end-effector drift with positional anchors. However, this hybrid approach still suffers from four key drawbacks: Mismatch between the sensory origin and the biological origin: There is a centimeter-level offset between the head-mounted display tracking center (such as the bridge of the nose) and the center of the human joint, which will interfere with rotation if not kinematically compensated. The accuracy of location mapping.

[0004] Geometric scale mismatch: Differences in bone proportions among different subjects lead to drastically different end displacements for the same amount of rotation. Existing methods lack explicit modeling of body proportions, resulting in excessive bending or unnatural elongation of limbs.

[0005] The impact of intermittent loss of observations: Hand anchor points appear frequently and intermittently due to the field of view (FOV) limitation of the head-mounted display or body occlusion. Existing solutions are prone to attitude jumps or accuracy collapse when anchor points are lost.

[0006] The selection of the heading calibration reference is suboptimal: the traditional system still uses the easily drifting pelvic IMU as the root node, failing to make full use of the high-precision, drift-independent VIO reference provided by the head-mounted display.

[0007] Therefore, how to effectively suppress the propagation of inertial errors, handle intermittent observations, and achieve global heading consistency using the position anchor points provided by the head-mounted display without sacrificing the system's lightweight and convenient features is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0008] This invention provides a real-time inertial motion capture method and system based on enhanced position anchors, which can solve the problems of end-point position divergence, long-range heading drift, geometric mismatch in multimodal fusion, and poor robustness of intermittent observations caused by insufficient observation information in existing sparse inertial motion capture systems.

[0009] The first aspect of this invention provides a real-time inertial motion capture method based on position anchor point enhancement, comprising: The system acquires acceleration and rotation signals output by the inertial measurement unit, as well as at least one three-dimensional spatial position anchor point of a limb end relative to the head reference frame and the visibility tag corresponding to the spatial position anchor point, obtained from the head-mounted display device. The three-dimensional spatial location anchor points, acceleration signals, and rotation signals are encoded into multimodal observation feature vectors; Obtain the user's human body shape parameters and the initial motion state summary of the current inference window, and construct a global structured condition vector, where the initial motion state summary includes the initial velocity, initial position, and initial pose; A frame-by-frame validity mask is generated based on the visibility label, and the spatial location anchor point channel in the multimodal observation feature vector is gated using the frame-by-frame validity mask. The gated multimodal observation feature vector is input into a transformer network containing multiple processing blocks. The activation values ​​of each processing block are scaled and offset by an adaptive layer normalization mechanism using the global structured condition vector, and the human posture parameters, joint position parameters, and motion velocity parameters are output. Using the visual inertial pose output by the head-mounted display device as a stable head orientation reference, the orientation of the inertial measurement unit is dynamically calibrated online to block the propagation of yaw error along the kinematic chain; Based on human posture parameters, forward kinematics is used to reconstruct the three-dimensional joint positions of the whole body and the human body mesh model in real time.

[0010] Optionally, encoding the three-dimensional spatial location anchor point, acceleration signal, and rotation signal into a multimodal observation feature vector includes: The multimodal observation feature vector is obtained by splicing together the three-dimensional displacement vectors of the left and right wrists of the head in the reference frame, the acceleration signals of all inertial measurement units, and the six-dimensional rotation representations of all inertial measurement units.

[0011] Optionally, constructing the global structured condition vector includes: The user's human body shape parameters and initial motion state summary are concatenated to obtain a global structured condition vector; where the human body shape parameters are used to define the proportion of bone length, and the initial motion state summary is used to provide the kinematic prior for the starting point of the inference window.

[0012] Optionally, scaling and offsetting the activation values ​​of each processing block using a global structured conditional vector through an adaptive layer normalization mechanism includes: Adaptive layer normalization is employed to inject the global structured conditional vector into each processing block of the transformer network, thereby controlling the scaling and offset adjustment of neuron activation values. This allows the network to adaptively correct the mapping relationship between rotation observation and spatial displacement according to the user's body proportions.

[0013] Optionally, generating a frame-by-frame validity mask based on the visibility tag includes: A mask matrix is ​​generated based on the time-varying validity matrix. The validity matrix contains binary labels for the visibility of left-hand anchors, binary labels for the visibility of right-hand anchors, and all-pass labels for all inertial measurement unit channels. When the anchor visibility label is valid, the corresponding anchor channel value is retained; when the anchor visibility label is invalid, the corresponding anchor channel value is set to zero.

[0014] Optionally, online dynamic calibration of the orientation of the inertial measurement unit includes: A stable head orientation reference is determined using the visual inertial pose output by the head-mounted display device; Based on the stable head-mounted reference, an orientation calibration relationship is established between the head reference coordinate system and the inertial measurement unit coordinate system. The heading of the head-mounted display device is used as an absolutely correct global reference. During real-time operation, the head is kept anchored. The inertial measurement units of other nodes are updated online through rotation diversity triggering rules, so that the orientation is aligned to the head-mounted reference system in real time.

[0015] Optionally, the method further includes: Training data is generated through virtual-real transfer simulation. During the simulation, a semi-ellipsoidal visibility model is constructed based on the field of view parameters of the head-mounted display device. When the spatial vector of the hand relative to the head satisfies the geometric constraints set by the semi-ellipsoidal visibility model, the corresponding anchor point visibility label is set to valid; otherwise, it is set to invalid.

[0016] A second aspect of the present invention provides a real-time inertial motion capture system based on enhanced position anchors, comprising: The acquisition module is used to acquire the acceleration and rotation signals output by the inertial measurement unit, as well as the three-dimensional spatial position anchor point of at least one limb end relative to the head reference frame and the visibility tag corresponding to the spatial position anchor point obtained from the head-mounted display device. The encoding module is used to encode three-dimensional spatial position anchor points, acceleration signals, and rotation signals into multimodal observation feature vectors; The condition construction module is used to obtain the user's human body shape parameters and the initial motion state summary of the current inference window, and construct a global structured condition vector. The mask control module is used to generate a frame-by-frame validity mask based on the visibility label, and to perform gating processing on the spatial location anchor point channel in the multimodal observation feature vector using the frame-by-frame validity mask; The network inference module inputs the gated multimodal observation feature vector into a transformer network containing multiple processing blocks. It uses the global structured conditional vector to scale and offset the activation value of each processing block through an adaptive layer normalization mechanism, and outputs human posture parameters, joint position parameters, and motion velocity parameters. The calibration module is used to perform online dynamic calibration of the orientation of the inertial measurement unit using the visual-inertial pose output by the head-mounted display device as a stable head orientation reference. The reconstruction module is used to reconstruct the three-dimensional joint positions of the whole body and the human body mesh model in real time through forward kinematics based on the human posture parameters, the joint position parameters and the motion velocity parameters.

[0017] A third aspect of the present invention provides a real-time inertial motion capture device based on enhanced position anchors, comprising: One or more processors; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time inertial motion capture method based on position anchor enhancement as described in any of the preceding claims.

[0018] A fourth aspect of the present invention provides a computer storage medium for storing a program, which, when executed, is used to implement the real-time inertial motion capture method based on position anchor point enhancement as described in any of the preceding claims.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) Significantly improved end-effector positioning accuracy. By introducing spatial position anchors for the limb end (such as the wrist) provided by a head-mounted display device into the sparse inertial kinematic chain, geometric constraints are directly applied to the coordinates of the limb end, effectively suppressing the end-effector position error caused by the cascade amplification of forward kinematics.

[0020] (2) Completely eliminate global yaw drift. By using the visual inertial odometry output of the head-mounted display as the absolute heading reference, the inertial measurement unit is dynamically calibrated online, and the whole-body attitude reference is transferred from the root IMU which is prone to drift to the stable head-mounted display. This structurally blocks the propagation of yaw error along the kinematic tree and achieves long-term global orientation consistency.

[0021] (3) The robustness to real-world environments is greatly enhanced. By generating a mask matrix based on visibility labels, strong geometric constraints are applied when the anchor point is valid and the corresponding channel input is masked when it is invalid. This enables the system to automatically identify and smoothly handle intermittent missing anchor point signals, avoiding attitude collapse or jitter caused by observation jumps.

[0022] (4) Cross-user geometric consistency. By constructing a global structured conditional vector with human body shape parameters and initial motion state summary, and injecting it into the transformer network with an adaptive layer normalization mechanism, the network can adaptively match the bone ratio of different users, eliminating unnatural distortions such as excessive bending or stretching of limbs caused by body shape differences.

[0023] (5) Lightweight and high precision, no external infrastructure required. The system relies only on the sensing capabilities of the sparse inertial measurement unit and the head-mounted display device itself, without the need to deploy external camera arrays, base stations or optical markers. It achieves high-precision motion capture while maintaining lightweight and portability, significantly reducing deployment costs and the barrier to entry. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating an exemplary application scenario provided by an embodiment of the present invention; Figure 2 A flowchart illustrating a real-time inertial motion capture method based on enhanced position anchor points, provided in an embodiment of the present invention; Figure 3 A schematic diagram of a real-time inertial motion capture system based on enhanced position anchor points, provided for an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0026] This invention provides a real-time inertial motion capture method and system based on enhanced position anchors. By introducing human shape parameters as geometric constraints and utilizing a hierarchical modulation architecture inspired by Diffusion Transformer (DiT), it achieves precise alignment of sparse inertial observations with spatial anchors on a physical scale.

[0027] To facilitate understanding, the application scenarios of the embodiments of the present invention will be introduced first.

[0028] Can be combined Figure 1 To understand, Figure 1 This is a schematic diagram illustrating an exemplary application scenario provided by an embodiment of the present invention. The user wears a head-mounted display device (e.g., Apple Vision Pro) and a motion capture suit integrated with a sparse inertial measurement unit (SMU) array. The motion capture suit can be deployed in two configurations: a 6-IMU tight-fitting strap layout (for scientific benchmark comparison) and a 10-IMU loose everyday clothing layout (concealing the sensors within a sports jacket for "disruptive" wear). The user can freely perform complex actions such as basketball, Tai Chi, boxing, and fine hand interactions (e.g., disconnecting cables, folding clothes) in open indoor and outdoor environments. The head-mounted display device outputs the six-DOF head pose in real time via a built-in visual inertial odometry system, and automatically generates the three-dimensional spatial position (i.e., position anchor point) of the left / right wrist relative to the bridge of the nose reference frame and its visibility label when the hand enters the field of view. The system's backend host synchronously acquires IMU acceleration and rotation signals, fuses the position anchor points, and then uses a deep learning inference engine to reconstruct a high-fidelity human body mesh model conforming to the SMPL standard in real time, outputting the results for VR collaboration, sports rehabilitation monitoring, or robot teleoperation.

[0029] The hardware platform of this invention aims to build an "unobtrusive motion capture suit" that combines high-precision posture perception with extremely low wear invasiveness, in order to achieve a deep balance between dynamic capture robustness and lightweight deployment. It consists of two parts working together: an unobtrusive motion capture suit and a head-mounted display device.

[0030] Motion capture clothing employs an inertial sensor network integrated with flexible fabric. Each sensor node utilizes a nine-axis inertial measurement unit (such as the Bosch BNO085) with on-chip signal preprocessing capabilities, whose miniaturization and low cost support high-density covert deployment. To address the radio frequency interference issues caused by overlapping limbs in traditional wireless sensor networks, each node is seamlessly sewn into the lining of the clothing using extremely fine flexible wires, forming a sensor network imperceptible to both sight and touch. Data streams converge to a central aggregation motherboard, which uses a wired polling mechanism to acquire the raw observation stream in real time and perform lightweight frame encapsulation. The system is physically divided into decoupled upper limb sensing subnets and lower limb sensing subnets. Each subnet is equipped with an independent low-power data processing unit, which transmits the preprocessed local inertial frames to the backend host in real time via Bluetooth Low Energy (BLE). The host uses a dual-channel serial receiver to concurrently monitor the global data streams of the upper and lower limbs and performs timing alignment based on sensor hardware timestamps. The system's global sampling rate is stabilized at 30Hz.

[0031] Head-mounted display devices: These employ devices with visual inertial odometry (VIO) capabilities (such as the Apple Vision Pro). This device not only outputs the head's six-DOF global pose but also automatically generates three-dimensional displacement vectors (i.e., 6D position anchors) of the left and right wrists relative to the head reference frame (typically defined near the bridge of the nose) when the hands enter the camera's field of view. It also provides a visibility label for each anchor point in real time (1 for visible, 0 for invisible).

[0032] See Figure 2 This figure is a schematic flowchart of a real-time inertial motion capture method based on position anchor point enhancement provided by an embodiment of the present invention. The real-time inertial motion capture method based on position anchor point enhancement provided by this embodiment of the present invention can be implemented, for example, through the following steps S201-207.

[0033] S201: Acquire acceleration and rotation signals output by the inertial measurement unit, as well as at least one three-dimensional spatial position anchor point of a limb end relative to the head reference frame and the visibility tag corresponding to the spatial position anchor point obtained from the head-mounted display device.

[0034] In this embodiment of the invention, the system synchronously acquires acceleration and rotation signals (represented using 6D continuous rotation) from all Inertial Measurement Unit (IMU) nodes. It obtains three-dimensional displacement vectors (left hand 3D, right hand 3D) of the left and right wrists relative to the bridge of the nose reference frame from the head-mounted display device, along with corresponding visibility labels (1 indicating visible, 0 indicating invisible).

[0035] In one implementation of this invention, no specific restrictions are placed on the data source and physical location expansion of the position anchor point. In this invention, the hand anchor point can also be provided by devices such as smartphones (using ARKit / ARCore to track limbs in real time), smartwatches and bracelets (providing the relative distance between joints based on UWB ultra-wideband or acoustic ranging), and lightweight fisheye / RGB-D cameras worn on the chest.

[0036] The anchor point location can be extended to the whole body (such as the lower limbs). Addressing the limitation of minimalist wrist anchors in directly constraining lower limb gait, the anchor point fusion mechanism of this invention (such as DiT modulation and CatMask masking) can be smoothly extended to lower limb anchor points. For example, smart insoles (using piezoresistive sensors to detect foot-ground contact as a zero-velocity anchor point), ankle UWB tags, or the 3D coordinates of the foot extracted by the head-mounted display's downward-facing camera can be used as "lower limb anchor points," as long as they provide a clear spatial reference for the inertial kinetic chain to reduce the zero-space dimension.

[0037] S202: Encode the three-dimensional spatial location anchor point, acceleration signal, and rotation signal into a multimodal observation feature vector.

[0038] In this embodiment of the invention, the three-dimensional displacement vectors of the left and right wrists in the head relative to the reference frame, the acceleration signals of all inertial measurement units, and the six-dimensional rotation representations of all inertial measurement units are spliced ​​together to obtain the multimodal observation feature vector.

[0039] Specifically, to simultaneously capture the dynamic characteristics and kinematic space constraints of the human body, the system encodes multi-source sensor signals into a unified observation feature vector in each frame. The algorithm input consists of three parts: the position anchor point in the head-oriented reference frame, linear acceleration, and rotational attitude. For a configuration containing N IMU sensors, the... Frame input features Represented as: ; in, Represents the three-dimensional displacement vectors of the left and right wrists relative to the center reference frame of the bridge of the nose, provided in real time by the head-mounted device; and These correspond to the acceleration signal captured by the inertial sensor and the orientation information processed by 6D rotation representation, respectively. This joint representation method ensures that the system can establish the correlation between "local rotation" and "end position" during the feature extraction stage, laying the data foundation for subsequent geometric consistency calculation.

[0040] S203: Obtain the user's human body shape parameters and the initial motion state summary of the current inference window, and construct a global structured condition vector, wherein the initial motion state summary includes the initial velocity, initial position and initial posture.

[0041] In this embodiment of the invention, the user's human body shape parameters and the initial motion state summary are concatenated to obtain a global structured condition vector; wherein, the human body shape parameters are used to define the proportion of bone length, and the initial motion state summary is used to provide the kinematic prior of the inference window starting point.

[0042] Specifically, in addressing the issue of universality across subjects, this invention incorporates human body shape parameters. A network architecture is introduced as a "geometric bridge" connecting rotation observations and position constraints. Due to the significant differences in skeleton proportions among different users, the end-effector spatial displacements corresponding to the same rotation increment are fundamentally different. Therefore, the algorithm constructs a globally structured condition vector. Used to dynamically adjust physical dimensions at a deep network level: ; in, The parameters define shape features such as bone length; This represents a state summary of the inference window's initial frame.

[0043] S204: Generate a frame-by-frame validity mask based on the visibility label, and use the frame-by-frame validity mask to perform gating processing on the spatial location anchor point channel in the multimodal observation feature vector.

[0044] In this embodiment of the invention, a mask matrix is ​​generated based on a time-varying validity matrix. The validity matrix includes left-hand anchor visibility binary labels, right-hand anchor visibility binary labels, and all-pass labels for all inertial measurement unit channels. When an anchor visibility label is valid, the corresponding anchor channel value is retained; when an anchor visibility label is invalid, the corresponding anchor channel value is set to zero.

[0045] Specifically, considering that in practical applications, the hand position anchor point often experiences frequent interruptions due to exceeding the head-mounted display's field of view (FOV) or being obstructed by the body, this invention designs a feature-level gate masking mechanism (CatMask). This mechanism defines a time-varying validity matrix. The input feature stream is dynamically filtered to explicitly distinguish between "missing data" and "zero values": ; in, and A binary label for the visibility of the corresponding hand anchor point (provided in real-time by the head-mounted display tracking status) is used. When the anchor point signal is valid, the gate opens, and the model applies strong geometric constraints to suppress the accumulated error of the IMU. When the anchor point signal is missing (e.g., the hand moves out of the viewpoint), the gate closes, the corresponding anchor point channel in the feature vector is masked, and the model gracefully degenerates into smooth inference relying on inertial temporal features and the first frame state summary. This mechanism greatly enhances the robustness of the system in real-world open-world applications, avoiding instantaneous attitude jitter or collapse caused by observation jumps.

[0046] S205: The gated multimodal observation feature vector is input into a transformer network containing multiple processing blocks. The activation values ​​of each processing block are scaled and offset by an adaptive layer normalization mechanism using a global structured conditional vector, and the human posture parameters, joint position parameters, and motion velocity parameters are output.

[0047] In this embodiment of the invention, adaptive layer normalization is employed to inject the global structured condition vector into each processing block of the transformer network, and to perform controlled scaling and offset adjustment on the neuron activation values, so that the network adaptively corrects the mapping relationship between rotation observation and spatial displacement according to the user's body proportion.

[0048] Specifically, an Adaptive Layer Normalization (AdaLN) mechanism is employed to inject a globally structured conditional vector into each processing block of the Transformer. By controlling the scaling and offset of neuron activation values, the network can adaptively adjust the rotation-position mapping according to the user's body proportions, ensuring that users of different heights and arm lengths can reconstruct physically reasonable poses. This design not only guarantees capture accuracy for users of different heights but also significantly suppresses non-physical distortions caused by long kinetic chains during FK propagation.

[0049] In one implementation of this invention, the feature fusion part of the multimodal fusion network architecture can also be implemented by a standard Transformer encoder-decoder architecture (such as using location anchors as queries and IMU features as keys / values ​​for cross-attention calculation), graph convolutional networks (GCN) or graph attention networks (GAT, which transmits information between nodes along the physical topology of the human skeleton), and a hybrid structure of long short-term memory networks (LSTM / GRU) and self-attention mechanisms. It can also achieve effective mapping of user body proportions and multimodal spatiotemporal features.

[0050] In terms of the catmask mechanism for handling intermittent anchor observations, in addition to using feature-dimensional gate masks to directly block invalid channels, adaptive attention fusion based on confidence weights (dynamically adjusting weights according to the tracking quality feedback from sensors), state prediction and update mechanism based on extended Kalman filter (EKF) (relying on inertial pure prediction when there are no observations, and performing state correction when there are observations), or introducing generative adversarial networks (GANs) and masked autoencoders (MAEs) to perform implicit interpolation and feature reconstruction on missing anchor frames. The above methods can be selected according to the specific system implementation and training data conditions. The purpose is to reduce the interference of invalid observations on the network inference results when anchor observations are missing or restored.

[0051] S206: Use the visual inertial odometry output of the head-mounted display device as the absolute heading reference to perform online dynamic calibration of the orientation of the inertial measurement unit, thereby blocking the propagation of yaw error along the kinematic chain.

[0052] In this embodiment of the invention, the drift rotation and installation offset of the head inertial measurement unit are forcibly set as a unit matrix, and the heading of the head-mounted display device is used as an absolutely correct global reference. During real-time operation, the head is kept anchored. The inertial measurement units of the remaining nodes are updated online through rotation diversity triggering rules, so that the orientation is aligned to the head-head reference frame in real time.

[0053] Specifically, addressing the most challenging issue in inertial motion capture—yaw drift—this invention proposes a head-anchored online calibration mechanism (TIC). Unlike traditional schemes that use the pelvic IMU as the root node, this invention shifts the calibration reference point to a head-mounted display device with stable VIO output. During calibration, the system forcibly sets the drift rotation of the head IMU. and installation offset For identity matrix: ; This means that the head-mounted display's heading is considered an absolutely correct global benchmark. In real-time operation, this mechanism keeps the head anchored and does not perform TIC updates, i.e.: .

[0054] Through this architectural improvement, yaw error is blocked at the head node, preventing it from spreading throughout the body along the kinematic chain. The IMUs of the remaining nodes are then updated online using rotation-diversity triggering rules, ensuring their orientation is aligned with a stable head-oriented reference frame in real time. This achieves motion consistency for several hours in a global coordinate system.

[0055] In one implementation of this invention, the stable reference source for global coordinate calibration strategy and output representation is not limited to head-mounted display devices. It can also be an independent visual odometry device (such as a tracker) mounted on the pelvis, or a fixed reference IMU (IMU) on the vehicle body / environment as the "root node" for coordinate system transformation in specific traffic / work scenarios. At the final output and supervision level of the model, in addition to regressing SMPL parameters, it is also possible to directly regress the 3D keypoint coordinate sequence of a standard human skeleton, the quaternions / 6D rotation vectors of independent joints, or use other parametric human models such as SMPL-X or STAR. Furthermore, during the training phase, contrastive learning, weak supervision, or kinematic physical losses (such as clipping penalties or constant bone length penalties) can be introduced for optimization.

[0056] S207: Based on human posture parameters, joint position parameters, and motion velocity parameters, forward kinematics is used to reconstruct the three-dimensional joint positions and human mesh model of the whole body in real time.

[0057] In this embodiment of the invention, based on the SMPL pose parameters output by the network and combined with shape parameters, the three-dimensional spatial positions of 24 key joints throughout the body are calculated using standard forward kinematics formulas, and a vertex-level human body mesh model is further generated. The output data also includes auxiliary information such as instantaneous joint velocities for use in upper-level applications.

[0058] In one implementation of this invention, at the system integration level, the AIP employs a sliding window buffer mechanism for streaming inference. The inference engine operates at a fixed scheduling frequency of 30 FPS to ensure that the physical meaning of time-difference signals (such as acceleration) remains consistent in timing. The system exports the pre-trained model to ONNX format and utilizes hierarchical linearization technology for efficient deployment on the CPU. Under standard Intel i7 hardware, the system can achieve a peak update rate of over 40Hz, reserving sufficient computational margin for multi-device communication. Finally, the pose parameters, joint positions, and velocity vectors output by the network are rendered in real-time into a 3D human body mesh model through the forward kinematics module, achieving an extremely low-latency interactive experience.

[0059] In one implementation of this invention, training data is generated through virtual-real migration simulation. During the simulation, a semi-ellipsoidal visibility model is constructed based on the field of view parameters of the head-mounted display device. When the spatial vector of the hand relative to the head satisfies the geometric constraints set by the semi-ellipsoidal visibility model, the corresponding anchor point visibility label is set to valid; otherwise, it is set to invalid.

[0060] Specifically, in order to ensure the applicability of the algorithm in real-world scenarios, this invention solves the inherent noise and drift problems of inertial sensors by constructing a "virtual-reality" transfer bridge.

[0061] Due to the extreme scarcity of motion capture data with positional anchors in the real world, this invention constructs a high-fidelity Sim-to-Real simulation layer based on the AMASS dataset. During the simulation, positional anchors are synthesized by simulating the physical characteristics of the head-mounted display: First, a nose bridge center reference frame is established on the surface of the SMPL model, and the 6D displacement vector of the hand relative to this reference frame is calculated based on forward kinematics (FK). To simulate the intermittent missing characteristics of real sensors, this invention introduces a semi-ellipsoidal visibility model. Given the spatial vector of the hand relative to the head... Anchor labels are set to valid only if they satisfy the following geometric constraints: ):

[0062]

[0063] ; in, Based on the azimuth angle of the head-mounted display's field of view ( ) and elevation angle ( ) setting. Furthermore, yaw angle drift is injected into the original inertial flow. and installation deviation This simulates the non-ideal characteristics of consumer-grade IMUs, ensuring that the trained model has extremely strong noise resistance.

[0064] In one implementation of this invention, in order to verify the motion reconstruction performance of the Anchored Inertial Poser (AIP) system in complex environments, a thorough quantitative evaluation and qualitative comparison were conducted on synthetic datasets and large-scale real-world datasets.

[0065] The evaluation of this system is primarily based on two types of datasets: first, Sim-to-Real data synthesized using the AMASS large-scale motion database, used for pre-training the model in ideal and noisy scenarios; and second, the AIP-DB real-world dataset independently collected by this invention. The AIP-DB dataset contains motion sequences from 10 subjects (covering different body types and genders) under various wearing conditions, with a total duration of approximately 5 hours (about 326,000 frames). The dataset is divided into two subsets: a controlled subset using a 6-IMU tight-fitting layout (comparable to the commercial system Noitom), and a real-world application subset using a 10-IMU loose-fitting clothing layout. The dataset covers 37 complex motion categories, including sports (basketball, combat sports), slow controlled movements (Tai Chi, Baduanjin), and fine hand interactions (folding clothes, unplugging cables), and utilizes an optical motion capture system to obtain millimeter-level global reference ground truth values.

[0066] In terms of algorithm training and implementation details, the system uses the MSE loss function for joint supervised training of joint velocity, position, and SMPL pose parameters. The optimizer chosen is Adam, and the learning rate is set to [value missing]. The batch size is 512, and the temporal inference window length T is set to 128. To enhance the robustness of the model, the initial state conditions are adjusted during training. A 10% random dropout was applied. In terms of hardware, the model was trained on a workstation equipped with an NVIDIA RTX GPU and exported to ONNX format for real-time inference testing on a standard Intel i7 CPU.

[0067] To comprehensively evaluate the quality of motion capture, this study selected five core evaluation indicators: (1) SIP error, used to measure the rotational error of the hip and shoulder; (2) angular error, the average Euler angle deviation of all joint postures; (3) position error, the three-dimensional coordinate offset of all joints in the global coordinate system; (4) mesh error, the average distance between vertices of the reconstructed human three-dimensional mesh; and (5) jitter, used to measure the temporal smoothness of the motion trajectory. This multi-dimensional evaluation system ensures that the system is not only accurate in static position but also conforms to the laws of human kinematics in dynamic continuity.

[0068] This invention provides a comprehensive comparison between the AIP system and current mainstream motion capture algorithms. The comparison covers two technical approaches: one is the method based on inertial sensors, including TIP, PIP, PNP and DynaIP; the other is the hybrid sensing method HMD-Poser, which has the closest hardware configuration to this system.

[0069] In the quantitative analysis based on the real-world AIP-DB-6IMUs dataset (see Table 1), AIP showed significant advantages across all core evaluation metrics.

[0070] Table 1: Quantitative comparison of AIP system and existing methods on the AIP-DB-6IMUs dataset

[0071] To further verify the system's generalization ability, this invention conducted comparative tests on the publicly available DIP dataset (see Table 2). Since the DIP dataset lacks real head-mounted display reference anchors, this invention, following industry standard practice, synthesized hand position observations using real motion values. Experimental results show that AIP maintains its leading position in synthetic data scenarios, demonstrating that the algorithm's fusion mechanism is not only adapted to specific hardware noise but also possesses general physical constraint capabilities across datasets.

[0072] Table 2 Quantitative comparison of AIP system and existing methods on the DIP dataset.

[0073] At the qualitative visual contrast level, AIP's reconstruction results exhibit extremely high fidelity at the extremities. In contrast, existing pure inertial baseline methods often exhibit significant joint offset and heading drift when performing vigorous movements or long-path reaching actions. By introducing head-oriented position anchors and online dynamic calibration, AIP ensures that the virtual character's hands accurately hit spatial targets and maintain consistency in the global coordinate system throughout long-term capture, effectively eliminating common motion capture artifacts such as "slipping" and "over-flexion of joints."

[0074] To further verify the contribution of each core component in the AIP system to the final reconstruction accuracy, this invention conducted ablation experiments, focusing on the necessity of human body shape parameters, temporal masking mechanisms, and initial state priors in dealing with complex motion capture environments.

[0075] Table 3 Comparison of ablation experiment results of AIP system on different datasets

[0076] First, human body shape parameters ( The injection mechanism of shape parameters has been proven to be crucial for maintaining geometric consistency. Experimental data shows that when conditional injection of shape parameters is removed, the system's positional error increases from 4.72 cm to 5.23 cm. From a physical logic perspective, without a precise understanding of the subject's bone length ratio, the model cannot correctly establish the constraint relationship between "joint rotation" and "end-effector alignment position." Qualitative analysis indicates that, in the absence of... In the case of parameters, in order to forcibly "explain" the positional anchor points, the model often causes unnatural bending of the elbow or shoulder, resulting in significant biological structural violations.

[0077] Secondly, the feature-level gate mask (CatMask) plays a crucial and robust role in handling missing observations. When the masking mechanism is removed and missing anchor observations are directly filled with zeros, the system performance deteriorates sharply, with the SIP error on the AIP-DB-6IMUs dataset increasing from [previous value]. Soaring to This result demonstrates that explicitly modeling the "data missing" state can effectively prevent the network from misinterpreting invalid zero values ​​as real geometric constraints, thereby ensuring the system's smooth degradation capability under extreme conditions such as the hand being removed from the field of view.

[0078] Finally, a summary of the initial window state ( This provides the necessary stability for long-term time-series inference. (Remove) Subsequently, especially in sequences with sparse anchor point observations, the drift phenomenon in window prediction is significantly aggravated. Experiments demonstrate that by providing compact kinematic priors at the window starting point, the search space of the model in the initial solution stage can be significantly reduced, thereby improving the coherence of real-time streaming inference. In summary, location anchor points are the main driving force for accuracy improvement, while the collaborative work of the aforementioned algorithm modules ensures the stable implementation of this accuracy improvement in real-world complex wearable scenarios.

[0079] Beneficial effects: (1) Significantly improved the spatial positioning accuracy of the joints at the extremities. This invention theoretically reduces the null-space dimension of the dynamic Jacobian matrix by introducing a very small number of position anchor points (such as the wrist positions) into a sparse inertial kinematic chain, thereby utilizing position constraints. Compared to traditional pure inertial schemes that rely on step-by-step calculations of forward kinematics (FK) and suffer from amplified errors, this invention can directly anchor the absolute spatial coordinates of distal joints such as the hand, effectively suppressing the accumulation of errors along the long "shoulder-elbow-wrist" kinematic chain. Experimental results show that this technology significantly improves the positioning accuracy of the wrist and ankle, enabling motion-captured characters to perform touch, grasp, and complex gesture interactions with higher fidelity.

[0080] (2) It completely solves the inherent global yaw drift problem of inertial motion capture. This invention innovatively proposes a dynamic online calibration mechanism of "head-mounted anchoring," migrating the reference benchmark for the whole-body posture from the root IMU, which is susceptible to magnetic field interference and rotational drift, to a head-mounted display device with a stable visual inertial odometry (VIO) reference. This architectural design structurally blocks the systematic propagation of yaw drift along the human dynamics chain. In long-term capture missions, the system can always maintain a heading reference consistent with the physical world, ensuring the long-range motion consistency of the virtual character in the global coordinate system and supporting continuous stable operation for several hours.

[0081] (3) Enhanced system robustness and smoothness in real complex environments. By leveraging a feature-level CatMask masking gate mechanism, this invention can accurately handle the intermittent loss of position anchors that frequently occurs in practical use (such as a hand moving out of or into the head-mounted display's field of view). The system can automatically identify the validity of the observed signal, providing strong geometric constraints when anchors are available, and performing smooth degradation inference based on inertial temporal features and the prior of the first frame state summary when anchors are lost. This design avoids attitude collapse or sudden jitter caused by signal jumps, greatly improving the system's practical reliability in open-world, uncontrolled scenarios.

[0082] (4) Achieved geometric consistency and physical plausibility across subjects. By employing a DiT-inspired structured conditional injection architecture, this invention deeply integrates SMPL human shape parameters into the modulation of each layer of the neural network. This design ensures that the model can automatically perceive and adapt to the bone length ratios of different subjects, achieving a high degree of unity between "inertial rotation" and "spatial position" on a physical scale. It effectively solves the unnatural distortions caused by joint over-flexion and limb stretching due to differences in body shape in traditional methods, ensuring that the reconstructed movements conform to kinematic constraints and possess extremely high biological realism.

[0083] (5) It balances motion capture performance with the convenience of everyday wear. This invention achieves professional-grade motion capture accuracy by cleverly reusing the sensing capabilities of existing head-mounted displays without adding additional external infrastructure (such as infrared camera arrays or base stations). The sensor nodes are lightweight and have a very small hardware footprint, and can be integrated into tight-fitting clothing or loose everyday garments. This "plug-and-play" feature not only reduces system deployment costs but also significantly optimizes the user's wearing experience, clearing the way for large-scale, low-barrier-to-entry high-precision motion capture applications.

[0084] Based on the methods provided in the above embodiments, this invention also provides a real-time inertial motion capture system based on enhanced position anchors. The following describes the real-time inertial motion capture system based on enhanced position anchors in conjunction with the accompanying drawings.

[0085] See Figure 3 The figure is a schematic diagram of the structure of a real-time inertial motion capture system based on position anchor point enhancement provided in an embodiment of the present invention.

[0086] The real-time inertial motion capture system 300 based on position anchor point enhancement provided in this embodiment of the invention includes: an acquisition module 301, an encoding module 302, a condition construction module 303, a network inference module 304, a mask control module 305, a calibration module 306, and a reconstruction module 307.

[0087] The acquisition module 301 is used to acquire the acceleration signal and rotation signal output by the inertial measurement unit, as well as the three-dimensional spatial position anchor point of at least one limb end relative to the head reference frame and the visibility tag corresponding to the spatial position anchor point obtained from the head-mounted display device. Encoding module 302 is used to encode three-dimensional spatial position anchor points, acceleration signals and rotation signals into multimodal observation feature vectors; The condition construction module 303 is used to obtain the user's human body shape parameters and the initial motion state summary of the current inference window, and construct a global structured condition vector. The mask control module 304 is used to generate a frame-by-frame validity mask based on the visibility label, and to perform gating processing on the spatial location anchor point channel in the multimodal observation feature vector using the frame-by-frame validity mask; The network inference module 305 inputs the gated multimodal observation feature vector into a transformer network containing multiple processing blocks. It uses a global structured conditional vector to scale and offset the activation value of each processing block through an adaptive layer normalization mechanism, and outputs human posture parameters, joint position parameters, and motion velocity parameters. The calibration module 306 is used to perform online dynamic calibration of the orientation of the inertial measurement unit using the visual inertial pose output by the head-mounted display device as a stable head orientation reference. Reconstruction module 307 is used to reconstruct the three-dimensional joint positions of the whole body and the human body mesh model in real time through forward kinematics based on human posture parameters, joint position parameters and motion velocity parameters.

[0088] In one possible implementation, the encoding module 302 is specifically used for: Encoding the three-dimensional spatial location anchor point, acceleration signal, and rotation signal into a multimodal observation feature vector includes: The multimodal observation feature vector is obtained by splicing together the three-dimensional displacement vectors of the left and right wrists of the head in the reference frame, the acceleration signals of all inertial measurement units, and the six-dimensional rotation representations of all inertial measurement units.

[0089] In one possible implementation, the condition construction module 303 is specifically used for: Constructing a global structured condition vector includes: The user's human body shape parameters and initial motion state summary are concatenated to obtain a global structured condition vector; where the human body shape parameters are used to define the proportion of bone length, and the initial motion state summary is used to provide the kinematic prior for the starting point of the inference window.

[0090] In one possible implementation, network inference module 304 is specifically used for: The activation values ​​of each processing block are scaled and offset using a global structured conditional vector through an adaptive layer normalization mechanism, including: Adaptive layer normalization is employed to inject the global structured conditional vector into each processing block of the transformer network, thereby controlling the scaling and offset adjustment of neuron activation values. This allows the network to adaptively correct the mapping relationship between rotation observation and spatial displacement according to the user's body proportions.

[0091] In one possible implementation, the mask control module 305 is specifically used for: Generating a mask matrix based on visibility tags includes: A mask matrix is ​​generated based on the time-varying validity matrix. The validity matrix contains binary labels for the visibility of left-hand anchors, binary labels for the visibility of right-hand anchors, and all-pass labels for all inertial measurement unit channels. When the anchor visibility label is valid, the corresponding anchor channel value is retained; when the anchor visibility label is invalid, the corresponding anchor channel value is set to zero.

[0092] In one possible implementation, calibration module 306 is specifically used for: Online dynamic calibration of the orientation of the inertial measurement unit includes: The drift rotation and installation offset of the head inertial measurement unit are forcibly set to the unit matrix, and the heading of the head-mounted display device is used as the absolutely correct global reference. The head is kept anchored in real time. The inertial measurement units of other nodes are updated online through rotation diversity triggering rules, so that the orientation is aligned to the head-head reference frame in real time.

[0093] In one possible implementation, a simulation training module is also included, specifically for: Training data is generated through virtual-real transfer simulation. During the simulation, a semi-ellipsoidal visibility model is constructed based on the field of view parameters of the head-mounted display device. When the spatial vector of the hand relative to the head satisfies the geometric constraints set by the semi-ellipsoidal visibility model, the corresponding anchor point visibility label is set to valid; otherwise, it is set to invalid.

[0094] Since the real-time inertial motion capture system 300 based on position anchor point enhancement is a system corresponding to the real-time inertial motion capture method based on position anchor point enhancement provided in the above method embodiments, the specific implementation of each unit of the real-time inertial motion capture system 300 based on position anchor point enhancement is based on the same concept as in the above method embodiments. Therefore, for the specific implementation of each unit of the real-time inertial motion capture system 300 based on position anchor point enhancement, please refer to the description of the real-time inertial motion capture method based on position anchor point enhancement in the above method embodiments, and it will not be repeated here.

[0095] This invention also provides a real-time inertial motion capture device based on enhanced position anchors, the device comprising: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to perform the real-time inertial motion capture method based on enhanced position anchors mentioned in the above embodiments.

[0096] It should be noted that the hardware structure of the real-time inertial motion capture device based on enhanced position anchor points provided in the embodiments of the present invention can be as follows: Figure 4 The structure shown, Figure 4This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0097] Please see Figure 4 As shown, device 400 includes: processor 410, communication interface 420, and memory 430. The number of processors 410 in device 400 can be one or more. Figure 4 Taking a processor as an example, in this embodiment of the invention, the processor 410, communication interface 420, and memory 430 can be connected via a bus system or other means. Figure 4 Taking the connection between China and Israel via the bus system 440 as an example.

[0098] Processor 410 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 410 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0099] The memory 430 may include volatile memory, such as random-access memory (RAM); the memory 430 may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 430 may also include a combination of the above types of memory.

[0100] Optionally, the memory 430 stores an operating system and programs, executable modules, or data structures, or subsets thereof, or extended sets thereof. The programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and handling hardware-based tasks. The processor 410 can read the programs in the memory 430 to implement the real-time inertial motion capture method based on position anchor point enhancement provided in this embodiment of the invention.

[0101] The bus system 440 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system 440 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0102] This invention also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the real-time inertial motion capture method based on enhanced position anchors mentioned in the above embodiments.

[0103] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the real-time inertial motion capture method based on enhanced position anchors mentioned in the above embodiments.

[0104] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A real-time inertial motion capture method based on position anchor point enhancement, characterized in that, include: Acceleration and rotation signals output by the inertial measurement unit are acquired, as well as at least one three-dimensional spatial position anchor point of a limb end relative to the head reference frame and the visibility tag corresponding to the spatial position anchor point are obtained from the head-mounted display device. The three-dimensional spatial position anchor point, the acceleration signal, and the rotation signal are encoded into a multimodal observation feature vector; Obtain the user's human body shape parameters and the initial motion state summary of the current inference window, and construct a global structured condition vector, wherein the initial motion state summary includes initial velocity, initial position and initial posture; A frame-by-frame validity mask is generated based on the visibility label, and the spatial location anchor point channel in the multimodal observation feature vector is gated using the frame-by-frame validity mask. The gated multimodal observation feature vector is input into a transformer network containing multiple processing blocks. The activation values ​​of each processing block are scaled and offset by an adaptive layer normalization mechanism using the global structured condition vector, and the human posture parameters, joint position parameters, and motion velocity parameters are output. Using the visual inertial pose output by the head-mounted display device as a stable head orientation reference, the orientation of the inertial measurement unit is dynamically calibrated online to block the propagation of yaw error along the kinematic chain; Based on the human posture parameters, joint position parameters, and motion velocity parameters, forward kinematics is used to reconstruct the three-dimensional joint positions of the whole body and the human body mesh model in real time.

2. The method according to claim 1, characterized in that, Encoding the three-dimensional spatial anchor point, the acceleration signal, and the rotation signal into a multimodal observation feature vector includes: The multimodal observation feature vector is obtained by concatenating the three-dimensional displacement vectors of the left and right wrists of the head in the reference frame, the acceleration signals of all inertial measurement units, and the six-dimensional rotation representations of all inertial measurement units.

3. The method according to claim 1, characterized in that, The construction of the global structured condition vector includes: The user's human body shape parameters and the initial motion state summary are concatenated to obtain the global structured condition vector; wherein, the human body shape parameters are used to define the bone length ratio, and the initial motion state summary is used to provide the kinematic prior for the inference window starting point.

4. The method according to claim 1, characterized in that, The process of scaling and offsetting the activation value of each processing block using the global structured conditional vector through an adaptive layer normalization mechanism includes: Adaptive layer normalization is employed to inject the global structured conditional vector into each processing block of the transformer network, thereby controlling the scaling and offset adjustment of neuron activation values. This allows the network to adaptively correct the mapping relationship between rotational observation and spatial displacement according to the user's body proportions.

5. The method according to claim 1, characterized in that, The step of generating a frame-by-frame validity mask based on the visibility tag includes: The mask matrix is ​​generated based on the time-varying validity matrix, which includes left-hand anchor visibility binary labels, right-hand anchor visibility binary labels, and all-pass labels for all inertial measurement unit channels. When the anchor visibility label is valid, the corresponding anchor channel value is retained; when the anchor visibility label is invalid, the corresponding anchor channel value is set to zero.

6. The method according to claim 1, characterized in that, The online dynamic calibration of the orientation of the inertial measurement unit includes: A stable head orientation reference is determined using the visual inertial pose output by the head-mounted display device; Based on the stable head-mounted reference, an orientation calibration relationship is established between the head reference coordinate system and the inertial measurement unit coordinate system. The heading of the head-mounted display device is used as an absolutely correct global reference. During real-time operation, the head is kept anchored. The inertial measurement units of other nodes are updated online through rotation diversity triggering rules, so that the orientation is aligned to the head-mounted reference system in real time.

7. The method according to claim 1, characterized in that, The method further includes: Training data is generated through virtual-real migration simulation. During the simulation, a semi-ellipsoidal visibility model is constructed based on the field of view parameters of the head-mounted display device. When the spatial vector of the hand relative to the head satisfies the geometric constraints set by the semi-ellipsoidal visibility model, the corresponding anchor point visibility label is set to valid; otherwise, it is set to invalid.

8. A real-time inertial motion capture system based on position anchor point enhancement, characterized in that, include: The acquisition module is used to acquire acceleration and rotation signals output by the inertial measurement unit, as well as at least one three-dimensional spatial position anchor point of a limb end relative to the head reference frame and the visibility tag corresponding to the spatial position anchor point obtained from the head-mounted display device. The encoding module is used to encode the three-dimensional spatial position anchor point, the acceleration signal, and the rotation signal into a multimodal observation feature vector; The condition construction module is used to obtain the user's human body shape parameters and the initial motion state summary of the current inference window, and construct a global structured condition vector. The mask control module is used to generate a frame-by-frame validity mask based on the visibility label, and to perform gating processing on the spatial location anchor point channel in the multimodal observation feature vector using the frame-by-frame validity mask; The network inference module inputs the gated multimodal observation feature vector into a transformer network containing multiple processing blocks. It uses the global structured conditional vector to scale and offset the activation value of each processing block through an adaptive layer normalization mechanism, and outputs human posture parameters, joint position parameters, and motion velocity parameters. The calibration module is used to perform online dynamic calibration of the orientation of the inertial measurement unit using the visual-inertial pose output by the head-mounted display device as a stable head orientation reference. The reconstruction module is used to reconstruct the three-dimensional joint positions of the whole body and the human body mesh model in real time through forward kinematics based on the human posture parameters, the joint position parameters and the motion velocity parameters.

9. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Including instructions that, when run on a computer, cause the computer to perform the method described in any one of claims 1-7 above.

Images