An action capture method for a markerless dynamic occlusion scene

Abstract

The application discloses a motion capture method for a non-marked dynamic occlusion scene, and relates to the technical field of computer vision, and comprises the following steps: a mixed sensing array composed of multi-view visual sensors and sparse inertial measurement units is configured; after multi-modal preprocessing, a four-dimensional space-time correlation graph model is constructed, current frame visual key points, historical three-dimensional trajectories, inertial motion estimation and parameterized human body model prior are jointly coded into graph nodes and edge weights; by solving a global energy minimization problem, cross-view key point matching, occluded part inference, identity consistency maintenance and three-dimensional posture optimization are simultaneously completed; finally, the optimization result is input into a parameterized human body model fitting module to generate a final mesh sequence. Through the multi-source information tight coupling and global optimization mechanism, the application effectively solves the problems of observation missing and identity confusion under dynamic occlusion, and ensures the robustness, continuity and anatomical rationality of the motion capture.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and more specifically, to a motion capture method for unmarked dynamic occlusion scenes. Background Technology

[0002] Label-free motion capture technology, as an important research direction at the intersection of computer vision and motion analysis, aims to eliminate reliance on wearable sensors or artificial markers, achieving high-precision reconstruction of the human body's three-dimensional motion state solely through visual sensing devices and intelligent algorithms. This technology, with its advantages of non-invasiveness, convenient deployment, and controllable cost, has shown significant application potential in various high-value application scenarios such as film and television special effects production, sports motion analysis, virtual reality interaction, and clinical rehabilitation assessment. With the continuous advancement of deep learning and multi-view geometry theory, label-free motion capture methods based on monocular or sparse multi-view vision systems have made significant progress in the past decade, especially in controlled laboratory environments where relatively stable human pose estimation and motion trajectory tracking can be achieved.

[0003] Specifically, current mainstream technologies generally employ 2D human keypoint detection models based on convolutional neural networks, combined with multi-view geometric constraints or temporal filtering mechanisms, to map image pixels to 3D keypoint coordinates. These methods typically construct a sequential processing flow of "detect first, then associate, then track": first, 2D pose estimation is performed independently at each viewpoint, and then a multi-viewpoint matching strategy is used to establish multi-view... Figure 1 Consistency is maintained through Kalman filtering or optical flow-guided temporal smoothing. This framework can effectively suppress noise and output coherent 3D motion sequences under ideal conditions of uniform illumination, simple background, and isolated individuals, thus becoming the basic architecture of most current commercial and academic systems.

[0004] However, with the continuous development of related technologies and the increasingly stringent performance requirements of application scenarios, some inherent characteristics of the aforementioned technical solutions at the principle level have gradually revealed deep-seated limitations when dealing with the typical complex situation of dynamic occlusion. Fundamentally, the problem does not stem simply from insufficient detection accuracy, but from structural defects in the information fusion mechanism and optimization paradigm of existing systems. When multiple people interact closely or environmental objects temporarily obstruct the viewpoints, key point observations from some perspectives will inevitably be missing, offset, or even mismatched. The sequential workflow cannot backtrack and correct errors in the early stages, causing errors to amplify step by step along the "detection-association-tracking" chain, ultimately leading to severe jitter or topological collapse in the 3D reconstruction results. Furthermore, due to the lack of strong prior constraints on the overall human morphology, when faced with highly blurred or incomplete observations, the system often tends to generate abnormal postures that violate anatomical rules, such as hyperextension of joints or distorted limb proportions, severely damaging the physical rationality and application credibility of the output results. In addition, existing models mostly rely on limited and static training datasets, making it difficult to generalize to the ever-changing clothing styles, lighting conditions, and unpredictable interaction behaviors in the real world, which greatly reduces the robustness of the system in open and uncontrolled scenarios.

[0005] Building upon this, a more subtle yet crucial contradiction gradually emerges: label-free systems relying solely on visual observation, while pursuing high degrees of freedom and low deployment barriers, actually sacrifice the ability to maintain motion continuity and structural consistency under extreme occlusion. This performance trade-off means that while the system performs exceptionally well in ideal scenarios, it fails at critical moments in real-world applications where reliability is most crucial. Therefore, how to construct a motion capture framework that combines global optimization capabilities, multi-source information fusion mechanisms, and strong physical prior guidance, enabling it to adaptively invoke redundant information and collaboratively infer the state of occluded parts when dynamic occlusion occurs, and ensuring that the reconstruction results conform to both observational data and human kinematics, has become a key challenge and an urgent technical problem for those skilled in the art. Summary of the Invention

[0006] The present invention provides a motion capture method for unmarked dynamic occlusion scenarios, in order to solve the technical problems of existing unmarked motion capture systems, such as 3D pose reconstruction distortion, identity mismatch and motion continuity break caused by sequential processing flow, lack of global optimization mechanism and physical prior constraints, under conditions of multi-person interaction or environmental object occlusion.

[0007] To achieve the above-mentioned objectives, this invention constructs a motion capture system architecture that integrates multimodal sensing data, is based on a spatiotemporal joint optimization framework, and incorporates a parameterized human body model with strong priors. Through three core technical paths—tight coupling of multi-source information, high-dimensional graph structure modeling, and synthetic data-driven training—it achieves robust, continuous, and anatomically sound high-precision reconstruction of the three-dimensional motion state of the human body in dynamically occluded scenes.

[0008] The motion capture method includes the following steps: First, during the deployment phase, a hybrid sensing array consisting of multiple synchronous visual sensors and several sparse inertial measurement units (IMUs) is configured. The visual sensors are arranged in a non-coplanar manner around the target area, forming multi-view coverage of the target human body. No more than five IMUs are attached to key rigid parts of the human body, including the head, mid-thoracic spine, sacrum, and the outer sides of both thighs, to collect local angular velocity and linear acceleration signals. Second, within each time frame, the system synchronously acquires the image frame sequence output by all visual sensors and the raw sensor data stream output by the IMUs, and performs multimodal preprocessing. Subsequently, a four-dimensional sensor array is constructed. The spatiotemporal graph model encodes the 2D keypoint detection results from all perspectives in the current frame, the 3D joint trajectories determined in historical frames, the local motion state estimation provided by the inertial measurement unit, and the joint rotation limits and limb length constraints defined by the parameterized human model into graph nodes and edge weights. Then, by solving the global energy minimization problem of this graph model, cross-view keypoint matching, occluded joint position inference, identity consistency maintenance, and 3D pose optimization are completed simultaneously. Finally, the optimized 3D joint coordinate sequence is input into the parameterized human model fitting module to generate a complete human mesh sequence that conforms to anatomical laws and outputs the final motion capture results.

[0009] Furthermore, the multimodal preprocessing operation includes a visual data processing sub-process and an inertial data processing sub-process. In the visual data processing sub-process, a two-dimensional human keypoint detector based on a deep convolutional neural network is used to perform forward inference on each viewpoint image, outputting a heatmap response containing the two-dimensional coordinates of seventeen standard keypoints and their confidence scores; for keypoints with confidence scores below a preset threshold, the system marks them as unreliable observations and records their corresponding viewpoints and timestamps; in the inertial data processing sub-process, the raw angular velocity and linear acceleration signals output by each inertial measurement unit are subjected to zero-bias correction and low-pass filtering, and then the filtered data is converted into a motion state description in the global reference frame through a transformation matrix from the local coordinate system to the global world coordinate system, and its reliability weight as a supplementary observation is calculated. This reliability weight is jointly determined by the motion amplitude and historical consistency of the sensor attachment point.

[0010] As a preferred embodiment of the present invention, the construction process of the four-dimensional spatiotemporal correlation graph model is as follows: Each node in the graph represents a set of two-dimensional keypoint observations of a potential human instance at a specific time frame and a specific viewpoint. The node attributes include the confidence vector of the observation, the viewpoint identifier, the timestamp, and the associated inertial measurement unit index. The edges in the graph are divided into three categories: the first category is cross-viewpoint edges, which connect nodes belonging to the same human instance at different viewpoints within the same time frame, and their edge weights are calculated by the multi-view geometric consistency error and the appearance similarity measure. The second category is cross-time edges, which connect nodes with the same identity identifier in adjacent time frames, and their edge weights are determined by the three-dimensional motion smoothness constraint and the inertial data prediction residual. The third category is prior constraint edges, which connect any node with its corresponding parameterized human model state variable, and their edge weights are determined by the degree of deviation between the three-dimensional pose implied by the node and the feasible anatomical domain of the model. All edge weights are normalized to ensure that different types of constraints are comparable in global optimization.

[0011] Furthermore, the solution to the global energy minimization problem employs an iterative reweighted least squares strategy, with its objective function consisting of a weighted sum of four terms: the first term is the multi-view reprojection error term, which measures the Euclidean distance between the projection of the current 3D pose assumption under each viewpoint and the actual 2D observation, and only applies to observation points with a confidence level higher than a threshold; the second term is the temporal dynamics smoothing term, which forces the displacement and velocity changes of joints between adjacent frames to satisfy the second-order continuity condition, and introduces the acceleration observation provided by the inertial measurement unit as an external excitation term; the third term is the parameterized human model constraint term, which maps the 3D joint coordinates to the joint rotation space of the SMPL human model, calculates its KL divergence relative to the prior distribution of the joint range of motion, and adds it as a penalty term to the objective function; the fourth term is the identity consistency regularization term, which applies Markov random field smoothing constraints to the cross-frame identity label allocation to prevent identity jumps caused by local occlusion.

[0012] In a preferred embodiment of the present invention, the parameterized human body model fitting module employs a two-stage optimization mechanism. The first stage is coarse fitting, using the 3D joint coordinates output by global optimization as input, and initializing the pose and shape parameters of the SMPL model using a linear regression method. The second stage is fine optimization, constructing a composite objective function that includes joint reprojection loss, limb length conservation loss, and surface normal consistency loss, and iteratively updating the model parameters using the Gauss-Newton method until convergence. During the optimization process, the system dynamically adjusts the weight coefficients of each loss. When a joint is detected to be in a state of continuous occlusion, the weight of its reprojection loss is automatically reduced, while the weight of the limb length conservation loss is increased to maintain the rationality of the overall human body structure.

[0013] Furthermore, to enhance the system's generalization ability under unknown occlusion modes, this invention constructs a procedural synthetic data generation pipeline. Based on a parameterized human motion database, this pipeline generates a sequence of virtual scenes containing complex dynamic occlusions by randomly sampling multi-user interaction scripts, environmental obstacle layouts, clothing textures, and lighting conditions. In each synthetic frame, the system precisely labels the visibility state, occlusion source type, and depth information of all key points, and simulates the noise model of a real visual sensor and the drift characteristics of an inertial measurement unit. The generated synthetic dataset is used to jointly train the initial parameters of the 2D keypoint detector and the global optimization module. The keypoint detector employs an occlusion-aware attention mechanism, enabling it to adaptively adjust feature response intensity based on local context during the inference phase. The global optimization module is implemented through an end-to-end differentiable graph neural network, whose message passing rules are designed to explicitly encode multi-view geometric constraints and temporal dynamic priors.

[0014] In a preferred embodiment of the present invention, the inertial measurement unit and the vision sensor in the hybrid sensing array achieve microsecond-level time alignment through a hardware-level time synchronization circuit. This circuit consists of a master clock generator, a distributed trigger distribution network, and timestamp latches built into each sensor. The vision sensor uses a global shutter CMOS image sensor with a frame rate of no less than 60 frames per second and a resolution of no less than one million pixels. The inertial measurement unit includes a three-axis gyroscope and a three-axis accelerometer with a sampling rate of no less than 200 Hz and a zero-bias instability better than 0.5 degrees per hour. All sensor data are transmitted to the central processing unit in real time via gigabit Ethernet. The central processing unit is equipped with a multi-core CPU and a dedicated AI acceleration card for parallel execution of key point detection, graph model construction, and global optimization solution.

[0015] Furthermore, the system implements a dynamic reliability assessment mechanism during operation. This mechanism continuously monitors the confidence sequence of keypoint detection, the motion consistency index of the inertial measurement unit, and the changing trend of the global optimization residuals under each viewpoint. When a body part is determined to be an unreliable observation within three consecutive frames, the system automatically activates the inertial data fusion channel corresponding to that part and increases the weight of the parameterized human body model constraint terms. If the occlusion is removed, the inertial data weight is gradually reduced and the reconstruction mode dominated by visual observation is restored. This switching process is achieved through an exponential smoothing filter to ensure that the continuity of the motion trajectory is not affected by the mode switching.

[0016] In a preferred embodiment of the present invention, the four-dimensional spatiotemporal correlation graph model supports incremental construction and local re-optimization. Under normal operation, the system maintains only the graph structure of the most recent ten frames and uses a sliding window strategy for efficient solution. When a sudden increase in global optimization residuals or identity consistency conflicts are detected, the system automatically expands the time window to the most recent thirty frames and performs local graph reconstruction and re-optimization on the affected human instances to correct early error propagation. This incremental construction and local re-optimization mechanism is implemented by maintaining a lightweight historical state buffer. The buffer adopts a circular queue structure to store the original observations, intermediate optimization variables, and final output results of each frame.

[0017] Furthermore, the specific implementation of the parameterized human model constraint terms is as follows: the joint rotation parameters of the SMPL model are represented as rotation vectors, and a hyperellipsoidal boundary of the physiological range of motion is defined for each joint; in each iteration, the Mahalanobis distance (representing the covariance distance of the data) of the current rotation vector relative to the boundary is calculated; if the distance is greater than a preset threshold, a secondary penalty is applied; the gradient of the penalty term is backpropagated to the three-dimensional joint coordinates through the chain rule, thereby guiding the optimization process away from anatomically infeasible regions; for parts with complex degrees of freedom of movement such as the shoulder joint and hip joint, a piecewise linear approximation method is used to construct their range of motion boundaries to balance computational efficiency and constraint accuracy.

[0018] In a preferred embodiment of the present invention, the multi-user interaction script in the synthetic data generation pipeline is driven by a physically based character controller. This controller integrates an inverse kinematics solver and a contact force model to ensure that the generated actions conform to biomechanical laws and include realistic occlusion behavior. Environmental obstacles include static furniture and dynamic handheld objects, whose geometric models are randomly selected from a standard 3D asset library and assigned random material properties. Clothing textures are created through a procedural generation algorithm, covering various styles such as tight-fitting, loose, and multi-layered, and simulating the impact of fabric occlusion on the visibility of key points. Lighting conditions are controlled by a physically based rendering engine, supporting multi-light source configurations and dynamic shadow casting to enhance the model's robustness to realistic lighting changes.

[0019] Furthermore, the output of the global optimization module is encapsulated into a standardized motion data format, including timestamps, identifiers, 3D key point coordinate sequences, confidence indices, and occlusion status markers. This data format supports both real-time streaming and offline playback modes and can be interfaced with downstream application systems via a network interface. The system also provides a visual debugging interface that displays in real time the key point detection results from each perspective, multi-view matching relationships, inertial data fusion status, and parametric human body model fitting effects, facilitating operators to monitor the system's operation and make necessary interventions.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] (1) Solved the problems of 3D reconstruction distortion and identity mismatch in dynamic occlusion scenarios: This invention constructs a hybrid tracking architecture that integrates sparse inertial sensing and multi-view visual observation, and establishes a global joint optimization model based on a four-dimensional spatiotemporal correlation graph model. Unlike the traditional sequential processing flow of "detect first, then correlate, then track", this invention integrates cross-view matching, occluded part inference, identity consistency maintenance and 3D pose optimization into a unified global energy minimization problem. It uses the local motion state estimation provided by the inertial measurement unit (IMU) to make up for the visual blind spot, and deeply integrates the parameterized human body model (such as SMPL) as a strong anatomical prior. This fundamentally solves the problems of trajectory breakage, identity confusion and abnormal pose generation that violate human physiological structure (such as limb proportion distortion and joint reverse bending) that are easily caused by the lack of visual observation due to multi-person interaction or environmental occlusion in the unmarked motion capture system, and significantly improves the robustness and accuracy of the system under complex working conditions.

[0022] (2) Adaptive weighting and smooth switching of multimodal data are achieved: This invention designs a dynamic confidence assessment mechanism and an adaptive weighting strategy. When visual confidence decreases (e.g., occlusion occurs), the system automatically increases the weights of inertial data and model priors; when the occlusion is removed, it smoothly transitions back to the vision-dominant mode. This mechanism avoids system crashes caused by single-modal failure and also prevents numerical jumps caused by instantaneous modal switching, ensuring the smoothness and continuity of the motion trajectory.

[0023] (3) Ensuring that the motion conforms to the laws of dynamics and the limitations of human anatomy: This invention introduces a temporal dynamics smoothing term based on inertial acceleration excitation and a parameterized model constraint term based on KL divergence or Mahalanobis distance into the objective function. Through a two-stage fitting mechanism (coarse fitting and fine optimization), not only is the accuracy of the posture guaranteed on a macroscopic level, but also, on a microscopic level, through the conservation of limb length and the loss of surface normal consistency, the output results are forced to strictly follow the biomechanical laws such as the invariance of human bone length and the limitation of joint range of motion, which greatly improves the physical rationality and usability of motion capture data.

[0024] (4) Improved system's ability to correct early errors and balance real-time performance: The four-dimensional spatiotemporal correlation graph model proposed in this invention supports incremental construction and local re-optimization. Through the sliding window and the historical backtracking mechanism triggered by anomaly detection, the system can perform a posteriori correction of errors caused by insufficient early observations while ensuring real-time operating efficiency, effectively preventing the cumulative propagation of errors along the time axis.

[0025] (5) Enhanced generalization ability of the model in uncontrolled environments: By constructing a synthetic data generation pipeline based on physical simulation, this invention can generate massive amounts of complex occlusion and interaction data with precise annotations for joint training. This enables the keypoint detector and graph optimization module to have the ability to "perceive" occlusion, significantly improving the generalization performance of the algorithm in the real world when faced with changes in lighting, complex backgrounds and unseen interactive actions.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the overall system architecture of the motion capture method for markerless dynamic occlusion scenes described in this invention.

[0029] Figure 2 This is a schematic diagram showing the deployment method of the hybrid sensor array, multi-view coverage, and attachment position of the inertial measurement unit in this invention.

[0030] Figure 3 This is a schematic diagram of the structure of the four-dimensional spatiotemporal relational graph model constructed in this invention, showing the node types and the connection relationships of the three types of edges.

[0031] Figure 4 This is a flowchart illustrating the four components and their interactions in the global energy minimization solution process of this invention.

[0032] Figure 5 This is a schematic diagram of the two-stage optimization mechanism and dynamic loss weight adjustment strategy of the parametric human body model fitting module of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0034] This invention provides a motion capture method for markerless dynamic occlusion scenes, such as... Figure 1As shown, the overall technical solution constructs a hybrid sensing system integrating multi-view visual sensing and sparse inertial measurement units (IMUs), and combines a four-dimensional spatiotemporal correlation graph model with prior constraints from a parameterized human body model to achieve high-precision, continuous, and anatomically sound reconstruction of the three-dimensional motion state of the human body under dynamic occlusion environments. The following will provide a complete, sufficient, and reproducible description of the technical solution of this invention, in conjunction with the accompanying drawings and specific engineering implementation details.

[0035] During the system deployment phase, such as Figure 2 As shown, a hybrid sensor array consisting of multiple synchronous visual sensors and several sparse inertial measurement units (IMUs) is first configured. The visual sensors are arranged in a non-coplanar manner around the target motion capture area, ensuring that at least three different viewpoints can observe the main torso and limb structures of the target human body at any given time. The visual sensors employ global shutter CMOS image sensors with a frame rate of at least 60 frames per second and a resolution of at least one megapixel to meet the temporal resolution and spatial detail requirements of high-speed motion capture. The number of inertial measurement units (IMUs) does not exceed five, and they are attached to key rigid parts of the human body, including an IMU at the top of the head (IMU1), an IMU near the seventh thoracic vertebra (IMU2), an IMU in the center of the sacrum (IMU3), and IMUs in the middle of the outer sides of both thighs (IMU4, IMU5). Each IMU includes a three-axis gyroscope and a three-axis accelerometer, with a sampling rate of at least 200 Hz and a zero-bias instability better than 0.5 degrees per hour, used to acquire local angular velocity and linear acceleration signals. All sensors achieve microsecond-level time alignment through a hardware-level time synchronization circuit. This circuit consists of a master clock generator, a distributed trigger distribution network, and timestamp latches built into each sensor, ensuring strict consistency of multi-source data in the time dimension. Sensor data is transmitted in real time to the central processing unit via gigabit Ethernet. This unit is equipped with a multi-core CPU and a dedicated AI acceleration card for parallel execution of subsequent data processing and optimization tasks.

[0036] Within each time frame, the system synchronously acquires the image frame sequence output by all visual sensors and the raw sensor data stream output by the inertial measurement unit, and performs multimodal preprocessing. This preprocessing operation is divided into visual data processing sub-processes and inertial data processing sub-processes. In the visual data processing sub-process, a two-dimensional human keypoint detector based on a deep convolutional neural network is used to perform forward inference on each viewpoint image, outputting a heatmap response containing the two-dimensional coordinates of seventeen standard keypoints and their confidence scores. This keypoint detector has been enhanced during the training phase through a procedural synthetic data generation pipeline, possessing robustness to occlusion, low illumination, and complex backgrounds. For keypoints with confidence scores below a preset threshold, the system marks them as unreliable observations and records their corresponding viewpoint identifiers and timestamps for selective use by the subsequent global optimization module. In the inertial data processing sub-process, the raw angular velocity and linear acceleration signals output by each inertial measurement unit are sequentially subjected to zero-bias correction and low-pass filtering to eliminate inherent sensor drift and high-frequency noise. Subsequently, using a pre-calibrated transformation matrix between the sensor attachment point and the global world coordinate system, the filtered data is converted into a motion state description in the global reference frame. Based on this, the system calculates the reliability weight of each inertial measurement unit as supplementary observation. This weight is determined by the motion amplitude and historical consistency of the sensor attachment point: if the motion amplitude of a certain part is small in consecutive frames or deviates significantly from the motion trend of adjacent frames, its weight is dynamically reduced, and vice versa.

[0037] Furthermore, the system constructs a four-dimensional spatiotemporal correlation graph model to integrate multi-source observation information from the current frame and historical frames, and embeds anatomical priors of a parameterized human body model. For example... Figure 3As shown, each node in this graph model represents a set of 2D keypoint observations of a potential human instance at a specific time frame and viewpoint. Node attributes include the confidence vector of the observation, viewpoint identifier, timestamp, and associated inertial measurement unit index. Edges in the graph are divided into three categories: the first category is cross-viewpoint edges, connecting nodes belonging to the same human instance at different viewpoints within the same time frame; their edge weights are calculated jointly by multi-view geometric consistency error and appearance similarity measure. Multi-view geometric consistency error calculates the 3D reprojection residuals of corresponding keypoints at two viewpoints using the triangulation principle, while the appearance similarity measure is based on the cosine similarity of depth features of local image patches of keypoints. The second category is cross-time edges, connecting nodes with the same identity identifier in adjacent time frames; their edge weights are determined jointly by 3D motion smoothness constraints and inertial data prediction residuals. The 3D motion smoothness constraint adopts a second-order difference form, which forces the joint trajectory to satisfy acceleration continuity between three adjacent frames. The inertial data prediction residual is obtained by comparing the attitude change obtained by integrating the angular velocity provided by the inertial measurement unit with the attitude change inferred by vision. The third type is the prior constraint edge, which connects any node to its corresponding parameterized human model state variable. The edge weight is determined by the degree of deviation between the implicit 3D attitude of the node and the feasible anatomical region of the model. All edge weights are normalized to ensure that different types of constraints are comparable in global optimization.

[0038] Subsequently, the system simultaneously completes cross-view keypoint matching, occluded joint position inference, identity consistency maintenance, and 3D pose optimization by solving the global energy minimization problem of the graph model. This global energy minimization problem is solved using an iterative reweighted least squares strategy, such as... Figure 4 As shown, its objective function consists of a weighted sum of four terms. The first term is the multi-view reprojection error term, defined as:

[0039]

[0040] in, Indicates the time frame index. Indicates the key point index. Indicates the view index. For the first Frame number The three-dimensional coordinates of each joint point For its perspective Two-dimensional observation below, From the perspective The camera projection function, This is the confidence weight for the observation, and it is positive only when the confidence is higher than a preset threshold, otherwise it is zero.

[0041] The second term is the temporal dynamics smoothing term, defined as:

[0042]

[0043] in, For frame interval, The global acceleration observations are derived from inertial measurement units attached to nearby locations. and These are adjustable weighting coefficients, which control the intensity of the acceleration excitation term and the velocity smoothing term, respectively.

[0044] The third term is the parametric human model constraint term, which calculates the KL divergence relative to the prior distribution of joint range of motion by mapping the 3D joint coordinates to the joint rotation space of the SMPL (Skinned Multi-Person Linear) human model. Specifically, let... For the SMPL model in the first The pose parameters of the frame (represented by rotation vectors) for each joint Its physiological activity range is defined as a hyperellipsoid. ,in, Indicates the SMPL model in the first... Frame time The rotation vectors of each joint; and Statistical results from a large-scale human motion database Let be the scaling factor for the safety boundary. Then the constraint term can be expressed as:

[0045]

[0046] For joints with complex degrees of freedom of movement, such as the shoulder and hip joints, a piecewise linear approximation method is used to construct their range of motion boundaries. This involves dividing the non-convex feasible region into several convex sub-regions, and activating the corresponding linear constraints in each iteration based on the sub-region to which the current pose belongs.

[0047] The fourth term is the identity consistency regularization term, which applies a Markov random field smoothing constraint to the cross-frame identity label allocation to prevent identity jumps caused by local occlusion. Let T be the current optimization time window length (number of frames). For the first The identity tag of a human instance in the frame. If the maximum number of people in the scene is given, then this item is defined as:

[0048]

[0049] in, This is used to indicate the penalty function, which determines whether the identity tags are consistent between adjacent frames. hour, ;when hour, The value is a preset penalty constant greater than zero. This term is used to force the identity label of human instances to remain constant over time, thereby preventing frequent identity jumps caused by partial occlusion.

[0050] The above four energy terms are weighted and combined using adaptive weighting coefficients to form the final objective function:

[0051]

[0052] in, The weight coefficient for the identity consistency regularization term is used to balance the intensity of the identity change penalty. and The time-dependent dynamic weights are adjusted in real time by a dynamic reliability assessment mechanism during system operation. This mechanism continuously monitors the confidence sequence of keypoint detection, the motion consistency index of the inertial measurement unit, and the changing trend of the global optimization residuals at each viewpoint; when a body part is judged as an unreliable observation within three consecutive frames, the system automatically upgrades the confidence level. And activate the corresponding inertial data fusion channel for that part; if the obstruction is removed, gradually reduce... It then restores the reconstruction mode dominated by visual observation. This switching process is achieved through an exponential smoothing filter, ensuring that the continuity of the motion trajectory is not affected by the mode switching.

[0053] As a preferred embodiment of the present invention, such as Figure 5 As shown, the parametric human model fitting module employs a two-stage optimization mechanism. The first stage is a coarse fitting, using the 3D joint coordinates output by global optimization as input, and initializing the pose parameters of the SMPL model through a linear regression method. With shape parameters Specifically, an affine transformation matrix is ​​constructed from SMPL joints to template joints, and the optimal solution is obtained using the least squares method. and The second stage involves fine-tuning, constructing a composite objective function that includes joint reprojection loss, limb length conservation loss, and surface normal consistency loss.

[0054]

[0055] in, The output of the global optimization phase is the first Three-dimensional joint coordinates (i.e., the fitting target); The first generation of the SMPL model The location of each joint. For a predefined set of limb connection pairs, The baseline limb length is determined by the initial fitting. Let k be a subset of the vertices of the human body mesh, where k is the index of a vertex in that subset. and These are the predicted and reference surface normals, respectively. Combined Figure 5 It can be seen that during the optimization process, the system dynamically adjusts the weight coefficients of each loss. When a joint is detected to be continuously occluded, automatically reduce... At the same time, improve To maintain the rationality of the overall human body structure, this optimization uses the Gauss-Newton method to iteratively update the model parameters until the objective function converges or the maximum number of iterations is reached.

[0056] To enhance the system's generalization ability under unknown occlusion modes, this invention constructs a procedural synthetic data generation pipeline. Based on parametric human motion databases such as AMASS and Human3.6M, this pipeline generates virtual scene sequences containing complex dynamic occlusion by randomly sampling multi-person interaction scripts, environmental obstacle layouts, clothing textures, and lighting conditions. The multi-person interaction scripts are driven by a physically based character controller that integrates an inverse kinematics solver and a contact force model, ensuring that the generated actions conform to biomechanical principles and include realistic occlusion behaviors, such as handshakes, hugs, and obstruction of vision. Environmental obstacles include static furniture (such as tables and chairs) and dynamic handheld objects (such as books and cups), whose geometric models are randomly selected from standard 3D asset libraries (such as ShapeNet and ModelNet) and assigned random material properties. Clothing textures are created using a procedural generation algorithm, covering various styles such as tight-fitting, loose-fitting, and multi-layered textures, and simulating the impact of fabric occlusion on the visibility of key points, such as loose cuffs obscuring the elbow joint. Lighting conditions are controlled by a physically based rendering engine (such as Unreal Engine or Blender Cycles renderer), supporting multi-light source configurations and dynamic shadow casting to enhance the model's robustness to realistic lighting changes. In each synthetic frame, the system accurately annotates the visibility state, occlusion source type, and depth information of all keypoints, and simulates noise models of real visual sensors (such as Poisson noise and lens distortion) and drift characteristics of inertial measurement units (such as random walk bias). The generated synthetic dataset is used to jointly train the initial parameters of the 2D keypoint detector and the global optimization module. The keypoint detector employs an occlusion-aware attention mechanism, enabling it to adaptively adjust feature response intensity based on local context during the inference phase; the global optimization module is implemented through an end-to-end differentiable graph neural network, whose message passing rules are designed to explicitly encode multi-view geometric constraints and temporal dynamic priors.

[0057] Furthermore, the four-dimensional spatiotemporal relational graph model supports incremental construction and local re-optimization. Under normal operating conditions, the system maintains only the graph structure of the most recent ten frames and employs a sliding window strategy for efficient solution, balancing computational overhead and utilization of historical information. When a sudden increase in global optimization residuals (e.g., exceeding three standard deviations) or identity consistency conflicts are detected (e.g., significant pose jumps of the same identity in adjacent frames), the system automatically expands the time window to the most recent thirty frames and performs local graph reconstruction and re-optimization on the affected human instances to correct early error propagation. This mechanism is implemented by maintaining a lightweight historical state buffer, which uses a circular queue structure to store the original observations, intermediate optimization variables (such as 3D keypoint estimation, identity labels, and weight coefficients), and final output results for each frame, ensuring rapid backtracking and reconstruction of the local graph structure when needed.

[0058] In a preferred embodiment of the present invention, the output of the global optimization module is encapsulated in a standardized motion data format, including timestamps, identifiers, three-dimensional keypoint coordinate sequences, confidence indices, and occlusion status markers. This data format conforms to OpenPose or BML specifications, supports both real-time streaming (via WebSocket or gRPC protocol) and offline playback (stored as JSON or HDF5 files), and can seamlessly interface with downstream application systems (such as virtual reality engines, motion analysis software, and rehabilitation training platforms) via network interfaces. The system also provides a visual debugging interface, displaying real-time keypoint detection results from various perspectives, multi-view matching relationships, inertial data fusion status, and parametric human model fitting effects. This facilitates operators in monitoring system operation and making necessary interventions, such as manually correcting identifiers or adjusting optimization weights.

[0059] To verify the technical effects of the present invention, a specific embodiment and a comparative example are provided for comparison and explanation.

[0060] In one specific embodiment, the system is deployed in a 4m x 4m indoor motion capture area, with six global shutter CMOS cameras (1280×1024 resolution, 60Hz frame rate) arranged in a hexagonal ring. Five inertial measurement units (200Hz sampling rate) are attached to the head, thoracic spine, sacrum, and both thighs of a subject, respectively. The subject performs a complex sequence of movements involving frequent self-occlusion (such as crossing arms, turning around) and environmental occlusion (such as walking around tables and chairs) for 30 seconds. The system runs the method described in this invention and outputs a complete 3D joint trajectory and SMPL human body mesh sequence.

[0061] In the comparison, the traditional sequential markerless motion capture method was adopted: first, multi-view key point detection and triangulation were performed independently in each frame, then cross-frame identity matching was performed using the Hungarian algorithm, and finally, trajectory smoothing was performed using Kalman filtering. No inertial sensing data was introduced, nor were parameterized human body model constraints embedded.

[0062] The two sets of output results were quantitatively evaluated using the following three indicators: (1) 3D joint reconstruction error (unit: cm), based on the ground truth (provided by the high-precision optical motion capture system Vicon); (2) identity mismatch rate (unit: %), defined as the proportion of frames with incorrect identity tag assignment to the total number of frames; (3) motion continuity score (unit: dimensionless), measured by calculating the reciprocal of the standard deviation of the joint velocity between adjacent frames, the higher the value, the smoother the motion.

[0063] The evaluation results are shown in the table below:

[0064] Evaluation indicators Specific Implementation Comparative Example 3D joint reconstruction error 2.1 5.8 Identity mismatch rate 0.7 12.3 Motion continuity score 0.94 0.68

[0065] As the data shows, this invention significantly outperforms traditional methods in dynamic occlusion scenarios. In a specific embodiment, by integrating inertial data and prior knowledge of the human model, even when the elbows and wrists are completely invisible due to crossed arms, the system can still infer the lower body motion state through the thigh IMU and reasonably infer the upper limb posture by combining the limb length constraints of the SMPL model, thereby maintaining low reconstruction error and high continuity. In contrast, the comparative method, lacking global optimization and prior constraints, suffers from keypoint triangulation failure and identity matching chaos when occlusion occurs, leading to trajectory breaks and posture distortion.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A motion capture method for markerless dynamic occlusion scenes, characterized in that, include: During the deployment phase, a hybrid sensing array consisting of multiple synchronous visual sensors and several sparse inertial measurement units is configured. The visual sensors are arranged around the target area in a non-coplanar manner, and the inertial measurement units are attached to key rigid parts of the human body. Within each time frame, the system synchronously acquires the image frame sequence output by all visual sensors and the raw sensing data stream output by the inertial measurement unit, and performs multimodal preprocessing operations including both visual and inertial channels. A four-dimensional spatiotemporal relational graph model is constructed, which encodes the 2D keypoint detection results from all perspectives in the current frame, the determined 3D joint trajectories in historical frames, the local motion state estimation provided by the inertial measurement unit, and the joint rotation limits and limb length constraints defined by the parameterized human model as graph nodes and edge weights. Nodes in the graph represent the set of 2D keypoint observations of potential human instances at a specific time frame and perspective. All edge weights need to be normalized to ensure comparability of different types of constraints in global optimization. Edges in the graph are defined as three categories: the first category is cross-viewpoint edges, connecting nodes belonging to the same human instance from different perspectives within the same time frame; their edge weights are calculated by multi-view geometric consistency error and appearance similarity measure. The second category is cross-temporal edges, connecting nodes with the same identity in adjacent time frames; their edge weights are determined by 3D motion smoothness constraints and inertial data prediction residuals. The third category is prior constraint edges, connecting any node with its corresponding parameterized human model state variables; their edge weights are determined by the degree of deviation between the node's implicit 3D pose and the model's anatomical feasible region. The global energy minimization problem of the four-dimensional spatiotemporal correlation graph model is solved by adopting an iterative reweighted least squares strategy, and cross-view key point matching, occluded joint position inference, identity consistency maintenance and three-dimensional pose optimization are completed simultaneously. The globally optimized 3D joint coordinate sequence is input into the parametric human body model fitting module to generate a complete human body mesh sequence that conforms to anatomical rules, and the final motion capture result is output.

2. The motion capture method according to claim 1, characterized in that, The multimodal preprocessing operations include: Visual data processing: A two-dimensional human keypoint detector based on a deep convolutional neural network is used to perform forward inference on each viewpoint image, output the two-dimensional coordinates and confidence scores of standard keypoints, and keypoints with confidence scores below a preset threshold are marked as unreliable observations; Inertial data processing: The raw angular velocity and linear acceleration signals output by each inertial measurement unit are subjected to zero bias correction and low-pass filtering. Using the transformation matrix between the pre-calibrated sensor attachment site and the global world coordinate system, the filtered data is converted into a motion state description in the global reference frame, and its reliability weight as a supplementary observation is calculated. This reliability weight is jointly determined by the motion amplitude of the sensor attachment site and historical consistency.

3. The motion capture method according to claim 1, characterized in that, The objective function of the global energy minimization problem consists of a weighted sum of four terms: The first term is the multi-view reprojection error term, which measures the Euclidean distance between the projection of the current three-dimensional attitude assumption under each view and the actual two-dimensional observation. It only applies to two-dimensional observation points with a confidence level higher than the threshold. The second term is a temporal dynamics smoothing term, which forces the displacement and velocity changes of joints between adjacent frames to satisfy the second-order continuity condition, and introduces global acceleration observations provided by the inertial measurement unit as an external excitation term to constrain the second-order difference of the three-dimensional joint trajectory. The third term is the parameterized human model constraint term. By mapping the coordinates of three-dimensional joint points to the joint rotation space of the parameterized human model, the KL divergence of the joint range of motion relative to the prior distribution is calculated and used as a penalty term. The fourth term is the identity consistency regularization term, which applies Markov random field smoothing constraints to cross-frame identity label allocation to prevent identity jumps caused by local occlusion.

4. The motion capture method according to claim 3, characterized in that, The parametric human body model fitting module employs a two-stage optimization mechanism: The first stage is a rough fitting, which uses the 3D joint coordinates output by the global energy minimization problem as input to construct an affine transformation matrix from the joints of the parameterized human model to the joints of the template, and initializes the pose and shape parameters of the model through a linear regression method. The second stage is fine optimization, which involves constructing a composite objective function that includes joint reprojection loss, limb length conservation loss and surface normal consistency loss, and using the Gauss-Newton method to iteratively update the model parameters. During the fine optimization process, the system dynamically adjusts the weight coefficients of various losses: when a joint is detected to be in a state of continuous occlusion, the weight of its reprojection loss is automatically reduced, while the weight of the limb length conservation loss is increased, in order to maintain the rationality of the overall human body structure.

5. The motion capture method according to claim 4, characterized in that, This also includes implementing a dynamic credibility assessment mechanism during operation: Continuously monitor the confidence sequence of key point detection, the motion consistency index of the inertial measurement unit, and the changing trend of the global optimization residual from each viewpoint; When a body part is determined to be an unreliable observation within a predetermined number of consecutive frames, the inertial data fusion channel corresponding to that body part is automatically activated, and the weight of the parameterized human body model constraint term is increased. If the occlusion is removed, the weight of the inertial data is gradually reduced through an exponential smoothing filter and the reconstruction mode dominated by visual observation is restored to ensure that the continuity of the motion trajectory is not affected by the mode switching.

6. The motion capture method according to claim 5, characterized in that, The four-dimensional spatiotemporal correlation graph model supports incremental construction and local re-optimization: Under normal operating conditions, only the graph structure within the most recent fixed-length time window is maintained, and a sliding window strategy is used for solving. When a sudden increase in global optimization residuals or identity consistency conflicts are detected, the time window is automatically extended to a longer historical range, and local graph reconstruction and re-optimization are performed on the affected human instances to correct early error propagation. The incremental construction and local re-optimization are achieved by maintaining a lightweight historical state buffer, which uses a circular queue structure to store the original observations, intermediate optimization variables and final output results of each frame.

7. The motion capture method according to claim 3, characterized in that, The specific implementation method of the parameterized human body model constraint terms is as follows: The joint rotation parameters of the parameterized human model are represented as rotation vectors, and the hyperellipsoidal boundary of the physiological range of motion of each joint is defined. In each iteration, the Mahalanobis distance of the current rotation vector relative to the boundary of the hyperellipsoid is calculated. If the distance is greater than a preset threshold, a secondary penalty is applied, and the gradient of the secondary penalty term is backpropagated to the three-dimensional joint coordinates through the chain rule. For joints with complex degrees of freedom, a piecewise linear approximation method is used to construct their range of motion boundaries, dividing the non-convex feasible region into several convex sub-regions, and activating the corresponding linear constraints according to the sub-region to which the current pose belongs.

8. The motion capture method according to claim 2, characterized in that, It also includes the step of jointly training the model parameters based on a programmatic synthetic data generation pipeline: The pipeline is based on a parameterized human motion database and generates a sequence of virtual scenes containing complex dynamic occlusions by randomly sampling multi-person interaction scripts, environmental obstacle layouts, clothing textures and lighting conditions. In each synthesized frame, the visibility status, occlusion source type, and depth information of the key points are labeled, and the noise model of the real vision sensor and the drift characteristics of the inertial measurement unit are simulated. A two-dimensional human keypoint detector and a graph neural network module for global energy minimization are jointly trained using the generated synthetic dataset, wherein the keypoint detector is trained to have an attention mechanism with occlusion awareness.

9. The motion capture method according to claim 1, characterized in that, The specific configuration of the hybrid sensing array is as follows: The visual sensor uses a global shutter CMOS image sensor to ensure that at any given time, the main torso and limb structures of the target human body can be observed from at least three different perspectives. The number of inertial measurement units is no more than five, which are respectively attached to the head, middle of the thoracic spine, sacrum and outer sides of both thighs of the human body, and achieve microsecond-level time alignment with the visual sensor through a hardware-level time synchronization circuit.

Images