Three-dimensional motion capture method and system based on multi-view consensus and kinematic constraints

Abstract

The present application relates to the field of computer vision and three-dimensional reconstruction technology, and particularly relates to a three-dimensional motion capture method and system based on multi-view consensus and kinematic constraint, multi-view image data is acquired and a two-dimensional key point set is obtained, multi-view space consistency solving is performed based on the two-dimensional key point set, and initial three-dimensional joint coordinates after fusion are generated; a kinematic constraint model is constructed based on rigid body prior, and a deviation measure between current skeleton features corresponding to the initial three-dimensional joint coordinates and reference skeleton features is calculated; geometric constraint correction is performed on the initial three-dimensional joint coordinates based on the deviation measure, and target three-dimensional joint data satisfying rigid constraint is output. The present application effectively solves the problems of poor robustness of existing markerless motion capture technology in a complex occlusion environment, and "breathing effect" and jitter caused by lack of physical constraint in single frame solving, and significantly improves the physical realism and stability of three-dimensional motion capture.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, and in particular to a 3D motion capture method and system based on multi-view consensus and kinematic constraints. Background Technology

[0002] Most existing label-free motion capture systems remain at the level of basic multi-camera video acquisition and deep learning extraction of 2D key points, and then combine multi-view geometric principles to reconstruct the 3D human skeleton. However, in real-world complex occlusion and dynamic motion scenarios, existing systems often struggle to output stable, highly realistic skeleton data that conforms to human kinematics.

[0003] Several patented solutions exist regarding 3D motion capture or pose reconstruction. For example, CN116152439A discloses a method and system for 3D human pose reconstruction based on multi-view human images. This method proposes acquiring multi-view images using multiple cameras and combining network-predicted depth with camera intrinsic and extrinsic parameters to transform 2D keypoints into a 3D camera coordinate system to determine the human's 3D pose. Another example is CN106843484A, which discloses a method for fusing indoor positioning data and motion capture data. This method uses a Kalman filter algorithm to filter the initial output displacement acquired by the motion capture sensor to remove flickering points generated during the correction process and form a smooth output. However, in practical high-precision motion capture applications with complex occlusion, these solutions still face many severe challenges.

[0004] Specifically, some solutions heavily rely on parameterized solutions from the pre-processor network and direct calculation of 2D line-of-sight. Lacking a multi-line-of-sight "spatial consensus and density clustering" mechanism in 3D space, when some cameras experience 2D detection drift due to limb occlusion or background clutter, incorrect lines of sight can directly lead to positional shifts in the calculated 3D keypoints, or even severe limb distortion. Other solutions employ post-processing smoothing algorithms such as Kalman filtering, which primarily focus on pure signal processing theory, treating human keypoints as independently moving particles and ignoring the core physical rigidity of the human skeleton's immutable length over short periods. This results in inconsistent skeleton lengths when the target is stationary or moving slowly, due to minute fluctuations in the detection algorithm itself, leading to an unnatural "breathing effect" or high-frequency micro-jitter.

[0005] Summarizing existing technical solutions, it can be found that, given the availability of hardware for multiple synchronized cameras, the bottleneck in achieving high-precision, robust, and markerless 3D motion capture lies in how to break away from the traditional principles of "direct spatial calculation susceptible to interference" and "pure signal smoothing lacking physical constraints," and construct a 3D skeleton modeling and constraint algorithm system that can adapt to complex occlusion environments, accurately identify and eliminate erroneous lines of sight, and strictly adhere to the rigid characteristics of human physical and biological kinematics. Summary of the Invention

[0006] To address the problems of severe interference from outliers in spatial calculation accuracy under occlusion environments and unnatural jitter and "breathing effect" in reconstructed skeletons due to lack of physical rigid constraints in existing technologies, this invention proposes a three-dimensional motion capture method and system based on multi-view consensus and kinematic constraints. Through multi-view spatial consensus and biological kinematic rigid constraints, it achieves high-precision three-dimensional motion capture that is markerless, highly stable, jitter-free, and free of breathing effect under complex occlusion.

[0007] To achieve the above objectives, the technical solution adopted is:

[0008] This invention provides a three-dimensional motion capture method based on multi-view consensus and kinematic constraints, comprising the following steps:

[0009] Acquire multi-view image data of the target object at the same time;

[0010] Feature extraction is performed on the multi-view image data to obtain a two-dimensional key point set for each view. Multi-view spatial consistency calculation is performed based on the two-dimensional key point set to generate the fused initial three-dimensional joint coordinates.

[0011] Based on the rigid body prior, a kinematic constraint model is constructed, and the deviation metric between the current bone features and the reference bone features corresponding to the initial three-dimensional joint coordinates is calculated.

[0012] Based on the deviation metric, geometric constraint correction is performed on the initial three-dimensional joint coordinates to output target three-dimensional joint data that satisfies rigid constraints.

[0013] According to the 3D motion capture method based on multi-view consensus and kinematic constraints of the present invention, the step of performing multi-view spatial consistency calculation based on the 2D keypoint set to generate fused initial 3D joint coordinates further includes:

[0014] Based on the multi-view imaging topology, for each type of human joint in the set of two-dimensional key points, the observation points under different views are back-projected to three-dimensional space to construct a corresponding three-dimensional candidate point cloud for the joint.

[0015] Calculate the spatial density distribution characteristics of a 3D candidate point cloud;

[0016] Outlier noise points are eliminated based on the spatial density distribution characteristics, and the centroid coordinates of points in the remaining high-density region are calculated as the initial three-dimensional joint coordinates of the joint.

[0017] According to the 3D motion capture method based on multi-view consensus and kinematic constraints of the present invention, the step of calculating the spatial density distribution characteristics of the 3D candidate point cloud and removing outliers based on the spatial density distribution characteristics includes:

[0018] Calculate the weighted geometric centroid of a 3D candidate point cloud : ,in This represents a 3D point in a 3D candidate point cloud. This represents the weights based on the confidence level of 2D keypoint detection from the corresponding camera's viewpoint.

[0019] Calculate each 3D point in the 3D candidate point cloud To the weighted geometric centroid Euclidean distance When Euclidean distance When the spatial consistency threshold is greater than the preset threshold, the corresponding candidate point is determined to be outlier noise and is removed.

[0020] According to the three-dimensional motion capture method based on multi-view consensus and kinematic constraints of the present invention, the reference skeleton feature further includes the reference skeleton length, and the steps for determining the reference skeleton length are as follows:

[0021] During the system initialization phase, multiple high-confidence frames that satisfy multi-view observation consistency are identified, and the average length of each bone segment is calculated as the baseline bone length. : ,in To effectively initialize the number of frames, and These are the three-dimensional coordinates of the parent and child nodes of the skeleton in the k-th frame, respectively.

[0022] According to the three-dimensional motion capture method based on multi-view consensus and kinematic constraints of the present invention, the calculation process of the deviation metric is further as follows:

[0023] Calculate the current bone vector formed by the coordinates of adjacent 3D joints and its magnitude, and use the magnitude as the current bone length. ;

[0024] Calculate the current bone length Compared with baseline bone length The absolute value of the difference .

[0025] According to the 3D motion capture method based on multi-view consensus and kinematic constraints of the present invention, the step of performing geometric constraint correction on the initial 3D joint coordinates based on the deviation metric further includes: utilizing a preset jitter tolerance threshold. and motion mutation threshold The absolute value of the difference Perform interval determination and execute a hierarchical correction strategy based on the determination result. The hierarchical correction strategy is as follows:

[0026] If the absolute value of the difference Greater than the threshold of motion change If it is determined to be abnormal observation noise, rigid locking is performed to keep the reference bone length unchanged, and the initial three-dimensional joint coordinates are corrected to the position that meets the reference bone length along the direction of the current bone vector.

[0027] If the absolute value of the difference Less than or equal to the jitter tolerance threshold If the movement is determined to be non-physical micro-motion, rigid locking is executed to keep the reference bone length unchanged, and the initial three-dimensional joint coordinates are corrected to a position that meets the reference bone length along the direction of the current bone vector.

[0028] According to the three-dimensional motion capture method based on multi-view consensus and kinematic constraints of the present invention, the method further includes an adaptive update step for the bone length reference:

[0029] If the absolute value of the difference Located at the jitter tolerance threshold With motion mutation threshold If the multi-view reprojection error of the current frame is less than the preset error threshold, it is determined to be a true deformation or system error accumulation; the reference bone length is updated by weighting the current bone length.

[0030] According to the three-dimensional motion capture method based on multi-view consensus and kinematic constraints of the present invention, before calculating the deviation metric between the current bone features corresponding to the initial three-dimensional joint coordinates and the reference bone features, the method further includes:

[0031] Obtain the initial 3D joint coordinates of the current frame and input them into the adaptive filter;

[0032] The cutoff frequency of the adaptive filter is dynamically adjusted based on the current motion speed of the target object, and the initial three-dimensional joint coordinates of the current frame are smoothed in the temporal domain.

[0033] The smoothed 3D joint coordinates of the current frame are used as inputs for subsequent deviation metric calculations and geometric constraint corrections.

[0034] According to the three-dimensional motion capture method based on multi-view consensus and kinematic constraints of the present invention, the method further includes: when it is determined that a certain joint is not visible in all views, the state extrapolation of the three-dimensional coordinates of the joint is performed based on a constant velocity motion model, and the predicted value is used as the alternative three-dimensional coordinates of the joint in the current frame.

[0035] When the joint is visually observed again in a subsequent frame, the observed values ​​of that frame are fused with the current predicted values ​​using the Kalman gain matrix to obtain the restored 3D joint coordinates.

[0036] Furthermore, the present invention also provides a three-dimensional motion capture system based on multi-view consensus and kinematic constraints to implement the above-mentioned method, the system comprising:

[0037] The data acquisition module is used to acquire multi-view image data of the target object at the same time.

[0038] The feature extraction module is used to extract features from the multi-view image data to obtain a set of two-dimensional key points for each view.

[0039] The spatial solution module is used to perform multi-view spatial consistency solution based on the two-dimensional key point set to generate the fused initial three-dimensional joint coordinates;

[0040] The constraint correction module is used to construct a kinematic constraint model based on the rigid body prior, calculate the deviation metric between the current bone features and the reference bone features corresponding to the initial three-dimensional joint coordinates, perform geometric constraint correction on the initial three-dimensional joint coordinates based on the deviation metric, and output target three-dimensional joint data that meets the rigid constraints.

[0041] The beneficial effects achieved by adopting the above technical solution are:

[0042] 1. Significantly improves the robustness of spatial solutions under complex occlusion environments: This invention abandons traditional global least squares optimization and innovatively proposes an outlier removal mechanism based on "multi-view spatial consensus." By directly measuring the consistency of the line of sight at the three-dimensional spatial level, this invention can accurately identify and physically isolate erroneous observation points caused by single-view occlusion, background interference, or local illumination changes. This ensures that the final fused three-dimensional coordinates are contributed only by reliable viewpoints, effectively avoiding the distortion of the overall result caused by individual false positives.

[0043] 2. Effectively eliminates non-physical jitter and the "breathing effect," ensuring the realism of the movement: This invention introduces "biokinematic constraints," utilizing the physical fact that the length of the human skeleton cannot change in a short time to construct a mandatory geometric projection correction algorithm. Unlike traditional filtering algorithms that simply "smooth" the signal, this invention, based on a dual-threshold state machine, strictly "locks" the skeleton length at a physical reference value when it is determined to be non-realistic movement. This method fundamentally eliminates the phenomenon of skeletal stretching when the target is stationary or slightly moving, making the output motion data smoother, more natural, and consistent with the laws of human kinematics.

[0044] 3. High-precision markerless motion capture without the need for expensive specialized equipment: Through the constraint correction and improvement at the underlying algorithm level, this invention can filter out a large amount of high-frequency noise caused by hardware or environment in conventional industrial cameras or even consumer-grade cameras, achieving stable accuracy close to that of traditional optical marker motion capture systems, and significantly reducing the threshold for using high-precision motion capture and the cost of hardware deployment. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0046] Figure 1 This is a flowchart illustrating the three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram illustrating the principle of spatial consensus for eliminating outliers in an embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of the biokinematic constraint correction process based on dual thresholds according to an embodiment of the present invention;

[0049] Figure 4 This is a verification data diagram of the non-physical jitter smoothing effect of the skeleton based on the dual-threshold kinematic state machine in an embodiment of the present invention;

[0050] Figure 5 This is a structural block diagram of a three-dimensional motion capture system based on multi-view consensus and kinematic constraints according to an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram comparing the 3D reconstruction results of the method described in this embodiment of the invention with existing technologies under occluded environments and lack of kinematic constraints; wherein, Figure 6 (a) is a schematic diagram of the reconstruction results of the existing technology. Figure 6 (b) is a schematic diagram of the reconstruction results of the present invention. Detailed Implementation

[0052] The exemplary solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art.

[0053] This invention discloses a three-dimensional motion capture method based on multi-view consensus and kinematic constraints, such as... Figure 1 As shown, the specific steps include the following:

[0054] Step S1: Acquire multi-view image data of the target object at the same time.

[0055] First, to address the image distortion issues caused by wide-angle lenses and establish a unified imaging geometry model, the system introduces distortion correction and virtual camera projection modules. Let the... The intrinsic parameter matrix of each camera is The rotation matrix is The translation vector is For any observation point on the image plane The system utilizes the distortion coefficient vector Perform distortion correction to map the observation points to the normalized plane, and then construct a virtual projection matrix. The linearized expression of this projection relationship is:

[0056]

[0057] in, Represents homogeneous three-dimensional coordinates in the world coordinate system. Image pixel coordinates, The scaling factor is used. In this embodiment, to address the issue of overfitting of the calibration data, a preset focal length estimate is used to reconstruct the intrinsic parameter matrix. Furthermore, the distortion coefficients are forced to zero to construct an ideal pinhole model, thereby improving the robustness of subsequent solutions.

[0058] Step S2: Feature extraction is performed on the multi-view image data to obtain a set of two-dimensional key points for each view. Multi-view spatial consistency calculation is then performed based on the set of two-dimensional key points to generate the fused initial three-dimensional joint coordinates. Specifically, as follows:

[0059] Step S201: Based on the multi-view imaging topology, for each type of human joint in the two-dimensional keypoint set, the observation points under different views are back-projected to the three-dimensional space to construct the corresponding three-dimensional candidate point cloud for the joint.

[0060] Specifically, the system acquires multi-view video streams and performs frame-level time synchronization, using a deep convolutional neural network to extract a set of two-dimensional key points in each frame. To address the occlusion problem between multiple views, this embodiment constructs a multi-view imaging topology and generates a three-dimensional candidate point cloud for each joint. Each camera perspective, constructing For any pair of observations, for any pair of cameras ( Solving the system of equations using the direct linear transformation algorithm The three-dimensional candidate point cloud of the j-th joint is obtained. :

[0061]

[0062] in, Indicates by the first and the Three-dimensional spatial points calculated from the intersection of the lines of sight of each camera. This is a valid set of camera indexes.

[0063] Step S202: Calculate the spatial density distribution characteristics of the three-dimensional candidate point cloud.

[0064] To eliminate outlier noise caused by false detections, the system performs spatial consistency calculation based on density clustering, such as... Figure 2 As shown, the system first calculates the weighted geometric centroid of the 3D candidate point cloud. :

[0065]

[0066] in, This represents a 3D point in a 3D candidate point cloud. This represents the weights based on the confidence level of 2D keypoint detection from the corresponding camera's viewpoint. and They represent the first The and the first Detection confidence of the two-dimensional key point from the perspective of each camera.

[0067] Step S203: Remove outlier noise points based on the spatial density distribution characteristics, and calculate the centroid coordinates of the points in the remaining high-density region as the initial three-dimensional joint coordinates of the joint.

[0068] Calculate each 3D point in the 3D candidate point cloud To the weighted geometric centroid Euclidean distance The system sets a spatial consistency threshold. ,when If a candidate point is found to be an invalid outlier, it is discarded. Finally, the centroid coordinates of the points within the remaining high-density region are used as the initial three-dimensional joint coordinates for that joint. .

[0069] To address the contradiction between "high-frequency jitter" and "tracking delay" in video motion capture, this embodiment introduces a velocity-adaptive low-pass filter to obtain the initial 3D joint coordinates of the current frame and input them into the adaptive filter. The cutoff frequency of the adaptive filter is dynamically adjusted based on the current motion velocity of the target object to perform temporal smoothing on the initial 3D joint coordinates of the current frame. The smoothed 3D joint coordinates of the current frame are used as input for subsequent deviation metric calculations and geometric constraint corrections. The adaptive filter employs a first-order recursive filtering form, and the filtered output... Defined as:

[0070]

[0071] in, These are the initial 3D joint coordinates for the current frame. The smoothing factor is the filtered output from the previous time step. From dynamic cutoff frequency Decision, and , This is a preset proportional coefficient. The current velocity of the joint; when the target is stationary ( When this happens, the system automatically reduces... To eliminate minute vibrations; to improve performance when the target is moving rapidly. To reduce latency.

[0072] Step S3: Construct a kinematic constraint model based on the rigid body prior, and calculate the deviation metric between the current bone features and the reference bone features corresponding to the initial three-dimensional joint coordinates.

[0073] After obtaining the filtered coordinates, a bio-kinematic constraint mechanism is introduced to correct non-rigid deformation; the reference skeleton features in this embodiment include the reference skeleton length. During the system initialization phase, multiple high-confidence frames that satisfy multi-view observation consistency are identified, and the average length of each skeleton segment is calculated as the reference skeleton length. :

[0074]

[0075] in, To effectively initialize the number of frames, and These are the 3D coordinates of the parent and child nodes of the skeleton in frame k. Calculate the current skeleton vector formed by adjacent 3D joint coordinates and its magnitude, using the magnitude as the current skeleton length. Calculate the current bone length Compared with baseline bone length The absolute value of the difference .

[0076] Step S4: Perform geometric constraint correction on the initial three-dimensional joint coordinates based on the deviation metric, and output the target three-dimensional joint data that satisfies the rigid constraints.

[0077] Based on the aforementioned baseline bone length, the system constructs a "three-level state machine based on dual thresholds" correction model during real-time operation. This model utilizes a preset jitter tolerance threshold. and motion mutation threshold The absolute value of the difference An interval determination is performed, and a graded correction strategy is implemented based on the determination result: when the determination is abnormal observation noise or non-physical micro-motion, the initial 3D joint coordinates are forcibly projected onto a sphere with the parent node as the center and the reference bone length as the radius; the correction strategy is specifically divided into the following three levels, such as Figure 3 As shown:

[0078] Level 1: Non-physical micro-motion elimination. When judged... The system determines that the current deviation originates from observation noise or algorithm micro-jitter; at this point, rigid locking is executed, maintaining the baseline bone length unchanged, and the initial 3D joint coordinates are corrected along the direction of the current bone vector to a position that satisfies the baseline bone length. The corrected child node coordinates... for:

[0079]

[0080] in, The direction unit vector of the current bone. The three-dimensional coordinates of the parent node of the current frame, such as Figure 4 As shown, this step eliminates the "breathing effect" of limbs commonly found in traditional motion capture from a physical perspective.

[0081] Level Two: Abnormal Mutation Suppression. When determining... The system determines that the current observation has a serious detection drift or topology connection error. At this point, the system also executes the first-level formula. The rigid constraint forces the child node to be pulled back to satisfy the condition. On the sphere, or in extreme cases, directly using the relative position of the previous frame, thereby suppressing abrupt changes in action.

[0082] Level 3: Adaptive Update of Bone Length Benchmark: If determined If the multi-view reprojection error of the current frame is less than a preset error threshold, the system determines that real human micro-deformation or system error accumulation may have occurred. At this point, the current observation value is accepted. The baseline bone length is then updated using a weighted average based on the current bone length.

[0083]

[0084] in, The updated baseline bone length, The baseline bone length before the update. The length of the skeleton observed in the current frame. This is the update rate coefficient. This mechanism gives the system the ability to adapt to subtle changes in the body size of different users.

[0085] This embodiment also includes an adaptive calibration step for the baseline bone length: establishing a long-term sliding window, and statistically analyzing the average stable bone length observations that satisfy the multi-view consistency constraint within the sliding window; calculating the long-term deviation between the current baseline bone length and the average stable bone length observations; if the long-term deviation continues to exceed a preset calibration threshold, replacing the current baseline bone length with the average stable bone length observations.

[0086] Furthermore, for extreme cases where complete occlusion leads to the total loss of visual data, this system additionally integrates a Kalman prediction module. When it is determined that a joint is invisible from all viewpoints, the system extrapolates the state of the joint's three-dimensional coordinates based on a constant velocity motion model, using the predicted value as the alternative three-dimensional coordinates of the joint in the current frame.

[0087]

[0088] in, The predicted 3D joint coordinates for the current frame. These are the filtered 3D joint coordinates from the previous frame. The time interval between two adjacent frames. This represents the estimated joint motion velocity from the previous moment. When the joint is visually observed again in a subsequent frame, the observed values ​​from that frame are fused with the current predicted values ​​using the Kalman gain matrix to obtain the restored 3D joint coordinates, achieving a smooth transition.

[0089] Corresponding to the above method, embodiments of the present invention also disclose a three-dimensional motion capture system based on multi-view consensus and kinematic constraints, such as... Figure 5 As shown, in terms of hardware deployment, the system includes a layout surrounding the shooting area. The system comprises a synchronized industrial camera and a high-performance computing workstation. Its core logic includes imaging geometry modeling, spatiotemporal joint observation, adaptive filtering, and hierarchical skeleton dynamics constraint correction. The system mainly consists of the following modules:

[0090] The data acquisition module is used to acquire multi-view image data of the target object at the same time.

[0091] The feature extraction module is used to extract features from the multi-view image data to obtain a set of two-dimensional key points for each view.

[0092] The spatial solution module is used to perform multi-view spatial consistency solution based on the two-dimensional key point set to generate the fused initial three-dimensional joint coordinates;

[0093] The constraint correction module is used to construct a kinematic constraint model based on the rigid body prior, calculate the deviation metric between the current bone features and the reference bone features corresponding to the initial three-dimensional joint coordinates, perform geometric constraint correction on the initial three-dimensional joint coordinates based on the deviation metric, and output target three-dimensional joint data that meets the rigid constraints.

[0094] To more intuitively illustrate the effects of this invention, please refer to [link / reference needed]. Figure 6 . Figure 6 This demonstrates a direct comparison between the 3D skeleton output by our system (i.e., the set of all 3D joint coordinates) and existing traditional multi-view reconstruction methods. For example... Figure 6 As shown in (a), in the traditional bundle adjustment global optimization scheme, when part of the target limb is occluded or affected by background interference, incorrect two-dimensional observation lines can lead to severe outlier noise in the calculated three-dimensional coordinates, such as the joint flying point phenomenon shown in Figure 601. Simultaneously, due to the lack of rigid constraints in the time dimension, measurement errors in single-frame calculations can cause the skeleton length to exhibit a fluctuating "breathing effect" or abnormal stretching, as shown in Figure 602, where the lower leg bone length is distorted.

[0095] In comparison, such as Figure 6 As shown in (b), after processing by the 3D motion capture method based on multi-view consensus and kinematic constraints described in this invention, due to the introduction of a density clustering-based spatial consensus mechanism in the spatial solution stage, the system successfully eliminates erroneous lines of sight exceeding the centroid distance, thereby effectively correcting the joint flying point phenomenon caused by occlusion and outputting physically reasonable joint positions, as shown in the attached figure 603; furthermore, in the time series processing, this system introduces a biokinematic state machine to perform constraint correction, when the current bone length deviation is identified... In the event of an abnormal mutation, the system will force a baseline bone length. By performing a geometric manifold projection on the radius, the coordinates are strictly anchored to the physically rigid sphere, thus completely eliminating unnatural bone length deformations. As shown in Figure 604, the leg bone length is strictly locked to a standard reference. The final output exhibits extremely high physical realism and robustness in both spatial consistency and temporal stability.

[0096] Finally, the system standardizes and outputs the target 3D skeleton data after the aforementioned geometric and kinematic corrections. The system defines the hip center as the root node, transforms all joint coordinates to a local coordinate system, and generates a standard BVH format data stream. Through these steps, this invention achieves robust removal of occlusion noise and physical-level suppression of unnatural bone jitter without relying on external markers, significantly improving the realism of the 3D reconstruction.

[0097] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A three-dimensional motion capture method based on multi-view consensus and kinematic constraints, characterized in that, Includes the following steps: Acquire multi-view image data of the target object at the same time; Feature extraction is performed on the multi-view image data to obtain a two-dimensional key point set for each view. Multi-view spatial consistency calculation is performed based on the two-dimensional key point set to generate the fused initial three-dimensional joint coordinates. Based on the rigid body prior, a kinematic constraint model is constructed, and the deviation metric between the current bone features and the reference bone features corresponding to the initial three-dimensional joint coordinates is calculated. Based on the deviation metric, geometric constraint correction is performed on the initial three-dimensional joint coordinates to output target three-dimensional joint data that satisfies rigid constraints.

2. The multi-view consensus and kinematic constraint based three-dimensional motion capture method according to claim 1, wherein, The step of performing multi-view spatial consistency calculation based on the two-dimensional keypoint set to generate the fused initial three-dimensional joint coordinates includes: Based on the multi-view imaging topology, for each type of human joint in the set of two-dimensional key points, the observation points under different views are back-projected to three-dimensional space to construct a corresponding three-dimensional candidate point cloud for the joint. Calculate the spatial density distribution characteristics of a 3D candidate point cloud; Outlier noise points are eliminated based on the spatial density distribution characteristics, and the centroid coordinates of points in the remaining high-density region are calculated as the initial three-dimensional joint coordinates of the joint.

3. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 2, characterized in that, The calculation of the spatial density distribution features of the 3D candidate point cloud, and the removal of outlier noise points based on the spatial density distribution features, includes: Calculate the weighted geometric centroid of a 3D candidate point cloud : ,in This represents a 3D point in a 3D candidate point cloud. This represents the weights based on the confidence level of 2D keypoint detection from the corresponding camera's viewpoint. Calculate each 3D point in the 3D candidate point cloud To the weighted geometric centroid Euclidean distance When Euclidean distance When the spatial consistency threshold is greater than the preset threshold, the corresponding candidate point is determined to be outlier noise and is removed.

4. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 1, characterized in that, The reference skeleton features include the reference skeleton length, and the steps for determining the reference skeleton length are as follows: During the system initialization phase, multiple high-confidence frames that satisfy multi-view observation consistency are identified, and the average length of each bone segment is calculated as the baseline bone length. : ,in To effectively initialize the number of frames, and These are the three-dimensional coordinates of the parent and child nodes of the skeleton in the k-th frame, respectively.

5. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 4, characterized in that, The calculation process for the deviation metric is as follows: Calculate the current bone vector formed by the coordinates of adjacent 3D joints and its magnitude, and use the magnitude as the current bone length. ; Calculate the current bone length Compared with baseline bone length The absolute value of the difference .

6. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 5, characterized in that, The step of performing geometric constraint correction on the initial three-dimensional joint coordinates based on the deviation metric includes: utilizing a preset jitter tolerance threshold. and motion mutation threshold The absolute value of the difference Perform interval determination and execute a hierarchical correction strategy based on the determination result. The hierarchical correction strategy is as follows: If the absolute value of the difference Greater than the threshold of motion change If it is determined to be abnormal observation noise, rigid locking is performed to keep the reference bone length unchanged, and the initial three-dimensional joint coordinates are corrected to the position that meets the reference bone length along the direction of the current bone vector. If the absolute value of the difference Less than or equal to the jitter tolerance threshold If the movement is determined to be non-physical micro-motion, rigid locking is executed to keep the reference bone length unchanged, and the initial three-dimensional joint coordinates are corrected to a position that meets the reference bone length along the direction of the current bone vector.

7. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 6, characterized in that, The method also includes an adaptive update step for the bone length baseline: If the absolute value of the difference Located at the jitter tolerance threshold With motion mutation threshold If the multi-view reprojection error of the current frame is less than the preset error threshold, it is determined to be a true deformation or system error accumulation; the reference bone length is updated by weighting the current bone length.

8. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 1, characterized in that, Before calculating the deviation metric between the current bone features corresponding to the initial three-dimensional joint coordinates and the reference bone features, the method further includes: Obtain the initial 3D joint coordinates of the current frame and input them into the adaptive filter; The cutoff frequency of the adaptive filter is dynamically adjusted based on the current motion speed of the target object, and the initial three-dimensional joint coordinates of the current frame are smoothed in the temporal domain. The smoothed 3D joint coordinates of the current frame are used as inputs for subsequent deviation metric calculations and geometric constraint corrections.

9. The three-dimensional motion capture method based on multi-view consensus and kinematic constraints according to claim 1, characterized in that, The method further includes: when it is determined that a joint is not visible in all viewpoints, extrapolating the state of the three-dimensional coordinates of the joint based on a constant velocity motion model, and using the predicted value as the alternative three-dimensional coordinates of the joint in the current frame; When the joint is visually observed again in a subsequent frame, the observed values ​​of that frame are fused with the current predicted values ​​using the Kalman gain matrix to obtain the restored 3D joint coordinates.

10. A three-dimensional motion capture system based on multi-view consensus and kinematic constraints, characterized in that, For implementing the method as described in any one of claims 1-9, the system comprises: The data acquisition module is used to acquire multi-view image data of the target object at the same time. The feature extraction module is used to extract features from the multi-view image data to obtain a set of two-dimensional key points for each view. The spatial solution module is used to perform multi-view spatial consistency solution based on the two-dimensional key point set to generate the fused initial three-dimensional joint coordinates; The constraint correction module is used to construct a kinematic constraint model based on the rigid body prior, calculate the deviation metric between the current bone features and the reference bone features corresponding to the initial three-dimensional joint coordinates, perform geometric constraint correction on the initial three-dimensional joint coordinates based on the deviation metric, and output target three-dimensional joint data that meets the rigid constraints.

Images