A human motion capture method based on visual and inertial feature manifold alignment

Abstract

The present disclosure provides a human motion capture method based on visual and inertial feature manifold alignment, comprising the following steps: S1, extracting spatial saliency regions from monocular visual sequences, and mapping image pixels to spatial feature vectors with global context information; S2, synchronously inputting sparse IMU data, and extracting acceleration and angular velocity time sequence features by using an IMU encoder; S3, designing a conditional feature alignment bridge operator, projecting visual features and IMU dynamic features to a unified associated manifold space, and generating mixed enhanced features; S4, inputting the mixed enhanced features to a monocular human body morphological parameterization deduction model, and predicting a human body three-dimensional posture and a parameterized human body model grid parameter; and S5, constructing a multi-objective loss function based on dynamic consistency constraints, and performing end-to-end training on the model. While significantly improving the physical robustness and spatiotemporal consistency of human motion capture, low-rank collaborative training of heterogeneous sensor data is realized.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and artificial intelligence, and more specifically, to a method for human motion capture based on the alignment of visual and inertial feature manifolds. Background Technology

[0002] In the field of computer vision, monocular human motion capture (MOV) technology, as an important means of acquiring human posture and mesh information, is widely used in virtual reality, motion analysis, film and television special effects, and other scenarios. Existing monocular motion capture methods mainly regress human key points or 3D mesh parameters directly from single-frame images or video sequences using deep neural networks. While such pure vision solutions have significant advantages such as being non-contact and having low deployment costs, they still face two major challenges in practical industrial applications: First, the inherent ambiguity of geometric projection leads to underdeterministic results. The monocular imaging process involves the loss of physical information in the depth dimension. In complex scenarios such as limb self-occlusion, significant overlap, or environmental lighting interference, pure vision algorithms struggle to obtain accurate depth inference from the spatial probability distribution, often inducing deviations in 3D posture calculations or instantaneous drifts in physical spatial position. Second, the lack of motion dynamics constraints weakens spatiotemporal consistency. Existing end-to-end regression models heavily rely on spatial representations and lack temporal modeling of the physical characteristics of human anatomical movement. When dealing with unsteady movements such as explosive motion or instantaneous impacts, the reconstruction results often produce unnatural joint vibrations and cannot guarantee the long-term stability of bone proportions. To address these issues, some studies have attempted to introduce optimization methods to iteratively correct the regression results; however, these optimization methods are computationally complex and have poor real-time performance, making them difficult to meet practical application requirements. Furthermore, multimodal motion capture methods that have emerged in recent years attempt to fuse inertial measurement unit (IMU) data with visual information, providing local acceleration and angular velocity information from sensor data to assist in 3D pose estimation, thereby overcoming the accuracy limitations of pure visual motion capture in scenarios with occlusion, lighting changes, and rapid motion. While these methods have improved accuracy, most existing methods rely on specific network architectures or specific task scenarios for design, lacking a universal fusion strategy that can be universally adapted to different monocular vision models. Therefore, how to construct a multimodal collaborative representation framework that can effectively eliminate projection ambiguity, enhance physical and dynamic constraints, and possess cross-architecture transfer characteristics without compromising the integrity of the original visual representation manifold has become a critical technical bottleneck that urgently needs to be overcome in the field of human motion capture. Summary of the Invention

[0003] The purpose of this disclosure is to provide a human motion capture method based on the alignment of visual and inertial feature manifolds. By constructing a deep coupling mechanism between visual priors and dynamic constraints, it solves the technical pain points of monocular visual motion capture, such as pose ambiguity, physical distortion and poor stability caused by an excessively large search space when facing rapid human movement, lack of depth information and complex occlusion.

[0004] In general, a human motion capture method based on visual and inertial feature manifold alignment is provided, including: Step S1: Receive the monocular vision sequence and extract the original visual feature manifold containing the human body topology through the static weighted visual processing branch; Step S2: Synchronously input sparse IMU data and use the dynamic coding branch to extract the temporal features of acceleration and angular velocity containing human dynamic constraints; Step S3: Design a conditional feature alignment bridging operator to project the original visual feature manifold and temporal features onto a unified associated manifold space, and achieve deep coupling between visual and inertial features through a zero-initialization residual mechanism. Step S4: Input the hybrid enhancement features into the monocular human morphology parameterization model, and use the rotation constraints provided by the IMU to perform nonlinear compensation on the monocular visual depth estimation to predict the human three-dimensional pose and the parameterized human model mesh parameters. Step S5: Construct a multi-objective loss function based on dynamic consistency constraints, and perform end-to-end training on each branch and operator.

[0005] The process of extracting acceleration and angular velocity temporal features using dynamic coding branches specifically includes the following steps: Step S21, Sparse IMU data sequence Preprocessing, where F is the number of sampling frames, and each frame of IMU data includes acceleration. and angular velocity This includes resampling alignment based on image sequence timestamps, and high-frequency denoising using smoothing filters; Step S22: Using a dynamic feature mapping operator composed of multi-layer linear projection and nonlinear activation units, the preprocessed IMU data is mapped to a feature dimension consistent with visual features to obtain IMU conditional features. .

[0006] The implementation method of the dynamic feature mapping operator is as follows: ,in The weight matrix is ​​a learnable matrix. For the corresponding bias vector, This represents a nonlinear activation operator (such as a linear rectifier function). This represents a feature normalization operator (such as a layer normalization function).

[0007] The implementation logic of the conditional feature alignment bridging operator specifically includes the following steps: Step S31: Transfer the IMU conditional features Through zero convolutional layer Perform weight transformation and combine with visual features Perform element-wise summation to obtain the connection characteristics. ; Step S32: Construct a trainable parallel branch that is symmetrical to the visual processing branch structure. The connection feature Input this parallel branch to extract multimodal coupling features ; Step S33: Couple the multimodal features Initialize the mapping layer with zero After processing, the residual connections are used to link the original visual features. Blending to generate the hybrid enhancement feature And input to the back-end decoder middle.

[0008] The connection features The way to obtain it is: ,in It is a visual feature. Represents a zero convolutional layer operator. It is a weight matrix initialized to zero.

[0009] The multimodal coupling feature The way to obtain it is, ,in This represents a trainable parallel network that is symmetrical to the visual processing branch structure.

[0010] The mapping layer is initialized with zero. Processed multimodal coupling features With visual features Mix them together and input them to the back-end decoder The specific method is as follows ,in It is a multilayer perceptron with all weights and biases initialized to zero. It is a hybrid enhancement feature.

[0011] The loss function method used in S5 is as follows: Supervision is applied to human pose, shape, and keypoints. The total loss function is a weighted sum of the losses of each component, as shown in the following formula: , Among them, attitude loss: , Shape loss: , Keypoint reprojection loss: .

[0012] The parameterized human body model includes shape parameters and posture parameters. The shape parameters describe the shape information of the human body model, such as height and body shape, while the posture parameters describe the relative rotational posture of the human body's joints.

[0013] The technical effects to be achieved by the embodiments of the present invention are as follows: 1. By introducing inertial measurement unit (IMU) data as a dynamic constraint source, the spatial representation capability of vision and the temporal rotation accuracy of inertia are aligned in a unified associated manifold space. This effectively alleviates the pose blurring and instability problems that are prone to occur in monocular vision methods in scenarios such as rapid movement, severe occlusion and limited viewpoint, and significantly improves the accuracy of 3D pose and mesh reconstruction.

[0014] 2. A conditional feature alignment bridging operator is used to inject IMU conditional features, ensuring that the model can fully retain the generalization ability of the pre-trained visual encoder in the early stages of training, avoiding gradient oscillations caused by the introduction of heterogeneous data. Simultaneously, through a parallel branching structure, a plug-in enhancement of the monocular motion capture model is achieved without disrupting the original model's decoding logic. This allows for generalization to existing mainstream monocular visual motion capture models, demonstrating good versatility and scalability.

[0015] 3. By adopting a decoupling strategy of static weight branching and local module training, the parameter space to be optimized is greatly compressed, and the training cycle is shortened. At the same time, by introducing dynamic consistency supervision into the loss function, the output parameterized human model not only conforms to visual anatomical structure but also has more realistic physical motion characteristics, effectively solving typical fault points such as feet crossing the ground or joints rotating in reverse. Attached Figure Description

[0016] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0017] Figure 1 This is a schematic diagram illustrating the architecture of a human motion capture method based on visual and inertial feature manifold alignment according to an embodiment of the present disclosure. Detailed Implementation

[0018] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0019] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0020] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0021] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0022] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0023] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0024] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0025] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0026] Figure 1 This is a schematic diagram illustrating a human motion capture method based on visual and inertial feature manifold alignment according to an embodiment of the present disclosure.

[0027] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 1 As shown.

[0028] Specifically, it includes 5 steps: Step S1: Receive the monocular vision sequence and extract the original visual feature manifold containing the human body topology through the static weighted visual processing branch; Step S2: Synchronously input sparse IMU data and use the dynamic coding branch to extract the temporal features of acceleration and angular velocity containing human dynamic constraints; Step S3: Using the conditional feature alignment bridging operator, the original visual feature manifold and temporal features are projected onto a unified associated manifold space, and a hybrid enhancement feature with motion continuity prior is generated through the residual injection mechanism. Step S4: Input the hybrid enhancement features into the monocular human morphology parameterization model, and use the rotation constraints provided by the IMU to perform nonlinear compensation on the monocular visual depth estimation to predict the human three-dimensional pose and the parameterized human model mesh parameters. Step S5: Construct a multi-objective loss function based on dynamic consistency constraints, and perform end-to-end training on each branch and operator.

[0029] Furthermore, the visual encoder is configured as a static feature anchoring unit. Through pre-defined parameterized spatial weights, it maps local images into stable high-dimensional spatial representations. During subsequent training, the mapping operator of this unit maintains constant weights, thereby anchoring the spatial geometric priors obtained through large-scale data pre-training and avoiding catastrophic forgetting when introducing heterogeneous data later. In addition, a geometric topological constraint branch is introduced. This branch extracts two-dimensional anatomical keypoints from local images to construct a self-supervised operator based on reprojection error.

[0030] Furthermore, the conditional feature alignment bridging operator couples the motion changes extracted by the IMU into the visual inference process through a residual injection mechanism, and its input / output interface is logically compatible with mainstream human morphology inference kernels.

[0031] Furthermore, the aforementioned parametric human body model includes shape parameters and posture parameters. The shape parameters describe the shape information of the human body model, such as height and body shape, while the posture parameters describe the relative rotational posture of the human body's joints.

[0032] Furthermore, step S3 specifically includes the following steps: Step S31: Transfer the IMU conditional features Through zero convolutional layer Perform weight transformation and combine with visual features Perform element-wise summation to obtain the connection characteristics. ; Step S32: Construct a trainable parallel branch that is symmetrical to the visual processing branch structure. The connection feature Input this parallel branch to extract multimodal coupling features ; Step S33: Couple the multimodal features Initialize the mapping layer with zero After processing, the residual connections are used to link the original visual features. Blending to generate the hybrid enhancement feature And input to the back-end decoder middle.

[0033] Furthermore, step S2 specifically includes the following steps: Step S21, Sparse IMU data sequence Preprocessing, where F is the number of sampling frames, and each frame of IMU data includes acceleration. and angular velocity This includes resampling alignment based on image sequence timestamps, and high-frequency denoising using smoothing filters; Step S22: Using a dynamic feature mapping operator composed of multi-layer linear projection and nonlinear activation units, the preprocessed IMU data is mapped to a feature dimension consistent with visual features to obtain IMU conditional features. .

[0034] Specifically, step S31 above connects the two types of features according to the following formula to obtain... ,in It is a visual feature. Represents a zero convolutional layer operator. It is a weight matrix initialized to zero: . Specifically, step S32 above obtains the multimodal coupling features according to the following formula. ,in Representing a trainable parallel branch with a symmetrical branching structure in visual processing: . Specifically, step S33 above is based on the following formula and the original visual features. Mixed input back-end decoder, where It is a multilayer perceptron with all weights and biases initialized to zero. It is a hybrid enhancement feature: . Specifically, step S22 above obtains the IMU conditional features according to the following formula. ,in The weight matrix is ​​a learnable matrix. For the corresponding bias vector, This represents a nonlinear activation operator (such as a linear rectifier function). Characteristic normalization operators (such as layer normalization functions): . Specifically, step S5 above involves supervising human pose, shape, and key points. The total loss function is a weighted sum of the losses of each component, as shown in the following formula: , Among them, attitude loss: , Shape loss: , Keypoint reprojection loss: . While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for human motion capture based on visual and inertial feature manifold alignment, characterized in that, Step S1: Extract spatially salient regions from the monocular visual sequence and extract the original visual feature manifold containing the human body topology through static weighted visual processing branches; Step S2: Synchronously input sparse IMU data and extract the temporal features of acceleration and angular velocity containing human dynamic constraints through the dynamic coding branch; Step S3: Using the conditional feature alignment bridging operator, the original visual feature manifold and temporal features are projected onto a unified associated manifold space, and a hybrid enhancement feature with motion continuity prior is generated through the residual injection mechanism. Step S4: Map the hybrid enhancement features to the three-dimensional human pose solution space, use the rotation constraints provided by the temporal features to perform nonlinear compensation for monocular visual projection ambiguity, and output the three-dimensional human pose parameters and parameterized human model mesh parameters. Step S5: Construct a multi-objective loss function based on dynamic consistency constraints, and perform end-to-end training on each branch and operator.

2. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 1, characterized in that: The process of extracting acceleration and angular velocity temporal features through dynamic coding branches specifically includes the following steps: Step S21, Sparse IMU data sequence Preprocessing, where F is the number of sampling frames, and each frame of IMU data includes acceleration. and angular velocity This includes resampling alignment based on image sequence timestamps, and high-frequency denoising using smoothing filters; Step S22: Using a dynamic feature mapping operator composed of multi-layer linear projection and nonlinear activation units, the preprocessed IMU data is mapped to a feature dimension consistent with visual features to obtain IMU conditional features. .

3. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 2, characterized in that: The implementation method of the dynamic feature mapping operator is as follows: ,in The weight matrix is ​​a learnable matrix. For the corresponding bias vector, This represents a nonlinear activation operator (such as a linear rectifier function). This represents a feature normalization operator (such as a layer normalization function).

4. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 1, characterized in that: The implementation logic of the conditional feature alignment bridging operator specifically includes the following steps: Step S31: Transfer the IMU conditional features Through zero convolutional layer Perform weight transformation and combine with visual features Perform element-wise summation to obtain the connection characteristics. ; Step S32: Construct a trainable parallel branch that is symmetrical to the visual processing branch structure. The connection feature Input this parallel branch to extract multimodal coupling features ; Step S33: Couple the multimodal features Initialize the mapping layer with zero After processing, the residual connections are used to link the original visual features. Blending to generate the hybrid enhancement feature And input to the back-end decoder middle.

5. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 4, characterized in that: The connection features The method of obtaining it is as follows: ,in It is a visual feature. Represents a zero convolutional layer operator. It is a weight matrix initialized to zero; The multimodal coupling feature The way to obtain it is, ,in This represents a trainable parallel network that is symmetrical to the visual processing branch structure.

6. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 4, characterized in that: The mapping layer is initialized with zero. Processed multimodal coupling features With visual features Mix them together and input them to the back-end decoder The specific method is as follows: ,in It is a multilayer perceptron with all weights and biases initialized to zero. It is a hybrid enhancement feature.

7. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 1, characterized in that: The loss function method used in step S5 is as follows: It is designed to supervise human pose, shape, and key points, and the total loss function is a weighted sum of the losses of each component, as shown in the following formula: Among them, attitude loss: Shape loss: Keypoint reprojection loss: .

8. The human motion capture method based on visual and inertial feature manifold alignment as described in claim 1, characterized in that: The parameterized human body model includes shape parameters and posture parameters. The shape parameters describe the shape information of the human body model, such as height and body shape, while the posture parameters describe the relative rotational posture of the human body's joints.

Images