Human motion capture method and device based on sparse imu

Abstract

The embodiment of the application provides a human motion capture method and device based on sparse IMU, inertia data measured by IMU worn at the lower left elbow, the lower right elbow, the lower left knee, the lower right knee and the head of a human body is collected, the inertia data of the five IMUs is input into a human posture estimation model, first human posture information and foot ground contact state information are obtained, and position information of the head and the left and right hands is detected, and second human posture information is determined according to the first human posture information, the foot ground contact state information, the position information of the head and the hands. The method can realize human posture capture by using the data of the five IMUs, and compared with the prior art, the waist IMU is reduced, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of human motion capture technology, and in particular to a human motion capture method and device based on sparse IMU. Background Technology

[0002] Human motion capture technology is a technique that detects the posture and movement trajectory of a human body in three-dimensional space and reproduces human movement in a virtual three-dimensional environment. It is widely used in film production, motion analysis, game development, and medical diagnosis.

[0003] Currently, human motion capture methods mainly fall into four categories: optical motion capture, inertial motion capture, visual motion capture, and hybrid motion capture. Except for inertial motion capture, the other three methods require additional equipment to be deployed in the scene, making them unsuitable for applications such as Virtual Reality (VR). Currently, mature inertial motion capture products on the market require a large number of inertial measurement units (IMUs), with some products requiring more than 17 IMUs. Existing products require at least 6 IMUs, which are worn on the left arm, right arm, lower left knee, lower right knee, head, and waist.

[0004] However, a waist-mounted IMU can severely impact the user experience. Summary of the Invention

[0005] This application provides a human motion capture method and device based on sparse IMUs, which can achieve human posture capture using data from 5 IMUs. Compared with the prior art, it reduces the number of waist IMUs and improves the user experience.

[0006] In a first aspect, embodiments of this application provide a human motion capture method based on sparse IMUs, the method comprising:

[0007] Inertial data is collected from inertial sensors (IMUs) worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head of the human body.

[0008] The inertial data is input into the human posture estimation model to obtain the first human posture information and foot contact state information;

[0009] Detect the position information of the head and left and right hands;

[0010] The second human posture information is determined based on the first human posture information, the foot contact state information, and the position information.

[0011] In some embodiments, inputting the inertial data into a human posture estimation model to obtain first human posture information and foot contact state information includes:

[0012] The orientation of each joint in the human body is determined based on the inertial data, and the joints of the human body include the lumbar joints;

[0013] Based on the orientation of each joint of the human body, the posture information of the first human body and the ground contact state information of the foot are determined.

[0014] In some embodiments, determining the orientation of each joint of the human body based on the inertial data includes:

[0015] Based on the inertial data, estimate the orientation of the root node of the human body;

[0016] Based on the orientation of the root node, the inertial data is regularized so that the orientation of the inertial data is aligned with the orientation of the root node.

[0017] Based on the inertial data after orientation alignment, estimate the 3D coordinates of the joints where each IMU is located;

[0018] Based on the 3D coordinates of the joints where each IMU is located, estimate the 3D coordinates of the remaining joints of the human body.

[0019] In some embodiments, determining the first human posture information and the foot contact state information based on the orientation of the joints of the human body includes:

[0020] Based on the estimated 3D coordinates of each joint of the human body, the posture information of the first human body is estimated;

[0021] Based on the inertial data after orientation alignment and the estimated 3D coordinates of each joint of the human body, a binary classification algorithm is used to determine the foot contact state information.

[0022] In some embodiments, the human pose estimation model is trained using a supervised learning method.

[0023] In some embodiments, the loss of the human pose estimation model is a weighted sum of the following six losses: a first loss, a second loss, a third loss, a fourth loss, a fifth loss, and a sixth loss;

[0024] The first loss is the mean square error loss between the true value and the estimated value of the orientation of the root node;

[0025] The second loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of each IMU joint;

[0026] The third loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of the remaining joints of the human body.

[0027] The fourth loss is the mean square error loss between the true value and the estimated value of the first human posture information;

[0028] The fifth loss is the temporal smoothing loss between data frames used for human motion capture;

[0029] The sixth loss is the binary cross-entropy (BCE) loss between the true value and the estimated value of the foot-touching state.

[0030] In some embodiments, the true value of the foot-touching state is calculated using the following two parameters: whether the foot is stationary and whether the height of the foot from the ground is less than a threshold.

[0031] In some embodiments, determining the second human posture information based on the first human posture information, the foot contact state information, and the position information includes:

[0032] Based on rigid body mechanics and physical constraints, an optimization algorithm is used to solve for the first human posture information, the foot contact state information, and the position information to obtain the second human posture information.

[0033] In some embodiments, the optimization algorithm employs the following optimization equation:

[0034]

[0035] st

[0036]

[0037] λ∈F

[0038]

[0039] in, Here, λ represents the first human posture information, the position information of the head and left and right hands, λ represents the ground contact force acting on the point of contact between the human body and the ground, and τ represents the torque at the joints of the human body. Let c be the Jacobian matrix of the point of contact between the person and the ground, εPD be the energy term of the PD controller, and ε reg To optimize the regularization term, M is the human body inertia matrix, h is the nonlinear term including Coriolis force, centrifugal force, and gravity, and F is the friction cone constraint set of ground contact forces. Let j be the linear velocity of the joint in contact with the ground, and ζ be the set of constraints on the linear velocity of the joint in contact with the ground.

[0040] In some embodiments, the IMUs at the lower left and right elbows are integrated into the handle.

[0041] On the other hand, embodiments of this application provide a human motion capture device based on sparse IMU, the device comprising:

[0042] The data acquisition module is used to acquire inertial data measured by inertial sensors (IMUs) worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head of the human body.

[0043] A human posture estimation model is used to estimate first human posture information and foot contact state information based on the inertial data.

[0044] The position detection module is used to detect the position information of the head and left and right hands;

[0045] The fusion estimation module is used to determine the second human posture information based on the first human posture information, the foot contact state information, and the position information.

[0046] On the other hand, embodiments of this application provide an electronic device, the electronic device including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in any of the above.

[0047] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in any of the foregoing descriptions.

[0048] On the other hand, embodiments of this application provide a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the method described in any of the above-mentioned embodiments.

[0049] The human motion capture method and device based on sparse IMUs provided in this application collects inertial data from IMUs worn at the lower left elbow, lower right elbow, lower left knee, lower right knee, and head. The inertial data from these five IMUs are input into a human posture estimation model to obtain first human posture information and foot contact state information. The position information of the head and left and right hands is also detected. Based on the first human posture information, foot contact state information, and head and hand position information, second human posture information is determined. This method achieves human posture capture using only data from five IMUs, reducing the need for a waist-mounted IMU compared to existing technologies and improving the user experience. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of a hardware system to which the method of the embodiments of this application is applicable;

[0052] Figure 2 A flowchart of the human motion capture method based on sparse IMU provided in Embodiment 1 of this application;

[0053] Figure 3 A schematic diagram of an algorithm architecture provided for this application;

[0054] Figure 4 A flowchart of the human motion capture method based on sparse IMU provided in Embodiment 2 of this application;

[0055] Figure 5 This is a schematic diagram of the structure of a human motion capture device based on a sparse IMU, provided in Embodiment 3 of this application;

[0056] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0059] This application provides a human motion capture method based on sparse IMU, which can be applied to head-mounted extended reality (XR) devices.

[0060] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR is also a collective term for various technologies including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). By integrating the visual interaction technologies of these three technologies, it provides users with a seamless and immersive experience between the virtual and real worlds. Head-mounted XR devices include, but are not limited to, head-mounted VR, AR, and MR devices.

[0061] VR: A technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0062] VR devices refer to terminals that achieve virtual reality effects. They can typically be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged as needed.

[0063] AR: An AR scene refers to a simulated scene in which at least one virtual object is superimposed on a physical scene or its representation. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or videos of the physical scene, which are representations of the physical scene. The system combines the images or videos with virtual objects and displays this combination on the opaque display. Individuals use the system to indirectly view the physical scene via images or videos of the physical scene and observe the virtual objects superimposed on the physical scene. When the system uses one or more image sensors to capture images of the physical scene and uses those images to present the AR scene on an opaque display, the displayed images are referred to as video pass-through. Alternatively, the electronic system for displaying the AR scene may have a transparent or semi-transparent display through which an individual can directly view the physical scene. The system may display virtual objects on the transparent or semi-transparent display, allowing an individual to observe the virtual objects superimposed on the physical scene using the system. As another example, the system may include a projection system that projects virtual objects onto the physical scene. Virtual objects can be projected, for example, onto a physical surface or as holograms, allowing individuals to observe virtual objects superimposed on a physical setting using the system. Specifically, a technique involves calculating the camera's pose parameters in the real world (or 3D world, the real world) in real time during image acquisition, and adding virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0064] MR (Mixed Reality): By presenting virtual scene information within a real-world setting, an interactive feedback loop is established between the real world, the virtual world, and the user to enhance the realism of the user experience. For example, computer-generated sensory input (e.g., virtual objects) is integrated with sensory input or its representation from a physical setting within a simulated scene. In some MR scenes, the computer-generated sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0065] This application provides an IMU-based human motion capture method, which can capture human motion based on inertial data from five IMUs. Figure 1 This is a schematic diagram of a hardware system to which the method of this application embodiment is applicable, such as... Figure 1As shown, the hardware system includes: one head-mounted display (HMD), two controllers, and two lower limb controllers. The HMD is worn on the user's head, the two controllers are worn on the user's left and right wrists respectively, and the two lower limb controllers are worn on the right and left ankles respectively. It is understood that the two lower limb controllers can be worn either below the right or left knee; this embodiment does not impose any limitation on this.

[0066] The HMD, two handheld controllers, and two lower limb controllers each have an IMU built into them. An IMU is a common inertial sensor that measures inertial data. The inertial data from these five IMUs can include triaxial acceleration and triaxial orientation data calculated internally by the sensors. The orientation can be represented by a quaternion.

[0067] In addition to an IMU, an HMD may include, but is not limited to, the following modules: 1) Detection (module): Detects user commands using various sensors and applies them to the virtual environment, such as continuously updating the image displayed on the screen based on the user's gaze, enabling interaction between the user and the virtual environment and scene, for example, continuously updating virtual reality content based on the detected direction of the user's head rotation; 2) Feedback (module): Receives data from sensors and provides real-time feedback to the user; 3) Sensors: On one hand, they receive user commands and apply them to the virtual environment; on the other hand, they provide the results of the operation to the user in various feedback forms; 4) Control module: Controls sensors and various input / output devices, including acquiring user data (such as actions and voice) and outputting sensory data, such as images, vibrations, temperature, and sound, which affect the user, the virtual environment, and the real world; 5) Modeling module: Constructs a three-dimensional model of the virtual environment, and may also include various feedback mechanisms such as sound and touch within the three-dimensional model.

[0068] Two controllers and two lower limb controllers can connect to the HMD via short-range communication technologies such as Bluetooth or WIFI to enable data transmission between the controllers, lower limb controllers, and the HMD.

[0069] Compared to existing technologies, the method in this application does not require a wearable device with a built-in IMU to be worn on the waist. It only requires data from 5 IMUs to achieve human motion capture, thus improving the user experience.

[0070] Figure 2 This is a flowchart of the human motion capture method based on sparse IMU provided in Embodiment 1 of this application. This embodiment uses the aforementioned HMD as the execution subject for illustration. Figure 2As shown, the human motion capture method based on sparse IMU includes the following steps.

[0071] S101. Collect inertial data from IMUs worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head of the human body.

[0072] The inertial data includes skeletal acceleration and orientation. The acceleration is the triaxial linear acceleration in the sensor's local coordinate system. The orientation is the sensor's orientation in a global inertial frame (typically NE-G) shared by the five sensors, represented by a rotation matrix or a quaternion.

[0073] After obtaining inertial data from four IMUs located below the left and right elbows, the left and right knees, this data is sent to the HMD. The IMUs located below the left and right elbows can be worn at any position on the lower part of the elbow, specifically on the hand, wrist, or forearm. For HMDs controlled by a handle, the IMUs can be integrated into the handle. The difficulty of attitude estimation varies depending on the wearing position; estimation at the hand and wrist is more difficult than estimation at the forearm because the wrist has significantly more degrees of freedom in rotation than the arm.

[0074] Optionally, in this embodiment, the inertial data of the IMU is sampled and synchronized at 60Hz, that is, 60 frames of data can be obtained per second, and each frame includes the raw acceleration of 5 IMUs and the orientation represented by the rotation matrix.

[0075] S102. Input the inertial data into the human posture estimation model to obtain the first human posture information and foot contact state information.

[0076] The main function of this human posture estimation model is to estimate the first human posture information and foot ground contact status information based on the acceleration and orientation of five IMUs. The first human posture information does not include the body's motion information, but only the posture information of the body's joints. The first human posture information includes the orientation information of the root node, and specifically, it can be the rotation matrix of each joint. The foot ground contact status information includes whether the left and right feet are in contact with the ground.

[0077] For example, the orientation of each joint of the human body is determined based on inertial data, including the lumbar joint. Based on the orientation of each joint of the human body, the first human posture information and foot contact state information are determined.

[0078] For example, taking the Skinned Multi-Person Linear (SMPL) model, the SMPL model includes 24 joints: head node, neck node, left collar node, right collar node, left shoulder node, right shoulder node, 3 cervical vertebra nodes, pelvis node, left elbow node, right elbow node, left wrist node, right wrist node, left hand, right hand, left hip node, right hip node, left knee node, right knee node, left ankle node, right ankle node, left foot, and right foot. The root node is a hypothetical node located in the pelvis, used to represent the human body's position in the world coordinate system.

[0079] The human skeleton is tightly connected by joints. The human skeleton consists of individual bones, which are linked together by joints to form the framework of the body. Joints are rotatable. In this skeletal model, the orientation of a joint can be represented by the rotation of its coordinate system. The rotation of a joint can be represented by a rotation matrix. Typically, joints at the elbow and knee are assumed to be hinge joints, meaning they can only rotate within a fixed plane. For example, for the elbow joint, the plane formed by the shoulder node, elbow node, and wrist node is the plane of rotation. All nodes except hinge joints can rotate arbitrarily.

[0080] In this embodiment, although the IMU did not collect the orientation information of the waist, the orientation of the waist joint can be estimated by using the inertial data of the other 5 IMUs.

[0081] S103. Detect the position information of the head and left and right hands.

[0082] The HMD integrates a positioning module that can detect the position information of the head and left and right hands. For example, this positioning module is a Simultaneous Localization and Mapping (SLAM) module, which can perform positioning based on data collected by the HMD's sensor modules such as cameras, gyroscopes, lasers, accelerometers, and infrared sensors. This SLAM module can also locate the position information of the left and right hands based on data measured from the handles. The position information of the head and left and right hands can be three-dimensional coordinates in the human body coordinate system.

[0083] S104. Determine the second human posture information based on the first human posture information, the foot contact status information, and the position information of the head and left and right hands.

[0084] For example, based on rigid body mechanics and physical constraints, the first human posture information, foot contact state information, head position information, and left and right hand position information can be solved to obtain the second human posture information. An optimization algorithm can be used to solve for the second human posture information.

[0085] The second human posture information may include the coordinate values ​​(xyz) of each joint of the human body, as well as the orientation and posture of the human body. The orientation and posture of the human body can be obtained by converting the coordinate values ​​of each joint.

[0086] The position information of the left and right hands can be obtained through the handle, allowing for data optimization based on the handle's 6 degrees of freedom (DoF) data, resulting in more accurate estimated hand postures. The 6DoF data for the joints includes 3DoF of rotational data and 3DoF of sliding data. The rotational data can be understood as the orientation of the joint, and the sliding data can be understood as the xyz translation of the joint.

[0087] The positional information of the head and left and right hands is easy to obtain, which makes the calculation of posture prediction in this step simpler and the obtained positional information more accurate, thereby improving the accuracy of human posture prediction.

[0088] In this embodiment, inertial data is collected from IMUs worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head. The inertial data from these five IMUs is input into a human posture estimation model to obtain first human posture information and foot contact state information. The position information of the head and left and right hands is also detected. Based on the first human posture information, foot contact state information, and head and hand position information, second human posture information is determined. This method uses data from only five IMUs to achieve human posture capture, reducing the need for a waist-mounted IMU compared to existing technologies and improving the user experience.

[0089] Figure 3 A schematic diagram of an algorithm architecture provided in this application, such as Figure 3 As shown, the architecture mainly consists of two parts: a human pose estimation model and post-processing. The human pose estimation model primarily functions to estimate two aspects of information based on inertial data measured by IMUs worn at the lower left and right elbows, lower left and right knees, and the head: first human pose information and foot ground contact information. The first human pose information includes the orientation information of the root node. The post-processing part primarily functions to solve for the final pose information by fusing physical constraints with the first human pose information, foot ground contact information, and the position information of the head and left and right hands, based on the output of the human pose estimation model.

[0090] like Figure 3As shown, the human posture estimation model adopts a coarse-to-fine estimation process, which starts from estimating the overall orientation of the human body -> estimating the 3D coordinates of the joint (robot joint) corresponding to the location of the IMU sensor -> estimating the 3D coordinates of the missing joints on the skeleton link -> IK solves the posture parameters. In addition, the foot contact state information is also estimated.

[0091] For example, the human pose estimation model includes six modules: root node orientation estimation module, regularization module, leaf node 3D coordinate estimation module, joint 3D coordinate estimation module, pose estimation module, and foot contact estimation module. Except for the regularization module, the other five modules can all use a standard recurrent neural network (RNN). Optionally, the standard RNN can include two linear layers and one two-layer unidirectional long short-term memory network (LSTM). The LSTM network can be replaced with a bidirectional recurrent neural network.

[0092] Root node orientation estimation module: This module estimates the orientation of the human root node based on the inertial data from five IMUs. Since inertial data from the lumbar IMU is lacking, this embodiment requires estimating the root node orientation using the inertial data from all five IMUs. The root node is assumed to be located in the pelvis, and can therefore be understood as a lumbar node. The root node orientation estimated by this module can be used not only for regularization of the IMU inertial data but also as the root node orientation output in the final attitude calculation.

[0093] The regularization module is used to regularize the inertial data based on the orientation of the root node, aligning the orientation of the inertial data to the orientation of the root node. The inertial data includes skeletal acceleration and orientation; aligning the orientation of the inertial data to the orientation of the root node can be understood as aligning the skeletal orientation and acceleration to the root node's coordinate system.

[0094] For example, the orientation of the root node can be kept unchanged, and the inertial data of other nodes can be transformed from the global world coordinate system to the root node coordinate system, thereby aligning the orientation of the inertial data with the orientation of the root node.

[0095] The orientation of the root node can be arbitrary, and this embodiment does not limit this.

[0096] Leaf node 3D coordinate estimation module: used to estimate the 3D coordinates of the joints where each IMU is located based on the inertial data after orientation alignment. The 3D coordinates of the joint where the IMU is located are the 3D coordinates of the joint corresponding to the location of the IMU. Here, the joints where the IMU is located include the wrist, ankle, and head joints. The wrist, ankle, and head are located at the ends of the human skeleton, so the joints where the IMU is located are also called leaf nodes or end nodes.

[0097] The Joint3D coordinate estimation module estimates the 3D coordinates of the remaining joints in the human body based on the 3D coordinates of the joints where each IMU is located, as estimated by the leaf node 3D coordinate estimation module. As can be seen from the human skeletal model, besides the end joints, there are many other joints. The function of the Joint3D coordinate estimation module is to complete the 3D coordinates of the remaining joints.

[0098] Pose estimation module: Based on the estimated 3D coordinates of each joint of the human body, this module estimates the first human pose information. Its function is similar to inverse kinematics (IK) solving, processing the 3D coordinates of all joints through the model to calculate the rotation matrices of all joints.

[0099] Foot contact estimation module: This module determines the foot contact status information using a binary classification algorithm based on the aligned inertial data and the estimated 3D coordinates of each joint in the human body. The binary classification algorithm produces two results; in this embodiment, the results are foot contact and foot not contacting the ground.

[0100] Human pose estimation models can be trained using large datasets of human motion capture data (containing a large number of known human poses and movements). Optionally, they can be trained using supervised learning methods, where the data is labeled and appears in the form of (x,t), where x is the input data and t is the label. Correct t labels are ground truth (GT), while incorrect labels are not.

[0101] The loss of the human pose estimation model is a weighted sum of the following six losses: the first loss, the second loss, the third loss, the fourth loss, the fifth loss, and the sixth loss. The estimation positions of each loss are shown in the figure. Figure 3 As shown, the definitions of each loss are as follows.

[0102] The first loss is the mean squared error loss (MES loss) between the true and estimated values ​​of the root node's orientation, also known as the orientation estimation loss. Here, the estimated value refers to the result estimated by the root node orientation estimation module, while the true value of the orientation is labeled in the model's input data.

[0103] The second loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of each IMU joint, also known as the leaf node estimation loss. Here, the estimated value is the result estimated by the leaf node 3D coordinate estimation module, while the true value is labeled in the input data of the model.

[0104] The third loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of the remaining joints of the human body, also known as the node estimation loss. Here, the estimated value is the result obtained by the Joint3D coordinate estimation module, while the true value is labeled in the input data of the model.

[0105] The fourth loss is the mean square error loss between the true value and the estimated value of the first human posture information, also known as the posture estimation loss. Here, the estimated value is the result estimated by the posture estimation module, while the true value is labeled in the input data of the model.

[0106] The fifth loss is the temporal smoothing loss between data frames used for motion capture. A common problem in pose estimation models is that while the estimation results perform well in a single image, they become jittery in a video.

[0107] To address this, this application proposes a time-dimensional attitude smoothing process. Since the input is continuous, the position, velocity, and acceleration values ​​can be calculated based on the changes in key point positions between adjacent frames. The jitter between adjacent frames can also be understood as an acceleration loss. Among these, the velocity and acceleration are closely related to the continuity of motion.

[0108] The sixth loss is the binary cross-entropy (BCE) loss between the true and estimated values ​​of the foot-touching state, also known as the foot-touching binary classification loss. Here, the estimated value is the result obtained by the foot-touching estimation module, and the calculation of the true value is crucial. The true value is calculated using the following two parameters: whether the foot is stationary and whether the height of the foot from the ground is less than a threshold. The true value calculated using these two parameters is more accurate.

[0109] based on Figure 3 The algorithm architecture shown in Embodiment 2 of this application provides a human motion capture method based on sparse IMU, employing... Figure 3 The algorithm architecture shown implements human motion capture. Figure 4 The flowchart of the human motion capture method based on sparse IMU provided in Embodiment 2 of this application is as follows: Figure 4 As shown, the method provided in this embodiment includes the following steps.

[0110] S201. Collect inertial data from IMUs worn on the left hand, right hand, left ankle, right ankle, and head of the human body.

[0111] S202. Input the inertial data into the human posture estimation model.

[0112] S203. Based on inertial data, estimate the orientation of the root node of the human body.

[0113] This step is estimated by the root node orientation estimation module of the human pose estimation model. The root node orientation estimation module estimates the orientation of the root node based on the inertial data of 5 IMUs.

[0114] S204. Based on the orientation of the root node, perform regularization on the inertial data so that the orientation of the inertial data is aligned with the orientation of the root node.

[0115] This step is handled by the regularization module mentioned above.

[0116] S205. Based on the inertial data after orientation alignment, estimate the 3D coordinates of the joints where each IMU is located.

[0117] This step uses the leaf node 3D coordinate estimation module to estimate the 3D coordinates of the head, two wrists, and two ankle joints based on the inertial data after orientation alignment.

[0118] S206. Estimate the 3D coordinates of the remaining joints of the human body based on the 3D coordinates of the joints where each IMU is located.

[0119] This step uses the Joint3D coordinate estimation module to estimate the 3D coordinates of the head, two wrists, and two ankles based on the leaf node 3D coordinate estimation module, and then estimates the 3D coordinates of the remaining joints of the human body.

[0120] S207. Estimate the first human posture information based on the estimated 3D coordinates of each joint of the human body.

[0121] This step uses the 3D coordinates of each joint of the human body estimated by the Joint3D coordinate estimation module to obtain the first human posture information through human inverse dynamics estimation.

[0122] S208. Based on the inertial data after orientation alignment and the estimated 3D coordinates of each joint of the human body, a binary classification algorithm is used to determine the foot contact state information.

[0123] This step uses the foot contact estimation module to estimate the foot contact state information based on the aligned inertial data and the 3D coordinates of each joint of the human body estimated by the Joint3D coordinate estimation module.

[0124] S209. Detect the position information of the head and left and right hands.

[0125] Referring to the relevant description in Example 1, it will not be repeated here.

[0126] S210. Based on rigid body mechanics and physical constraints, an optimization algorithm is used to solve for the first human posture information, foot contact state information, and head and left and right hand position information to obtain the second human posture information.

[0127] This step is implemented by the post-processing section, such as... Figure 3 As shown, the input to the post-processing section includes two parts: the first human pose information and foot contact information output by the human pose estimation model, and the position information of the head and left and right hands. For example, the optimization equation used in this optimization algorithm is:

[0128]

[0129] st

[0130]

[0131] λ∈F

[0132]

[0133] in, Here, λ represents the initial human posture information, the position information of the head and left and right hands, λ represents the ground contact force acting on the point of contact between the human body and the ground, and τ represents the torque at the joints of the human body. Let c be the Jacobian matrix of the point of contact between the person and the ground, εPD be the energy term of the PD controller, and ε reg To optimize the regularization term, M is the human body inertia matrix, h is the nonlinear term including Coriolis force, centrifugal force, and gravity, and F is the friction cone constraint set of ground contact forces. Let j be the linear velocity of the joint in contact with the ground, and ζ be the set of constraints on the linear velocity of the joint in contact with the ground.

[0134] As shown in the above formula, the parameters that need to be solved mainly include three categories: (1) the first human posture + the position information of the head and left and right hands. (2) The contact force (i.e., support force) λ of the ground on the human body, (3) The torque τ of the joint. Among them, the latter two types of parameters can be understood as intermediate results, and their specific values ​​are not constrained during optimization (except for the direction). The first part of minimization is to make the estimated posture as close as possible to the model output result, and the estimated position information as close as possible to the xyz position of the head and hands (i.e., HDM and handle). The second part is a regularization term, which contains the following constraint 3. The constraint terms of the equation include: (1) force balance, i.e. constraint 2; (2) tribological constraint, i.e. constraint 4; (3) when the foot contacts the ground, the velocity should be close to 0, i.e. constraint 1.

[0135] Constraint 1: When the model determines that the foot is in contact with the ground and the confidence level is high, the speed of the foot relative to the ground should be close to 0.

[0136] Constraint 2: The human body as a whole must maintain a balance of forces.

[0137] Constraint 3: The supporting force from the ground must be upward and can only be applied to the key points of contact with the ground.

[0138] Constraint 4: The supporting force from the ground must conform to tribological constraints (the angle between the supporting force and the vertical y-axis must be less than a certain angle).

[0139] In this embodiment, inertial data from five IMUs worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head are collected. This inertial data is input into a human posture estimation model. Based on the inertial data, the orientation of the root node is estimated. Further, the 3D coordinates of the joints where each IMU is located are estimated based on the orientation of the root node. The 3D coordinates of the remaining joints are then estimated based on the 3D coordinates of the joints where each IMU is located. Based on the estimated 3D coordinates of each joint, the first human posture information and foot contact state information are estimated. Based on rigid body mechanics and physical constraints, an optimization algorithm is used to solve for the first human posture information, foot contact state information, and the position information of the head and left and right hands, resulting in the second human posture information. This embodiment uses data from only five IMUs to achieve human posture capture. Compared to existing technologies, this reduces the number of waist IMUs while ensuring the accuracy of the prediction results, thereby improving the user experience.

[0140] To facilitate better implementation of the human motion capture method based on sparse IMU in the embodiments of this application, the embodiments of this application also provide a human motion capture device based on sparse IMU. Figure 5 This is a schematic diagram of a human motion capture device based on a sparse IMU, as provided in Embodiment 3 of this application. Figure 5 As shown, the sparse IMU-based human motion capture device 100 may include:

[0141] The acquisition module 11 is used to acquire inertial data measured by inertial sensors (IMUs) worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head of the human body.

[0142] Human posture estimation model 12 is used to estimate first human posture information and foot contact state information based on the inertial data;

[0143] Position detection module 13 is used to detect the position information of the head and left and right hands;

[0144] The fusion estimation module 14 is used to determine the second human posture information based on the first human posture information, the foot contact state information, and the position information.

[0145] In some embodiments, the human pose estimation module 12 is specifically used for:

[0146] The orientation of each joint in the human body is determined based on the inertial data, and the joints of the human body include the lumbar joints;

[0147] Based on the orientation of each joint of the human body, the posture information of the first human body and the ground contact state information of the foot are determined.

[0148] In some embodiments, the human pose estimation module 12 is specifically used for:

[0149] Based on the inertial data, estimate the orientation of the root node of the human body;

[0150] Based on the orientation of the root node, the inertial data is regularized so that the orientation of the inertial data is aligned with the orientation of the root node.

[0151] Based on the inertial data after orientation alignment, estimate the 3D coordinates of the joints where each IMU is located;

[0152] Based on the 3D coordinates of the joints where each IMU is located, estimate the 3D coordinates of the remaining joints of the human body.

[0153] In some embodiments, the human pose estimation module 12 is specifically used for:

[0154] Based on the estimated 3D coordinates of each joint of the human body, the posture information of the first human body is estimated;

[0155] Based on the inertial data after orientation alignment and the estimated 3D coordinates of each joint of the human body, a binary classification algorithm is used to determine the foot contact state information.

[0156] In some embodiments, the human pose estimation model is trained using a supervised learning method.

[0157] In some embodiments, the loss of the human pose estimation model is a weighted sum of the following six losses: a first loss, a second loss, a third loss, a fourth loss, a fifth loss, and a sixth loss;

[0158] The first loss is the mean square error loss between the true value and the estimated value of the orientation of the root node;

[0159] The second loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of each IMU joint;

[0160] The third loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of the remaining joints of the human body.

[0161] The fourth loss is the mean square error loss between the true value and the estimated value of the first human posture information;

[0162] The fifth loss is the temporal smoothing loss between data frames used for human motion capture;

[0163] The sixth loss is the binary cross-entropy (BCE) loss between the true value and the estimated value of the foot-touching state.

[0164] In some embodiments, the true value of the foot-touching state is calculated using the following two parameters: whether the foot is stationary and whether the height of the foot from the ground is less than a threshold.

[0165] In some embodiments, the fusion estimation module 14 is specifically used for:

[0166] Based on rigid body mechanics and physical constraints, an optimization algorithm is used to solve for the first human posture information, the foot contact state information, and the position information to obtain the second human posture information.

[0167] In some embodiments, the optimization algorithm employs the following optimization equation:

[0168]

[0169] st

[0170]

[0171] λ∈F

[0172]

[0173] in, Here, λ represents the first human posture information, the position information of the head and left and right hands, λ represents the ground contact force acting on the point of contact between the human body and the ground, and τ represents the torque at the joints of the human body. Let c be the Jacobian matrix of the point of contact between the person and the ground, εPD be the energy term of the PD controller, and ε reg To optimize the regularization term, M is the human body inertia matrix, h is the nonlinear term including Coriolis force, centrifugal force, and gravity, and F is the friction cone constraint set of ground contact forces. Let j be the linear velocity of the joint in contact with the ground, and ζ be the set of constraints on the linear velocity of the joint in contact with the ground.

[0174] In some embodiments, the IMUs at the lower left and right elbows are integrated into the handle.

[0175] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0176] The apparatus 100 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0177] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application, as shown below. Figure 6 As shown, the electronic device 200 may include:

[0178] The system includes a memory 21 and a processor 22. The memory 21 stores computer programs and transfers the program code to the processor 22. In other words, the processor 22 can retrieve and run the computer programs from the memory 21 to implement the methods described in the embodiments of this application.

[0179] For example, the processor 22 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0180] In some embodiments of this application, the processor 22 may include, but is not limited to:

[0181] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0182] In some embodiments of this application, the memory 21 includes, but is not limited to:

[0183] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0184] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0185] like Figure 6 As shown, the electronic device 200 may further include a transceiver 23, which can be connected to the processor 22 or the memory 21.

[0186] The processor 22 can control the transceiver 23 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 23 may include a transmitter and a receiver. The transceiver 23 may further include antennas, and the number of antennas may be one or more.

[0187] Understandable, although Figure 6 As not shown in the diagram, the electronic device 200 may also include a camera module, a Wi-Fi module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be described in detail here.

[0188] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0189] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0190] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the corresponding processes in the sparse IMU-based human motion capture method described in the embodiments of this application. For simplicity, these details will not be elaborated further here.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0192] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for human motion capture based on sparse IMU, characterized in that, include: Inertial data is collected from inertial sensors (IMUs) worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head of the human body. The inertial data is input into the human posture estimation model to obtain first human posture information and foot contact state information, including: determining the orientation of each joint of the human body based on the inertial data, wherein each joint of the human body includes the lumbar joint; and determining the first human posture information and the foot contact state information based on the orientation of each joint of the human body. Detect the position information of the head and left and right hands; The second human posture information is determined based on the first human posture information, the foot contact state information, and the position information.

2. The method according to claim 1, characterized in that, Determining the orientation of each joint of the human body based on the inertial data includes: Based on the inertial data, estimate the orientation of the root node of the human body; Based on the orientation of the root node, the inertial data is regularized so that the orientation of the inertial data is aligned with the orientation of the root node. Based on the inertial data after orientation alignment, estimate the 3D coordinates of the joints where each IMU is located; Based on the 3D coordinates of the joints where each IMU is located, estimate the 3D coordinates of the remaining joints of the human body.

3. The method according to claim 2, characterized in that, The step of determining the first human posture information and the foot contact state information based on the orientation of each joint of the human body includes: Based on the estimated 3D coordinates of each joint of the human body, the posture information of the first human body is estimated; Based on the inertial data after orientation alignment and the estimated 3D coordinates of each joint of the human body, a binary classification algorithm is used to determine the foot contact state information.

4. The method according to claim 3, characterized in that, The human pose estimation model was trained using a supervised learning method.

5. The method according to claim 4, characterized in that, The loss of the human pose estimation model is a weighted sum of the following six losses: first loss, second loss, third loss, fourth loss, fifth loss, and sixth loss; The first loss is the mean square error loss between the true value and the estimated value of the orientation of the root node; The second loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of each IMU joint; The third loss is the mean square error loss between the true and estimated values ​​of the 3D coordinates of the remaining joints of the human body. The fourth loss is the mean square error loss between the true value and the estimated value of the first human posture information; The fifth loss is the temporal smoothing loss between data frames used for human motion capture; The sixth loss is the binary cross-entropy (BCE) loss between the true value and the estimated value of the foot-touching state.

6. The method according to claim 5, characterized in that, The true value of the foot contact state is calculated using the following two parameters: whether the foot is stationary and whether the height of the foot from the ground is less than a threshold.

7. The method according to any one of claims 1-6, characterized in that, The step of determining the second human posture information based on the first human posture information, the foot contact state information, and the position information includes: Based on rigid body mechanics and physical constraints, an optimization algorithm is used to solve for the first human posture information, the foot contact state information, and the position information to obtain the second human posture information.

8. The method according to claim 7, characterized in that, The optimization equation used in the optimization algorithm is: in, This includes the first human body's posture information, and the position information of the head and left and right hands. The ground contact force acting at the point of contact between the human body and the ground. For the torque at the joints of the human body, Let c be the Jacobian matrix of the point of contact between the person and the ground, and let c be the point of contact between the person and the ground. For the PD controller energy item, To optimize the regularization term, M is the human body inertia matrix, h is the nonlinear term including Coriolis force, centrifugal force, and gravity, and F is the friction cone constraint set of ground contact forces. Let j be the linear velocity of the joint in contact with the ground, and j be the velocity of the joint in contact with the ground. This is the set of linear velocity constraints for the ground-contact joint.

9. The method according to any one of claims 1-6, characterized in that, The IMUs located under the left and right elbows are integrated into the handle.

10. A human motion capture device based on sparse IMU, characterized in that, include: The data acquisition module is used to acquire inertial data measured by inertial sensors (IMUs) worn on the lower left elbow, lower right elbow, lower left knee, lower right knee, and head of the human body. A human posture estimation model is used to estimate first human posture information and foot contact state information based on the inertial data, including: determining the orientation of each joint of the human body based on the inertial data, wherein each joint of the human body includes the lumbar joint; and determining the first human posture information and the foot contact state information based on the orientation of each joint of the human body. The position detection module is used to detect the position information of the head and left and right hands; The fusion estimation module is used to determine the second human posture information based on the first human posture information, the foot contact state information, and the position information.

11. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.

Images