Real-time human and environment interactive motion reconstruction method and system based on sparse inertial sensor, and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to a real-time human and environment interactive motion reconstruction method and system based on a sparse inertial sensor and electronic equipment. The method comprises the following steps: collecting a human body motion data set of a target object, and determining first root node measurement data of a root node corresponding to each human body motion data and a first root node gravity direction; based on a data driving mode, optimizing the first root node measurement data and the gravity direction of the first root node to obtain second root node measurement data and the gravity direction of the second root node, and estimating local posture data and global motion data of the human body; and according to the local posture data and the global motion data of the human body, simulating and optimizing the physical dynamics when a physical model corresponding to the target object interacts with the environment, and according to the optimized human body motion data, realizing motion reconstruction when the target object interacts with the environment. By adopting the scheme, accurate three-dimensional interactive motion reconstruction can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a real-time human-environment interaction motion reconstruction method, system, and electronic device based on sparse inertial sensors. Background Art

[0002] With the rapid development of fields such as virtual reality (VR), augmented reality (AR), robotics, video games, and human-computer interaction (HCI), the demand for digital human motion has been increasing. Traditional human motion capture technologies generally rely on devices such as optical sensors and cameras to reconstruct human postures and motion trajectories by analyzing images or videos. However, these vision-based methods face many limitations, including camera occlusion, limitations of fixed capture spaces, and high costs, which greatly limit their applications in dynamic and uncontrolled environments.

[0003] To address these problems, motion capture technologies based on inertial measurement units (IMUs) have gradually become a research direction that has received much attention. However, current motion reconstruction technologies based on inertial sensors face the technical problems that sensor data noise affects the reconstruction accuracy of global human motion (including global translation and global orientation), especially the difficulty of motion capture in the z-axis (vertical) direction. Summary of the Invention

[0004] The present disclosure aims to at least partly solve one of the technical problems in the related art.

[0005] To this end, the first object of the present disclosure is to propose a real-time human-environment interaction motion reconstruction method based on sparse inertial sensors to achieve accurate three-dimensional interaction motion reconstruction.

[0006] The second object of the present disclosure is to propose a real-time human-environment interaction motion reconstruction system based on sparse inertial sensors.

[0007] The third object of the present disclosure is to propose an electronic device.

[0008] The fourth object of the present disclosure is to propose a computer-readable storage medium.

[0009] The fifth object of the present disclosure is to propose a computer program product.

[0010] To achieve the above object, the first aspect embodiment of the present disclosure proposes a real-time human-environment interaction motion reconstruction method based on sparse inertial sensors, including:

[0011] Collect a set of human motion data of a target object through a set of inertial measurement units, and determine the first root node measurement data and the first root node gravity direction of the root node corresponding to each human motion data in the set of human motion data, wherein the position of the root node corresponds one-to-one with the position of the inertial measurement unit on the target object;

[0012] Based on a data-driven method, optimize the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimate human local pose data and global motion data;

[0013] According to the human local pose data and the global motion data, simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment, obtain optimized human motion data, and implement motion reconstruction when the target object interacts with the environment according to the human motion data.

[0014] Optionally, the determining the first root node measurement data and the first root node gravity direction of the root node corresponding to each human motion data in the set of human motion data includes:

[0015] Vector splice the rotation matrix data, angular velocity data, and acceleration data in the human motion data corresponding to each root node in the set of human motion data to obtain the first root node measurement data of the root node corresponding to each human motion data relative to the inertial measurement unit;

[0016] Convert the first global direction in the human motion data corresponding to each root node in the set of human motion data from the world coordinate system to the root node coordinate system to obtain the first root node gravity direction.

[0017] Optionally, the optimizing the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimating human local pose data and global motion data includes:

[0018] Optimize the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimate human local pose data;

[0019] Estimate global motion data according to the second root node measurement data, the second root node gravity direction, and the human local pose data.

[0020] Optionally, the optimization of the measurement data of the first root node and the gravity direction of the first root node to obtain the measurement data of the second root node and the gravity direction of the second root node includes:

[0021] Controlling the first long short-term memory network to reconstruct the leaf joint position relative to the root node according to the measurement data of the first root node and the gravity direction of the first root node, and optimizing the gravity direction of the first root node to obtain the gravity direction of the third root node;

[0022] Optimizing the first global direction according to the minimum angle rotation matrix between the gravity direction of the first root node and the gravity direction of the three root nodes to obtain the second global direction, and correcting the measurement data of the first root node according to the second global direction to obtain the measurement data of the third root node;

[0023] Controlling the second long short-term memory network to estimate the full joint position according to the measurement data of the third root node, the gravity direction of the third root node and the leaf joint position, and optimizing the gravity direction of the third root node to obtain the gravity direction of the second root node;

[0024] Optimizing the second global direction according to the minimum angle rotation matrix between the gravity direction of the third root node and the gravity direction of the second root node to obtain the third global direction, and correcting the measurement data of the third root node according to the third global direction to obtain the measurement data of the second root node.

[0025] Optionally, the estimation of the human body local pose data includes:

[0026] Controlling the third long short-term memory network to estimate the human body local pose data according to the measurement data of the second root node, the gravity direction of the second root node and the full joint position.

[0027] Optionally, the estimation of the global motion data according to the measurement data of the second root node, the gravity direction of the second root node and the human body local pose data includes:

[0028] Controlling the fourth long short-term memory network to estimate the velocity of the first root node and the joint static probability corresponding to each target joint in the target joint set according to the measurement data of the second root node, the gravity direction of the second root node and the human body local pose data, wherein the velocity of the first root node includes a first orthogonal component and a second orthogonal component, the first orthogonal component is parallel to the gravity direction of the second root node, and the second orthogonal component is perpendicular to the gravity direction of the second root node;

[0029] Determine a set of stationary joints in the target joint set according to the joint stationary probability, and optimize the first root node velocity with the goal of keeping the stationary joints in the set of stationary joints immobile and minimizing the adjustment value of the first root node velocity, to obtain a second root node velocity;

[0030] Determine global motion data according to the second root node velocity.

[0031] Optionally, the simulating and optimizing the physical dynamics during the interaction between the physical model corresponding to the target object and the environment according to the human body local pose data and the global motion data to obtain optimized human motion data includes:

[0032] Without setting external contact forces, use the physical model corresponding to the target object to pre-track the human body local pose data and the global motion data to obtain the first joint torques of the physical model, where the first joint torques include the root node virtual forces that need to be explained by contact forces;

[0033] According to human kinetics, determine a set of three-dimensional contact points that can explain the root node virtual forces from the set of stationary joints in the target joint set, and determine the contact forces corresponding to each three-dimensional contact point in the set of three-dimensional contact points;

[0034] Set the contact forces corresponding to the three-dimensional contact points as external contact forces at the positions corresponding to the three-dimensional contact points in the physical model, and use the physical model corresponding to the target object to re-track the human body local pose data and the global motion data to obtain optimized human motion data, where the human motion data includes optimized human body local pose data, optimized global motion data, and second joint torques.

[0035] Optionally, the implementing motion reconstruction during the interaction between the target object and the environment according to the human motion data includes:

[0036] Perform real-time rendering on the optimized human body local pose data, the optimized global motion data, the second joint torques, the set of three-dimensional contact points, the contact forces, the physical model, and the interaction proxy object during the interaction between the target object and the environment, to implement motion reconstruction during the interaction between the target object and the environment.

[0037] To achieve the above object, a real-time human-environment interaction motion reconstruction system based on a sparse inertial sensor according to a second aspect embodiment of the present disclosure includes:

[0038] A data acquisition unit, configured to collect a set of human motion data of a target object through a set of inertial measurement units, and determine first root node measurement data and a first root node gravity direction of a root node corresponding to each piece of human motion data in the set of human motion data, where the position of the root node corresponds one-to-one to the position of the inertial measurement unit on the target object;

[0039] A data estimation unit, configured to optimize the first root node measurement data and the first root node gravity direction based on a data-driven method to obtain second root node measurement data, a second root node gravity direction, and estimate human local pose data and global motion data;

[0040] A data optimization unit, configured to simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment according to the human local pose data and the global motion data to obtain optimized human motion data, and perform motion reconstruction when the target object interacts with the environment according to the human motion data.

[0041] To achieve the above object, an embodiment of the third aspect of the present disclosure provides an electronic device, including:

[0042] A memory, configured to store executable program code;

[0043] A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method shown in any one of the foregoing first aspects.

[0044] To achieve the above object, an embodiment of the fourth aspect of the present disclosure provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed, the method shown in any one of the foregoing first aspects is implemented.

[0045] To achieve the above object, an embodiment of the fifth aspect of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method shown in any one of the foregoing first aspects is implemented.

[0046] In summary, the method, system, and electronic device provided by the present disclosure can effectively reduce the drift problem caused by the accumulation of inertial sensor measurement errors in traditional methods by introducing the constraint of gravity on human pose and motion in interactive reconstruction; secondly, by combining the dual constraints of data-driven and physical optimization, more accurate global motion reconstruction can be achieved under the condition of only using inertial measurement units, especially in terms of global movement and global orientation estimation in three-dimensional space, overcoming the planar ground assumption limitation and global pose drift problem in traditional methods, and accurate three-dimensional interactive motion reconstruction can be achieved.

[0047] Additional aspects and advantages of the present disclosure will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present disclosure. Description of the Drawings

[0048] The above and / or additional aspects and advantages of the present disclosure will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 is a schematic flowchart of a real-time human-environment interaction motion reconstruction method based on a sparse inertial sensor provided by an embodiment of the present disclosure;

[0050] Figure 2 is a flowchart for estimating human body local pose data provided by an embodiment of the present disclosure;

[0051] Figure 3 is a flowchart for estimating global motion data provided by an embodiment of the present disclosure;

[0052] Figure 4 is a flowchart for optimizing physical dynamics provided by an embodiment of the present disclosure;

[0053] Figure 5 is a schematic structural diagram of a real-time human-environment interaction motion reconstruction system based on a sparse inertial sensor provided by an embodiment of the present disclosure. Detailed Description of the Embodiments

[0054] Embodiments of the present disclosure will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0055] It should be noted that in the IMU-based motion capture technology, the IMU sensor can measure information such as acceleration and angular velocity, and can capture the dynamic motion of the human body in real time, and is not affected by external factors such as light and perspective changes. Therefore, it has broad application potential in various complex environments. The IMU sensor not only has strong portability, but also has a low cost, can provide long-term stable motion capture effects, and has higher flexibility and economy.

[0056] However, despite the significant advantages of IMU technology in motion capture, there are still many challenges. IMU sensors have measurement noise and drift, which particularly affect the estimation of global human motion (such as translation and rotation). Specifically, the measurement errors of IMUs accumulate over time, leading to a gradual decline in the estimation accuracy of global motion. In addition, IMUs cannot directly provide global spatial position information, so they face great difficulties in capturing global human displacement.

[0057] To improve the accuracy of IMU technology, researchers have proposed various methods to fuse different types of sensor data. For example, some studies have used cameras, Ultra Wide Band (UWB) sensors, or external positioning systems in combination with IMU data to compensate for the limitations of IMU sensors. Although these methods can improve the accuracy of motion capture, they also face many limitations, such as strong environmental dependence, complex calibration requirements, and strict requirements for sensor placement.

[0058] In recent years, some researchers have proposed using only IMU sensors for human motion capture, but most rely on the assumption of a fixed planar ground and detect two-dimensional contact points between the person and the ground to limit the cumulative error of human motion estimation. Therefore, these methods can only capture flat-ground motion and cannot handle complex three-dimensional spatial motion. In addition, the rotational measurement errors of IMUs easily lead to drift of the global human pose. Existing methods usually rely on the direct measurement values of IMUs at the root node to estimate the global orientation of the human body, resulting in a decline in accuracy during long-term tracking.

[0059] The following will explain the present disclosure in detail with specific embodiments.

[0060] In the first embodiment, as Figure 1 shown, Figure 1 is a schematic flowchart of a real-time human-environment interaction motion reconstruction method based on sparse inertial sensors provided by an embodiment of the present disclosure. This method can be implemented depending on a computer program and can run on a system for real-time human-environment interaction motion reconstruction based on sparse inertial sensors. This computer program can be integrated into an application or run as an independent tool-class application.

[0061] Among them, the real-time human-environment interaction motion reconstruction method based on sparse inertial sensors can be executed by an electronic device.

[0062] Exemplarily, the real-time human-environment interaction motion reconstruction method based on sparse inertial sensors includes the following steps:

[0063] S101. Collect a set of human motion data of a target object through an inertial measurement unit set, and determine the first root node measurement data of the root node corresponding to each human motion data in the set of human motion data and the first root node gravity direction;

[0064] According to some embodiments, the target object refers to an object that interacts with the environment and requires interactive motion reconstruction.

[0065] In some embodiments, the position of the root node corresponds one-to-one with the position of the inertial measurement unit on the target object.

[0066] According to some embodiments, the first root node measurement data refers to the measurement data of the root node relative to the IMU, that is, the direct measurement value of the IMU at the root node.

[0067] In some embodiments, the first root node gravity direction refers to the gravity direction of the human motion data in the root node coordinate system.

[0068] S102. Optimize the first root node measurement data and the first root node gravity direction based on a data-driven method to obtain the second root node measurement data, the second root node gravity direction, and estimate the human local pose data and the global motion data;

[0069] It should be noted that estimating the full-body pose from sparsely arranged and noisy IMU data is essentially underdetermined. Previous studies have estimated the local pose in the human root node coordinate system by using a deep neural network, aiming to reduce this underdetermination by modeling the local pose prior. However, these methods ignore the influence of the human global pose on the local pose. This embodiment believes that there is a strong correlation between the local pose and the global pose of the human body. For example, a person lying globally is unlikely to make a local walking pose. More precisely, the global rotation around the gravity axis is independent of the local pose of the body (because a person can make the same pose facing different directions), while the rotation in other degrees of freedom (reflecting the body's tilt) is related to the local pose. That is to say, the gravity direction in the root node coordinate system can be used as a reliable indicator of this global pose, because it remains unchanged when rotating around the gravity axis, but changes with the change of the body's tilt. Due to this correlation, this embodiment proposes to model the joint prior distribution of the gravity direction and the local pose in the root node coordinate system by simultaneously reconstructing the local pose and optimizing the gravity. On the one hand, the accurate gravity direction provides additional information relative to the IMU to enhance the local pose estimation. On the other hand, the local pose estimation helps to reduce the noise of the global pose measured by the root node IMU. Therefore, this embodiment introduces a more informative prior by integrating the gravity prior into the pose estimation, which can improve the accuracy of local pose and global pose estimation.

[0070] S103. Simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment according to the local human body pose data and the global motion data, obtain the optimized human body motion data, and implement motion reconstruction when the target object interacts with the environment according to the human body motion data.

[0071] In summary, for the method provided in this embodiment, by introducing the constraint of gravity on human body poses and motions in interactive reconstruction, the drift problem caused by the accumulation of inertial sensor measurement errors in traditional methods can be effectively reduced. Secondly, through the dual constraints of combining data-driven and physical optimization, more accurate global motion reconstruction can be achieved under the condition of only using inertial measurement units. Especially in terms of global movement and global orientation estimation in three-dimensional space, the limitations of the planar ground assumption and the global pose drift problem in traditional methods are overcome, and accurate three-dimensional interactive motion reconstruction can be realized.

[0072] This embodiment also provides another real-time human-environment interactive motion reconstruction method based on sparse inertial sensors. This method can be executed by an electronic device.

[0073] Exemplarily, the real-time human-environment interactive motion reconstruction method based on sparse inertial sensors may include the following steps:

[0074] S201. Collect the human body motion data set of the target object through a set of inertial measurement units;

[0075] According to some embodiments, each IMU may include devices such as an accelerometer, a gyroscope, and a magnetometer. Among them, the accelerometer is used to provide acceleration data, the gyroscope is used to provide angular velocity data, and the magnetometer is used to provide magnetic induction intensity data.

[0076] In some embodiments, when obtaining IMU data, the output signals of the accelerometer, gyroscope, and magnetometer may be collected in real time. First, by adopting high-frequency data sampling, sufficiently fine-grained motion information can be obtained. For example, the sampling frequency is 200 Hz to 800 Hz. Then, during the data collection process, the noise of the sensor can be filtered out through filtering processing. The filtering methods adopted include but are not limited to Kalman Filter, Complementary Filter, etc., so as to effectively remove noise and estimate the orientation of the sensor, and obtain the acceleration data, angular velocity data, and orientation measurement values of each IMU in the world coordinate system. Finally, the filtered signals of multiple IMUs can be synchronized and downsampled to 60 Hz to obtain the human body motion data set.

[0077] According to some embodiments, in order to achieve high-precision motion reconstruction, different parts of the target object, including key parts such as the head and limbs, can be used as root nodes, and multiple IMUs are respectively installed on these different parts. For example, at least 6 IMUs can be used and placed on these parts (1 IMU is placed on each part).

[0078] S202, splice the rotation matrix data, angular velocity data, and acceleration data in the human motion data corresponding to each root node in the human motion data set to obtain the first root node measurement data of each root node relative to the inertial measurement unit for each human motion data;

[0079] According to some embodiments, the rotation matrix data can be obtained from the orientation measurement values in the human motion data.

[0080] S203, convert the first global direction in the human motion data corresponding to each root node in the human motion data set from the world coordinate system to the root node coordinate system to obtain the first root node gravity direction;

[0081] According to some embodiments, the global direction is defined as the root node rotation, and the first global direction can be obtained from the orientation measurement values in the human motion data.

[0082] In some embodiments, the first root node gravity direction g″ is the gravity direction in the root node coordinate system and can be determined according to the following formula:

[0083] g″ = (R″ root ) T g M

[0084] where R″ root is the first global direction; g M is the gravity direction in the world coordinate system.

[0085] S204, based on a data-driven approach, optimize the first root node measurement data and the first root node gravity direction to obtain the second root node measurement data, the second root node gravity direction, and estimate the human local pose data;

[0086] According to some embodiments, the human local pose data estimation method provided in this embodiment includes three stages: first, estimate the positions of the leaf joints, then estimate the positions of the joints of the whole body, and finally estimate the local pose of the human body. Different from the PhysicalNon-inertial Poser (PNP) network, in this embodiment, on the basis of the PNP network, the gravity direction in the root node coordinate system is additionally input in each stage, and the network is required to optimize the gravity as an additional output at the same time. In this way, the network can learn the joint prior distribution of the local pose and the global direction.

[0087] In some embodiments, Figure 2 is a flowchart for estimating human body local pose data provided by an embodiment of the present disclosure. As Figure 2 shown, it includes the following steps:

[0088] S2041, control the first long short-term memory network (PL) to reconstruct the leaf joint position p relative to the root node according to the first root node measurement data x″ and the first root node gravity direction g″ leaf and optimize the first root node gravity direction g″ to obtain the third root node gravity direction g′;

[0089] S2042, optimize the first global direction R″ according to the minimum angle rotation matrix R{g′→g″} between the first root node gravity direction g″ and the three root node gravity direction g′ roaot to obtain the second global direction R′ root , and correct the first root node measurement data x″ according to the second global direction R′ root to obtain the third root node measurement data x′;

[0090] Among them, the minimum angle rotation matrix R{g′→g"} refers to the rotation matrix with the smallest angle required to rotate the angle corresponding to the first root node gravity direction g" to the angle corresponding to the third root node gravity direction g′.

[0091] Among them, the first global direction R" can be optimized according to the following formula root :

[0092] R′ root = R″ root R{g′→g″}

[0093] S2043, control the second long short-term memory network (PA) to estimate the full joint position p according to the third root node measurement data x′, the third root node gravity direction g′ and the leaf joint position p leaf and optimize the third root node gravity direction g′ to obtain the second root node gravity direction g; all Among them, the third root node measurement data x′, the third root node gravity direction g′ and the leaf joint position p

[0094] can be concatenated and input into PA to estimate the full joint position p leaf and optimize the third root node gravity direction g′. all

[0095] S2044, according to the minimum angle rotation matrix between the third root node gravity direction g′ and the second root node gravity direction g, for the second global direction R′ rootOptimize to obtain the third global direction R root and correct the measurement data x' of the third root node according to the third global direction R root to obtain the measurement data x of the second root node;

[0096] Among them, the formula used for optimizing the second global direction R' root can refer to step S2042 and will not be repeated here.

[0097] S2045. Control the third long short-term memory network (RA) to estimate the local human body pose data θ according to the measurement data x of the second root node, the gravity direction g of the second root node, and the full joint position p all Estimate the local human body pose data θ.

[0098] Among them, the measurement data x of the second root node, the gravity direction g of the second root node, and the full joint position p all can be concatenated and input into RA to regress the local human body pose data θ.

[0099] Among them, the local human body pose data θ is defined as the joint rotation of a skinned multi-person linear (SMPL) human body three-dimensional model.

[0100] It should be noted that based on the data-driven method, by using a long short-term memory (LSTM) network to construct a PNP network to denoise the measurement data x″ of the first root node and the gravity direction g" of the first root node, and estimating the local human body pose including the angles and positions of each joint and the global direction from the denoised measurement data x of the second root node and the denoised gravity direction g of the second root node, the accuracy of estimating the local human body pose data can be improved.

[0101] S205. Estimate the global motion data according to the measurement data of the second root node, the gravity direction of the second root node, and the local human body pose data;

[0102] According to some embodiments, in order to effectively incorporate gravity perception, the gravity direction in the optimized root coordinate system, that is, the gravity direction g of the second root node, can be input, and the network is required to reconstruct two orthogonal components of the root node velocity: one parallel to the gravity direction and the other perpendicular to the gravity direction. In this way, the relationship between the gravity direction and the global velocity can be learned. Subsequently, global motion optimization is performed to obtain the global motion data.

[0103] In some embodiments, Figure 3 is a flowchart for estimating global motion data provided by an embodiment of the present disclosure. As Figure 3 shown, it includes the following steps:

[0104] S2051, control the fourth long short - term memory network (OV) to estimate the velocity v of the first root node and the joint rest probability s corresponding to each target joint in the target joint set according to the measurement data x of the second root node, the gravity direction g of the second root node, and the human body local pose data θ;

[0105] Among them, the velocity v of the first root node includes a first orthogonal component v / / and a second orthogonal component v ⊥ , the first orthogonal component v / / is parallel to the gravity direction g of the second root node, and the second orthogonal component v ⊥ is perpendicular to the gravity direction g of the second root node.

[0106] Among them, the measurement data x of the second root node, the gravity direction g of the second root node, and the human body local pose data θ can be concatenated and input into the OV to estimate the velocity v of the first root node and estimate the joint rest probability s.

[0107] Among them, the target joints in the target joint set are the selected key joints related to movement. For example, five joints including both hands, both feet, and the pelvis can be selected in the target joint set, so as to obtain a five - dimensional joint rest probability.

[0108] S2052, determine the set of static joints in the target joint set according to the joint rest probability, and optimize the velocity v of the first root node with the goal that the static joints in the set of static joints remain stationary and the adjustment value of the velocity v of the first root node is the lowest, to obtain the velocity of the second root node

[0109] According to some embodiments, the target joints in the target joint set with a joint rest probability greater than the rest probability threshold can be used as static joints to obtain the set of static joints.

[0110] In some embodiments, the human body pose and position can be mapped to joint coordinates, and the static joint information can be used to optimize the velocity v of the first root node, as shown in the following formula:

[0111]

[0112] Among them, Δt is the frame interval, the superscript t represents the value of the t - th frame, the subscript i represents the i - th target joint, and FK(·) is the forward dynamics function.

[0113] Among them, the above formula aims to find a root node velocity such that all static joints remain stationary as much as possible, and at the same time make the root node velocity as close as possible to the velocity v of the first root node.

[0114] S2053. Determine the global motion data according to the speed of the second root node.

[0115] According to some embodiments, the global motion data includes the speed of the second root node. The speed of the second root node can also be directly used as the global motion data.

[0116] In some embodiments, the first long short - term memory network (PL), the second long short - term memory network (PA), the third long short - term memory network (RA), and the fourth long short - term memory network (OV) can adopt LSTM networks with the same structure.

[0117] It should be noted that this embodiment is superior to the existing methods in both the capture accuracy of local postures and global motions, and can accurately estimate the motions in three - dimensional space without relying on the ground. Among them, by using the human local posture estimation results and IMU measurement values, the static joints of the human body and the global motion information of the human body are estimated, and the global motion state of the human body is jointly optimized by using the static constraint and the gravity direction, which can reduce the drift problem of the global position and orientation.

[0118] S206. According to the human local posture data and the global motion data, simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment, and obtain the optimized human motion data.

[0119] It should be noted that the goal of the interactive physical optimization method is to estimate the three - dimensional (3D) contact situation in the interactive motion and perform physical optimization on this basis to improve the estimated human actions. This process is particularly important because traditional physical optimization methods usually assume that the ground is flat, which limits their ability to estimate 3D actions (such as going up stairs).

[0120] Specifically, the biggest challenge in performing physical optimization in three - dimensional space is the lack of scene knowledge, resulting in the inability to accurately detect the contact points. Therefore, it is crucial to design a method that can infer 3D contact points. By observing human actions, the contact points can be inferred from certain actions. For example, if we see a person walking upwards, we will naturally assume that there are stairs under his feet exerting a contact force. This is based on our understanding of the physical world: 1) The supporting force is necessary to prevent the human body from falling; 2) The supporting force usually acts on the stationary foot. According to this idea, this embodiment designs a dual - tracking algorithm to determine the 3D contact points and perform physical optimization.

[0121] According to some embodiments, Figure 4 This is an optimization flowchart of the physical dynamics provided by the embodiments of the present disclosure. As Figure 4 shown, it includes the following steps:

[0122] S2061. Without setting an external contact force, pre-track the local human body pose data θ and the global motion data using the physical model corresponding to the target object to obtain the first joint torque τ of the physical model;

[0123] It should be noted that the purpose of pre-tracking is to estimate the joint torque required for the physical model to track the reference motion without considering the contact force. According to the PNP method, the physical model can track the reference human body pose and motion estimated in steps S204 and S205, and assuming no external contact force, the first joint torque τ of the physical model can be obtained.

[0124] Among them, during the interactive physical optimization process, a role physical model with a joint-rigid body structure controlled by torque can be used as the physical model corresponding to the target object. The skeleton structure and degrees of freedom of this model are the same as those of the SMPL model, and its body mass, center of gravity, and inertia matrix are extracted from the average body shape of the SMPL model, and the density is assumed to be 1000 kg / m 3 . Different from the SMPL model which is driven by joint rotation, this physical model can be driven by joint torque and external contact force.

[0125] Among them, the first joint torque τ includes the virtual force of the root node that needs to be explained by the contact force. For example, in the case where IMUs are only set at six parts including the head and limbs, the first joint torque τ only targets these six joints. The first six dimensions in the first joint torque τ, that is, τ :6 , are the virtual forces of the root node that need to be explained by the contact force.

[0126] S2062. According to human dynamics, determine the three-dimensional contact point set that can explain the virtual force of the root node from the static joint set in the target joint set, and determine the contact force corresponding to each three-dimensional contact point in the three-dimensional contact point set;

[0127] It should be noted that the goal of this step is to find the three-dimensional contact points that can explain the virtual force. The static joints obtained in step S205 can be used as candidate contact points, and the actual physical contact points should be a subset of the candidate contact points. Therefore, for each possible subset of candidate contact points, the contact force used to explain the virtual force of the root node can be calculated by solving the following equation:

[0128]

[0129] s.t. λ ∈ C

[0130] where λ is the contact force, J is the Jacobian matrix of the contact point, β = 0.4 is the weight controlling the regularization term, and C is the contact force friction cone constraint used in PNP.

[0131] That is to say, for different candidate contact point subsets, the quadratic programming problem can be established and solved separately to obtain the optimal solution and the corresponding objective function value. Subsequently, find the candidate contact point subset that can minimize the objective function, which is the three-dimensional contact point set that can explain the virtual force of the root node, and the corresponding contact force can also be output simultaneously.

[0132] S2063, set the contact force corresponding to the three-dimensional contact point in the physical model as the external contact force at the position corresponding to the three-dimensional contact point, and use the physical model corresponding to the target object to re-track the local human body posture data and the global motion data to obtain the optimized human body motion data;

[0133] According to some embodiments, after obtaining the three-dimensional contact points and contact forces, by re-tracking the reference action, similar to the pre-tracking stage, in the re-tracking stage, by adding the contact points and contact forces as external forces to the physical model on the basis of pre-tracking, it can more accurately simulate the human interaction motion, and finally obtain the optimized human body motion data. Therefore, the optimization method combined with physical constraints can achieve high-precision three-dimensional human global motion estimation by modeling three-dimensional contact and gravity information, and optimize the local human body posture estimation on this basis, so as to achieve accurate three-dimensional interaction motion reconstruction.

[0134] Among them, the optimized human body motion data includes optimized local human body posture data, optimized global motion data, and the second joint torque.

[0135] It is easy to understand that the three-dimensional contact point estimation method between humans and the environment based on physical optimization provided in this embodiment uses a physical character to track and estimate the action. First, assume that there are no contact points at this time. In order to perform tracking, a large virtual force is allowed to be applied to the root joint of the human body. These virtual forces do not actually exist and must be explained by contact forces; after pre-tracking, the estimated static joints are used as candidate contact points, and a minimum subset of points is selected from them. The virtual forces required in pre-tracking are explained by the contact forces at these points, so as to obtain the 3D real contact points; finally, use the identified contact points and contact forces to re-track the human body action to obtain a more physically accurate result. Therefore, introducing the principle of physics into the processing of sparse inertial sensor data, according to human body dynamics, solving the three-dimensional contact points of human-environment interaction through physical algorithms, simulating and optimizing the physical dynamics of human-environment interaction, can ensure the naturalness and realism of interaction motion reconstruction, and solve the limitation that traditional methods cannot effectively capture three-dimensional global motion; at the same time, the innovative dual-tracking algorithm extends the traditional method and can accurately estimate and optimize contact points in three-dimensional space.

[0136] At S207, the optimized human body local pose data, the optimized global motion data, the second joint torque, the three-dimensional contact point set, the contact force, the physical model, and the interaction proxy object during the interaction between the target object and the environment are rendered in real time to achieve motion reconstruction during the interaction between the target object and the environment.

[0137] According to some embodiments, by rendering the optimized human motion data and the interaction proxy object as a 3D model and outputting the final motion reconstruction and mechanical analysis results, it can be used for visualization, analysis, or further processing.

[0138] In summary, the method provided by the embodiments of the present disclosure proposes a physically-driven sparse inertial sensor real-time human interaction motion reconstruction technology, belonging to the fields of computer graphics, inertial sensor technology, human motion capture, human motion physical simulation, etc. By obtaining the data of multiple IMUs and combining advanced deep learning and physical optimization technologies, high-precision full-body pose estimation, global motion estimation, and physical optimization of human-environment interaction are achieved. Finally, accurate 3D motion reconstruction results are generated. While ensuring high precision, this method can effectively cope with environmental complexity and dynamic changes in actions, can expand the application boundaries of motion capture technology, and can be widely applied to fields such as robotics, virtual reality, augmented reality, game development, and human-computer interaction.

[0139] Secondly, by introducing three-dimensional contact and gravity information, a global motion reconstruction in the full three-dimensional space is realized using a physical optimization method without relying on external devices. Compared with existing methods, the present disclosure can not only handle complex three-dimensional motions (such as going up and down stairs, etc.) that cannot be covered by the ground assumption, but also effectively reduce the drift problem of IMU sensors in global pose estimation. It has high precision, real-time performance, and feasibility under sparse IMU configurations, can improve the accuracy and applicability of motion capture while reducing hardware cost and complexity, and has broad application prospects and important technical value.

[0140] In addition, the present disclosure can not only estimate the local pose of the human body, but also simultaneously estimate additional physics-related information, such as physical information like three-dimensional contact points, contact forces, joint torques, and interaction proxy objects. These additional information has important application values in fields such as robot control and human-computer interaction, and can provide more support for applications in fields such as human-computer interaction and robot control.

[0141] To implement the above embodiments, the present disclosure also proposes a real-time human-environment interaction motion reconstruction system based on sparse inertial sensors.

[0142] Exemplarily, Figure 5 is a schematic structural diagram of a real-time human-environment interaction motion reconstruction system based on sparse inertial sensors provided by the embodiments of the present disclosure. As Figure 5As shown, the real-time human-environment interaction motion reconstruction system 500 based on sparse inertial sensors includes:

[0143] A data acquisition unit 501, configured to collect a set of human motion data of a target object through an inertial measurement unit set, and determine first root node measurement data and a first root node gravity direction of a root node corresponding to each human motion data in the set of human motion data, where the position of the root node corresponds one-to-one with the position of the inertial measurement unit on the target object;

[0144] A data estimation unit 502, configured to optimize the first root node measurement data and the first root node gravity direction based on a data-driven method to obtain second root node measurement data, a second root node gravity direction, and estimate human local pose data and global motion data;

[0145] A data optimization unit 503, configured to simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment according to the human local pose data and the global motion data to obtain optimized human motion data, and realize motion reconstruction when the target object interacts with the environment according to the human motion data.

[0146] Optionally, when the data acquisition unit 501 is configured to determine the first root node measurement data and the first root node gravity direction of the root node corresponding to each human motion data in the set of human motion data, it is specifically configured to:

[0147] Vector splice the rotation matrix data, angular velocity data, and acceleration data in the human motion data corresponding to each root node in the set of human motion data to obtain first root node measurement data of the root node corresponding to each human motion data relative to the inertial measurement unit;

[0148] Convert the first global direction in the human motion data corresponding to each root node in the set of human motion data from the world coordinate system to the root node coordinate system to obtain the first root node gravity direction.

[0149] Optionally, when the data estimation unit 502 is configured to optimize the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimate human local pose data and global motion data, it is specifically configured to:

[0150] Optimize the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimate human local pose data;

[0151] Estimate global motion data according to the second root node measurement data, the second root node gravity direction, and the human local pose data.

[0152] Optionally, when the data estimation unit 502 is used to optimize the first root node measurement data and the first root node gravity direction to obtain the second root node measurement data and the second root node gravity direction, it is specifically used for:

[0153] Controlling the first long short-term memory network to reconstruct the leaf joint positions relative to the root node according to the first root node measurement data and the first root node gravity direction and optimize the first root node gravity direction to obtain the third root node gravity direction;

[0154] Optimizing the first global direction according to the minimum angle rotation matrix between the first root node gravity direction and the three root node gravity directions to obtain the second global direction, and correcting the first root node measurement data according to the second global direction to obtain the third root node measurement data;

[0155] Controlling the second long short-term memory network to estimate the full joint positions according to the third root node measurement data, the third root node gravity direction and the leaf joint positions and optimize the third root node gravity direction to obtain the second root node gravity direction;

[0156] Optimizing the second global direction according to the minimum angle rotation matrix between the third root node gravity direction and the second root node gravity direction to obtain the third global direction, and correcting the third root node measurement data according to the third global direction to obtain the second root node measurement data.

[0157] Optionally, when the data estimation unit 502 is used to estimate the human body local pose data, it is specifically used for:

[0158] Controlling the third long short-term memory network to estimate the human body local pose data according to the second root node measurement data, the second root node gravity direction and the full joint positions.

[0159] Optionally, when the data estimation unit 502 is used to estimate the global motion data according to the second root node measurement data, the second root node gravity direction and the human body local pose data, it is specifically used for:

[0160] Controlling the fourth long short-term memory network to estimate the first root node velocity and the joint rest probability corresponding to each target joint in the target joint set according to the second root node measurement data, the second root node gravity direction and the human body local pose data, where the first root node velocity includes a first orthogonal component and a second orthogonal component, the first orthogonal component is parallel to the second root node gravity direction, and the second orthogonal component is perpendicular to the second root node gravity direction;

[0161] Determine the set of stationary joints in the target joint set according to the joint stationary probability, and optimize the speed of the first root node with the goal of keeping the stationary joints in the set of stationary joints stationary and minimizing the adjustment value of the speed of the first root node, to obtain the speed of the second root node;

[0162] Determine the global motion data according to the speed of the second root node.

[0163] Optionally, when the data optimization unit 503 is used to simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment according to the human body local pose data and the global motion data, and obtain the optimized human body motion data, it is specifically used for:

[0164] Without setting an external contact force, use the physical model corresponding to the target object to pre-track the human body local pose data and the global motion data to obtain the first joint torque of the physical model, where the first joint torque includes the virtual force of the root node that needs to be explained by the contact force;

[0165] According to human body dynamics, determine the set of three-dimensional contact points that can explain the virtual force of the root node from the set of stationary joints in the target joint set, and determine the contact force corresponding to each three-dimensional contact point in the set of three-dimensional contact points;

[0166] Set the contact force corresponding to the three-dimensional contact point as the external contact force at the position corresponding to the three-dimensional contact point in the physical model, and use the physical model corresponding to the target object to re-track the human body local pose data and the global motion data to obtain the optimized human body motion data, where the human body motion data includes the optimized human body local pose data, the optimized global motion data, and the second joint torque.

[0167] Optionally, when the data optimization unit 503 is used to perform motion reconstruction when the target object interacts with the environment according to the human body motion data, it is specifically used for:

[0168] Perform real-time rendering on the optimized human body local pose data, the optimized global motion data, the second joint torque, the set of three-dimensional contact points, the contact force, the physical model, and the interaction proxy object when the target object interacts with the environment, so as to realize the motion reconstruction when the target object interacts with the environment.

[0169] It should be noted that the foregoing explanation of the embodiments of the real-time human-environment interaction motion reconstruction method based on sparse inertial sensors also applies to the real-time human-environment interaction motion reconstruction system based on sparse inertial sensors of this embodiment, and will not be elaborated here.

[0170] In summary, the system provided by the embodiments of the present disclosure can effectively reduce the drift problem caused by the accumulation of measurement errors of inertial sensors in traditional methods by introducing the constraints of gravity on human postures and movements in interactive reconstruction. Secondly, by combining the dual constraints of data-driven and physical optimization, more accurate global motion reconstruction can be achieved under the condition of only using inertial measurement units. Especially in terms of global movement and global orientation estimation in three-dimensional space, it overcomes the limitations of the planar ground assumption and the global posture drift problem in traditional methods and can achieve accurate three-dimensional interactive motion reconstruction.

[0171] To implement the above embodiments, the present disclosure also provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the foregoing embodiments.

[0172] To implement the above embodiments, the present disclosure also provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method provided by the foregoing embodiments when executed by a processor.

[0173] To implement the above embodiments, the present disclosure also provides a computer program product including a computer program, which implements the method provided by the foregoing embodiments when executed by a processor.

[0174] The collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved in the present disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0175] It should be noted that personal information from users should be collected for legal and reasonable purposes and not shared or sold outside of these legal uses. In addition, such collection / sharing should be carried out after obtaining the informed consent of the user, including but not limited to notifying the user to read the user agreement / user notice and signing an agreement / authorization including authorizing relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0176] The present disclosure anticipates providing embodiments that allow users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of the user.

[0177] In the description of the foregoing embodiments, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0178] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0179] Any process or method description in a flowchart or described otherwise herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in an order opposite to that shown or discussed, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure pertain.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then stored in a computer memory.

[0181] It should be understood that various parts of the present disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0182] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0183] In addition, each functional unit in various embodiments of the present disclosure may be integrated into a processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0184] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A real-time human-environment interaction motion reconstruction method based on sparse inertial sensors, characterized in that, Including: Collecting a set of human motion data of a target object through a set of inertial measurement units, and determining first root node measurement data and a first root node gravity direction of a root node corresponding to each human motion data in the set of human motion data, wherein the position of the root node corresponds one-to-one with the position of the inertial measurement unit on the target object; Optimizing the first root node measurement data and the first root node gravity direction based on a data-driven method to obtain second root node measurement data, a second root node gravity direction, and estimating human local pose data and global motion data; According to the human local pose data and the global motion data, simulating and optimizing the physical dynamics when the physical model corresponding to the target object interacts with the environment to obtain optimized human motion data, and realizing motion reconstruction when the target object interacts with the environment according to the human motion data.

2. The method according to claim 1, wherein The determining the first root node measurement data and the first root node gravity direction of the root node corresponding to each human motion data in the set of human motion data includes: Vector splicing the rotation matrix data, angular velocity data, and acceleration data in the human motion data corresponding to each root node in the set of human motion data to obtain first root node measurement data of the root node relative to the inertial measurement unit corresponding to each human motion data; Converting the first global direction in the human motion data corresponding to each root node in the set of human motion data from the world coordinate system to the root node coordinate system to obtain the first root node gravity direction.

3. The method according to claim 1, characterized in that, The optimizing the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimating human local pose data and global motion data includes: Optimizing the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction, and estimating human local pose data; Estimating global motion data according to the second root node measurement data, the second root node gravity direction, and the human local pose data.

4. The method according to claim 3, characterized in that The optimizing the first root node measurement data and the first root node gravity direction to obtain second root node measurement data, a second root node gravity direction includes: Controlling a first long short-term memory network to reconstruct the position of the leaf joint relative to the root node according to the first root node measurement data and the first root node gravity direction and optimize the first root node gravity direction to obtain a third root node gravity direction; Optimizing the first global direction according to the minimum angle rotation matrix between the first root node gravity direction and the three root node gravity directions to obtain a second global direction, and correcting the first root node measurement data according to the second global direction to obtain third root node measurement data; Control the second long short-term memory network to estimate the full joint position based on the third root node measurement data, the gravity direction of the third root node, and the leaf joint position, and optimize the gravity direction of the third root node to obtain the gravity direction of the second root node; Optimize the second global direction according to the minimum angle rotation matrix between the gravity direction of the third root node and the gravity direction of the second root node to obtain the third global direction, and correct the third root node measurement data according to the third global direction to obtain the second root node measurement data.

5. The method according to claim 4, wherein The estimated human body local pose data includes: Control the third long short-term memory network to estimate the human body local pose data based on the second root node measurement data, the gravity direction of the second root node, and the full joint position.

6. The method according to claim 3, characterized in that, The estimating of the global motion data according to the second root node measurement data, the gravity direction of the second root node, and the human body local pose data includes: Control the fourth long short-term memory network to estimate the first root node velocity and the joint rest probability corresponding to each target joint in the target joint set based on the second root node measurement data, the gravity direction of the second root node, and the human body local pose data, where the first root node velocity includes a first orthogonal component and a second orthogonal component, the first orthogonal component is parallel to the gravity direction of the second root node, and the second orthogonal component is perpendicular to the gravity direction of the second root node; Determine the static joint set in the target joint set according to the joint rest probability, and optimize the first root node velocity with the goal that the static joints in the static joint set remain stationary and the adjustment value of the first root node velocity is the lowest to obtain the second root node velocity; Determine the global motion data according to the second root node velocity.

7. The method according to claim 1, characterized in that, The simulating and optimizing of the physical dynamics during the interaction between the physical model corresponding to the target object and the environment according to the human body local pose data and the global motion data to obtain the optimized human motion data includes: Without setting an external contact force, use the physical model corresponding to the target object to pre-track the human body local pose data and the global motion data to obtain the first joint torque of the physical model, where the first joint torque includes the root node virtual force that needs to be explained by the contact force; According to human body dynamics, determine the three-dimensional contact point set that can explain the root node virtual force from the static joint set in the target joint set, and determine the contact force corresponding to each three-dimensional contact point in the three-dimensional contact point set; Set the contact force corresponding to the three-dimensional contact point as the external contact force at the position corresponding to the three-dimensional contact point in the physical model, and use the physical model corresponding to the target object to re-track the human body local pose data and the global motion data to obtain the optimized human motion data, where the human motion data includes the optimized human body local pose data, the optimized global motion data, and the second joint torque.

8. The method according to claim 7, characterized in that, Implementing the motion reconstruction when the target object interacts with the environment according to the human motion data includes: Performing real-time rendering on the optimized human local pose data, the optimized global motion data, the second joint torque, the three-dimensional contact point set, the contact force, the physical model, and the interaction proxy object when the target object interacts with the environment, so as to implement the motion reconstruction when the target object interacts with the environment.

9. A real-time human-environment interaction motion reconstruction system based on sparse inertial sensors, characterized in that, It includes: A data acquisition unit, configured to collect a set of human motion data of a target object through a set of inertial measurement units, and determine the first root node measurement data and the first root node gravity direction of the root node corresponding to each human motion data in the set of human motion data, wherein the position of the root node corresponds one-to-one with the position of the inertial measurement unit on the target object; A data estimation unit, configured to optimize the first root node measurement data and the first root node gravity direction based on a data-driven method to obtain the second root node measurement data and the second root node gravity direction, and estimate the human local pose data and the global motion data; A data optimization unit, configured to simulate and optimize the physical dynamics when the physical model corresponding to the target object interacts with the environment according to the human local pose data and the global motion data, obtain the optimized human motion data, and implement the motion reconstruction when the target object interacts with the environment according to the human motion data.

10. An electronic device, characterized in that, The electronic device includes: A memory, configured to store executable program code; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

Images