Human body motion capture real-time rendering method and system

Abstract

The invention provides a human body motion capture real-time rendering method and system, and belongs to the technical field of three-dimensional human body motion, and the method comprises the following steps: initializing sensors in inertial measurement units on a plurality of human bodies, carrying out initial alignment on the initialized sensors of the inertial measurement units on the human bodies, obtaining an attitude relationship between the inertial measurement unit coordinate system and the limb coordinate system; based on the posture relation between the inertial measurement unit coordinate system and the limb coordinate system, a sensor of the inertial measurement unit on the human body after initial alignment is adopted for data collection; inputting the collected data into a human body posture prediction model to obtain rotation data and global displacement of a plurality of joint points; and correspondingly capturing the rotation data and the global displacement of the plurality of joint points to obtain a real-time motion capture rendering task. According to the method, the real-time performance of the human body motion capture system is ensured, and the capture reliability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional human body motion, and in particular relates to a method and system for real-time rendering of human body motion capture. Background Art

[0002] Human motion capture aims to reconstruct three-dimensional human motion and plays an important role in various applications such as games, sports, medicine, VR / AR (Virtual Reality / Augmented Reality), and filmmaking. So far, vision-based motion capture solutions are the main solutions for human motion capture. The first category requires optical markers to be installed on the human body and multiple cameras to track the markers for motion capture. Marker-based systems such as Vicon are widely used and are considered accurate enough for industrial use. However, this method requires expensive infrastructure and invasive equipment, which makes them unsuitable for general consumer-level use. Another classification focuses on using some RGB or RGB-D (Red, Green, Blue, Depth) cameras for motion capture. Although this method is much lighter, it is limited to the appearance characteristics of the human body because it needs to extract distinguishable features from the image.

[0003] Therefore, in recent years, the use of sensor-based motion capture has attracted more and more attention. According to the different sensor models, common human motion capture systems can be mainly divided into mechanical, acoustic, optical, electromagnetic and inertial sensor types. Among them, the mechanical human motion capture system mainly uses mechanical devices to track and measure the trajectory of human motion. The advantages of this method are low cost and high accuracy, but because the mechanical equipment has large restrictions on the wearer's movements, it is mainly used for capturing static postures and determining key frames. The acoustic motion capture system is usually composed of a transmitter, a receiver and a processing unit. The main principle is to measure the time or phase difference of the sound wave from the transmitter to the receiver. The acoustic equipment has a low cost, but it has a high delay in capturing human motion and is easily interfered by factors such as occlusions and noise. The optical human motion capture system mainly uses computer vision principles and relies on multiple precise and complex high-speed cameras to track and calculate the target feature points of the human body from different angles to achieve real-time human motion capture. However, the high-speed camera is expensive and has restrictions on the site due to the influence of light. Electromagnetic motion capture system is also a commonly used motion capture device, usually composed of a transmitter, a receiving sensor and a data processing unit. This method is fast, practical and relatively low-cost, but has strict requirements on the environment and the magnetic field is easily disturbed, resulting in inaccurate data. Based on the above description, the above human motion capture systems have gradually failed to meet the high requirements of current motion capture systems.

[0004] Thanks to the rapid development of micro-electromechanical system technology, the cost of inertial sensors has been continuously reduced, and the integration has been continuously improved, resulting in a smaller size and a greatly improved measurement accuracy, which has gradually attracted everyone's attention and research on the human motion capture system based on inertial sensors. This method uses multiple inertial sensors installed on various key parts of the human body for measurement. After the data collected by the sensors are fused and solved and converted between coordinate systems, the data is transmitted in real time and the human body model is driven to achieve real-time capture of human motion. Generally, the inertial sensor-based human motion capture system uses an inertial measurement unit (IMU) containing a gyroscope, accelerometer and magnetometer. Each IMU can directly measure three-dimensional posture data. Summary of the invention

[0005] In view of the deficiencies of the prior art, a method and system for real-time rendering of human motion capture is provided.

[0006] In a first aspect, the present application proposes a method for real-time rendering of human motion capture, comprising:

[0007] Initializing sensors in multiple inertial measurement units on the human body, wherein the multiple inertial measurement units on the human body include: a head inertial measurement unit, a waist inertial measurement unit, a left forearm inertial measurement unit, a right forearm inertial measurement unit, a left calf inertial measurement unit, and a right calf inertial measurement unit;

[0008] Performing initial alignment on the sensor of the inertial measurement unit on the initialized human body to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system;

[0009] Based on the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, data collection is performed using a sensor of the inertial measurement unit on the human body after initial alignment;

[0010] The collected data is input into the human posture prediction model to obtain the rotation data of multiple joints and the global displacement;

[0011] The rotation data and global displacement of the multiple joint points are correspondingly captured to obtain a real-time motion capture rendering task.

[0012] The sensors of the inertial measurement unit include: a gyroscope, an accelerometer and a magnetometer.

[0013] Initializing sensors in inertial measurement units on multiple human bodies includes:

[0014] Based on the initial data of the accelerometer, the initial pitch angle and the initial roll angle are calculated;

[0015] Converting the initial pitch angle and the initial roll angle into values ​​in a spatial physical coordinate system;

[0016] According to the value in the space physical coordinate system, the initial yaw angle is calculated;

[0017] According to the initial pitch angle, initial roll angle and initial yaw angle, the initial quaternion of the sensor collected data relative to the physical coordinates is calculated.

[0018] The initial pitch angle and the initial roll angle are converted into values ​​in the spatial physical coordinate system, and the calculation formula is as follows:

[0019]

[0020] in, is the x-axis measurement data of the inertial measurement unit, is the y-axis measurement data of the inertial measurement unit, is the z-axis measurement data of the inertial measurement unit, is the x-axis value in the space physical coordinate system, is the y-axis value in the spatial physical coordinate system, is the z-axis value in the spatial physical coordinate system, φ(0) is the initial roll angle, and θ(0) is the initial pitch angle.

[0021] The initial yaw angle is calculated based on the values ​​in the space physical coordinate system, and the calculation formula is as follows:

[0022]

[0023] in, is the x-axis value in the space physical coordinate system, is the y-axis value in the spatial physical coordinate system, is the z-axis value in the spatial physical coordinate system, yaw is the yaw angle, and ψ(0) is the initial yaw angle.

[0024] The initial pitch angle, initial roll angle and initial yaw angle are used to calculate the initial quaternion of the sensor acquisition data relative to the physical coordinates. The calculation formula is as follows:

[0025]

[0026] in, is the initial quaternion, θ(0) is the initial roll angle, θ(0) is the initial pitch angle, and ψ(0) is the initial yaw angle.

[0027] The initial alignment of the sensor of the inertial measurement unit on the initialized human body to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system includes:

[0028] The wearer of the inertial measurement unit stands still in a T-position;

[0029] The wearer's waist inertial measurement unit is selected as the human body reference coordinate system, and the inertial measurement unit coordinate system and the limb coordinate system are aligned using the initial quaternion to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system.

[0030] The initial quaternion is used to align the inertial measurement unit coordinate system with the limb coordinate system to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system. The calculation formula is as follows:

[0031] The initial quaternion of the sensor in each inertial measurement unit relative to the human limb joint coordinate system is calculated as follows:

[0032]

[0033] During the initialization of the sensors in the inertial measurement unit, at the initial moment, each joint A of the human body i The relationship between the relative physical coordinate system E is calculated as follows:

[0034]

[0035] The posture relationship between the inertial measurement unit coordinate system and the limb coordinate system is calculated as follows:

[0036]

[0037] in, is the initial moment of each joint A of the human body in each limb coordinate system i The quaternion of is the physical coordinate matrix in the i-th limb coordinate system at the initial moment, is the physical coordinate matrix in the i-th limb coordinate system at any time, is the initial moment at A i The physical coordinate matrix in the limb coordinate system, is the physical coordinate system matrix of the ith limb at any time, is the physical coordinate system matrix of the i-th limb at the initial moment.

[0038] The human body posture prediction model includes: an RNN prediction model, a node position model, an action prediction model and a joint node speed prediction model.

[0039] The collected data includes acceleration data and displacement data.

[0040] The collected data is input into the human body posture prediction model to obtain the rotation data of multiple joints and the global displacement, including:

[0041] Filter the collected data to remove the noise in the data and obtain filtered data;

[0042] Input the filtered data into the RNN prediction model to obtain leaf node position information;

[0043] Input the leaf node position information and the calibration superposition vector into the node position model to obtain the position information of all joint nodes;

[0044] Inputting the position information of all joint nodes and the filtered data into the motion prediction model to obtain the relative joint rotation data of the non-root joint point relative to the root joint point;

[0045] The position information of all joint nodes and the probability of the human body contacting the ground are input into the joint node velocity prediction model to obtain the joint node velocity;

[0046] The global displacement is calculated according to the joint node velocity.

[0047] The calibration superposition vector is calculated as follows:

[0048] x=[a root , ..., a rarm , R rooot , ..., R rarm ]

[0049] Where x is the calibration stacking vector, a root is the acceleration data of the waist measured by the inertial measurement unit during the initial alignment process, a rarm is the acceleration data of the right forearm of the waist measured by the inertial measurement unit during the initial alignment process, R root is the waist direction data measured by the inertial measurement unit during the initial alignment process, R rarm It is the orientation data of the right forearm measured by the inertial measurement unit during the initial alignment process.

[0050] In a second aspect, the present application proposes a human motion capture and real-time rendering system, comprising:

[0051] A data acquisition module is used to initialize sensors in multiple inertial measurement units on a human body, wherein the multiple inertial measurement units on the human body include: a head inertial measurement unit, a waist inertial measurement unit, a left forearm inertial measurement unit, a right forearm inertial measurement unit, a left calf inertial measurement unit, and a right calf inertial measurement unit; the sensors of the inertial measurement units on the human body after initialization are initially aligned to obtain a posture relationship between the inertial measurement unit coordinate system and the limb coordinate system; based on the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, the sensors of the inertial measurement units on the human body after initial alignment are used to perform data acquisition;

[0052] The central calculation module is used to input the collected data into the human posture prediction model to obtain the rotation data of multiple joints and the global displacement;

[0053] The data rendering module is used to correspondingly capture the rotation data and global displacement of the multiple joint points to obtain a real-time motion capture rendering task.

[0054] Beneficial effects:

[0055] The present application proposes a method and system for real-time rendering of human motion capture, which captures the acceleration information and joint direction information of six key motion joints to predict the motion data of 24 joints of the whole body to achieve motion capture, and uses the captured motion sequence data for real-time rendering tasks. The method of the present application ensures the real-time performance of the human motion capture system and improves the reliability of capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flow chart of a method for real-time rendering of human motion capture according to an embodiment of the present application;

[0057] Figure 2 Schematic diagram of calculating roll angle and pitch angle in an embodiment of the present application; wherein (a) is a schematic diagram of roll angle, and (b) is a schematic diagram of pitch angle;

[0058] Figure 3 Schematic diagram of calculating yaw angle in an embodiment of the present application; wherein (a) is a schematic diagram of a magnetometer coordinate system, and (b) is a schematic diagram of a yaw angle;

[0059] Figure 4 Schematic diagram of the initial T-pose and coordinate definition of the embodiment of the present application;

[0060] Figure 5 A schematic diagram of a method for real-time rendering of human motion capture according to an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the principle of a human motion capture and real-time rendering system according to an embodiment of the present application;

[0062] Figure 7 Schematic diagram of a human body posture prediction model according to an embodiment of the present application;

[0063] Figure 8 Motion capture effect diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0064] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0065] The real-time human motion capture system of the present application uses sparse IMU sensors, and uses the acceleration and direction sequence data collected by the IMU inertial sensor as model input. The model predicts the rotation data of the human motion joints and the global displacement data of the joints, and captures the human motion sequence data in real time. The present application proposes a real-time rendering method and system for human motion capture, and designs a comprehensive management architecture to ensure the coordinated work of the inertial sensor data acquisition and processing system and the Unity3D motion rendering system, thereby ensuring the real-time and reliability of the human motion capture system. The present application focuses on the study of the real-time rendering system for motion capture based on inertial sensors, and mainly introduces the composition and implementation technology of the inertial sensor data acquisition and processing system, which provides a research basis and assistance for the real-time rendering system for motion capture, and is of great significance.

[0066] First, this application studies the motion capture algorithm based on IMU inertial sensor and proposes an inertial sensor data acquisition and processing system to improve the reliability of prediction. The inertial sensor data acquisition and processing system includes a motion prediction module and a node speed prediction module, which predict joint rotation information and global displacement information respectively. At the same time, the central calculation module is used to receive and transmit motion sequence data. Afterwards, experiments show that the method of this application is better than the current optimal method. Finally, a data rendering module is designed based on Unity3D to realize the real-time rendering function of the motion sequence.

[0067] Embodiment 1:

[0068] This embodiment proposes a method for real-time rendering of human motion capture, such as Figure 1 As shown, including:

[0069] Step S1: Initializing sensors in multiple inertial measurement units on a human body, wherein the multiple inertial measurement units on a human body include: a head inertial measurement unit, a waist inertial measurement unit, a left forearm inertial measurement unit, a right forearm inertial measurement unit, a left calf inertial measurement unit, and a right calf inertial measurement unit;

[0070] In this embodiment, the human motion capture system based on inertial sensors uses inertial sensors to measure posture data, and then undergoes data conversion processing between coordinate systems to finally obtain corresponding human posture information. However, because a single sensor is easily affected by noise and interference when used alone, the measurement accuracy is low. Therefore, using a multi-sensor data fusion method to improve the accuracy of measurement data has become one of the main methods of human motion capture systems. The present invention uses the gyroscope, accelerometer and magnetometer inside the IMU (inertial measurement unit) sensor for data fusion to improve the system posture solution accuracy.

[0071] Initializing sensors in inertial measurement units on multiple human bodies includes:

[0072] Step S1.1: Based on the initial data of the accelerometer, calculate the initial pitch angle and the initial roll angle;

[0073] In this embodiment, first, the pitch angle and roll angle can be calculated using the initial data of the accelerometer. When the inertial sensor is in a stationary state, it is only affected by gravity. Therefore, the measured values ​​of the accelerometer can be used to calculate the components of gravity acceleration in the sagittal plane and the coronal plane, and the pitch angle and roll angle can be calculated from these components by using appropriate functions. The principle of calculating the roll angle and the pitch angle using gravity acceleration is as follows: Figure 2 As shown, Figure 2 (a) is a schematic diagram of the roll angle. Figure 2 (b) is a schematic diagram of the pitch angle; the calculation formula is as follows:

[0074]

[0075]

[0076] Among them, θ(0) and θ(0) are the calculated initial roll angle and initial pitch angle values, respectively. are the measured values ​​of the accelerometer on the x, y, and z axes respectively.

[0077] Step S1.2: converting the initial pitch angle and the initial roll angle into values ​​in a spatial physical coordinate system;

[0078] Step S1.3: Calculate the initial yaw angle according to the value in the space physical coordinate system;

[0079] After calculating the initial roll angle and initial pitch angle values, the initial yaw angle is calculated using the magnetometer unit of the inertial sensor. The yaw angle includes the magnetic inclination and the magnetic declination. The magnetic inclination refers to the angle between the magnetic needle and the horizontal direction, or the angle between the horizontal component of the geomagnetic field and the total geomagnetic field. The magnetic inclination changes with the latitude of the location of the magnetic needle. Lines with the same magnetic inclination are called isomagnetic lines, and the points with a magnetic inclination of 0° are connected together and are called the geomagnetic equator. Generally speaking, when the north end of the magnetic needle points to the opposite side, the magnetic inclination is positive; when it points to the sky, the magnetic inclination is negative. The magnetic declination is the angle between the magnetic meridian and the geographic meridian at any point on the earth's surface. In simple terms, it is the angle between the projection of the geomagnetic field on the horizontal plane and the true north direction of the sensor location. According to our regulations, when the magnetic pole points to the east of the North Pole, the magnetic declination is positive; when it points to the west, the magnetic declination is negative. The magnetic declination represents the angle between the north that the magnetic needle points to when it is stationary and the true north. The specific schematic diagram is as follows Figure 3 As shown, Figure 3 (a) is a schematic diagram of the magnetometer coordinate system. Figure 3 (b) is a schematic diagram of the yaw angle;

[0080] Since the inertial sensor magnetometer measurement data is a measurement value under the sensor coordinate system standard, it is necessary to convert the measurement data into a value under the spatial physical coordinate system. The specific conversion calculation formula is shown in Formula 3.

[0081]

[0082] in, is the x-axis measurement data of the inertial measurement unit, is the y-axis measurement data of the inertial measurement unit, is the z-axis measurement data of the inertial measurement unit, is the x-axis value in the space physical coordinate system, is the y-axis value in the spatial physical coordinate system, is the z-axis value in the spatial physical coordinate system, φ(O) is the initial roll angle, and θ(O) is the initial pitch angle.

[0083] According to the yaw angle calculation formula as shown in formula (4), the yaw angle value range is shown in formula (5), and the initial quaternion of the sensor acquisition data relative to the physical coordinates can be calculated according to formula (6):

[0084]

[0085]

[0086] in, is the x-axis value in the space physical coordinate system, is the y-axis value in the spatial physical coordinate system, is the z-axis value in the spatial physical coordinate system, yaw is the yaw angle, and ψ(O) is the initial yaw angle.

[0087] Step S1.4: According to the initial pitch angle, the initial roll angle and the initial yaw angle, the initial quaternion of the sensor acquisition data relative to the physical coordinates is calculated.

[0088]

[0089] in, is the initial quaternion.

[0090] Step S2: Initially align the sensor of the inertial measurement unit on the initialized human body to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, including:

[0091] Step S2.1: The wearer of the inertial measurement unit stands still in a T-shaped posture;

[0092] Step S2.2: Select the wearer's waist inertial measurement unit as the human body reference coordinate system, use the initial quaternion to align the inertial measurement unit coordinate system with the limb coordinate system, and obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system.

[0093] In this embodiment, in order to use multiple IMU sensor nodes to capture human motion, it is necessary to align the IMU with each limb part of the human body so that the IMU sensor node can output the posture information of each limb part. Since the human body is not a standard geometric shape, and the soft tissue between the human skeleton and the human epidermis is prone to deformation, it is impossible to make the IMU sensor truly worn along a certain limb direction, so it is necessary to perform sensor alignment and calibration to obtain the posture relationship between the IMU coordinate system and the limb coordinate system. In addition, in order to be able to reconstruct the posture of the human body, it is necessary to unify the posture of each part into a reference coordinate system, and usually the posture of this reference coordinate system is fixed relative to the geographic coordinate system. The present invention uses the classic static alignment posture "T-pose" for initialization and calibration.

[0094] The static T-pose alignment posture calibration method requires the tester wearing the sensor device to maintain a T-shaped posture at the initial moment, and complete the alignment operation between the sensor and the physical coordinate system based on the data conversion between the coordinates in a static posture. The wearer of the sensor device is required to stand still in the T-pose, and then select the waist as the human body reference coordinate system using the sensor initial quaternion Align the IMU sensor with each limb of the human body. Coordinate system A of each node of the human body iCoincident with the physical coordinate system E, the initial quaternion of each joint point sensor relative to the human limb joint coordinate system is calculated as formula (7):

[0095]

[0096] At the initialization moment, the transformation relationship between each joint of the human body and the physical coordinate system is as shown in formula (8).

[0097]

[0098] From the calculation formula (8), we can get the initial moment of each joint A of the human body: i The relationship between the IMU and the relative physical coordinate system E is that the position of the IMU will not change during the movement after it is bound to the human body joints, so:

[0099]

[0100] Where E is the physical coordinate system, 0 is the initial moment, i is the i-th IMU sensor, qi is the quaternion of the physical coordinate system of the i-th IMU sensor, Si represents the limb coordinate system of the i-th joint, is the initial moment of each joint A of the human body in each limb coordinate system i The quaternion of is the physical coordinate matrix in the i-th limb coordinate system at the initial moment, is the physical coordinate matrix in the i-th limb coordinate system at any time, is the initial moment at A i The physical coordinate matrix in the limb coordinate system, is the physical coordinate system matrix of the ith limb at any time, is the physical coordinate system matrix of the i-th limb at the initial moment.

[0101] The above static T-pose alignment posture calibration method is used to complete the alignment of the IMU and each joint of the human body, and the alignment relationship between the IMU and the human joints of each node is recorded. The quaternion of the human joint relative to the physical coordinate system is obtained through conversion, where i corresponds to the sensor number. The initial T-pose posture and coordinate system are defined as follows Figure 4 shown.

[0102] Step S3: Based on the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, the sensor of the inertial measurement unit on the human body after initial alignment is used to collect data;

[0103] In this embodiment, the inertial sensor data acquisition and processing system for predicting joint rotation data and speed information and the Unity3D real-time rendering system for real-time rendering effects are closely combined to complete the real-time capture rendering task. The inertial sensor transmits the collected motion data to the data acquisition and processing end through serial communication, and the joint acceleration and joint direction sequence data collected by the six node sensors are used as the input of the prediction model to output the joint rotation data and speed data of the 24 joints of the human body model. Among them, the 24 joints include: Pelvis: the central reference point of the human body, connecting the spine and lower limbs. Lower spine: connecting the pelvis and the middle of the spine. Middle spine: connecting the lower spine and the upper spine. Upper spine: connecting the middle spine and the neck. Neck: connecting the upper spine and the head. Head: the top of the human body, including eyes, ears, etc. Left shoulder: connecting the left arm and the trunk. Left Elbow: Connects the left upper arm and the left forearm. Left Wrist: Connects the left forearm and the left hand. Left Hand: The end of the left hand. Right Shoulder: Connects the right arm and the torso. Right Elbow: Connects the right upper arm and the right forearm. Right Wrist: Connects the right forearm and the right hand. Right Hand: The end of the right hand. Left Hip: Connects the left leg and the pelvis. Left Knee: Connects the left thigh and the left calf. Left Ankle: Connects the left calf and the left foot. Left Foot: The end of the left foot. Right Hip: Connects the right leg and the pelvis. Right Knee: Connects the right thigh and the right calf. Right Ankle: Connects the right calf and the right foot. Right Foot: The end of the right foot. Left Clavicle: Connects the left shoulder and torso. Right Clavicle: Connects the right shoulder and torso. The integrated management system is used to establish a communication connection with the inertial sensor data acquisition and processing system, and distributes data through AMQP technology to ensure that the client obtains complete and effective joint motion sequence data after prediction and processing. After the client obtains the motion sequence data, it binds the data of the 24 joint points of the human body to the corresponding bone nodes, and completes the real-time rendering task of human motion capture after processing by the rendering system. The overall process of the system is as follows: Figure 5 The detailed structure of the system is shown in Figure 6 The integrated management architecture and Unity3D rendering system proposed in the present invention are both main components of the human motion capture real-time rendering system, ensuring the integrity of the system.

[0104] This system uses IMU (Inertial Measurement Unit) sensors for human motion capture. Compared with traditional motion capture sensors, IMU sensors have higher flexibility and portability. In addition, IMU sensors can detect and measure human motion postures in real time. Compared with traditional vision-based motion capture systems, IMU sensors do not need to process and analyze the collected videos or images, but can directly provide real-time motion data. The measurement data of IMU sensors is more accurate and has higher data acquisition stability. Its measurement results can provide more precise posture tracking accuracy and maintain stability in various environments. Therefore, IMU sensors are very suitable for the content of this study. The use of IMU can bring more reliable and accurate human motion capture capabilities to this system.

[0105] The inertial sensor data acquisition and processing system uses six IMU sensors to measure the acceleration and rotation data of the six joints of the personnel waist, head, left forearm, right forearm, left calf, and right calf. The waist is the reference IMU measurement node, and the acceleration data of each node ∈R 3 , the rotation data of each node ∈ R 9 .

[0106] The IMU sensor collects the acceleration and direction data of the six joints through serial communication and transmits them to the human posture prediction model as the input of the human posture prediction model system. After being processed by the human posture prediction model, the velocity data and rotation data of the 24 joints of the human body are output. The following mainly introduces the human posture prediction model of the inertial sensor data acquisition and processing system.

[0107] Step S4: input the collected data into a human posture prediction model to obtain rotation data of multiple joint points and global displacement;

[0108] In this embodiment, the human body posture prediction model includes: RNN prediction model, node position model, action prediction model and joint node speed prediction model. The collected data includes: acceleration data and direction data.

[0109] The collected data is input into the human body posture prediction model to obtain the rotation data of multiple joints and the global displacement, such as Figure 7 As shown, including:

[0110] Step S4.1: filtering the collected data to remove noise in the data and obtain filtered data;

[0111] Step S4.2: input the filtered data into the RNN prediction model to obtain leaf node position information;

[0112] In this embodiment, the structure of the RNN prediction model includes:

[0113] Input layer: Input data: IMU rotation matrix and acceleration

[0114] RNN layer: Use bidirectional LSTM (biLSTM) to process action sequence data.

[0115] Fully connected layer: maps the output of the RNN to the target space (such as joint position or rotation).

[0116] Step S4.3: Input the leaf node position information and the calibration superposition vector into the node position model to obtain the position information of all joint nodes;

[0117] In this embodiment, the structure of the node location model includes:

[0118] Input: IMU rotation and acceleration data;

[0119] Middle layer: bidirectional LSTM layer;

[0120] Fully connected layer: maps the LSTM output to 15 dimensions.

[0121] Output: Node location.

[0122] Step S4.4: inputting the position information of all joint nodes and the filtered data into the motion prediction model to obtain the relative joint rotation data of the non-root joint point relative to the root joint point;

[0123] In this embodiment, the root node refers to the waist, and the non-root joint points refer to the right forearm, the left forearm, the left calf, the right calf, and the head. The data received in this embodiment is the data of the sensors of 6 parts, and then the rotation position relative to the root node is predicted based on the approximate relative position of the human body.

[0124] Step S4.5: input the position information of all joint nodes and the probability of the human body contacting the ground into the joint node velocity prediction model to obtain the joint node velocity;

[0125] Step S4.6: Calculate the global displacement based on the joint node velocity.

[0126] In this embodiment, the model uses RNN as the basic network model, not only estimating the root joint velocity, but also estimating the velocity of all related nodes to obtain a more accurate global displacement estimation result. The human posture prediction model is divided into the following two models:

[0127] 1) Action prediction model: Through the RNN prediction model, the leaf node position information P is predicted using the IMU measurement data leaf , Rleaf Together with the IMU measurement data, it is used as the input of the motion prediction model to regress and predict the joint position information and joint rotation information.

[0128] The structure of the action prediction model includes:

[0129] Input: Node position model results as input;

[0130] Bidirectional LSTM layer: 2 layers, with 128 hidden units in each layer.

[0131] Fully connected layer: maps the LSTM output to 132 dimensions (rotations of 22 non-root joints).

[0132] Output: Rotation representations of all non-root joints.

[0133] 2) Joint node velocity prediction model: The joint node position information and IMU measurement data are used to predict the joint node movement speed, and the joint node movement speed is used to predict the global displacement.

[0134] The structure of the joint node velocity prediction model includes:

[0135] Input: position information of all joint nodes and filtered data.

[0136] Unidirectional LSTM layer: 2 layers, with 256 hidden units in each layer.

[0137] Fully connected layer: maps LSTM output to 3 dimensions

[0138] Output: The velocity of the root joint in its own coordinate system.

[0139] There is ambiguity in motion capture using sparse sensors. For example, due to the sparsity of sensors, it is difficult to distinguish between standing still and sitting for a long time, because the sensor measurement data is basically the same. In order to resolve the ambiguity, the state information in the historical data frame must be captured. Therefore, we use RNN to retain the complete historical state information.

[0140] At the beginning, a T-pose calibration is performed, and during the processing, the acceleration data and rotation data measured by the IMU are superimposed into a single input vector x = [a root , ..., a rarm , R root , ..., R rarm ], a is acceleration information, and R is direction information. We use the recurrent neural network RNN ​​model and LSTM to learn the mapping relationship from IMU measurement data to leaf joint positions. For LSTM we have:

[0141]

[0142] in, Use vector x regression to predict the leaf node position information P leaf ∈R 15 Then combine P leaf and x form a joint vector [P leaf x] as the input of the model for predicting the position of all joint nodes, and output the position information P of all joint nodes all ∈R 3J (where J = 24, representing the number of human motion joints); then all joint node position information and IMU measurement data are combined into a joint vector [P all x] is used as the input of the motion prediction model and the joint velocity prediction model.

[0143] The input of the motion prediction model is the concatenated vector of all joint node position information and inertial measurement values. The input vector is input into the RNN prediction model to predict the relative joint rotation data of the non-root joint point relative to the root joint point. The rotation data is represented in 6D form as R∈R 6(J-1) , the root node rotation data information can be measured by the sensor. Finally, the root node measured rotation data information and the predicted non-root joint node rotation data information are combined into the action posture data θ∈R 9J .

[0144] The main task of the joint node velocity prediction module is to predict the linear velocity of all joint nodes of the human body's actions. Since the IMU sensor cannot directly measure distance information, it is very challenging to predict the global displacement. Here, the RNN node velocity prediction model is used to complete the global displacement prediction task. The linear velocity information of the human body's joint nodes is predicted mainly by using the body position information predicted by the joint point position prediction module and the IMU sensor measurement data. The joint node velocity prediction module uses the RNN recurrent neural network to predict the global displacement velocity of all joint nodes based on the joint position information of the position prediction model and the basic probability of human body contact with the ground. The RNN model outputs the joint node velocity in the sensor coordinates, and converts it into the velocity in the global coordinates according to the rotation data of each joint in the body posture θ. Because predicting the joint node velocity requires more information from previous frames, using RNN can ensure that there is sufficient window size to meet the needs of the prediction model when running. The prediction model outputs the velocity information of the joint nodes in the global coordinate system, and uses the joint node velocity to predict the global displacement.

[0145] By predicting the posture data θ and the posture root node velocity v, the action sequence data is finally predicted to complete the motion capture task. The relevant data sets used in this model training are as follows:

[0146] 1) DIP-IMU, including IMU measurement data of 10 people wearing 17 IMU sensors and performing 90 minutes of exercise;

[0147] 2) TotalCapture, including global displacement data, posture parameters, and IMU measurement-related data of 5 people wearing 13 IMU sensors for approximately 50 minutes of exercise.

[0148] Through Bluetooth serial communication, the inertial sensor transmits the collected data to the data acquisition and processing system. After receiving the data collected by the IMU sensor, the data acquisition and processing system passes it to the human posture prediction model for posture prediction. This is done to ensure the stability of data collection and the reliability of motion prediction data, and to provide real-time data consumption capabilities for the rendering system.

[0149] Step S5: Capture the rotation data of the multiple joint points and the global displacement accordingly to obtain a real-time motion capture rendering task.

[0150] In this embodiment, unity3D is used to render the captured action in real time. First, the tester binds the IMU inertial sensors to the waist, head, left forearm, right forearm, left calf, and right calf to ensure that the T-Pose initialization is completed. After the initialization calibration operation is completed, the data acquisition and processing system obtains the sensor collected action sequence data. The human body posture prediction model predicts and outputs the joint rotation data and displacement data based on the sensor collected data. The action rendering system binds the joint data accordingly to complete the real-time action capture rendering task. Action capture real-time rendering such as Figure 8 As shown, the hand and leg movements are accurately captured in real time and the rendering task is completed.

[0151] Embodiment 2:

[0152] This embodiment proposes a human motion capture real-time rendering system. Figure 6 As shown, including:

[0153] A data acquisition module is used to initialize sensors in multiple inertial measurement units on a human body, wherein the multiple inertial measurement units on the human body include: a head inertial measurement unit, a waist inertial measurement unit, a left forearm inertial measurement unit, a right forearm inertial measurement unit, a left calf inertial measurement unit, and a right calf inertial measurement unit; the sensors of the inertial measurement units on the human body after initialization are initially aligned to obtain a posture relationship between the inertial measurement unit coordinate system and the limb coordinate system; based on the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, the sensors of the inertial measurement units on the human body after initial alignment are used to perform data acquisition;

[0154] The central calculation module is used to input the collected data into the human posture prediction model to obtain the rotation data of multiple joints and the global displacement;

[0155] The data rendering module is used to correspondingly capture the rotation data and global displacement of the multiple joint points to obtain a real-time motion capture rendering task.

[0156] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0157] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A method for real-time rendering of human motion capture, characterized in that: include: Initializing sensors in multiple inertial measurement units on the human body, wherein the multiple inertial measurement units on the human body include: a head inertial measurement unit, a waist inertial measurement unit, a left forearm inertial measurement unit, a right forearm inertial measurement unit, a left calf inertial measurement unit, and a right calf inertial measurement unit; Performing initial alignment on the sensor of the inertial measurement unit on the initialized human body to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system; Based on the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, data collection is performed using a sensor of the inertial measurement unit on the human body after initial alignment; The collected data is input into the human posture prediction model to obtain the rotation data of multiple joints and the global displacement; The rotation data and global displacement of the multiple joint points are correspondingly captured to obtain a real-time motion capture rendering task.

2. A method for real-time rendering of human motion capture according to claim 1, characterized in that: The sensors of the inertial measurement unit include: a gyroscope, an accelerometer and a magnetometer.

3. The method for real-time rendering of human motion capture according to claim 1, characterized in that: Initializing sensors in inertial measurement units on multiple human bodies includes: Based on the initial data of the accelerometer, the initial pitch angle and the initial roll angle are calculated; Converting the initial pitch angle and the initial roll angle into values ​​in a spatial physical coordinate system; According to the value in the space physical coordinate system, the initial yaw angle is calculated; According to the initial pitch angle, initial roll angle and initial yaw angle, the initial quaternion of the sensor collected data relative to the physical coordinates is calculated.

4. A method for real-time rendering of human motion capture according to claim 3, characterized in that: The initial pitch angle and the initial roll angle are converted into values ​​in the spatial physical coordinate system, and the calculation formula is as follows: in, is the x-axis measurement data of the inertial measurement unit, is the y-axis measurement data of the inertial measurement unit, is the z-axis measurement data of the inertial measurement unit, is the x-axis value in the space physical coordinate system, is the y-axis value in the spatial physical coordinate system, is the z-axis value in the spatial physical coordinate system, φ(0) is the initial roll angle, and θ(0) is the initial pitch angle.

5. The method for real-time rendering of human motion capture according to claim 3, characterized in that: The initial yaw angle is calculated based on the values ​​in the space physical coordinate system, and the calculation formula is as follows: in, is the x-axis value in the space physical coordinate system, is the y-axis value in the spatial physical coordinate system, is the z-axis value in the spatial physical coordinate system, yaw is the yaw angle, and ψ(0) is the initial yaw angle.

6. The method for real-time rendering of human motion capture according to claim 3, characterized in that: The initial pitch angle, initial roll angle and initial yaw angle are used to calculate the initial quaternion of the sensor acquisition data relative to the physical coordinates. The calculation formula is as follows: in, is the initial quaternion, φ(0) is the initial roll angle, θ(0) is the initial pitch angle, and ψ(0) is the initial yaw angle.

7. The method for real-time rendering of human motion capture according to claim 1, characterized in that: The initial alignment of the sensor of the inertial measurement unit on the initialized human body to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system includes: The wearer of the inertial measurement unit stands still in a T-position; The wearer's waist inertial measurement unit is selected as the human body reference coordinate system, and the inertial measurement unit coordinate system and the limb coordinate system are aligned using the initial quaternion to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system.

8. The method for real-time rendering of human motion capture according to claim 7, characterized in that: The initial quaternion is used to align the inertial measurement unit coordinate system with the limb coordinate system to obtain the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system. The calculation formula is as follows: The initial quaternion of the sensor in each inertial measurement unit relative to the human limb joint coordinate system is calculated as follows: During the initialization of the sensors in the inertial measurement unit, at the initial moment, each joint A of the human body i The relationship between the relative physical coordinate system E is calculated as follows: The posture relationship between the inertial measurement unit coordinate system and the limb coordinate system is calculated as follows: in, is the initial moment of each joint A of the human body in each limb coordinate system i The quaternion of is the physical coordinate matrix in the i-th limb coordinate system at the initial moment, is the physical coordinate matrix in the i-th limb coordinate system at any time, is the initial moment at A i The physical coordinate matrix in the limb coordinate system, is the physical coordinate system matrix of the ith limb at any time, is the physical coordinate system matrix of the i-th limb at the initial moment.

9. The method for real-time rendering of human motion capture according to claim 1, characterized in that: The human body posture prediction model includes: an RNN prediction model, a node position model, an action prediction model, and a joint node speed prediction model; the collected data includes: acceleration data and displacement data; The collected data is input into the human body posture prediction model to obtain the rotation data of multiple joints and the global displacement, including: Filter the collected data to remove the noise in the data and obtain filtered data; Input the filtered data into the RNN prediction model to obtain leaf node position information; Input the leaf node position information and the calibration superposition vector into the node position model to obtain the position information of all joint nodes; Inputting the position information of all joint nodes and the filtered data into the motion prediction model to obtain the relative joint rotation data of the non-root joint point relative to the root joint point; The position information of all joint nodes and the probability of the human body contacting the ground are input into the joint node velocity prediction model to obtain the joint node velocity; The global displacement is calculated according to the joint node velocity.

10. A human motion capture real-time rendering system, characterized in that: include: A data acquisition module is used to initialize sensors in multiple inertial measurement units on a human body, wherein the multiple inertial measurement units on the human body include: a head inertial measurement unit, a waist inertial measurement unit, a left forearm inertial measurement unit, a right forearm inertial measurement unit, a left calf inertial measurement unit, and a right calf inertial measurement unit; the sensors of the inertial measurement units on the human body after initialization are initially aligned to obtain a posture relationship between the inertial measurement unit coordinate system and the limb coordinate system; based on the posture relationship between the inertial measurement unit coordinate system and the limb coordinate system, the sensors of the inertial measurement units on the human body after initial alignment are used to perform data acquisition; The central calculation module is used to input the collected data into the human posture prediction model to obtain the rotation data of multiple joints and the global displacement; The data rendering module is used to correspondingly capture the rotation data and global displacement of the multiple joint points to obtain a real-time motion capture rendering task.

Images