A 3D human pose estimation method based on multi-modal information

Abstract

The present invention discloses a method for estimating three-dimensional human posture based on multimodal information. The method is designed to simulate the force conditions between the human body and the clothing when the human body is wearing clothing, and to record this data. In addition, a method for obtaining the visual true value of human posture that is adapted to pressure sensor data is designed. When a person wears a pressure data acquisition device, they perform actions under the video data acquisition system designed by the present invention, and the three-dimensional human posture true value is obtained through the video. Finally, the pressure data and the three-dimensional human posture true value are multimodally aligned to generate a multimodal data set, which is input into the three-dimensional human posture estimation network designed by the present invention to train the model. This method has a long-term application prospect, and the use of pressure data between the human body and clothing for human posture estimation has great development potential.

Description

Technical Field

[0001] This paper relates to multimodal human posture estimation technology. It simulates the pressure of people wearing clothing and proposes a method for acquiring simulated pressure data. Due to the limitations of pressure data, this paper also proposes a method for acquiring and optimizing the true value of human posture suitable for estimation using pressure sensors. Finally, a human posture estimation model using pressure data as input is proposed. Based on the theoretical foundation of multimodal human posture estimation, this paper designs a method for aligning pressure data with the true value of human posture, and ultimately designs an efficient network based on LSTM. Background Art

[0002] Human pose estimation has widespread applications in many fields, particularly in VR gaming, human motion recognition, and human anomaly analysis. In VR gaming, wearable sensors and cameras can be used to estimate human poses and display them in-game, immersing users in the experience. Human pose recognition can also be applied to social security, such as identifying fallen passengers on buses and identifying road violations. In short, human pose estimation can be used across various industries as a solution to numerous problems.

[0003] Currently, most research on human pose estimation is still based on visual data. While visual human pose estimation has reached a high level, it still faces the following common challenges: First, human pose estimation based on visual data is affected by the environment and is difficult to perform in the dark; second, human pose estimation based on visual data is easily affected by occlusion, which affects its accuracy; third, human pose estimation based on visual data is difficult to implement in daily life, as cameras are not available everywhere; and finally, human pose estimation based on visual data often requires high-quality images as a foundation, which is not lightweight. However, it is feasible to use visual data to obtain the true value of human pose in the presence of minimal occlusion, because in this case, the human pose estimated by visual methods is generally higher than that obtained by non-visual methods.

[0004] Due to these issues with visual human pose estimation, a growing number of non-visual human pose estimation methods are emerging. Thanks to the rapid advancement of sensor technology, the accuracy of various sensors is constantly improving. Some non-visual human pose estimation methods that rely solely on sensor data have achieved prediction accuracy comparable to visual methods. Non-visual methods can effectively address some of the issues encountered by visual methods. First, sensor data is not subject to occlusion and is not significantly affected by environmental factors. Furthermore, sensor data is much lighter than images. Summary of the Invention

[0005] A three-dimensional human body posture estimation method based on multimodal information comprises the following steps:

[0006] S1 obtains pressure data of the human body when wearing clothes;

[0007] Deploy a thin film pressure sensor on the surface of a human body, and set a protective gear outside the thin film pressure sensor;

[0008] The pressure growth rate is represented by the ratio of the current pressure value and the initial pressure value measured by the thin film pressure sensor as pressure data; the initial pressure value is the pressure value measured by the thin film pressure sensor when the human body is standing still after the deployment is completed.

[0009] S2 obtains visual data of human posture; two parallel cameras are used to collect video data of the human body where the thin film pressure sensor is deployed, and after obtaining the two-dimensional posture in the video data using the Openpose algorithm, the triangulation method is used to perform dimensionality upgrade to obtain the three-dimensional posture;

[0010] Optimizing the three-dimensional posture based on pressure data

[0011] S3. Establish a data set, align the pressure data and the visual posture true value according to time, and generate a multimodal information data set;

[0012] S4 constructs a neural network and trains the neural network using the coordinates corresponding to the three-dimensional posture as supervision values

[0013] S5 uses the trained network to perform 3D human posture estimation based on pressure data and the height and weight of the human body.

[0014] Preferably, in S1, the deployment method of the thin film pressure sensor is: one side of the thin film pressure sensor is attached to the surface of the human body, and an iron sheet is provided on the other side of the thin film pressure sensor.

[0015] Preferably, the thin film pressure sensors are attached to the surface of the human body at the following positions: one on each side 2 cm below the elbow, four on each side 10 cm above the elbow of each arm when the arms are placed naturally, one on each side 2 cm below the knee, and four on each side 15 cm above the knee of each thigh when standing still.

[0016] Preferably, said S2 comprises the following steps:

[0017] Step 1: Place the two cameras in parallel and align them with the shooting area, and perform stereo calibration on the two cameras using a chessboard calibration method;

[0018] Step 2: The two cameras simultaneously record video data of the thin film pressure sensor deployed on the human body following the human body to complete a preset operation process, and respectively record the time when each frame in the two video data is obtained.

[0019] Step 3: Use the Openpose algorithm to process the two videos separately, obtain the 2D human pose estimation results in each frame, and intercept the 3D coordinates of the first 15 joints output by the Openpose algorithm. The 15 joints are, in order: nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, mid-hip, left hip, left knee, left ankle, right hip, right knee, and right ankle.

[0020] Step 4: Based on the stereo calibration parameters of the two cameras, the two-dimensional human posture estimation result of each frame is upgraded to three dimensions through the triangulation method to obtain a three-dimensional posture.

[0021] Step 5: Set the length of each limb segment of all the obtained three-dimensional postures to a fixed value, and set the three-dimensional coordinate origin of all the three-dimensional postures to the coordinates of a certain joint point, so that the coordinates of all other joint points represent their relative positions to the joint point that serves as the three-dimensional coordinate origin.

[0022] Preferably, in S4, the structure of the neural network includes a linear layer, an LSTM layer, and a linear layer in order, with a ReLU activation and a batch normalization performed between every two layers; the loss of the network is defined as the L2 distance between the network output and the true value; and the training is stopped until the estimated error of the network converges.

[0023] Preferably, in said S3, in the multimodal information data set, one out of every five data is extracted as a test set, and the remaining four are used as a training set.

[0024] Preferably, in step five of step S2, the joint point serving as the origin of the three-dimensional coordinates is set to be the second node of the human body joint point representing the joint point at the neck.

[0025] This invention simulates the pressure data generated between a person and clothing and uses it in conjunction with human posture data derived from video data to estimate human posture. In daily life, due to muscle stress or the pull of clothing, the body and clothing exert a certain amount of pressure on each other. However, since it is impossible to estimate the specific position of human joints solely based on pressure data, and the specific length of limbs cannot be determined using pressure data, estimating human posture based on pressure has certain limitations. This requires changes in the acquisition and processing of visual truth values to be suitable for estimating human posture using pressure data.

[0026] In the invention, pressure data is obtained through a hardware system designed by us, and we have designed a video-based true value acquisition and optimization method suitable for estimating human posture using pressure sensor data. We then conduct multimodal human posture estimation training using data from both video and pressure modalities, and finally achieve the effect of estimating human posture directly based on pressure data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of pressure sensor deployment and human joints.

[0028] Figure 2 Schematic diagram of the process of obtaining the initial 3D pose truth value.

[0029] Figure 3 Schematic diagram of the deep learning network structure designed for this invention.

[0030] Figure 4 This is a schematic diagram of the estimated effect of actual use of the present invention. DETAILED DESCRIPTION

[0031] The present invention simulates the situation of a person wearing clothing, designs a method for collecting pressure data and obtaining the true value of human posture in the form of video, and a network structure that mainly uses pressure data as input. The present invention will be described in detail below.

[0032] (1) First, we will introduce in detail how to obtain the pressure of the simulated person wearing clothes. The specific process is as follows:

[0033] Step 1: Install the thin film pressure sensor and stick it on an iron sheet.

[0034] Step 2: Fix the installed thin film pressure sensor with the sensor side facing the body directly with tape on the key positions on the human body surface. The specific key positions we designed are: one on each side 2cm below the elbow, four on each side 10cm above the elbow of each arm when the arm is naturally placed, one on each side 2cm below the knee, and four on each side 15cm above the knee of each thigh when standing still. The approximate positions of the sensors are as follows: Figure 1 As shown in the circled area.

[0035] Step 3: Connect all the thin film pressure sensors of the upper body and all the thin film pressure sensors of the lower body to two FSR thin film pressure sensor controller test boards respectively, and the FSR thin film pressure sensor controller test boards are connected to the computer.

[0036] Step 4: Open the host computers of two FSR film pressure sensor controller test boards on the computer to record the pressure data of the upper body and the pressure data of the lower body respectively.

[0037] Step 5. Tie the corresponding protective gear and Velcro on the outside of the sensor to allow the sensor to reach a certain initial value.

[0038] Step 6: The final recorded pressure data is the ratio of the current pressure value to the initial pressure value.

[0039] (2) Figure 1 As shown in the figure, the specific process of the method for obtaining the true value of human posture vision adapted to pressure sensor data is as follows: Figure 2 As shown:

[0040] Step 1: Deploy two cameras in parallel and align them with the shooting area, then use the chessboard calibration method to perform stereo calibration on the two cameras.

[0041] Step 2: The experimenter wears the pressure sensor data acquisition device described above and performs a set of actions designed by us in the shooting area. The two cameras will record the video separately and record the time of each frame acquisition.

[0042] Step 3: Use Openpose to process the two videos separately and obtain the two-dimensional human posture estimation results in each frame. The present invention intercepts the first 15 joints of Openpose, which include nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, middle hip, left hip, left knee, left ankle, right hip, right knee, and right ankle in the order of the joints in the data. The specific joint positions are as follows: Figure 1 As shown by the black solid circle in .

[0043] Step 4: Use the previously obtained stereo calibration parameters to directly upgrade the two-dimensional human posture estimation results of each frame to three dimensions through the Triangulation method.

[0044] Step 5. Since pressure data cannot be used to estimate the specific position of human joints and limb lengths, the present invention fixes the limb lengths of all obtained true values of human postures, and sets the origin of the three-dimensional coordinates of all three-dimensional human postures to the coordinates of a certain joint point of the human body, that is, the coordinates of all other joint points are their relative positions to the origin joint. The lengths of the limbs of each segment of the obtained initial three-dimensional human posture are adjusted to fixed values, specifically: the distance from the nose to the neck is 20cm, the distance from the neck to the shoulder is 23cm, the distance from the shoulder to the elbow is 28cm, the distance from the elbow to the wrist is 28cm, the distance from the neck to the middle buttock is 56cm, the distance from the middle buttock to the left and right buttocks is 17.5cm respectively, the distance from the buttocks to the knees is 40cm, and the distance from the knees to the ankles is 40cm. And the origin of the three-dimensional coordinates of all three-dimensional human postures is set to the second node of the human joint point (the joint point at the neck).

[0045] (3) Next, we will introduce in detail a 3D human pose estimation network based on multimodal information, which uses pressure sensor data as input and visual information as the ground truth:

[0046] Step 1: When collecting data, the time when each piece of pressure data and video data was generated was recorded. The posture truth value aligns the previously collected pressure data and visual posture truth value according to time to generate a multimodal information dataset.

[0047] Step 2: The network structure used is as follows Figure 3 As shown in the figure, the sequence includes linear layer, LSTM layer, and linear layer, with a ReLU activation and batch normalization between every two layers. All collected data are integrated, and one out of every five is selected as the test set, and the remaining four are used as the training set.

[0048] Step 3: Use the training set data as input for training. The network loss is defined as the L2 distance between the network output and the true value. Training is stopped until the network's estimated error converges.

[0049] Step 4: Input the test set data into the trained network to view the results.

[0050] The specific implementation is as follows: First, configure the pressure sensors. Each pressure sensor is attached to a metal sheet. Then, use tape to attach them to the skin at pre-designed locations, with the pressure sensor facing the body. Then, connect the upper and lower body pressure sensors to two FSR thin-film pressure sensor controller test boards, allowing the pressure values of each pressure sensor to be viewed directly on a computer. Then, attach Velcro and protective gear to the outside of the sensors and adjust them to ensure that each sensor has a specific initial value. Then, perform exercises while wearing the pressure sensors in front of two calibrated cameras. In our implementation, we designed the following upper body movements: hands down, hands raised, arms on hips, chest out; and lower body movements: squat, left lunge, right lunge, leg splits, and standing still. A total of 20 upper and lower body movements are combined in pairs. After performing the exercises, the pressure data is converted into a ratio of current pressure to initial pressure, and the video data is then processed. The video data is input into the OpenPose algorithm for processing to obtain a two-dimensional posture, and then the two-dimensional posture is upgraded to a three-dimensional posture using the Triangulation method. The obtained three-dimensional posture and pressure data are matched by time, and each matched three-dimensional posture is optimized as described above. In practice, we collected data from three people in total. After putting all the data together, we extracted one out of every five data as a test set, and the remaining four were used as a training set. The training set data method is used to train the network described above. The estimated results of the trained network with the test set data as input are as follows Figure 4shown.

Claims

1. A three-dimensional human posture estimation method based on multimodal information, characterized in that: The following steps are involved: S1 obtains pressure data of the human body when wearing clothes; Deploy a thin film pressure sensor on the surface of a human body, and set a protective gear outside the thin film pressure sensor; The pressure growth rate is represented by the ratio of the current pressure value and the initial pressure value measured by the thin film pressure sensor as the pressure data; the initial pressure value is the pressure value measured by the thin film pressure sensor when the human body is standing still after the thin film pressure sensor is deployed; S2 obtains the true value of the visual posture; two parallel cameras are used to collect video data of the human body where the thin film pressure sensor is deployed, and after obtaining the two-dimensional posture in the video data using the Openpose algorithm, the triangulation method is used to perform dimensionality upgrade to obtain the three-dimensional posture; S3 establishes a data set, aligns the pressure data and the true value of the visual posture according to time, and generates a multimodal information data set; in S3, in the multimodal information data set, one out of every five data is extracted as a test set, and the remaining four are used as a training set; S4 builds a neural network and uses the training set data as input for training. The network loss is defined as the L2 distance between the network output and the true value of the visual pose. S5 inputs the test set data into the trained network to perform 3D human pose estimation.

2. A three-dimensional human posture estimation method based on multimodal information according to claim 1, characterized in that: In the S1, The deployment method of the thin film pressure sensor is as follows: one side of the thin film pressure sensor is attached to the surface of the human body, and an iron sheet is provided on the other side of the thin film pressure sensor.

3. The method for estimating a three-dimensional human body posture based on multimodal information according to claim 2, wherein: The thin film pressure sensors are attached to the human body surface at the following positions: one on each side 2 cm below the elbow, four on each side 10 cm above the elbow on each arm when the arms are placed naturally, one on each side 2 cm below the knee, and four on each side 15 cm above the knee on each thigh when standing still.

4. The method for estimating a three-dimensional human body posture based on multimodal information according to claim 1, wherein: Said S2 comprises the following steps, Step 1: Place the two cameras in parallel and align them with the shooting area, and perform stereo calibration on the two cameras using a chessboard calibration method; Step 2: The two cameras simultaneously record video data of the thin film pressure sensor deployed on the human body following the human body to complete a preset operation process, and respectively record the time when each frame of the two video data is acquired; Step 3: Use the Openpose algorithm to process the two videos separately, obtain the 2D human pose estimation results in each frame, and intercept the 3D coordinates of the first 15 joints output by the Openpose algorithm. The 15 joints are, in order: nose, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, mid-hip, left hip, left knee, left ankle, right hip, right knee, and right ankle. Step 4: Based on the stereo calibration parameters of the two cameras, the two-dimensional human pose estimation result of each frame is upgraded to three dimensions through the triangulation method to obtain the three-dimensional pose; Step 5: Set the length of each limb segment of all the obtained three-dimensional postures to a fixed value, and set the three-dimensional coordinate origin of all the three-dimensional postures to the coordinates of a certain joint point, so that the coordinates of all other joint points represent their relative positions to the joint point that serves as the three-dimensional coordinate origin.

5. The method for estimating a three-dimensional human body posture based on multimodal information according to claim 4, wherein: In S4, the structure of the neural network includes a linear layer, an LSTM layer, and a linear layer in order, and a ReLU activation and a batch normalization are performed between every two layers.

6. The method for estimating a three-dimensional human body posture based on multimodal information according to claim 4, wherein: In step five of step S2, the joint point serving as the origin of the three-dimensional coordinates is set as the second node of the human body joint point representing the joint point at the neck.

Images