Mark-free motion capture system integrating pressure sensor array and depth camera

Abstract

The invention discloses an unmarked motion capture system integrating a pressure sensor array and a depth camera, which belongs to the field of computer vision and comprises a depth camera module, a plantar pressure sensor array module, a data summarization module, a joint point calibration module, a three-dimensional force prediction module and a posture simulation module. The system collects and summarizes positions of human body joint points and pressure distribution data of planta pedis, extracts foot feature points, calibrates the human body joint points to obtain kinematics data, predicts ground reaction force in combination with planta pedis pressure distribution, and uses the calibrated kinematics data and predicted three-dimensional ground reaction force data to predict the three-dimensional ground reaction force. And carrying out human kinematics and dynamics simulation. The system provided by the invention can accurately position the spatial position of the human body joint point, accurately collect the pressure distribution and the ground reaction force in the human body movement process, and improve the accuracy of kinematics and dynamics analysis of the human body.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a markerless motion capture system integrating a pressure sensor array and a depth camera. Background Art

[0002] In recent years, with the advancement of technology, motion capture technology has continued to develop and has been widely used in film production, game development, virtual reality, sports training, medical rehabilitation and other fields. The traditional optical motion capture system uses multiple cameras to record multiple markers attached to the body to record the whole body's motion trajectory. This method is considered to have high accuracy and high recognition, but it is also limited by factors such as large space occupation, high cost, high technical requirements, and time consumption, making it difficult to popularize.

[0003] There are also methods of binding inertial sensors, gyroscopes, etc. to the user's body surface to achieve motion tracking. For example, the invention application with publication number CN113220119A discloses an inertial sensor motion capture device, which uses inertial sensors and somatosensory devices to obtain human body motion data and perform data fusion processing, and converts the fused and extracted motion data into human joint motion characteristics to drive the movement of virtual people. Although it solves the problems of complex data processing algorithms, large amount of calculations, and inability to work when the marker points are confused or blocked when measuring human motion in the current optical tracking system, this contact motion capture system may affect the user's normal movement, and the data quality is easily affected by soft tissue noise and shedding.

[0004] Contactless depth cameras are a potential alternative. They measure the distance information of objects based on the time delay, phase difference and other characteristics of light, thereby achieving three-dimensional modeling of scenes and human bodies. Their non-contact, low-cost and ease of use make them very suitable for motion capture in home scenes, providing possibilities for markerless motion capture technology in general scenes.

[0005] For example, the invention patent with publication number CN102842148A discloses a markerless motion capture and scene reconstruction method and device, which provides a human body model and a floor model, uses multiple depth cameras to shoot handheld, spatially transforms the depth map returned by the camera to obtain a three-dimensional point cloud set, extracts scene feature points from color images of adjacent frames of the same depth camera, and defines a joint optimization model based on information about surface grid points in the human body model, floor points in the floor model, and scene feature points in three-dimensional space, matches the three-dimensional point cloud points with the surface grid points, introduces the matching results into the joint optimization model for solution and transformation, and obtains motion capture results and scene reconstruction results. Although the present invention can perform motion capture and scene reconstruction more flexibly and conveniently by using a depth camera, the depth camera has a high degree of unreliability when capturing lower limb movements, and a correction method is required to improve the accuracy of the position of the lower limb joints.

[0006] In clinical motion analysis, the motion capture data of the depth camera can be used to calculate kinematic parameters such as joint angles, but it is difficult to evaluate the dynamics of the human body. By inputting the three-dimensional ground reaction force and motion data into the human skeleton model, the calculation and analysis of dynamic parameters can be achieved. However, the commonly used three-dimensional ground reaction force measurement instruments are expensive and bulky, and are not suitable for home scenes. The pressure sensor array is inexpensive, easy to carry, and more suitable for home scenes. However, the pressure sensor array can only obtain the vertical distribution information of mechanics, and the accuracy of human motion capture data is poor, which limits the application of motion capture data in the human skeleton model. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a markerless motion capture system that integrates a pressure sensor array and a depth camera to accurately locate the spatial position of human joints, accurately collect the pressure distribution and three-dimensional ground reaction force during human movement, and improve the accuracy of kinematic and dynamic analysis of the human body.

[0008] To achieve the above-mentioned purpose of the invention, an embodiment provides a markerless motion capture system integrating a pressure sensor array and a depth camera, comprising: The depth camera module is used to collect the depth point cloud data of the human body, estimate the position of the joint points based on the point cloud data and store it; The foot pressure sensor array module includes a flexible pressure sensor array and an acquisition circuit. The flexible pressure sensor acquires the pressure distribution of the foot of the human body, and the acquisition circuit converts the pressure distribution into a digital signal for transmission; A data aggregation module is used to obtain the position and pressure distribution of the joint points, unify the time coordinates and space coordinates of the joint points and pressure distribution, and obtain the aggregated joint points and pressure distribution; A joint point calibration module is used to extract foot feature points through the summarized pressure distribution, calibrate the foot joint points, ankle joint points and knee joint points in the human body joint points based on the foot feature points, and fuse the calibrated foot joint points, ankle joint points and knee joint points with the original summarized joint points to obtain the calibrated joint points; A three-dimensional force prediction module is used to extract the pressure characteristics and kinematic characteristics of the sole of the foot through the summarized pressure distribution and the calibrated joint points, and output the three-dimensional ground reaction force by fusing the pressure characteristics and kinematic characteristics; The posture simulation module is used to build a human skeleton model, integrate the calibrated joint points and three-dimensional ground reaction forces, obtain the kinematic and dynamic parameters of the human body, and complete the marker-free motion capture of the human body.

[0009] In one embodiment, the extraction of foot feature points through summarized pressure distribution includes: obtaining a pressure distribution image through summarized pressure distribution and constructing a footprint positioning template, changing the angle of the footprint positioning template and convolving it with the pressure distribution image, determining the foot position and foot direction in the pressure distribution image to obtain a footprint, and extracting foot feature points based on the pressure size of the footprint, wherein the foot feature points include metatarsophalangeal joints and calcaneal landmarks on the sole of the foot.

[0010] In one embodiment, the footprint positioning template includes a high reward area, a low reward area and a no reward area. The high reward area is the high pressure distribution area of ​​the calcaneus and metatarsals, the low reward area is the area of ​​the foot outside the high pressure distribution area of ​​the calcaneus and metatarsals, and the no reward area is the area outside the foot.

[0011] In one embodiment, the extraction of foot feature points based on pressure size of the footprint includes: the inner point of the row with the maximum pressure value in the front of the footprint is the first metatarsophalangeal joint point, the outer point of the row with the maximum pressure value in the front of the footprint is the fifth metatarsophalangeal joint point, and the point with the maximum pressure value in the back of the footprint is the calcaneal landmark point.

[0012] In one embodiment, the calibrating foot joints, ankle joints and knee joints among human body joints based on foot feature points includes: calibrating the foot joints using the midpoints of the first metatarsophalangeal joints and the fifth metatarsophalangeal joints to obtain calibrated foot joints; Use a point above the calcaneal landmark to calibrate the ankle joint point to obtain the calibrated ankle joint point; Based on the summarized joint points and the calibrated ankle joint points, the knee joint points are calibrated to obtain the calibrated knee joint points, wherein the summarized joint points include the hip joint points and the knee joint points.

[0013] In one embodiment, the calibration optimization objectives of the knee joint point calibration are thigh length consistency, calf length consistency and lower limb plane invariance: ,in, , , is the three-dimensional coordinate of the knee joint point before calibration, is the calf length, is the thigh length, is the ankle joint position after calibration, is the summarized hip joint position, represents the knee joint angle, The plane equation determined by the aggregated joint points and the calibrated ankle joint points, , and is the weight item.

[0014] In one embodiment, in the three-dimensional force prediction module, 19 joint points are converted into three graph structures, and the three graph structures are established according to the connection relationship between adjacent joint points, the connection relationship between joint points separated by one node, and the connection relationship between joint points separated by two nodes respectively; Among them, the node parameters in the graph structure include the position, velocity and acceleration of the joint points.

[0015] In one embodiment, based on the summarized pressure distribution, the pressure characteristics of the sole are extracted through a three-dimensional convolutional network; based on the calibrated joint points, the kinematic characteristics of the calibrated joint points are extracted through a graph convolutional network and a long short-term memory network.

[0016] In one embodiment, the human skeleton model is constructed, including 14 body segments and 18 joints, with the trunk as the root body segment, and other body segments are sequentially based on the body segment close to the trunk as the parent body segment to establish a chain structure.

[0017] In one embodiment, the obtaining of kinematic parameters and dynamic parameters of a human body includes: calculating kinematic parameters of a body segment center of mass according to kinematic parameters of joint points at both ends of each body segment as kinematic parameters of the human body; The joint forces and joint moments of each body segment are calculated based on the Newton-Euler equation as the dynamic parameters of the human body.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) The joint point calibration module is introduced to calibrate the captured human joint points through the plantar pressure distribution information, which can more accurately determine the relative position of the foot joint points and the lower limb joint points of the human body, thereby improving the accuracy of lower limb joint point recognition; (2) A three-dimensional force prediction module is introduced to convert the two-dimensional pressure distribution information into three-dimensional pressure distribution information based on the kinematic characteristics and pressure characteristics of the calibrated joint points, thereby improving the accuracy of human kinematics and dynamics simulation and analysis; (3) The markerless motion capture system provided by the present invention is suitable for sports training, rehabilitation therapy and virtual reality sports scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0020] Figure 1 It is a structural schematic diagram of the markerless motion capture system provided by the present invention.

[0021] Figure 2 It is a scenario of kinematics acquisition and pressure acquisition.

[0022] Figure 3 It is a schematic diagram of the relationship between joint point data and joint point connections.

[0023] Figure 4 It is a schematic diagram of pressure distribution data.

[0024] Figure 5 It is a schematic diagram of the footprint positioning template.

[0025] Figure 6 It is the structural diagram of the three-dimensional force prediction module.

[0026] Figure 7 This is a schematic diagram of the human skeleton model.

[0027] Figure 8 It is a flow chart for calculation of kinematic and dynamic parameters.

[0028] Fig. 9 It is a comparison chart of joint angles before and after lower limb kinematics calibration.

[0029] Fig.10 It is a comparison chart of the predicted value and actual value of the three-dimensional ground reaction force. DETAILED DESCRIPTION

[0030] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0031] In order to accurately locate the spatial position of human joints, accurately collect the pressure distribution and ground reaction force during human movement, and improve the accuracy of human kinematic and dynamic analysis, an embodiment provides a markerless motion capture system integrating a pressure sensor array and a depth camera, as shown in the attached figure. Figure 1 As shown, it includes: a depth camera module, a plantar pressure sensor array module, a data aggregation module, a joint point calibration module, a three-dimensional force prediction module and a posture simulation module.

[0032] In the embodiment, the depth camera module is used to collect the depth point cloud data of the human body, estimate the position of the joint points based on the point cloud data and store it. The plantar pressure sensor array module includes a flexible pressure sensor array and an acquisition circuit. The flexible pressure sensor collects the pressure distribution of the plantar of the human body, and the acquisition circuit converts the pressure distribution into a digital signal for transmission.

[0033] Specifically, as attached Figure 2 As shown, the depth camera is facing the front of the human body, within a range of 1.5 to 5 meters from the human body, and the human feet are located above the pressure sensor array. Test actions such as standing still and standing up are performed, and the kinematic data of the human joints and the plantar pressure distribution data are recorded on the host computer.

[0034] Next, the position and pressure distribution of the joint points are obtained through the data aggregation module, and the time coordinates and spatial coordinates of the joint points and pressure distribution are unified to obtain the aggregated joint points and pressure distribution.

[0035] Specifically, 19 key human joints are identified from the human depth point cloud data captured by the depth camera, and the connection relationship between each joint is determined, as shown in the attached figure. Figure 3 As shown, including: head joints, neck joints, pelvic joints, shoulder joints (left and right), elbow joints (left and right), wrist joints (left and right), hand joints (left and right), hip joints (left and right), knee joints (left and right), ankle joints (left and right), and foot joints (left and right).

[0036] The schematic diagram of pressure distribution obtained by the plantar pressure sensor array module is shown in the attached Figure 4 The plantar pressure sensor array contains 32×32 pressure sensor units, so the obtained pressure distribution image is 32×32 pixels, and the value of each pixel is between 0-255. The larger the pixel value, the greater the vertical pressure measured by the corresponding sensor unit, and the closer the color on the pressure distribution image is to red, and vice versa, the closer it is to blue.

[0037] In the embodiment, the joint point calibration module is used to extract foot feature points through the summarized pressure distribution, and the foot joint points, ankle joint points and knee joint points in the human body joint points are calibrated based on the foot feature points. The calibrated foot joint points, ankle joint points and knee joint points are merged with the original summarized joint points to obtain the calibrated joint points. Different from the prior art that usually relies on marker points or external sensors to obtain the joint point position, the present invention introduces a joint point calibration module to extract foot feature points from the plantar pressure distribution, and calibrate the position of the lower limb joint points. Through the plantar pressure distribution information, the relative position of the foot and the lower limb joint points can be determined more accurately, thereby compensating for the low recognition accuracy of the depth camera for the lower limb joints and meeting its use for kinematic and dynamic analysis.

[0038] Specifically, as attached Figure 5 As shown, the embodiment proposes a footprint positioning template, which is a 32×32 pixel image, including a high reward area, a low reward area, and a no reward area, and the corresponding values ​​are 3, 1, and 0. The high reward area defines the high pressure distribution area of ​​the calcaneus and metatarsals, the low reward area is the foot area outside the high pressure distribution area of ​​the calcaneus and metatarsals, and the no reward area is the area outside the foot.

[0039] The size of the high reward area corresponding to the calcaneal region is 4×3 pixels, and the size of the low reward area is 6×5 pixels. The size of the high reward area corresponding to the metatarsal region is 4×3 pixels, and the size of the low reward area is 4×6 pixels. The length of the arch of the low reward area corresponding to the arch region is pixels, and the width is 4 pixels. For each frame of pressure distribution image , change the angle of the footprint positioning template And with the pressure distribution image Convolution, get the angle of the footprint positioning template when the convolution value is maximum The length of the arch is , the position of the footprint in the footprint positioning template is determined by the position of the maximum convolution value. The specific formula is as follows: , in, The optimal angle for the footprint positioning template, For the optimal arch length, represents the convolution operation, The angle of the footprint positioning template and arch length Determined footprint positioning template, , .

[0040] Based on the footprint, the medial point with the maximum pressure value in the front of the footprint is determined by the pressure size as the first metatarsophalangeal joint point, the lateral point with the maximum pressure value in the front of the footprint is the fifth metatarsophalangeal joint point, and the point with the maximum pressure value in the back of the footprint is the calcaneal landmark point.

[0041] In the embodiment, for the left and right feet, the midpoints of the first metatarsophalangeal joint point (MTP1) and the fifth metatarsophalangeal joint point (MTP1) are used to calibrate the foot joint points to obtain the calibrated foot joint points: , , , in, , , is the three-dimensional coordinate of the calibrated foot joint point, , , is the three-dimensional coordinate of the first metatarsophalangeal joint, , , is the three-dimensional coordinate of the fifth metatarsophalangeal joint.

[0042] Use the point 0.0235 times the body height above the calcaneal landmark to replace the calibrated ankle point and obtain the calibrated ankle point: , , , in, , , is the three-dimensional coordinate of the ankle joint point after calibration, , , is the three-dimensional coordinate of the calcaneal landmark point.

[0043] The knee joint point is calibrated based on the hip joint point, knee joint point and calibrated ankle joint point to obtain the calibrated knee joint point. The calibration objectives are thigh length consistency, calf length consistency and lower limb plane invariance: ,in, , , is the three-dimensional coordinate of the knee joint point before calibration, is the calf length, is the thigh length, is the ankle joint position after calibration, is the summarized hip joint position, represents the knee joint angle, The plane equation determined by the aggregated joint points and the calibrated ankle joint points, , and is the weight item.

[0044] In the embodiment, the three-dimensional force prediction module is used to extract the pressure characteristics and kinematic characteristics of the sole of the foot through the summarized pressure distribution and the calibrated joint points, and output the three-dimensional ground reaction force by fusing the pressure characteristics and the kinematic characteristics. Different from the pressure sensor array in the prior art, which can usually only obtain the vertical pressure distribution and cannot obtain the complete three-dimensional component of the ground reaction force, the present invention converts the two-dimensional pressure distribution information into three-dimensional pressure information through machine learning model prediction, thereby improving the accuracy of human kinematics and dynamics simulation and analysis.

[0045] Specifically, the three-dimensional force prediction module proposed in the embodiment includes a graph convolutional network, a long short-term memory network, a three-dimensional convolutional network and a fusion layer, and its structure is as shown in the attached figure. Figure 6 As shown in the figure, the 19 joint points in each frame captured by the depth camera are converted into three graph structures. In the graph structure, the node parameters are the position, velocity and acceleration information of the joint points, and the edges are the connection relationships of the joint points. The connection relationships of the three graph structures are different, which are respectively established between adjacent nodes, nodes separated by one node, and nodes separated by two nodes according to the connection relationship of the joint points.

[0046] Based on the constructed graph structure, the calibrated joint points are used as input, and preliminary extraction is performed through the graph convolution network. The output of the graph convolution network is used as the input of the long short-term memory network to extract kinematic features again. Based on the constructed graph structure, the foot is divided by the pressure distribution image of continuous frames. Based on the result of the foot division, the right foot is flipped to obtain the foot features of the left and right feet, which are used as the input of the three-dimensional convolution network to extract the pressure features of the sole. The kinematic spatiotemporal features and pressure features are output through the fully connected layer, and then merged in the fusion layer. Finally, the predicted three-dimensional ground reaction force of both feet is output through the fully connected layer.

[0047] Finally, motion capture is performed through a posture simulation module, which is used to build a human skeleton model, fuse the calibrated joint points and three-dimensional ground reaction forces, obtain the kinematic parameters and dynamic parameters of the human body, and complete the marker-free motion capture of the human body.

[0048] Specifically, the human skeleton model includes 14 body segments and 18 joints. Figure 7 As shown in the figure. With the trunk as the root segment, other segments take the segments close to the trunk as the parent segments in turn to establish a chain structure. Figure 8As shown, the kinematic parameters of the center of mass of the body segment are calculated according to the motion parameters of the joint points at both ends of each body segment, and the kinematic parameters of the human body are obtained; the external forces on the outermost head, hands and feet are known, and the Newton-Euler equation is derived from the outside to the inside to calculate the joint force and joint moment of each body segment, and the dynamic parameters of the human body are obtained. Based on the kinematic parameters and dynamic parameters, the markerless motion capture of the human body is completed.

[0049] By collecting the deep point cloud data and plantar pressure distribution of the human body, a footprint positioning template is constructed, and the collected foot, ankle and hip joint positions are calibrated. The three-dimensional ground reaction force is predicted based on the calibrated lower limb kinematic information and plantar pressure distribution information through a deep learning model composed of a graph neural network, a long short-term memory network and a three-dimensional convolutional network. By establishing a human skeleton model and using the calibrated kinematic data and the predicted three-dimensional ground reaction force data, kinematic and dynamic simulations are performed. This can accurately locate the spatial position of the human joints, accurately collect the pressure distribution and ground reaction force during human movement, and improve the accuracy of the human kinematic and dynamic analysis.

[0050] In order to verify the effectiveness of the markerless motion capture system proposed in this invention that integrates the pressure sensor array and the depth camera, the standing-up action of three healthy subjects was tested, and the data were recorded by a high-precision Vicon motion capture system and a three-dimensional force platform as actual measurement values ​​for comparison.

[0051] As attached Fig. 9 The figure shows the joint angle comparison before and after the lower limb kinematic calibration. Through calibration, the difference between the joint angles based on the depth camera and the Vicon motion capture system was reduced, and the correlation coefficient of the hip joint angle increased from 0.60 to 0.95, the correlation coefficient of the knee joint angle increased from 0.95 to 0.99, and the correlation coefficient of the ankle joint angle increased from 0.10 to 0.85.

[0052] As attached Fig.10 The figure shows the comparison of the predicted and actual values ​​of the three-dimensional ground reaction force. In the vertical direction, the correlation coefficients between the predicted and actual values ​​of the ground reaction force of the left and right feet were 0.82 and 0.81, respectively; in the anterior-posterior direction, the correlation coefficients between the predicted and actual values ​​of the ground reaction force of the left and right feet were 0.98 and 0.97, respectively. The results show that the predicted three-dimensional ground reaction force and the actual value show high consistency in the vertical direction and the anterior-posterior direction, that is, the main direction of movement.

[0053] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A markerless motion capture system integrating a pressure sensor array and a depth camera, characterized in that: include: The depth camera module is used to collect the depth point cloud data of the human body, estimate the position of the joint points based on the point cloud data and store it; The foot pressure sensor array module includes a flexible pressure sensor array and an acquisition circuit. The flexible pressure sensor acquires the pressure distribution of the foot of the human body, and the acquisition circuit converts the pressure distribution into a digital signal for transmission; A data aggregation module is used to obtain the position and pressure distribution of the joint points, unify the time coordinates and space coordinates of the joint points and pressure distribution, and obtain the aggregated joint points and pressure distribution; A joint point calibration module is used to extract foot feature points through the summarized pressure distribution, calibrate the foot joint points, ankle joint points and knee joint points in the human body joint points based on the foot feature points, and fuse the calibrated foot joint points, ankle joint points and knee joint points with the original summarized joint points to obtain the calibrated joint points; A three-dimensional force prediction module is used to extract the pressure characteristics and kinematic characteristics of the sole of the foot through the summarized pressure distribution and the calibrated joint points, and output the three-dimensional ground reaction force by fusing the pressure characteristics and kinematic characteristics; The posture simulation module is used to build a human skeleton model, integrate the calibrated joint points and three-dimensional ground reaction forces, obtain the kinematic and dynamic parameters of the human body, and complete the marker-free motion capture of the human body.

2. The markerless motion capture system according to claim 1, characterized in that: The method of extracting foot feature points through summarized pressure distribution includes: obtaining a pressure distribution image through summarized pressure distribution and constructing a footprint positioning template, changing the angle of the footprint positioning template and convolving it with the pressure distribution image, determining the foot position and foot direction in the pressure distribution image to obtain a footprint, and extracting foot feature points based on the pressure size of the footprint, wherein the foot feature points include the metatarsophalangeal joint points and calcaneal landmark points on the sole of the foot.

3. The markerless motion capture system according to claim 2, characterized in that: The footprint positioning template includes a high reward area, a low reward area and a no reward area. The high reward area is the high pressure distribution area of ​​the calcaneus and metatarsals, the low reward area is the foot area outside the high pressure distribution area of ​​the calcaneus and metatarsals, and the no reward area is the area outside the foot.

4. The markerless motion capture system according to claim 2, characterized in that: The method of extracting foot feature points based on pressure size of the footprint includes: the inner point of the row with the maximum pressure value in the front of the footprint is the first metatarsophalangeal joint point, the outer point of the row with the maximum pressure value in the front of the footprint is the fifth metatarsophalangeal joint point, and the point with the maximum pressure value in the back of the footprint is the calcaneal landmark point.

5. The markerless motion capture system according to claim 4, characterized in that: The method of calibrating the foot joints, ankle joints and knee joints of the human body joints based on the foot feature points includes: calibrating the foot joints using the midpoints of the first metatarsophalangeal joints and the fifth metatarsophalangeal joints to obtain calibrated foot joints; Use a point above the calcaneal landmark to calibrate the ankle joint point to obtain the calibrated ankle joint point; Based on the summarized joint points and the calibrated ankle joint points, the knee joint points are calibrated to obtain the calibrated knee joint points, wherein the summarized joint points include the hip joint points and the knee joint points.

6. The markerless motion capture system according to claim 5, characterized in that: The calibration optimization objectives of the knee joint point calibration are thigh length consistency, calf length consistency and lower limb plane invariance: ， in, , , is the three-dimensional coordinate of the knee joint point before calibration, is the calf length, is the thigh length, is the ankle joint position after calibration, is the summarized hip joint position, represents the knee joint angle, The plane equation determined by the aggregated joint points and the calibrated ankle joint points, , and is the weight item.

7. The markerless motion capture system according to claim 1, characterized in that: In the three-dimensional force prediction module, 19 joints are converted into three graph structures, which are established based on the connection relationship between adjacent joints, the connection relationship between joints separated by one node, and the connection relationship between joints separated by two nodes. Among them, the node parameters in the graph structure include the position, velocity and acceleration of the joint points.

8. The markerless motion capture system according to claim 1, characterized in that: Based on the summarized pressure distribution, the pressure characteristics of the sole of the foot are extracted through a three-dimensional convolutional network; Based on the calibrated joint points, the kinematic features of the calibrated joint points are extracted through graph convolutional networks and long short-term memory networks.

9. The markerless motion capture system according to claim 1, characterized in that: The human skeleton model comprises 14 body segments and 18 joints, with the trunk as the root body segment, and other body segments are sequentially based on the body segments close to the trunk as the parent body segments to establish a chain structure.

10. The markerless motion capture system according to claim 9, characterized in that: The obtaining of the kinematic parameters and dynamic parameters of the human body includes: calculating the kinematic parameters of the centroid of the body segment according to the kinematic parameters of the joint points at both ends of each body segment as the kinematic parameters of the human body; The joint forces and joint moments of each body segment are calculated based on the Newton-Euler equation as the dynamic parameters of the human body.

Images