Human body motion capture system and method

Abstract

The embodiment of the invention provides a human motion capture system and method, the system comprises a virtual reality device and at least one inertial sensing module, and the inertial sensing module is in communication connection with the virtual reality device; wherein the inertial sensing module is used for acquiring first posture information of each key node, except the head, of a human body and sending the first posture information to the virtual reality equipment; and the virtual reality equipment is used for acquiring second posture information of the head of the human body, and determining a human body motion capture result according to the first posture information and the second posture information in combination with a pre-constructed posture prediction model. According to the system, the virtual reality equipment is used for replacing head inertia equipment and an external calculation module, so that the portability of the whole set of equipment is improved; and meanwhile, the six-degree-of-freedom function of the virtual reality equipment is used as reference data of the inertial sensing module, so that the precision of the inertial motion capture system is improved, and the problems that the original inertial motion capture precision is insufficient and inertial motion capture data drifts are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion capture, and in particular to a human motion capture system and method. Background Art

[0002] Motion capture technology is a technology used to track and record the motion of objects or human bodies. Motion capture technology is used in film and television animation, game entertainment, virtual reality, industrial simulation, life sciences, drone research, education, animation production, motion analysis, sports training or rehabilitation medicine. Motion capture technology usually uses a motion capture system to determine the posture and motion trajectory of an object or human body, and these motion data can be used to generate the posture of an object or human body in a virtual reality environment. At present, the commonly used motion capture systems are: motion capture systems based on optical devices, motion capture systems based on laser base stations, and motion capture systems based on inertial elements.

[0003] However, the above motion capture systems all have corresponding problems. For example, the motion capture system based on optical devices has strong dependence on the environment, data loss or errors caused by occlusion, high equipment cost and complex large-area layout; the motion capture system based on laser has strong line of sight dependence, resolution limitation, environmental limitation and high equipment cost; the motion capture system based on inertial devices has problems such as cumulative error, lack of external reference, magnetic interference, complex calibration process and motion limitation. Summary of the invention

[0004] The present invention provides a human motion capture system and method, which can improve the portability of the entire device and enhance the accuracy of the inertial motion capture system.

[0005] In a first aspect, an embodiment of the present invention provides a human motion capture system, comprising: a virtual reality device and at least one inertial sensor module, wherein the inertial sensor module is communicatively connected to the virtual reality device;

[0006] The inertial sensor module is used to obtain the first posture information of each key node of the human body except the head and send it to the virtual reality device;

[0007] The virtual reality device is used to obtain second posture information of the human head, and determine the human motion capture result based on the first posture information and the second posture information in combination with a pre-built posture prediction model.

[0008] In a second aspect, an embodiment of the present invention provides a human motion capture method, which is performed by the human motion capture system described in the embodiment of the first aspect, and the method includes:

[0009] The inertial sensing module of the human motion capture system is used to obtain first posture information of each key node of the human body except the head and send the first posture information to the virtual reality device;

[0010] The second posture information of the human head is obtained through the virtual reality device of the human motion capture system, and the human motion capture result is determined according to the first posture information and the second posture information in combination with a pre-built posture prediction model.

[0011] The embodiment of the present invention provides a human motion capture system and method, the system includes a virtual reality device and at least one inertial sensing module, the inertial sensing module is communicatively connected to the virtual reality device; wherein the inertial sensing module is used to obtain first posture information of each key node of the human body except the head and send it to the virtual reality device; the virtual reality device is used to obtain second posture information of the human head, and determine the human motion capture result according to the first posture information and the second posture information, combined with a pre-built posture prediction model. The above technical scheme uses a virtual reality device to replace the head inertial device and the external computing module, thereby improving the portability of the entire set of equipment; at the same time, the 6-degree-of-freedom function of the virtual reality device is used as the reference data of the inertial sensing module to improve the accuracy of the inertial motion capture system, thereby solving the problems of insufficient accuracy of the original inertial motion capture and drift of the inertial motion capture data.

[0012] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 It is a working schematic diagram of a motion capture system based on optical equipment in the prior art;

[0015] Figure 2 It is a working schematic diagram of a motion capture system based on a laser base station in the prior art;

[0016] Figure 3 It is a working schematic diagram of a motion capture system based on an inertial unit in the prior art;

[0017] Figure 4A schematic diagram of the structure of a human motion capture system provided in Embodiment 1 of the present invention;

[0018] Figure 5 A schematic diagram of the structure of another human motion capture system provided in Embodiment 1 of the present invention;

[0019] Figure 6 This is a structural example diagram of a human motion capture system in an application scenario provided by the first embodiment of the present invention;

[0020] Figure 7 A schematic diagram of the structure of another human motion capture system provided in Embodiment 1 of the present invention;

[0021] Figure 8 A schematic diagram of a human skeleton model in a human motion capture system provided in Embodiment 1 of the present invention;

[0022] Fig. 9 A schematic diagram of the structure of an initial prediction model in a human motion capture system provided in Embodiment 1 of the present invention;

[0023] Fig.10 A schematic flow chart of a method for capturing human motion provided in accordance with the second embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "original", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] At present, the commonly used motion capture systems are: motion capture systems based on optical devices, motion capture systems based on laser base stations, and motion capture systems based on inertial elements. Among them, the motion capture system based on optical devices is based on the principle of computer vision. Multiple high-speed cameras monitor and track target feature points from different angles, and combine skeleton solving algorithms to complete motion capture. In theory, for any point in space, as long as it can be seen by more than two cameras at the same time, the 3D position of the point in space at that moment can be determined. Figure 1 FIG. 1 is a schematic diagram of a motion capture system based on an optical device in the prior art. Figure 1 As shown, the motion capture system based on optical devices includes multiple external cameras, such as camera A, camera B, camera C and camera D, a wearable optical marker device configured on a person, and an image processing system (generally composed of a PC or a server).

[0027] The key working steps of the motion capture system based on optical devices are as follows: 11) Installation of cameras and markers. Multiple cameras are fixed at different locations in the scene. Optical markers are installed at key parts of the human body or object (such as joints and limb ends). These markers are usually fixed by sticking or strapping. 12) Camera calibration. Before starting to capture, the camera needs to be calibrated. The purpose of calibration is to determine the exact position and viewing angle of each camera in the capture space so as to accurately calculate the three-dimensional position of the marker. 13) Data acquisition. The captured object performs a series of actions as required, and multiple cameras simultaneously record the position of the markers in their respective fields of view. The two-dimensional image data captured by each camera is processed by software to calculate the exact position of the marker in three-dimensional space. 14) Data processing and analysis. The computer integrates the marker data captured by all cameras, and calculates the object's motion trajectory and posture through triangulation and kinematic algorithms. These data can be used for animation production, motion analysis, etc.

[0028] However, motion capture systems based on optical devices have the following problems: 21) Strong environmental dependence: The optical system relies on the field of view of the camera, and is therefore easily affected by factors such as ambient light, occlusion, and reflection. In complex or low-light environments, the accuracy and reliability of the system will decrease. 22) Occlusion problem: When the human body or other objects block the camera's line of sight, the optical system cannot correctly capture the action, resulting in data loss or errors. 23) High equipment cost: The optical system usually requires multiple high-precision cameras and markers, with high equipment and maintenance costs, and requires professional installation and calibration. 24) Complex layout over large areas: For large-scale motion capture, the optical system needs to deploy multiple cameras, which are more complicated to install and synchronize, and have certain space restrictions.

[0029] Among them, the laser base station-based motion capture system is a technology that uses laser emitters and sensors to track and record the movement of objects or human bodies. Unlike traditional optical motion capture systems, this system mainly relies on laser base stations (usually called "lighthouses" or "base stations") and sensors installed on the captured objects to determine their position and posture in three-dimensional space. Figure 2 FIG. 1 is a working diagram of a motion capture system based on a laser base station in the prior art. Figure 2 As shown, the motion capture system based on the laser base station includes a laser transmitting tower A and a laser transmitting tower B. The character is equipped with a laser receiver, which communicates with the edge server via wireless / Bluetooth.

[0030] The key steps of the motion capture system based on laser base stations are as follows: 31) Laser base stations are usually devices installed in a fixed position in the room or capture space. Each base station emits a rotating laser beam and spreads it into space. Usually at least two laser base stations are required to ensure sufficient coverage and accuracy. These base stations work in coordination with each other, and the laser beam emitted by each base station can be received by the sensor on the captured object. 32) Sensors are usually installed on the object or human body to be captured. These sensors can detect the light signals emitted by the laser base station and convert them into time data. By accurately measuring the time difference between the laser beam reaching the sensor, the position of the sensor relative to the base station can be calculated. 33) Time difference measurement and triangulation. The laser beams emitted by the laser base station form a time series in space. The sensor can calculate its distance from the base station by capturing these laser beams and recording their arrival time. Multiple base stations emit laser beams at the same time and are received by multiple sensors. These data can be used to deduce the precise position of the sensor in three-dimensional space through triangulation. 34) Data processing and posture derivation. The captured sensor position data is transmitted to the computer, and the computer calculates the posture and motion trajectory of the object or human body installed with the sensor through complex algorithms. This data is often used to generate user poses in virtual reality environments, or to record and analyze movements.

[0031] However, motion capture systems based on optical devices have the following problems: 41) Line of sight dependence: The laser device also needs to maintain line of sight with the target object. If it is blocked or interfered by reflective surfaces, the capture effect will be affected. 42) Resolution limitation: The resolution of laser scanning is limited. When capturing fast or complex movements, it may not provide enough details, resulting in inaccurate motion capture. 43) Environmental limitations: The laser system is highly dependent on the environment. In dusty or reflective environments, the laser beam may be scattered or interfered, affecting the accuracy of the data. 44) High equipment cost: High-precision laser scanners and related control systems are expensive and require regular maintenance, which increases the overall cost of the system.

[0032] Among them, the motion capture system based on inertial devices is a technology that uses an inertial measurement unit to record and analyze human motion. An inertial measurement unit is usually composed of an accelerometer, a gyroscope, and a magnetometer, and can measure the acceleration, angular velocity, and orientation of an object. Compared with traditional optical motion capture systems, systems based on inertial units do not require external cameras or markers, and are more portable and environmentally adaptable. Figure 3 FIG. 1 is a schematic diagram of the operation of a motion capture system based on an inertial unit in the prior art. Figure 3 As shown, the head node, wrist node, ankle node and waist node are equipped with inertial measurement units, and the sensed information is sent to the edge server via Bluetooth communication, and the edge server determines the human motion capture result.

[0033] The key steps of the motion capture system based on the laser base station are as follows: 51) Sensor layout: Inertial measurement units are usually fixed to different parts of the human body, such as arms, legs, torso, etc. These sensors need to be reasonably arranged according to the target area of ​​the movement to ensure that the motion data of each key part can be accurately captured. Calibration: Before starting the capture, the sensor needs to be calibrated. The calibration process usually includes the initial posture calibration and alignment of the sensor to ensure that the reference coordinate system of all sensors is consistent. 52) Data acquisition: Real-time acquisition: During the movement, the inertial measurement unit (IMU) sensor will continuously collect acceleration, angular velocity and magnetic field data for each axis. These data will be transmitted to the central processing unit (usually a computer or embedded system) in real time for processing. Data synchronization: The data of multiple sensors needs to be synchronized to ensure that the movement of all parts can be analyzed on the same time axis. 53) Data processing: Attitude estimation: By fusing the data of the accelerometer, gyroscope and magnetometer, the attitude of the sensor in space (i.e., the tilt angle, rotation angle, etc. of the sensor) can be calculated. Commonly used algorithms include complementary filtering, Kalman filtering, etc. Motion trajectory calculation: Based on the posture data of each sensor and combined with the human body model, the motion trajectory of each joint of the entire human body is calculated. Error correction: Since IMU is easily affected by cumulative errors (such as drift), additional algorithms are required for error correction to improve the accuracy of the motion trajectory. 54) Data output and application, visualization: The processed data can be visualized in a three-dimensional environment to generate skeletal animation or human motion model. Analysis and application: These motion data can be applied to fields such as motion analysis, sports training, rehabilitation medicine, virtual reality and game development.

[0034] However, motion capture systems based on inertial devices have the following problems: 61) Accumulated error (drift): Inertial measurement units are susceptible to accumulated errors, especially when used for a long time. This error will cause the motion trajectory to shift, affecting the accuracy of the data. 62) Lack of external reference: Inertial systems rely on measurement data from internal sensors and have no external reference points for calibration and correction. Therefore, absolute position data cannot be directly obtained, and only relative motion can be inferred. 63) Magnetic interference: The magnetometer in the system is easily interfered by external magnetic fields, especially in environments with dense metal or electromagnetic interference, which may lead to inaccurate attitude estimation. 64) Complex calibration process: In order to ensure accuracy, inertial devices require complex and frequent calibration, especially after environmental changes or device movement. The calibration process may affect the ease of use of the system. 65) Motion limitations: Although inertial systems can capture a wide range of movements, in extreme fast movements or rotational movements, the sensors may not be able to accurately track all details, resulting in motion capture distortion.

[0035] Therefore, a motion capture system is needed to solve the above problems.

[0036] Embodiment 1

[0037] Figure 4 This is a schematic diagram of the structure of a human motion capture system provided in Embodiment 1 of the present invention. The system is applicable to capturing human motions, and the system can be implemented in the form of hardware and / or software.

[0038] In order to more clearly describe the human motion capture system provided by the present invention, some words are explained below. Artificial intelligence model refers to a computer program or system used to perform specific tasks, such as language understanding, image recognition, predictive analysis, etc. These models are trained based on data to learn the patterns or logic required to perform tasks. Virtual reality equipment is a hardware tool that simulates a virtual environment through computer technology to immerse users. It is mainly composed of a head-mounted display and a controller. Among them, the head-mounted display is the core component of the virtual reality device, which contains one or two display screens to display images of the 3D virtual world. The head-mounted display is usually also equipped with built-in sensors, such as accelerometers, gyroscopes and magnetometers, for tracking the user's head movement, thereby updating the perspective in the virtual environment in real time. At the same time, it is also a high-performance computing platform that uses powerful computing power to render the virtual world and process various sensor data. An inertial sensing module is a device for measuring the acceleration, angular velocity and magnetic field strength of an object. The inertial sensing module is mainly composed of three sensors: accelerometers, gyroscopes and magnetometers, which can provide information on the motion state and direction of an object. Inertial sensing modules are widely used in various fields that require accurate measurement of motion states, including but not limited to the following: aerospace, consumer electronics, robotics, virtual reality and augmented reality.

[0039] like Figure 4 As shown, the human motion capture system provided in the first embodiment may specifically include: a virtual reality device 10 and at least one inertial sensor module 20, wherein the inertial sensor module 20 is communicatively connected with the virtual reality device 10;

[0040] The inertial sensor module 20 is used to obtain the first posture information of each key node of the human body except the head and send it to the virtual reality device 20;

[0041] The virtual reality device 10 is used to obtain second posture information of the human head, and determine the human motion capture result according to the first posture information and the second posture information in combination with a pre-built posture prediction model.

[0042] It should be noted that, different from the structure of the motion capture system based on inertial devices in the prior art, in this embodiment, a virtual reality device is used to replace the inertial unit of the head, and the edge computing function is migrated to the virtual reality device. The human motion capture system provided in this embodiment is mainly composed of a virtual reality device 10 and an inertial sensor module 20. In this embodiment, the number of inertial sensor modules 20 can be determined according to actual conditions, and the nodes on the human body that need to wear the inertial sensor modules 20 are selected. Preferably, each inertial sensor module 20 is worn on the waist, wrist and ankle of the human body, and the virtual reality device 10 is worn on the head of the human body.

[0043] Among them, the joint points other than the head that are used to clearly characterize human body movements are recorded as key nodes. The inertial sensor module 20 is worn at the key nodes of the human body other than the head to capture the displacement and rotation information of each key node, which is recorded as the first posture information. Its working principle is the same as the principle of the existing motion capture system based on inertial devices, which will not be repeated here. Exemplarily, the inertial sensor module 20 is worn on the human waist, wrists, and ankles to capture the rotational movement of the joints as the first posture information of each key node. After the first posture information of the key node is obtained through the inertial sensor module 20, the first posture information will be sent to the virtual reality device 10. Exemplarily, each inertial sensor module 20 transmits the first posture information to the virtual reality device 10 via Bluetooth transmission.

[0044] In this embodiment, the virtual reality device 10 integrates the function of obtaining head posture information and the calculation function of motion capture, which can realize the portability of the whole system. The virtual reality device 10 is integrated with a binocular camera and an inertial sensor module. Based on the perception information of the surrounding environment by the binocular camera and the inertial sensor module, based on the Simultaneous Localization And Mapping (SLAM) algorithm, the movement information such as the displacement and rotation information of the human head can be determined, which is recorded as the second posture information. It can be understood that compared with the inertial sensor module, the SLAM algorithm using the camera can provide accurate position and rotation information, and serve as an external precise position reference, thereby eliminating the accumulated error of the inertial sensor module.

[0045] In this embodiment, an artificial intelligence model is used to train a human body model that conforms to the principles of dynamics, and the constructed model is recorded as a posture prediction model. By inputting the first posture information of each key node into the posture prediction model, the position and rotation information of other nodes of the human body can be predicted, thereby obtaining the overall posture of the human body. After the human body posture is inferred by the posture prediction model, the human head movement information is combined with the accurate human head movement information measured by the SLAM algorithm, and the human body posture is corrected using the human head movement information. Based on the posture information of each node of the human body after correction, the timestamp is aligned, and a more accurate action sequence can be obtained as the human motion capture result.

[0046] The above technical solution uses virtual reality equipment to replace the head inertial equipment and the external computing module, thereby improving the portability of the entire equipment. At the same time, the 6-degree-of-freedom function of the virtual reality equipment is used as reference data for the inertial sensing module to improve the accuracy of the inertial motion capture system, thereby solving the problems of insufficient accuracy of the original inertial motion capture and drift of the inertial motion capture data.

[0047] Figure 5A schematic diagram of the structure of another human motion capture system provided in the first embodiment of the present invention is shown in FIG. Figure 5 As shown, as a first optional embodiment of the present invention, based on the above embodiment, this first optional embodiment can optimize the virtual reality device 10, including:

[0048] A head positioning module 11, used to determine second posture information of the human head according to a synchronous positioning and map building algorithm;

[0049] The posture reasoning module 12 is used to input the first posture information into the posture prediction model, obtain the posture information of each node other than the key node of the human body, and record the first posture information of the key node and the posture information of the other nodes as the posture information of each node of the human body;

[0050] The motion capture module 13 is used to correct the posture information of each node of each human body based on the second posture information to determine the human body motion capture result.

[0051] Specifically, the virtual reality device 10 is integrated with the function of obtaining the motion information of the head and the function of motion capture calculation. Based on this, the virtual reality device 10 includes a head positioning module 11, a posture reasoning module 12 and a motion capture module 13. Among them, the head positioning module 11 uses a binocular camera and an inertial sensor module to determine the motion information such as the displacement and rotation information of the human head as the second posture information of the human head through the SLAM algorithm. It can be understood that the entire environment scene will be modeled through the SLAM algorithm, the image data is obtained based on the binocular camera sensing the surrounding environment, the acceleration, angular velocity and magnetic field data are obtained based on the inertial sensor module, and the second posture information of the human head can be obtained based on the fusion processing of the above data.

[0052] It should be noted that, usually, accurate motion capture requires more than ten or even dozens of key node information of the human body. However, if multiple inertial sensing modules are used, the equipment cost is relatively high, so a posture prediction model based on sparse inertial data is used in this embodiment. Specifically, only the key node information such as the head, wrist, ankle, waist, etc. is used to predict the other joint information of the human body, and then according to the physical characteristics of the human body, accurate human body movements are finally predicted. Specifically, the posture reasoning module 12, based on the posture prediction model, predicts the posture information of each other node of the human body except the key node through the first posture information of each key node, and records the first posture information of the key node and the posture information of other nodes as the posture information of each node of the human body, thereby obtaining the overall posture of the human body.

[0053] Since the inertial sensing module has cumulative errors, and the position information in the second posture information of the human head is obtained based on external information as a reference, the motion capture module 13 is used to correct the predicted human posture information based on the second posture information of the human head, and obtain the corrected posture information of each node of the human body, so that the human body motion sequence can be obtained as the human body motion capture result.

[0054] The above technical solution specifies that the virtual reality device includes a head positioning module, a posture reasoning module and a motion capture module, and uses the SLAM algorithm to obtain accurate movement information of the human head. At the same time, based on the posture information of key nodes, the posture information of other nodes of the human body is predicted to obtain the overall posture of the human body. The movement information of the human head is then used as a standard to correct the posture information of each node of the human body, and more accurate posture information of each node of the human body is obtained, thereby eliminating the cumulative error of the inertial sensor module.

[0055] Further, based on the first optional embodiment, the head positioning module 11 may be optimized, specifically for:

[0056] a1) Acquire first perception data and second perception data of the environment in which the human body is located through a binocular camera and an inertial sensor module respectively.

[0057] Specifically, the surrounding environment is sensed by a binocular camera, and image data of the human body's environment is obtained as the first perception data. The human head and the surrounding environment are sensed by an inertial sensor module, and the acceleration, angular velocity and magnetic field data of the human head in each axis are obtained. By fusing the accelerometer, gyroscope and magnetic field data, the posture of the sensor in space (i.e., the sensor's tilt angle, rotation angle, etc.) can be calculated and recorded as the second perception data.

[0058] b1) Based on the synchronous positioning and map building algorithm, the first perception data and the second perception data are processed to determine the second posture information of the human head.

[0059] In this embodiment, a SLAM algorithm is used to perform positioning and map construction based on the first perception data and the second perception data, thereby obtaining movement information of the human head, which is recorded as the second posture information of the human head.

[0060] The above technical solution provides the system with the displacement and rotation information of the head through the SLAM algorithm. Compared with the inertial sensor, the SLAM algorithm using the camera can provide accurate position and rotation information, and serve as an external precise position reference to eliminate the accumulated errors of the inertial sensor.

[0061] Further, based on the first optional embodiment, the motion capture module 12 may be optimized, specifically for:

[0062] a3) Based on a set data fusion algorithm, the second posture information is fused with the posture information of each node of the human body to obtain corrected posture information of each node of the human body.

[0063] In this embodiment, since the inertial sensing module has cumulative errors and the second posture information of the human head is based on external information as a reference, the predicted human posture information is corrected based on the second posture information of the human head to obtain the corrected posture information of each node of the human body.

[0064] b3) According to the corrected posture information of each node of the human body, a human body motion sequence is determined as a human body motion capture result.

[0065] In this embodiment, timestamp alignment is performed based on the corrected human body posture information, so that a human body motion sequence can be obtained as a human body motion capture result.

[0066] For example, Figure 6 This is a structural example diagram of a human motion capture system in an application scenario provided by the first embodiment of the present invention, such as Figure 6 As shown, the posture information of the waist, wrists, and ankles is obtained through the inertial sensor module 20 and transmitted to the posture reasoning model of the virtual reality device to obtain the human posture information. Since the human body movements are continuously performed, the posture reasoning module 12 includes a motion tracking optimizer and a dynamic state updater. The head positioning module 11 obtains the posture information of the head through the SLAM algorithm of multi-sensor fusion of the left camera, the right camera and the IMU, and inputs the head posture information and the human body posture information into the motion capture module 13 for correction, and obtains the corrected overall posture information of the human body, thereby determining the human body motion capture result.

[0067] The above technical solution specifies the steps of the motion capture module, and uses the posture information of the human head to correct the human body posture to obtain more accurate human body movements. It improves the accuracy of the inertial motion capture system, makes up for the original inertial motion capture accuracy and drift problems after long-term use, and makes the inertial system better applied in scientific research, education, and the gaming industry; at the same time, the virtual reality device integrates the external computing device of the original system, improves the integration and ease of use of the inertial motion capture system, and is more suitable for configuration and use by ordinary users, expanding the user group and market.

[0068] Figure 7 A schematic diagram of the structure of another human motion capture system provided in the first embodiment of the present invention is shown in FIG. Figure 7 As shown, as a second optional embodiment of the present invention, based on the above embodiment, this second optional embodiment can optimize the system further including: a cloud server 30, the cloud server 30 includes:

[0069] A model building module 31 is used to build an initial prediction model based on the human body structure and human physical properties;

[0070] The model training module 32 is used to train the initial prediction model based on the posture data sample set to obtain a general prediction model.

[0071] The posture data sample set includes at least one set of posture data sample pairs, and the posture data sample pairs include posture sample data of key nodes of the human body and posture sample data of other nodes of the human body.

[0072] In this embodiment, the system also includes a cloud server 30, which can be considered as a model training platform. The cloud server 30 includes a model building module 31 and a model training module 32. Usually, accurate motion capture requires more than ten or even dozens of key human node information. If multiple inertial sensing modules are used, the equipment cost is high. Therefore, in this embodiment, a prediction model based on sparse inertial data is used. Specifically, only the node information of the wrist, ankle, waist, etc. is used to predict the information of other joints of the human body, and then according to the physical property constraints of the human body model, accurate human body movements are finally predicted.

[0073] In this embodiment, the model construction module 31 is used to construct a human skeleton model including a hierarchical joint chain of human bones and joints according to the human body structure, construct a human prediction model based on the human skeleton model, and construct a joint torque control model according to the physical properties of the human body, and construct an initial prediction model based on the above models.

[0074] The model training module 32 is used to input the posture sample data of the key nodes of the human body into the initial prediction model, obtain the prediction output results of other nodes of the human body, substitute the prediction output results of other nodes of the human body and the posture sample data of other nodes of the human body into the preset loss function, obtain the loss value, adjust the parameters of the initial prediction model based on the loss value, optimize the preset loss function by adjusting the hyperparameters, and obtain the trained prediction model, which is recorded as the general prediction model.

[0075] Further, based on the second optional embodiment, the model building module 31 may be optimized, specifically for:

[0076] a4) Based on the human body structure, a human skeleton model including a hierarchical joint chain of human bones and joints is constructed.

[0077] In this embodiment, the human skeleton model organizes bones and joints in a tree-like hierarchical relationship according to the human body structure, and there is a parent-child hierarchical relationship between bones. Except for the root bone, each bone has a unique parent bone. For example, Figure 8A schematic diagram of a human skeleton model in a human motion capture system provided in Embodiment 1 of the present invention, such as Figure 8 As shown in the figure, 0, 1, 2, 3, etc. represent different nodes. The movement of a bone can be regarded as its rotational movement relative to the parent bone. By simplifying the human body into a hierarchical joint chain composed of several joints and bones, the movement of the human body can be described by the translation and rotation of the root bone and the rotational movement of each joint.

[0078] b4) Based on the human skeleton model, a human body prediction model is constructed using the long short-term memory network as the neural network structure.

[0079] In this embodiment, the human body prediction model is based on the skeleton model and predicts the positions of all skeleton points through some key point information. First, the initial state of the skeleton model is preset, and then the rotation, speed, and acceleration information of the head, wrist, ankle, and waist skeleton points are collected, combined with the chain structure of the skeleton model and the long short-term memory network (LSTM) timing prediction structure in the neural network to predict the motion information of each skeleton node.

[0080] c4) Construct a joint torque control model based on the physical properties of the human body.

[0081] In this embodiment, the joint torque control model includes the physical properties of the human body, and after being controlled by the joint torque and having physical effects with the environment, human body movements that conform to real physical laws can be generated.

[0082] d4) Combining the human body prediction model and the joint torque control model as the initial prediction model.

[0083] In this embodiment, the human body prediction model is connected to the joint torque control model, and the joint torque control model is used as a physical constraint of the output result of the human body prediction model, so as to obtain an initial prediction model. Fig. 9 A schematic diagram of the structure of an initial prediction model in a human motion capture system provided in the first embodiment of the present invention is shown in FIG. Fig. 9 As shown in the figure, the human kinematics model is based on the skeleton model and predicts the posture of all skeleton points through the information of some key points. First, the initial state of the skeleton model is preset, and then the rotation, speed, and acceleration information of the head, wrist, ankle, and waist skeleton points are collected as input data. Combining the chain structure of the skeleton model and the LSTM timing prediction structure in the neural network, the recurrent neural network adopts the LSTM structure and is connected through a fully connected layer. The output of the human prediction model predicts the motion information of each skeleton node through the constraints of the joint torque control model as the output data. The joint torque control model includes a differential controller and a tracking optimizer.

[0084] The above technical solution concretizes the functions of the model construction module, specifically the human body model includes a human skeleton model, a human body prediction model and a joint torque control model, and realizes the construction of an initial prediction model based on sparse inertial data.

[0085] Further, based on this second optional embodiment, the model training module 32 can be optimized, specifically used to: train the initial prediction model based on the posture data sample set, adjust the hyperparameters, optimize the preset loss function, and obtain a general prediction model.

[0086] Among them, the preset loss function can be set according to actual conditions. In this embodiment, the posture data sample set includes at least one group of posture data sample pairs, and the posture data sample pairs include posture sample data of key nodes of the human body and posture sample data of other nodes of the human body. Specifically, the posture sample data of the key nodes of the human body are input into the initial prediction model to obtain the prediction output results of other nodes of the human body, and the prediction output results of other nodes of the human body and the posture sample data of other nodes of the human body are substituted into the preset loss function to obtain the loss value, and the parameters of the initial prediction model are adjusted based on the loss value. By adjusting the hyperparameters, the preset loss function is optimized to obtain a trained prediction model, which is recorded as a general prediction model. Since the cloud server GPU has powerful performance and is suitable for large-scale data processing and training, a public data set is used to train the basic model, and at the same time, the training is optimized by adjusting the hyperparameters, such as the learning rate, the training batch size, etc., and the loss function is optimized to finally obtain the basic model.

[0087] The above technical solution specifies the training steps of the general prediction model.

[0088] Continue to refer Figure 7 Further, based on the second optional embodiment, the virtual reality device 10 can be optimized to also include a model optimization module 14, which is specifically used for:

[0089] a5) Obtain a general prediction model.

[0090] The virtual reality device 10 also includes a model optimization module 14, which can be regarded as an edge server. The general prediction model is downloaded from the cloud server, and the basic model is trained on the edge using local user data through transfer learning to obtain a fine-tuned model. It is understandable that since motion capture requires high accuracy and each person's skeletal model is quite different, it is very difficult to train a general model with good generalization ability. Therefore, for users of different body shapes, the user data is collected and the general model is retrained to obtain accurate models for different users. Since only the general model is fine-tuned, the scale of training data is small, and training can be performed on a small edge server to reduce data transmission delay and improve real-time performance. In this embodiment, it can be considered that the small edge server is integrated into the virtual reality device 10, which is recorded as the model optimization module 14. This step is used to obtain a basic general prediction model.

[0091] b5) Adjust the parameters of the human skeleton model according to the user's body shape.

[0092] It is understandable that the parameters of the human skeleton model corresponding to different user's body shapes are also different. In this step, the parameters of the human skeleton model are adjusted to correspond to the user's body shape. The skeleton model data is adjusted according to the user's body shape, and the user's motion data over a period of time is collected.

[0093] c5) Based on the user's posture data sample set and the adjusted parameters of the human skeleton model, the general prediction model is trained to obtain an adjusted posture prediction model for the user.

[0094] In this embodiment, based on the posture data sample set of the user and the parameters of the adjusted human skeleton model, the basic universal prediction model is trained to obtain the trained universal prediction model as the adjusted posture prediction model for the user.

[0095] It should be noted that when users use the device, a large amount of data will be generated, including collected inertial sensor data, data generated by local model prediction, and real data of head posture measured by virtual reality equipment. These data will be regularly transmitted to the cloud server as Zeng Guang Data to retrain the existing general prediction model and further improve the performance of the basic model.

[0096] The above technical solution adds a model optimization function, realizes the training of corresponding posture prediction models for different user body types, and provides a basis for subsequent accurate prediction of human body posture.

[0097] Embodiment 2

[0098] Fig.10The flowchart of a method for capturing human motion provided by the second embodiment of the present invention is shown in FIG. The method is applicable to capturing human motion, and the method can be performed by a human motion capture system. The system can be implemented in the form of hardware and / or software. Fig.10 As shown, the human motion capture method provided in the second embodiment may specifically include:

[0099] S201. Obtain first posture information of key nodes of the human body except the head through an inertial sensing module of a human motion capture system and send the information to a virtual reality device.

[0100] It should be noted that, different from the structure of the motion capture system based on inertial devices in the prior art, in this embodiment, a virtual reality device is used to replace the inertial unit of the head, and the edge computing function is migrated to the virtual reality device. The human motion capture system provided in this embodiment is mainly composed of a virtual reality device and an inertial sensing module. In this embodiment, the number of inertial sensing modules can be determined according to actual conditions, and the nodes on the human body that need to wear inertial sensing modules are selected. Preferably, each inertial sensing module is worn on the waist, wrist and ankle of the human body, and the virtual reality device is worn on the human head.

[0101] Among them, the joint points other than the head that are used to clearly characterize human body movements are recorded as key nodes. The inertial sensor module is worn at the key nodes of the human body other than the head to capture the displacement and rotation information of each key node, which is recorded as the first posture information. Its working principle is the same as the principle of the existing motion capture system based on inertial devices, which will not be repeated here. Exemplarily, the inertial sensor module is worn on the human body's waist, wrists, and ankles to capture the rotational movement of the joints as the first posture information of each key node. After the first posture information of the key node is obtained through the inertial sensor module, the first posture information will be sent to the virtual reality device. Exemplarily, the first posture information is transmitted to the virtual reality device via Bluetooth transmission.

[0102] S202. Acquire second posture information of the human head through a virtual reality device of the human motion capture system, and determine a human motion capture result based on the first posture information and the second posture information in combination with a pre-built posture prediction model.

[0103] In this embodiment, the virtual reality device integrates the function of acquiring head posture information and the calculation function of motion capture, which can realize the portability of the whole system. The virtual reality device is integrated with a binocular camera and an inertial sensor module. Based on the perception information of the surrounding environment by the binocular camera and the inertial sensor module, based on the SLAM algorithm, the movement information such as the displacement and rotation information of the human head can be determined, which is recorded as the second posture information. It can be understood that compared with the inertial sensor module, the SLAM algorithm using the camera can provide accurate position and rotation information, and serve as an external precise position reference, thereby eliminating the accumulated error of the inertial sensor module.

[0104] In this embodiment, an artificial intelligence model is used to train a human body model that conforms to the principles of dynamics, and the constructed model is recorded as a posture prediction model. By inputting the first posture information of each key node into the posture prediction model, the position and rotation information of other nodes of the human body can be predicted, thereby obtaining the overall posture of the human body. After the human body posture is inferred by the posture prediction model, the human head movement information is combined with the accurate human head movement information measured by the SLAM algorithm, and the human body posture is corrected using the human head movement information. Based on the posture information of each node of the human body after correction, the timestamp is aligned, and a more accurate action sequence can be obtained as the human motion capture result.

[0105] The above technical solution uses virtual reality equipment to replace the head inertial equipment and the external computing module, thereby improving the portability of the entire equipment. At the same time, the 6-degree-of-freedom function of the virtual reality equipment is used as reference data for the inertial sensing module to improve the accuracy of the inertial motion capture system, thereby solving the problems of insufficient accuracy of the original inertial motion capture and drift of the inertial motion capture data.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A human motion capture system, characterized in that: include: A virtual reality device and at least one inertial sensor module, wherein the inertial sensor module is communicatively connected with the virtual reality device; wherein: The inertial sensor module is used to obtain the first posture information of each key node of the human body except the head and send it to the virtual reality device; The virtual reality device is used to obtain second posture information of the human head, and determine the human motion capture result based on the first posture information and the second posture information in combination with a pre-built posture prediction model.

2. The system according to claim 1, characterized in that The virtual reality device comprises: A head positioning module, used for determining second posture information of the human head according to a synchronous positioning and map building algorithm; A posture reasoning module, used for inputting the first posture information into the posture prediction model, obtaining the posture information of each node of the human body except the key node, and recording the first posture information of the key node and the posture information of the other nodes as the posture information of each node of the human body; The motion capture module is used to correct the posture information of each node of the human body based on the second posture information to determine the human body motion capture result.

3. The system according to claim 2, characterized in that The head positioning module is specifically used for: Acquire first perception data and second perception data of the environment in which the human body is located through a binocular camera and an inertial sensor module respectively; Based on the synchronous positioning and mapping algorithm, the first perception data and the second perception data are processed to determine the second posture information of the human head.

4. The system according to claim 2, characterized in that The motion capture module is specifically used for: Based on a set data fusion algorithm, the second posture information is fused with the posture information of each node of the human body to obtain corrected posture information of each node of the human body; According to the corrected posture information of each node of the human body, a human motion sequence is determined as the human motion capture result. 。 5. The system according to claim 1, characterized in that Also included is a cloud server, the cloud server comprising: A model building module is used to build an initial prediction model based on the human body structure and physical properties; A model training module is used to train the initial prediction model based on a posture data sample set to obtain a general prediction model, wherein the posture data sample set includes at least one group of posture data sample pairs, and the posture data sample pairs include posture sample data of key nodes of the human body and posture sample data of other nodes of the human body.

6. The system according to claim 5, characterized in that The model building module is specifically used for: According to the human body structure, a human skeleton model including a hierarchical joint chain of human bones and joints is constructed; Based on the human skeleton model, a human prediction model is constructed using a long short-term memory network as a neural network structure; Constructing a joint torque control model according to the physical properties of the human body; The human body prediction model and the joint torque control model are combined as the initial prediction model.

7. The system according to claim 5, characterized in that The model training module is specifically used for: The initial prediction model is trained based on the posture data sample set, and hyperparameters are adjusted to optimize a preset loss function to obtain the general prediction model.

8. The system according to claim 5, characterized in that The virtual reality device also includes a model optimization module, which is specifically used to: Acquire the general prediction model; Adjust the parameters of the human skeleton model according to the user's body shape; Based on the user's posture data sample set and the adjusted parameters of the human skeleton model, the general prediction model is trained to obtain an adjusted posture prediction model for the user.

9. The system according to claim 1, characterized in that The inertial sensor modules are worn on the waist, wrist and ankle of the human body, and the virtual reality device is worn on the head of the human body.

10. A method for capturing human motion, characterized in that: The method is performed by the human motion capture system according to any one of claims 1 to 9, and comprises: The inertial sensing module of the human motion capture system is used to obtain first posture information of each key node of the human body except the head and send the first posture information to the virtual reality device; The second posture information of the human head is obtained through the virtual reality device of the human motion capture system, and the human motion capture result is determined according to the first posture information and the second posture information in combination with a pre-built posture prediction model.

Images