Inertial body motion capture method and device based on non-inertial system dynamics modeling

Abstract

This application proposes an inertial human motion capture method and apparatus based on non-inertial frame dynamics modeling. The method includes: acquiring raw IMU measurements at specific joints of the human body and preprocessing the raw IMU measurements; converting the preprocessed IMU measurements from a global coordinate system to a root joint relative coordinate system; estimating the inertial acceleration of the human body using an autoregressive estimator; compensating the converted IMU measurements based on the estimated inertial acceleration; estimating the local posture and global motion of the human body using the compensated IMU measurements; and visualizing the posture and position on a human body model based on the estimated local posture and global motion. This application corrects the IMU measurement signals in the non-inertial coordinate system by modeling inertial forces, ensuring that Newton's laws of motion are satisfied in the non-inertial coordinate system, and achieving accurate modeling of the relationship between acceleration and posture, thereby improving the precision and accuracy of human motion capture.

Description

Technical Field

[0001] This application relates to the field of human motion capture, and in particular to an inertial human motion capture method and device based on non-inertial frame dynamics modeling. Background Technology

[0002] In the field of human motion capture, while traditional visual motion capture technology excels in accuracy and precision, it still suffers from some inherent drawbacks. First, visual motion capture technology is susceptible to occlusion; when there are obstructions between or around the subject, the camera cannot accurately capture the motion information of all key points, leading to inaccurate pose estimation. Second, the capture range of traditional visual motion capture systems is limited by the camera's field of view and placement, requiring the subject to move within the camera's field of view, thus restricting the freedom of motion capture and its application scope. Furthermore, visual motion capture systems are expensive, requiring numerous cameras and computing resources, which places high demands on equipment for some real-time motion analysis applications.

[0003] To overcome these shortcomings of visual motion capture technology, inertial sensor-based motion capture technology has been adopted. Inertial sensors can accurately measure the acceleration and angle of human movement without being limited by the field of view, thus enabling more free and flexible motion capture. Compared to visual motion capture systems, inertial sensor technology is less expensive, the equipment is more portable and easier to wear, and it is suitable for various environments and scenarios. Furthermore, inertial sensors are unaffected by occlusion and can accurately capture the motion information of the measured object, making attitude estimation more accurate and reliable.

[0004] Based on these advantages, inertial sensors are widely used in the field of human motion capture, such as virtual reality, motion analysis, and medical rehabilitation. However, traditional inertial motion capture technology has a common problem: when estimating human posture, the root node coordinate system of the human body is usually treated as an inertial coordinate system, ignoring the potential impact of non-inertial effects on posture estimation. This approach may lead to inaccuracies and instabilities in the motion capture results.

[0005] Specifically, for ease of modeling and computation, previous studies typically decoupled this task into two components: local human pose estimation and global motion estimation. To estimate local pose, past work projected inertial measurements onto the human root-node coordinate system. However, due to human acceleration and turning, the human root-node coordinate system is usually a non-inertial reference frame. In this case, projecting inertial measurements from the world coordinate system to the non-inertial root-node coordinate system requires introducing inertial forces, such as centrifugal and Coriolis forces, to compensate for the raw acceleration measurements obtained from the sensors. Previous studies neglected the presence of inertial forces, resulting in "incorrect" acceleration measurements during the projection process that do not match the actual human motion as seen from the root-node coordinate system. Therefore, previous models failed to learn the relationship between acceleration and human motion well, effectively discarding most of the acceleration information. While these studies can still estimate many local poses using only sensor orientation measurements, they face difficulties in handling specific acceleration-related movements, such as raising an arm or leg, because the system cannot distinguish such acceleration movements simply by the rotation of sensors located on the forearm or lower leg. Summary of the Invention

[0006] This application aims to at least partially address one of the technical problems in the related art.

[0007] Therefore, the first objective of this application is to propose an inertial human motion capture method based on non-inertial frame dynamics modeling. By considering the possible linear acceleration or rotation at the root of the human body, the root node coordinate system is regarded as a non-inertial coordinate system, thereby more accurately modeling the non-inertial effects in human motion. Furthermore, by simulating inertial forces to correct the measurement signals of inertial sensors, it is ensured that Newton's laws of motion are satisfied in the non-inertial coordinate system.

[0008] The second objective of this application is to propose an inertial human motion capture device based on non-inertial frame dynamics modeling.

[0009] The third objective of this application is to propose an electronic device.

[0010] The fourth objective of this application is to provide a computer-readable storage medium.

[0011] To achieve the above objectives, the first aspect of this application proposes an inertial human motion capture method based on non-inertial frame dynamics modeling, comprising:

[0012] Obtain raw IMU measurements at specific joints of the human body, and preprocess the raw IMU measurements;

[0013] The preprocessed IMU measurements are converted from the global coordinate system to the root joint relative coordinate system. The inertial acceleration of the human body is estimated based on the autoregressive estimator. The IMU measurements converted to the root joint relative coordinate system are then compensated based on the estimated inertial acceleration.

[0014] Estimation of local human posture and global motion is performed using compensated IMU measurements;

[0015] Based on the estimated local pose and global motion, the pose and position are visualized on the human body model.

[0016] Optionally, acquiring IMU measurements at specific joint points of the human body includes:

[0017] Multiple sparse inertial measurement units are installed at different specific joint points, including the forearm, lower leg, head, and pelvis.

[0018] Record the raw IMU measurements of each sparse inertial measurement unit, which include acceleration, angular velocity and magnetic field measurement data at specific joint points.

[0019] Optionally, the preprocessing of the raw IMU measurements includes:

[0020] The raw IMU measurements are cleaned to remove any outliers or erroneous data points.

[0021] Noise processing is performed on the raw IMU measurements after cleaning, and filtering techniques are used to calculate the sensor attitude.

[0022] The optimal static hypothesis testing algorithm is used to perform zero-speed correction on the original IMU measurements after filtering and denoising.

[0023] The original IMU measurements were converted to units after correction.

[0024] Optional, also includes:

[0025] Any node with position p in the human root node coordinate system RL and velocity v RL The leaf joints L are all subjected to inertial force f. fic The influence of inertial force f fic The calculation formula is:

[0026]

[0027] Where m is mass, [·] × It is an antisymmetric matrix used for the cross product of vectors, a RR It is the acceleration of the root joint, ω RR It is the angular velocity of the root joint. It is position p RL The speed after taking the derivative with respect to time, It is angular velocity ω RR Angular acceleration after taking the derivative with respect to time.

[0028] Optionally, the training process of the autoregressive estimator includes:

[0029] The root node dynamics and leaf joint dynamics are used as network inputs, whereby the root node dynamics include the root node's acceleration 'a'. RR angular velocity ω RR and angular acceleration The leaf joint dynamics include the position p of the leaf node. RL ,speed acceleration a RL and direction R RL The root node is dynamically read directly from the preprocessed IMU measurement value, and the position p in the leaf joint dynamics is... RL and speed It is derived from the estimated inertial acceleration of the previous frame;

[0030] The estimated inertial acceleration of all leaf joints is used as the network output, expressed as:

[0031] a fic =f fic / m

[0032] Among them, a fic This represents an estimate of inertial acceleration;

[0033] The L2 loss is used as the loss function of the autoregressive estimator;

[0034] Based on the calculation formula of the inertial force and the preprocessed IMU measurement value, the real inertial acceleration is synthesized, and the real inertial acceleration is used as the supervised learning target.

[0035] Optionally, for local attitude estimation, the estimation of local attitude and global motion using compensated IMU measurements includes:

[0036] For local attitude estimation, {R} in the compensated IMU measurements RL ,a RL +a fic The connection vector, which is used as input, is used to calculate the leaf joint position, all joint positions, and all joint rotations in sequence through three long short-term memory recurrent neural networks with IMU input jump connections.

[0037] For global motion estimation, global motion is estimated by regressing joint velocities and foot-ground contact probabilities.

[0038] Optionally, the step of visualizing the pose and position on the human body model based on the estimated local pose and global motion includes:

[0039] The estimated local pose and global motion are used as inputs for post-processing.

[0040] For the local human body posture, the geometry of the human body surface is obtained by using a parametric human body model or skeletal skinning technology, based on the estimated human body joint rotation and the global orientation of the root node measured by the sparse inertial measurement unit at the root node.

[0041] Using 3D visualization technology, the human body model is rendered in real time based on the geometry of the human body surface;

[0042] The post-processed absolute pose and global motion are applied to the rendered human body model to update the pose and position of the human body model in real time.

[0043] To achieve the above objectives, a second aspect of this application provides an inertial human motion capture device for non-inertial frame dynamics modeling, comprising:

[0044] The data acquisition and preprocessing module is used to acquire raw IMU measurement values ​​of specific joint points of the human body and preprocess the raw IMU measurement values.

[0045] The inertial force estimation module is used to convert the preprocessed IMU measurements from the global coordinate system to the root joint relative coordinate system, estimate the inertial acceleration of the human body based on the autoregressive estimator, and compensate the IMU measurements converted to the root joint relative coordinate system based on the estimated inertial acceleration.

[0046] The pose and motion estimation module is used to estimate the local pose and global motion of the human body using compensated IMU measurements.

[0047] The results post-processing and visualization module is used to visualize the pose and position of the human body model based on the estimated local pose and global motion.

[0048] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0049] The memory stores computer-executed instructions;

[0050] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects above.

[0051] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of the first aspects above.

[0052] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0053] By considering the potential linear acceleration or rotation at the root of the human body, the root node coordinate system is treated as a non-inertial coordinate system, correcting the erroneous handling of the root node coordinate system in traditional inertial motion capture technology. This allows for more accurate modeling of non-inertial effects in human motion. By simulating inertial forces to correct the measurement signals of inertial sensors, it ensures that Newton's laws of motion are satisfied in the non-inertial coordinate system. By using an autoregressive estimator to accurately model the relationship between acceleration and human posture, deterministic learning of the relationship between acceleration and body motion is achieved. This allows for training using neural networks to achieve better motion capture results, effectively improving the accuracy and stability of human motion capture.

[0054] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0055] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0056] Figure 1 This is a text flowchart illustrating an inertial human motion capture method for non-inertial frame dynamics modeling according to an embodiment of this application;

[0057] Figure 2 This is an image flowchart illustrating an inertial human motion capture method for non-inertial frame dynamics modeling according to an embodiment of this application;

[0058] Figure 3 This is a comparison diagram showing the motion results of whether or not inertial effects are modeled in the local posture estimation of the human body, according to the embodiments of this application;

[0059] Figure 4 This is a block diagram of an inertial human motion capture device for non-inertial frame dynamics modeling, as shown in an embodiment of this application.

[0060] Figure 5 It is a block diagram of an electronic device. Detailed Implementation

[0061] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0062] The following description, with reference to the accompanying drawings, describes an inertial human motion capture method and apparatus for non-inertial frame dynamics modeling according to embodiments of this application.

[0063] Figure 1 and Figure 2 These are text flowcharts and image flowcharts illustrating an inertial human motion capture method based on non-inertial frame dynamics modeling, according to embodiments of this application. Figure 1 As shown, the method includes the following steps:

[0064] Step 101: Obtain the raw IMU measurement values ​​of specific joints of the human body and preprocess the raw IMU measurement values.

[0065] In one embodiment of this application, multiple sparse inertial measurement units are installed at different specific joints, including the forearm, lower leg, head, and pelvis. The raw IMU measurements of each sparse inertial measurement unit are then recorded. These data will be used in subsequent attitude and motion estimation processes. The raw IMU measurements include acceleration, angular velocity, and magnetic field measurements at the specific joints.

[0066] Understandably, in the solution of this application, only a minimum of 6 inertial measurement units are required to achieve real-time estimation of human movements.

[0067] It should be noted that during installation, the placement of sparse inertial measurement units (IMUs) needs to be precisely determined to ensure that each IMU can accurately measure the acceleration, angular velocity, etc. at specific joint points, and that there is minimal slippage during movement.

[0068] It should be noted that when recording the raw IMU measurements of each sparse inertial measurement unit, it is necessary to ensure that the data acquisition time of all sparse inertial measurement units is synchronized and the data of each IMU is aligned for subsequent data processing and analysis.

[0069] Then, the raw IMU measurements are preprocessed, including but not limited to data cleaning, noise reduction, filtering, zero-degree correction, and unit conversion.

[0070] Specifically, as one possible implementation, the acquired raw IMU measurement data is cleaned to remove potential outliers or erroneous data points, ensuring data quality and accuracy. Next, noise is processed on the cleaned raw IMU measurements, and filtering techniques are used for sensor attitude calculation. Specifically, Error-State Kalman Filter (ESKF) is used for attitude calculation, and Stance Hypothesis Optimal Detection (SHOE) is used to detect IMU stationary states. When stationary is detected, the Zero-Velocity Update (ZUPT) algorithm is run to improve the accuracy of IMU attitude calculation. Subsequently, the IMU measurements are unit-transformed, converting acceleration units to meters per second² in the global coordinate system while removing the gravitational acceleration component, and converting angular velocity units to radians per second in the global coordinate system.

[0071] Step 102: Convert the preprocessed IMU measurements from the global coordinate system to the root joint relative coordinate system, estimate the inertial acceleration of the human body using an autoregressive estimator, and compensate the IMU measurements converted to the root joint relative coordinate system based on the estimated inertial acceleration.

[0072] It should be noted that there is an inherent coupling relationship between local and global human motion in sensor signals. This application decomposes the estimation of local human posture and global motion into separate tasks. To estimate local posture, embodiments of this application convert IMU measurements from the global coordinate system to the root joint relative coordinate system based on the orientation readings of the root joint sensors. Although the global coordinate system is inertial, the root joint coordinate system is usually non-inertial due to body motion. When IMU measurements, especially acceleration signals, are projected from the inertial coordinate system to the non-inertial coordinate system, inertial forces need to be introduced for compensation.

[0073] Figure 3 This illustrates the necessity of modeling non-inertial effects in local human pose estimation. In the absence of inertial acceleration, such as... Figure 3 As shown in the left and middle figures, two different motions have the same observed acceleration values ​​in the root node coordinate system. However, considering inertial acceleration, as... Figure 3 As shown in the middle right figure, the acceleration was correctly observed.

[0074] It should be noted that in a non-inertial human root node coordinate system, inertial forces affect inertial measurements and must be considered when estimating the human posture relative to the root node. Formally, this application expresses the acceleration and angular velocity of the root joint as a... RR and ω RRFor any point p in the human root node coordinate system RL and velocity v RL The leaf joints L are all subjected to inertial force f. fic The influence of inertial force f fic The calculation formula is:

[0075]

[0076] Where m is mass, [·] × It is an antisymmetric matrix used for the cross product of vectors, a RR It is the acceleration of the root joint, ω RR It is the angular velocity of the root joint. It is position p RL The speed after taking the derivative with respect to time, It is angular velocity ω RR Angular acceleration after differentiation with respect to time, inertial force f fic It includes four terms: linear inertial force, centrifugal force, Coriolis force, and Euler force.

[0077] Inertial force f fic The calculation formula shows that when viewed from the root node coordinate system, the acceleration of the leaf joint is not equal to the original IMU measurement in the inertial world coordinate system. Instead, it undergoes acceleration caused by the inertial force f. fic The additional acceleration caused is referred to in this application as inertial acceleration.

[0078] It is understandable that inertial acceleration depends on the root joint acceleration a. RR and angular velocity ω RR and leaf joint position p RL and speed Root joint terms can be read from the corresponding sparse inertial measurement unit, while leaf joint terms are not directly measured by the sensor and must be calculated using the estimated body posture.

[0079] Therefore, in one embodiment of this application, an autoregressive neural network is trained to estimate inertial acceleration, using root node dynamics and leaf joint dynamics as network inputs, wherein the root node dynamics include the root node's acceleration 'a'. RR angular velocity ω RR and angular acceleration Leaf joint dynamics include the position p of the leaf node. RL ,speed acceleration a RL and direction R RL The root node dynamics are directly read from the preprocessed IMU measurements, and the position p in the leaf joint dynamics is... RL and speed The neural network implicitly learns how to translate the estimated inertial acceleration from the previous frame to the current frame; the estimated inertial acceleration of all leaf joints is used as the network output, expressed as: a fic =f fic / m, where a fic This indicates an estimate of inertial acceleration.

[0080] Furthermore, in the implementation process, this application uses a fusion network for all five leaf joints, i.e., the input is the dynamics of the root joint and all leaf joints, and the output is their inertial acceleration. In one embodiment of this application, the autoregressive estimator is implemented as a fully connected neural network trained with L2 loss, and the above formula is used to synthesize real inertial acceleration from motion capture data as a supervised learning target.

[0081] Step 103: Use the compensated IMU measurements to estimate the local posture and global motion of the human body.

[0082] Based on the estimated inertial acceleration a fic It can correctly convert the input IMU measurements into the root node coordinate system of the human body for local pose estimation.

[0083] In one embodiment of this application, for local attitude estimation, {R} in the compensated IMU measurements RL ,a RL +a fic The connection vector, which is used as input, contains the leaf joint rotation matrix and acceleration in the non-inertial root node coordinate system. The leaf joint position, all joint positions, and all joint rotations are calculated sequentially through three long short-term memory recurrent neural networks with IMU input jump connections.

[0084] It is understandable that the leaf joint position p RL Intermediate estimates, and their time derivatives calculated using finite differences. It is fed back to the autoregressive inertial force estimator for use in the next frame.

[0085] In another embodiment of this application, for global motion estimation, existing methods are followed to regress joint velocities and foot-ground contact probabilities to estimate global motion.

[0086] It should be noted that, unlike local attitude estimation, inertial forces are not required in global motion estimation because this application regresses the global motion relative to a static world coordinate system, which is an inertial reference coordinate system. Finally, this application performs physics-based motion optimization to ensure that the captured motion is physically correct.

[0087] Step 104: Based on the estimated local pose and global motion, visualize the pose and position on the human body model.

[0088] In this embodiment, the estimated local pose and global motion are used as inputs for post-processing.

[0089] For local human body pose, the human body surface geometry is obtained by using parametric human body model or skeletal skinning technology based on the estimated human body joint rotation and the root node global orientation measured by sparse inertial measurement units at the root node. Then, 3D visualization technology is used to render the human body model in real time based on the human body surface geometry. Finally, the post-processed absolute pose and global motion are applied to the rendered human body model to update the human body model's pose and position in real time.

[0090] As one possible approach, when rendering and updating the model, auxiliary information such as environmental scenes and action descriptions can be added to improve the visualization effect and enable real-time rendering of the human body model's motion trajectory, displaying the overall motion state and posture changes.

[0091] In one possible embodiment, the parameterized human body model is SMPL, and the 3D visualization technology is OpenGL or other graphics libraries.

[0092] This application's embodiments, by considering the possible linear acceleration or rotation at the root of the human body, treat the root node coordinate system as a non-inertial coordinate system, correcting the erroneous handling of the root node coordinate system in traditional inertial motion capture technology, thereby more accurately modeling the non-inertial effects in human motion; by simulating inertial forces to correct the inertial sensor measurement signals, it ensures that Newton's laws of motion are satisfied in the non-inertial coordinate system; by using an autoregressive estimator to accurately model the relationship between acceleration and human posture, it achieves deterministic learning of the relationship between acceleration and body motion, and then uses neural networks for training to achieve better motion capture results, effectively improving the accuracy and stability of human motion capture.

[0093] Figure 4 This is a block diagram of an inertial human motion capture device 10 based on non-inertial frame dynamics modeling, according to an embodiment of this application, comprising:

[0094] The data acquisition and preprocessing module 100 is used to acquire raw IMU measurement values ​​of specific joints of the human body and preprocess the raw IMU measurement values.

[0095] The inertial force estimation module 200 is used to convert the preprocessed IMU measurement values ​​from the global coordinate system to the root joint relative coordinate system, estimate the inertial acceleration of the human body according to the autoregressive estimator, and compensate the IMU measurement values ​​converted to the root joint relative coordinate system according to the estimated inertial acceleration.

[0096] The attitude and motion estimation module 300 is used to estimate the local attitude and global motion of the human body using compensated IMU measurements.

[0097] The result post-processing and visualization module 400 is used to visualize the pose and position on the human body model based on the estimated local pose and global motion.

[0098] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0099] Figure 5 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0100] like Figure 5 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0101] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the voice command response method. For example, in some embodiments, the voice command response method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the voice command response method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the voice command response method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0108] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for inertial human motion capture based on non-inertial frame dynamics modeling, characterized in that, include: Obtain raw IMU measurements at specific joints of the human body, and preprocess the raw IMU measurements; The preprocessed IMU measurements are converted from the global coordinate system to the root joint relative coordinate system. The inertial acceleration of the human body is estimated based on the autoregressive estimator. The IMU measurements converted to the root joint relative coordinate system are then compensated based on the estimated inertial acceleration. Estimation of local human posture and global motion is performed using compensated IMU measurements; Based on the estimated local pose and global motion, the pose and position are visualized on the human body model. Wherein, any node has a position in the human root node coordinate system and speed leaf joint Both are subject to inertial forces The influence of inertial force The calculation formula is: in, It's about quality. It is the antisymmetric matrix used for the cross product of vectors. It is the acceleration of the root joint. It is the angular velocity of the root joint. It is a location The speed after taking the derivative with respect to time, It is angular velocity The angular acceleration differentiated with respect to time; the training process of the autoregressive estimator includes: The network input consists of root node dynamics and leaf joint dynamics, where the root node dynamics include the root node's acceleration. angular velocity and angular acceleration The leaf joint dynamics include the position of the leaf node. ,speed acceleration and direction The root node is dynamically read directly from the preprocessed IMU measurements, and the position of the leaf joint in the dynamics is... and speed It is derived from the estimated inertial acceleration of the previous frame; The estimated inertial acceleration of all leaf joints is used as the network output, expressed as: in, This represents an estimate of inertial acceleration; The L2 loss is used as the loss function of the autoregressive estimator; Based on the calculation formula of the inertial force and the preprocessed IMU measurement value, the real inertial acceleration is synthesized, and the real inertial acceleration is used as the supervised learning target.

2. The method according to claim 1, characterized in that, The acquisition of IMU measurements at specific joints of the human body includes: Multiple sparse inertial measurement units are installed at different specific joint points, including the forearm, lower leg, head, and pelvis. Record the raw IMU measurements of each sparse inertial measurement unit, which include acceleration, angular velocity and magnetic field measurement data at specific joint points.

3. The method according to claim 2, characterized in that, The preprocessing of the raw IMU measurements includes: The raw IMU measurements are cleaned to remove any outliers or erroneous data points. Noise processing is performed on the raw IMU measurements after cleaning, and filtering techniques are used to calculate the sensor attitude. The optimal static hypothesis testing algorithm is used to perform zero-speed correction on the original IMU measurements after filtering and denoising. The original IMU measurements were converted to units after correction.

4. The method according to claim 3, characterized in that, The estimation of local attitude and global motion using compensated IMU measurements includes: For local attitude estimation, the compensated IMU measurements are used. The connection vector, which is the input, is used to calculate the leaf joint position, all joint positions, and all joint rotations in sequence through three long short-term memory recurrent neural networks with IMU input jump connections. For global motion estimation, global motion is estimated by regressing joint velocities and foot-ground contact probabilities.

5. The method according to claim 4, characterized in that, The visualization of pose and position on the human body model based on the estimated local pose and global motion includes: The estimated local pose and global motion are used as inputs for post-processing. For the local human body posture, the geometry of the human body surface is obtained by using a parametric human body model or skeletal skinning technology, based on the estimated human body joint rotation and the global orientation of the root node measured by the sparse inertial measurement unit at the root node. Using 3D visualization technology, the human body model is rendered in real time based on the geometry of the human body surface; The post-processed absolute pose and global motion are applied to the rendered human body model to update the pose and position of the human body model in real time.

6. An inertial human motion capture device based on non-inertial frame dynamics modeling, characterized in that, include: The data acquisition and preprocessing module is used to acquire raw IMU measurement values ​​of specific joint points of the human body and preprocess the raw IMU measurement values. The inertial force estimation module is used to convert the preprocessed IMU measurements from the global coordinate system to the root joint relative coordinate system, estimate the inertial acceleration of the human body based on the autoregressive estimator, and compensate the IMU measurements converted to the root joint relative coordinate system based on the estimated inertial acceleration. The pose and motion estimation module is used to estimate the local pose and global motion of the human body using compensated IMU measurements. The result post-processing and visualization module is used to visualize the pose and position of the human body model based on the estimated local pose and global motion. Wherein, any node has a position in the human root node coordinate system and speed leaf joint Both are subject to inertial forces The influence of inertial force The calculation formula is: in, It's about quality. It is the antisymmetric matrix used for the cross product of vectors. It is the acceleration of the root joint. It is the angular velocity of the root joint. It is a location The speed after taking the derivative with respect to time, It is angular velocity The angular acceleration differentiated with respect to time; the training process of the autoregressive estimator includes: The network input consists of root node dynamics and leaf joint dynamics, where the root node dynamics include the root node's acceleration. angular velocity and angular acceleration The leaf joint dynamics include the position of the leaf node. ,speed acceleration and direction The root node is dynamically read directly from the preprocessed IMU measurements, and the position of the leaf joint in the dynamics is... and speed It is derived from the estimated inertial acceleration of the previous frame; The estimated inertial acceleration of all leaf joints is used as the network output, expressed as: in, This represents an estimate of inertial acceleration; The L2 loss is used as the loss function of the autoregressive estimator; Based on the calculation formula of the inertial force and the preprocessed IMU measurement value, the real inertial acceleration is synthesized, and the real inertial acceleration is used as the supervised learning target.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

Images