A human motion capture method, system and storage medium based on IMU

By collecting human motion data using an IMU and performing a series of processing steps, the accuracy problem of inertial human motion capture methods has been solved, enabling precise capture and recognition of human motion and providing a more scientific basis for diagnosis.

CN115868967BActive Publication Date: 2024-11-26HANGZHOU ROBOCT TECH DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310033123.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-01-10
Publication Date
2024-11-26
Estimated Expiration
2043-01-10

AI Technical Summary

Technical Problem

Inertial motion capture methods have limitations in terms of accuracy, resulting in significant discrepancies between joint range of motion and gait data and actual data, which restricts their application in the field of human motion capture.

Method used

Human motion data is acquired by IMU, and coordinate system transformation, low-pass filtering, quaternion normalization, mirror pose transformation, frame rate interpolation and matrix transformation are performed to calculate joint angles, thereby achieving accurate capture and recognition of human motion.

Benefits of technology

It enables precise capture and recognition of human movements, providing a more scientific diagnostic basis for patient treatment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115868967B_ABST
    Figure CN115868967B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of robot equipment, and specifically to an IMU-based human motion capture method, system and storage medium. The present invention calculates joint angles by performing coordinate system conversion, low-pass filtering, quaternion normalization processing, conversion to mirror posture, frame rate interpolation processing, and matrix transformation on human motion data collected by the IMU to recursively infer the posture of active joints. The joint angles of all joints in the human motion process can be calculated based on the data acquired by the IMU, which is used to achieve accurate capture and recognition of human motion, thereby providing a more scientific diagnostic basis for the treatment of patients.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robotic equipment, and specifically to a method, system and storage medium for capturing human motion based on IMU. Background Art

[0002] At present, related products that can capture human motion can be divided into two types: optical and inertial due to different detection methods. The inertial human motion capture system can obtain the motion information of the object being measured through its internal sensors. It has no restrictions on usage scenarios and good real-time performance. Although its accuracy is lower than that of the optical system, it can meet the requirements of continuous collection of key information of motion nodes and biological analysis in medical and health application scenarios.

[0003] At present, the inertial human motion capture method has defects in accuracy, resulting in large discrepancies between the obtained joint range of motion data and gait data and the actual data, which seriously restricts the application of inertial measurement methods in the field of human motion capture. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned technical solutions and provide a human motion capture method, system and storage medium based on IMU, which are used to achieve accurate capture and recognition of human motion, thereby providing a more scientific diagnostic basis for the treatment of patients.

[0005] In order to achieve the above object, according to one aspect of the present invention, a method for capturing human motion based on IMU comprises the following steps:

[0006] Step 1: Collect human motion data through IMU;

[0007] Step 2: Perform coordinate system conversion on the human motion data collected by the IMU;

[0008] Step 3: low-pass filtering the human body motion posture data represented by the quaternion obtained in step 2;

[0009] Step 4: Perform quaternion normalization processing on the human body motion posture data obtained in step 3;

[0010] Step 5: Import the data obtained in step 4 into the Unity software to obtain the posture of the human body model, and convert the posture of the human body model into a mirror posture;

[0011] Step 6: performing frame rate interpolation processing on the human body motion posture data obtained in step 4;

[0012] Step 7: Recursively infer the posture of the active joint according to the matrix transformation;

[0013] Step 8: Calculate all joint angles based on the postures of all joints obtained in step 7.

[0014] Specifically, the IMU is an inertial measurement system, which includes different types of IMU components such as a gyroscope, an accelerometer, and a magnetometer, through which human motion data is collected;

[0015] Specifically, the data collected by the IMU is data obtained in a right-handed coordinate system, and the data in the unity coordinate system is obtained through coordinate system conversion;

[0016] Furthermore, the coordinate system conversion of the data collected by the IMU specifically includes:

[0017] Step 2.1: converting the human motion data collected by the IMU into human motion posture data represented by quaternions;

[0018] Since the quaternion representation method can be faster and more efficient in the calculation process, the present embodiment selects quaternion as the representation method of human motion posture when performing human motion capture;

[0019] Furthermore, the step 2.1 specifically includes: according to the formula f(q U )=Qf(q B ) to obtain the human motion data represented by quaternion, where Q is the mapping matrix between the IMU coordinate system and the unity coordinate system, q U is the human body's motion posture expressed by quaternion in the unity coordinate system, q B is the human body's motion posture expressed by quaternion in the geodetic coordinate system of IMU, and the function f(q U ) represents human motion data represented by quaternion;

[0020] Step 2.2: Obtain an expression of the human motion data represented by the quaternion;

[0021] Specifically, 2.2 is as follows: take any point q in the human body motion process, set q = (wxyz), wxyz are the four basic elements of the standard quaternion, then

[0022]

[0023] Where f(q) is the human motion data represented by quaternion;

[0024] Step 2.3: According to the steps 2.1 and 2.2, human body motion posture data represented by quaternion is obtained by solving;

[0025] Specifically, the step 2.3 specifically includes: by formula:

[0026]

[0027] Solve and obtain human body motion posture data represented by quaternion;

[0028] In the formula, q B0 ,q B1 They are the posture of the IMU on the human body at the zero position and real-time position respectively; q m0 ,q m1 are the human body postures at zero position and real-time position, respectively, U0 is the posture of the human body model in the initial state, q U1 The pose of the human model collected in real time;

[0029] Specifically, the expression of the low-pass filtering is:

[0030] q (k+1) =λq (k+1) +(1-λ)q (k)

[0031] In the formula, q (k+1) is the quaternion of the k+1th sampling point, λ is the filter coefficient, q (k) is the quaternion of the kth sampling point;

[0032] Specifically, the quaternion normalization processing formula is:

[0033]

[0034] Specifically, the step 5 is as follows:

[0035] Assuming that the posture of a certain part of the human body model is q0, the posture after mirroring is q1

[0036] q1=AR y (π)

[0037] In the formula, A is the intermediate matrix, R y (π) is a function related to q0, and the specific functional relationship is known in the art and will not be discussed in detail here;

[0038] Furthermore, A is obtained by solving the matrix B, where B = f(q0), and the elements of the matrices A and B satisfy the following relationship:

[0039] b 13 =-a 13 b 23 =-a 23

[0040] b 31 =-a 31 b 32 =-a32

[0041] In order to ensure the continuity of the action and the smoothness of the software model, it is necessary to perform three-dimensional dynamic frame rate interpolation calculations on the posture.

[0042] Specifically, the formula for the frame rate interpolation operation is:

[0043] p=slerp(q n ,q n+1 ,n)

[0044] p(i)=aq n (i)+bq n+1 (i),i=1,2,3,4

[0045]

[0046]

[0047] θ=cos -1 (<q0.q1>)

[0048] Where slerp is the spherical interpolation function, q n and q n+1 are two adjacent quaternion attitude data, n is the interpolation number, p is the quaternion sequence after interpolation, i is the i-th element of the quaternion; a, b, θ are the three intermediate calculation parameters for solving the interpolation quaternion;

[0049] Since the number of joints in the human body model is much greater than the number of installed IMUs, it is not guaranteed that both the parent and child systems of each joint are equipped with IMUs, so it is necessary to recursively deduce the parent and child postures of each motion joint;

[0050] Specifically, the step 7 specifically includes:

[0051] Step 7.1: Set the parent posture q of a certain motion joint at the zero position of the human body model f0 , the zero-point posture q of the imu installed closest to the parent s0 , the real-time posture is q s1 , the joint posture is q j , so the recursive node formula for each joint can be established as:

[0052] q r =f -1 ((f(q f0 )) T (f(q s0 ))(f(q j )) T )

[0053] Step 7.2: Obtain the posture of the joint parent system;

[0054] Specifically, the formula for obtaining the posture of the joint parent system is:

[0055] q f1 =f -1 ((f(q s1 ))(f(q r )))

[0056] Specifically, step 8 specifically includes: all parts equipped with IMU use the above-mentioned coordinate mapping, model posture solution, and mirror transformation to obtain real-time posture, and then assume that the real-time posture of the subsystem of the joint is q b1 , then, the joint angle calculation formula is:

[0057] q=f -1 ((f(q f1 )) T (f(q b1 )))

[0058] According to another aspect of the present invention, the present invention also includes a human motion capture system based on IMU, wherein the system includes:

[0059] Several IMUs to obtain human motion data;

[0060] The processor is used to execute the above-mentioned IMU-based human motion capture method.

[0061] According to another aspect of the present invention, this embodiment includes a computer-readable storage medium, on which a data processing program is stored, and the data processing program is executed by a processor to implement the above-mentioned IMU-based human motion capture method.

[0062] Based on the above technical solution, the present application provides an IMU-based human motion capture method, system and computer-readable storage medium, which have the following technical effects:

[0063] The present invention calculates the joint angles by performing coordinate system conversion, low-pass filtering, quaternion normalization, conversion to mirror posture, frame rate interpolation, and matrix transformation on the human motion data collected by IMU. The joint angles of all joints in the human motion process can be calculated based on the data acquired by IMU, which is used to achieve accurate capture and recognition of human motion, thereby providing a more scientific diagnostic basis for the treatment of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings constituting a part of this document are used to provide a further understanding of this document. The exemplary embodiments and descriptions of this document are used to explain this document and do not constitute an improper limitation on this document. In the accompanying drawings:

[0065] Figure 1 A flowchart of a human motion capture method based on IMU provided in an embodiment of the present application;

[0066] Figure 2 A flowchart of performing coordinate system conversion on the data collected by the IMU provided in an embodiment of the present application;

[0067] Figure 3 A flowchart of recursively inferring the posture of an active joint according to matrix transformation provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of a human motion capture system based on IMU is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of this article clearer, the technical solutions in the embodiments of this article will be clearly and completely described below in conjunction with the drawings in the embodiments of this article. Obviously, the described embodiments are part of the embodiments of this article, not all of the embodiments. Based on the embodiments in this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article. It should be noted that the embodiments in this article and the features in the embodiments can be combined with each other arbitrarily without conflict.

[0070] Unless the context clearly requires otherwise, throughout the specification and claims, the words "include", "comprising" and similar words should be interpreted in an inclusive sense rather than an exclusive or exhaustive sense; that is, in the sense of "including but not limited to".

[0071] Embodiment 1

[0072] The human motion capture method provided in this embodiment performs data fusion and filtering on the original acceleration and angular velocity data, thereby outputting the posture data of the IMU, thereby calculating the joint angle.

[0073] like Figure 1 As shown, a human motion capture method based on IMU includes the following steps:

[0074] Step 1: Collect human motion data through IMU;

[0075] Specifically, the IMU is an inertial measurement system, which includes different types of IMU components such as a gyroscope, an accelerometer, and a magnetometer, through which human motion data is collected;

[0076] Step 2: Perform coordinate system conversion on the human motion data collected by the IMU;

[0077] Specifically, the data collected by the IMU is data obtained in a right-handed coordinate system, and the data in the unity coordinate system is obtained through coordinate system conversion;

[0078] Furthermore, if Figure 2 As shown, the coordinate system conversion of the data collected by the IMU specifically includes:

[0079] Step 2.1: converting the human motion data collected by the IMU into human motion posture data represented by quaternions;

[0080] Since the quaternion representation method can be faster and more efficient in the calculation process, the present embodiment selects quaternion as the representation method of human motion posture when performing human motion capture;

[0081] Furthermore, the step 2.1 specifically includes: according to the formula f(q U )=Qf(q B ) to obtain the human motion data represented by quaternion, where Q is the mapping matrix between the IMU coordinate system and the unity coordinate system, q U is the human body's motion posture expressed by quaternion in the unity coordinate system, q B is the human body's motion posture expressed by quaternion in the geodetic coordinate system of IMU, and the function f(q U ) represents human motion data represented by quaternion;

[0082] Step 2.2: Obtain an expression of the human motion data represented by the quaternion;

[0083] Specifically, 2.2 is as follows: take any point q in the human body motion process, set q = (wxyz), wxyz are the four basic elements of the standard quaternion, then

[0084]

[0085] Where f(q) is the human motion data represented by quaternion;

[0086] Step 2.3: According to the steps 2.1 and 2.2, human body motion posture data represented by quaternion is obtained by solving;

[0087] Specifically, the step 2.3 specifically includes: by formula:

[0088]

[0089] Solve and obtain human body motion posture data represented by quaternion;

[0090] In the formula, q B0 ,q B1 They are the posture of the IMU on the human body at the zero position and real-time position respectively; q m0 ,q m1 are the human body postures at zero position and real-time position, respectively, U0 is the posture of the human body model in the initial state, q U1 The pose of the human model collected in real time;

[0091] Step 3: low-pass filtering the human body motion posture data represented by the quaternion obtained in step 2;

[0092] Specifically, the expression of the low-pass filtering is:

[0093] q (k+1) =λq (k+1 )+(1-λ)q (k)

[0094] In the formula, q (k+1) is the quaternion of the k+1th sampling point, λ is the filter coefficient, q (k) is the quaternion of the kth sampling point;

[0095] Step 4: Perform quaternion normalization processing on the human body motion posture data obtained in step 3; specifically, the quaternion normalization processing formula is:

[0096]

[0097] Step 5: Import the data obtained in step 4 into the unity software to obtain the posture of the human body model, and convert the posture of the human body model into a mirror posture;

[0098] Specifically, the step 5 is as follows:

[0099] Assuming that the posture of a certain part of the human body model is q0, the posture after mirroring is q1

[0100] q1=AR y (π)

[0101] In the formula, A is the intermediate matrix, R y (π) is a function related to q0, and the specific functional relationship is known in the art and will not be discussed in detail here;

[0102] Furthermore, A is obtained by solving the matrix B, where B = f(q0), and the elements of the matrices A and B satisfy the following relationship:

[0103] b 13 =-a 13 b 23 =-a 23

[0104] b 31 =-a 31 b 32 =-a 32

[0105] Step 6: performing frame rate interpolation processing on the human body motion posture data obtained in step 4;

[0106] In order to ensure the continuity of the action and the smoothness of the software model, it is necessary to perform three-dimensional dynamic frame rate interpolation calculations on the posture.

[0107] Specifically, the formula for the frame rate interpolation operation is:

[0108] p=slerp(q n ,q n+1 ,n)

[0109] p(i)=aq n (i)+bq n+1 (i),i=1,2,3,4

[0110]

[0111]

[0112] θ=cos -1 (<q0.q1> )

[0113] Where slerp is the spherical interpolation function, q n and q n+1 are two adjacent quaternion attitude data, n is the interpolation number, p is the quaternion sequence after interpolation, i is the i-th element of the quaternion; a, b, θ are the three intermediate calculation parameters for solving the interpolation quaternion;

[0114] Step 7: Recursively infer the posture of the active joint according to the matrix transformation;

[0115] Since the number of joints in the human body model is much greater than the number of installed IMUs, it is not guaranteed that both the parent and child systems of each joint are equipped with IMUs, so it is necessary to recursively deduce the parent and child postures of each motion joint;

[0116] Specifically, Figure 3 As shown, the step 7 specifically includes:

[0117] Step 7.1: Establish the recursive node formula for each joint;

[0118] Specifically, let the parent posture q of a certain motion joint at the zero position of the human body model be f0 , the zero-point posture q of the imu installed closest to the parent s0 , the real-time posture is q s1 , the joint posture is q j , so the recursive node formula for each joint can be established as:

[0119] q r =f -1 ((f(q f0 )) T (f(q s0 ))(f(q j )) T )

[0120] Step 7.2: Obtain the posture of the joint parent system;

[0121] Specifically, the formula for obtaining the posture of the joint parent system is:

[0122] q f1 =f -1 ((f(q s1 ))(f(q r )))

[0123] Step 8: Calculate all joint angles according to the postures of all joints obtained in step 7;

[0124] Specifically, step 8 specifically includes: all parts equipped with IMU use the above-mentioned coordinate mapping, model posture solution, and mirror transformation to obtain real-time posture, and then assume that the real-time posture of the subsystem of the joint is q b1 , then, the joint angle calculation formula is:

[0125] q=f -1 ((f(q f1 )) T (f(q b1 )))

[0126] Therefore, through the above steps 1-8, the joint angles of all joints during human movement can be calculated based on the data obtained by the IMU, which can be used to accurately capture and identify human movements, thereby providing a more scientific diagnostic basis for the patient's treatment.

[0127] Embodiment 2

[0128] like Figure 4As shown, this embodiment also includes a human motion capture system based on IMU, wherein the system includes:

[0129] Several IMUs to obtain human motion data;

[0130] The processor is used to execute the IMU-based human motion capture method of the first embodiment.

[0131] Embodiment 3

[0132] This embodiment includes a computer-readable storage medium, on which a data processing program is stored. The data processing program is executed by a processor to implement the IMU-based human motion capture method of embodiment 1.

[0133] It should be understood by those skilled in the art that the embodiments of this article can be provided as methods, devices (equipment), or computer program products. Therefore, this article can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this article can adopt the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data), including but not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media generally contain computer-readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0134] This document is described with reference to flowcharts and / or block diagrams of methods, apparatuses (devices) and computer program products according to the embodiments of this document. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] Although the preferred embodiments of this invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of this invention.

[0137] Obviously, those skilled in the art can make various changes and modifications to this article without departing from the spirit and scope of this article. Thus, if these modifications and variations of this article fall within the scope of the claims and their equivalents, then the intention of this article also includes these modifications and variations.

Claims

1. A human motion capture method based on IMU, comprising the following steps: Step 1: Collect human motion data through IMU; Step 2: Perform coordinate system conversion on the human motion data collected by the IMU; The coordinate system conversion of the data collected by the IMU specifically includes: Step 2.1: converting the human motion data collected by the IMU into human motion posture data represented by quaternion; the step 2.1 specifically includes: according to the formula f(q U )=Qf(q B ) to obtain the human motion data represented by quaternion, where Q is the mapping matrix between the IMU coordinate system and the unity coordinate system, q U is the human body's motion posture expressed by quaternion in the unity coordinate system, q B is the human body's motion posture expressed by quaternion in the geodetic coordinate system of IMU, and the function f(q U ) represents human motion data represented by quaternion; Step 2.2: Obtain an expression of the human motion data represented by the quaternion; the step 2.2 specifically includes: taking any point q in the human motion process, setting q = (w, x, y, z), w, x, y, z are the four basic elements of the standard quaternion, then Where f(q) is the human motion data represented by quaternion; Step 2.3: According to the steps 2.1 and 2.2, human body motion posture data represented by quaternion is obtained by solving; Step 3: low-pass filtering the human body motion posture data represented by the quaternion obtained in step 2; Step 4: Perform quaternion normalization processing on the human body motion posture data obtained in step 3; Step 5: Import the data obtained in step 4 into the unity software to obtain the posture of the human body model, and convert the posture of the human body model into a mirror posture; Step 6: performing frame rate interpolation processing on the human body motion posture data obtained in step 4; The formula for the frame rate interpolation operation is: p=slerp(q n ,q n+1 ,n) p(i)=aq n (i)+bq n+1 (i),i=1,2,3,4 θ=cos -1 (q0.q1) Where slerp is the spherical interpolation function, q n and q n+1 are two adjacent quaternion attitude data, n is the interpolation number, p is the quaternion sequence after interpolation, i is the i-th element of the quaternion; a, b, θ are the three intermediate calculation parameters for solving the interpolation quaternion; Step 7: Recursively infer the posture of the active joint according to the matrix transformation; The step 7 specifically includes: Step 7.1: Set the parent posture q of a certain motion joint at the zero position of the human body model f0 , the zero-point posture q of the imu installed closest to the parent s0 , the real-time posture is q s1 , the joint posture is q j , thus establishing the recursive node formula for each joint: q r =f -1 ((f(q f0 )) T (f(q s0 ))(f(q j )) T ) Step 7.2: Obtain the posture of the joint parent system; Specifically, the formula for obtaining the posture of the joint parent system is: q f1 =f -1 ((f(q s1 ))(f(q r ))); Step 8: Calculate all joint angles according to the postures of all joints obtained in step 7; The step 8 specifically includes: all parts equipped with IMU use coordinate mapping, model posture solving, and mirror transformation to obtain real-time posture, and then assume that the real-time posture of the subsystem of the joint is q b1 , then, the joint angle calculation formula is: q=f -1 ((f(q f1 )) T (f(q b1 )))。 2. The method for capturing human motion based on IMU according to claim 1, characterized in that: The IMU is an inertial measurement system, which includes different types of IMU components such as a gyroscope, an accelerometer, and a magnetometer, through which human motion data is collected.

3. The method for capturing human motion based on IMU according to claim 1, characterized in that: The step 2.3 specifically includes: by formula: q m1 =f -1 (f(q U1 )(f(q U0 )) T f(q m0 )) =f -1 (Qf(q B1 )(Qf(q B0 )) T f(q m0 )) Solve and obtain human body motion posture data represented by quaternion; In the formula, q B0 ,q B1 They are the posture of the IMU on the human body at the zero position and real-time position respectively; q m0 ,q m1 are the human body postures at zero position and real-time position, respectively, U0 is the posture of the human body model in the initial state, q U1 The pose of the human model collected in real time.

4. A human motion capture system based on IMU, characterized in that: The system includes: Several IMUs to obtain human motion data; A processor, used to execute the IMU-based human motion capture method described in any one of claims 1-3.

5. A computer-readable storage medium having a data processing program stored thereon, wherein the data processing program is executed by a processor to implement the IMU-based human motion capture method according to any one of claims 1 to 3.