Human motion monitoring method and device based on multi-source information fusion

Abstract

The present invention relates to a human motion monitoring method based on multi-source information fusion, wherein the method takes into account the different physiological structural characteristics of the human body of individuals, collects multi-source information, including anthropometric information, plantar pressure distribution data, actual inertial measurement unit attachment position and collected data, so that the human skeleton model used for calculation is more consistent with the real test subject. By establishing an objective function and constraint that improves the accuracy of the inertial measurement unit and satisfies the physiological significance of human motion, and performing minimum optimization calculation on the objective function, joint motion parameters can be obtained. The method can reduce the uncertainty problem and error of the human model caused by factors such as sensor error, mechanical joint assumption, and skin deformation, and improve the monitoring accuracy of joint motion. Since there is no need to arrange a detection environment, high-precision and physiologically significant human motion monitoring can be easily realized in a daily sports environment, including human motion analysis under pathology with high precision requirements.

Description

Technical Field

[0001] The present disclosure relates to the fields of biological and new medical technology - medical instrument technology, equipment and medical-specific software, and in particular to a human motion monitoring method and device based on multi-source information fusion. Background Art

[0002] Human motion monitoring plays an important role in rehabilitation therapy, sports, human-computer interaction and other fields. In the past, optical motion capture technology was widely used, but this method has problems such as expensive optical cameras, fixed monitoring scenes, and easy occlusion of parts. In recent years, wearable sensing devices represented by IMU (Inertial Measurement Unit) have been widely used in fields such as motion monitoring and medical rehabilitation because of their advantages such as easy wearing, unlimited movement space and low cost. For example, the invention patent CN201710442846.4 "Angle detection method and joint motion angle detection system" uses two angle detectors containing IMU to detect the posture angle generated by the movement of the corresponding joint arm, and then obtains the degree of the joint angle based on the posture angle.

[0003] However, there are two problems with IMU in use. On the one hand, it is easy to accumulate errors and data drift, and IMU has poor response to high-dynamic posture movements, so the data acquisition accuracy is limited. Invention patent CN202010633536.2 "A real-time full-body motion capture system and method based on camera and IMU data mixing" aims at this phenomenon by fusing camera equipment and IMU information to obtain joint motion posture, and supplementing the prediction of human joint model according to the color and depth information of video frame images to correct the motion posture of human joints obtained by IMU. In addition, this method also requires the use of external environment to arrange camera equipment to obtain correction information, and there is also the problem of limited monitoring scenes. On the other hand, the current tracking of human joint motion trajectory based on IMU is generally based on the inverse solution of a simple IMU rigid body motion chain, without considering the uncertainty of the human model caused by sensor errors, mechanical joint assumptions of the human body, and skin deformation errors. Therefore, the use of IMU equipment alone can usually only be used to obtain low-complexity kinematic information, and is not suitable for high-precision human motion analysis under pathology. Summary of the invention

[0004] In view of this, the main purpose of this application is to propose a human motion fusion monitoring method based on multi-source information, the purposes of which include: (1) improving the accuracy of human motion monitoring using inertial measurement units. The method will take into account the different physiological structure characteristics of the human body, as well as the uncertainty and errors of the human body model caused by sensor errors, mechanical joint assumptions, skin deformation and other factors, and incorporate human physiological motion criteria and additional mechanical sensor data to compensate for motion information; (2) removing the limitations of motion monitoring scenarios, and realizing high-precision, physiologically significant human motion monitoring in daily sports environments without the need to set up a detection environment.

[0005] Based on the above objectives, the present invention proposes the following technical solutions:

[0006] In a first aspect, the present invention proposes a method for monitoring human motion based on multi-source information fusion, the method comprising the following steps:

[0007] S1. Acquire multi-source information of the test subject, wherein the multi-source information includes anthropometric information, plantar pressure distribution data, actual inertial measurement unit attachment position and collected data;

[0008] S2. Based on multi-source information, the universal human skeleton model is adapted and adjusted to obtain a human skeleton model to be tested;

[0009] S3, using the human skeleton model to be tested and the multi-source information, establishing an objective function with joint motion parameters as decision variables, and obtaining the joint motion parameters by minimizing the objective function;

[0010] The objective function includes the sum of the first term and the second term;

[0011] The first item is used to measure the position error between the human skeleton model to be tested and the person being tested with respect to the inertial measurement unit;

[0012] The second item is used to measure all or part of the calculations related to energy metabolism during human joint movement.

[0013] Preferably, in the method, the human skeleton model to be tested is obtained by the following steps:

[0014] S21, scaling the universal human skeleton model using the anthropometric information to obtain a human skeleton model to be tested;

[0015] S22, using the plantar pressure distribution data, calculating the action center position and action concentration force of the ground reaction force, and associating the corresponding action center position and action concentration force as contact points and contact forces to the human skeleton model to be tested;

[0016] S23. According to the actual inertial measurement unit attachment position, define a virtual inertial measurement unit and rigidly connect it to the human skeleton model to be measured.

[0017] Preferably, in the method, the objective function is as follows:

[0018]

[0019]

[0020] Where:

[0021] A is the joint motion parameter,

[0022] q is the generalized position vector of the monitored joint, is the first-order derivative of q with respect to time t; is the second-order derivative of q with respect to time t;

[0023] The elements of q include the displacement and angle of the joint movement, where the displacement ranges from [-0.02, 0.02] in meters, the angle ranges from [-π, π] in radians, and θ IMU and are the angle vectors of the inertial measurement unit obtained by simulation and actual measurement respectively; t1 is the start time of the movement, t2 is the end time of the movement; M is the generalized force vector composed of force component and torque component, G is the gravity vector, C is the Coriolis force and centrifugal force vector; F Act is the generalized active force vector driving the joint motion; F PE The generalized passive force vector generated by the joint stiffness set in the human skeleton model to be tested; F external is the generalized external force vector; f energy It is all or part of the calculation formula related to energy metabolism during human joint movement.

[0024] Preferably, in the method, the inertial measurement unit attachment positions include the following positions: head, neck, thoracic spine, lumbar spine, pelvis, upper arm, forearm, hand, thigh, calf, foot.

[0025] Preferably, in the method, the joints include shoulder joints, elbow joints, wrist joints, hip joints, knee joints, ankle joints, cervical vertebrae joints, and lumbar vertebrae joints.

[0026] Preferably, in the method, the anthropometric information includes: geometric lengths of limbs and trunk, height, and weight.

[0027] Preferably, in the method, the second item includes exercise physiological mechanical energy and metabolic rate.

[0028] Preferably, in the method, the plantar pressure distribution data is obtained through a plantar pressure insole.

[0029] In the second aspect, the present invention proposes a human motion monitoring device based on multi-source information fusion, characterized in that the device includes an inertial measurement unit, a plantar pressure insole, a memory and a processor; the memory stores data collected by the inertial measurement unit and the plantar pressure insole, and a computer program that can be loaded by the processor and execute any of the above methods; the inertial measurement unit is used to be attached to the body of the test person to record the motion trajectory of the inertial measurement unit; the plantar pressure insole is used to obtain plantar pressure distribution data.

[0030] Preferably, in the device, the device also includes a data storage package for storing real-time data collected by the inertial measurement unit and the plantar pressure insole, and then transmitting and exporting the data via wired or wireless means.

[0031] Compared with the prior art, the present invention has the following beneficial technical effects:

[0032] (1) The present invention takes into account the different physiological structure characteristics of the human body and uses multi-source information to adapt and adjust the universal human skeleton model, so that the human skeleton model to be tested can more realistically simulate the movement state of the tested person and restore the force borne by each joint, thereby making the monitored human joint movement more realistic and accurate.

[0033] (2) A human skeleton model that can highly reproduce the motion state of the subject is used to establish an objective function with joint motion parameters as decision variables. The objective function limits the error in the correlation between the variable to be solved and the actual information source to improve the accuracy of using an inertial measurement unit to monitor human motion; and limits the physiological principles of human motion to the greatest extent possible during the motion process.

[0034] (3) By removing the restrictions on motion monitoring scenarios and without setting up a detection environment, high-precision, physiologically significant human motion monitoring can be achieved in daily exercise environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 , one A schematic diagram of a device deployed on a subject in an embodiment;

[0037] Figure 2 , schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0039] In Example 1, the following information of the test subject is first obtained:

[0040] (11) Anthropometric information, including the geometric lengths of the limbs and trunk, height, and weight. The specific measurement methods shall refer to anthropometric standards.

[0041] (12) Plantar pressure distribution data. An optimal and convenient way to obtain the data is to place a plantar pressure insole in the shoe of the testee, and then measure and record the plantar pressure distribution data through the plantar pressure insole. Alternatively, the plantar pressure can be tested by using socks, shoes, floor mats specifically used to collect plantar pressure, or other methods.

[0042] (13) Inertial measurement unit attachment position. Preferably, a nine-axis inertial measurement unit containing an accelerometer, a gyroscope, and a magnetometer is used, which is attached to the target body segment of the subject to record the motion trajectory of the inertial measurement unit in space. Figure 1 In the given embodiment, the human body segments are divided into: head, neck, thoracic vertebrae, lumbar vertebrae, pelvis, upper arm, forearm, hand, thigh, calf, foot, a total of 19 segments, and each segment is fixed with an IMU sensor using tape or a strap to form an IMU acquisition system. The specific sensor model and measurement equipment are not limited.

[0043] Schematic diagram of the attachment of plantar pressure insoles and inertial measurement units to the human body. Figure 1 shown.

[0044] The data from the inertial measurement unit and the plantar pressure distribution data can be transmitted in real time after the signal is acquired, or stored in a built-in data storage package and then exported wirelessly or wired when the data is extracted.

[0045] Secondly, the multi-source information obtained above is combined with the general human skeleton model for adaptation and adjustment, so as to obtain the human skeleton model to be tested. The specific adaptation and adjustment includes:

[0046] (21) Using the anthropometric information, the universal human skeleton model is scaled to obtain the human skeleton model to be tested.

[0047] (22) The plantar pressure distribution data is used to calculate the center of action and the concentrated force of the ground reaction force, and the corresponding center of action and concentrated force are used as contact points and contact forces to associate with the human skeleton model to be tested.

[0048] (23) According to the actual inertial measurement unit attachment position, a virtual inertial measurement unit is defined and rigidly connected to the human skeleton model to be tested.

[0049] In the above process, the general human skeleton model can be constructed in musculoskeletal simulation modeling software or existing ones, such as the open source musculoskeletal modeling software OpenSim and its database, which provide many human musculoskeletal models. The selected model must contain at least the human skeleton segment to which the inertial measurement unit is attached.

[0050] Third, the obtained multi-source information and the human skeleton model to be tested are used as input data for the following calculations.

[0051] S31, define the decision variable as the joint motion parameter A of the human body, where

[0052] q is the generalized position matrix of the monitored joint, and the elements of q include the displacement and angle of the joint movement. The range of the displacement is [-0.02, 0.02], and the unit is meter; the range of the angle is [-π, π], and the unit is radian.

[0053] is the first-order derivative of q with respect to time t, indicating the rotation speed of the joint, including linear velocity and angular velocity, It is the second-order derivative of q with respect to time t, which represents the joint rotation acceleration, including linear acceleration and angular acceleration.

[0054] S32. Establish an objective function to be optimized.

[0055] The objective function is, on the one hand, the error in the association between the variable to be solved and the actual information source, and on the other hand, the total or partial metric calculation value related to energy metabolism during joint movement. The metric calculation value can be the physiological mechanical energy of exercise, the metabolic rate, or other metric calculation methods for human energy conversion or consumption.

[0056] In the measurement calculation, the chemical energy contained in the food used by the human body and the heat energy finally converted and the external work done obey the law of conservation of energy. In the measurement calculation, all energy can be taken into account, such as measuring the heat generated and the external work done, and calculating the energy metabolism rate. The trend change of the energy metabolism rate is the same as the trend change of the energy metabolism rate in the joint movement process. Therefore, the second term of the objective function can be specifically the overall energy metabolism rate of human body function. In the measurement calculation, only the work done by the joint force in the joint movement process can be calculated, that is, the measurement calculation is the partial calculation related to energy metabolism in the joint movement process of the human body, and the heat required by the human body in this process is not calculated.

[0057] Specifically, the objective function can be expressed in the following form:

[0058]

[0059] Where:

[0060] θ IMU and are the position vectors of the simulated and actually measured inertial measurement units, respectively. The inertial measurement unit is approximately rigidly connected to the human body segment and is a function of one of the joint kinematic parameters q in the model, which can be obtained by default from the musculoskeletal software model. IMU The simulation value of can be calculated by the following formula:

[0061] θ IMU (q) = D·q

[0062] Where: D represents the coordinate transformation relationship from the local three-dimensional rectangular coordinate system of the bone segment to which the IMU is attached in the model to the local three-dimensional rectangular coordinate system of the IMU itself, that is, D is the coordinate transformation matrix.

[0063] t1 is the start time of the movement, t2 is the end time of the movement, and during the time t from the initial time t1 to the end time t2 of the movement, the first term of the objective function is used to calculate the cumulative error between the simulated and measured inertial measurement unit position variables.

[0064] The first term of the objective function is f energy It is all or part of the calculation formula related to energy metabolism during human joint movement.

[0065] When measuring and calculating the way of energy conversion or consumption of human body, taking physiological mechanical energy as an example, that is, the work done by joint force in the process of joint movement, the parameters in the above objective function also need to satisfy the dynamic balance of bones under the conditions of Newton's second law:

[0066]

[0067] M is the generalized force vector composed of force component and torque component, G is the gravity vector, and C is the Coriolis force and centrifugal force vector. PE The generalized passive force vector generated by the joint stiffness set in the human skeleton model to be tested. external is the generalized external force vector. Act is the generalized active force vector that drives the joint motion, including force and torque, generated by muscle tension. Optionally, is the sum of the squares of the active generalized forces of all joint degrees of freedom, including forces and moments.

[0068] In the above objective function solving process, the joint motion parameters are obtained by making the objective function obtain the minimum value, thereby realizing the joint motion monitoring.

[0069] The solution method can be the variational method or the collocation method. Taking the collocation method as an example, since the instantaneous joint position variable q and the velocity variable Acceleration variable The three satisfy the particle motion equation and have the following differential relationship, namely:

[0070]

[0071]

[0072] In the integral calculation, the continuous time curve is discretized into a finite time series, and q, the velocity variable, is further obtained. Acceleration variable The relationship between the three at two configuration points, that is, by constraining the instantaneous position and related position variables to follow the laws of particle motion, the dynamic consistency of the global motion is achieved, so that the above objective function can be solved.

[0073] The significance of the above objective function is to minimize the cumulative error between the simulated IMU position variables and the measured IMU position variables while minimizing the physiological mechanical energy. The physiological significance of minimizing the physiological mechanical energy is that the active joint force of the human body is generated by muscle contraction. Since human movement generally follows the "economic optimal principle", that is, people will subconsciously choose the most labor-saving mode to move during normal exercise, which is converted into a mathematical formula, that is, minimizing the sum of the squares of the generalized active joint forces in the inverse dynamics calculation, which is equivalent to the minimum mechanical energy.

[0074] The schematic diagram of the whole method process is as follows Figure 2 shown.

[0075] In the method of the present invention, the human skeleton model to be tested is obtained by adapting and adjusting the universal human skeleton model using the multi-source information of the testee, so that the modeling calculation behind the model is closer to the individual characteristics of the testee. Based on the actual inertial measurement unit information, the human skeleton model to be tested is attached in the same position, and the cumulative error of the real position and the simulated position is taken into account, and it is optimized as a part of the objective function, and the human physiological motion criterion is incorporated into the optimization calculation, and the constraints such as mechanical information, human dynamic balance and global dynamic consistency are added to compensate for the position information, so as to compensate for the uncertainty and error of the human skeleton model caused by factors such as inertial measurement unit sensor error, mechanical joint assumption, and skin deformation. Moreover, the method of the present invention removes the limitation of motion monitoring scene, and can realize high-precision human motion monitoring with physiological significance in daily sports environment without arranging the detection environment, which has great practical value.

[0076] In Example 2, a human motion monitoring device based on multi-source information fusion is used to monitor human motion. The device includes an inertial measurement unit, a plantar pressure insole, a memory and a processor; the memory stores data collected by the inertial measurement unit and the plantar pressure insole, and a computer program that can be loaded by the processor and execute the method in Example 1; the inertial measurement unit is used to be attached to the body of the test subject to record the motion trajectory of the inertial measurement unit; the plantar pressure insole is used to obtain plantar pressure distribution data. In order to facilitate the storage of data collected by the inertial measurement unit and the plantar pressure insole for calculation and use. The device also includes a data storage package, which is transmitted and exported by wired or wireless means when data calculation is required. The above device is simple and easy to implement and operate, and can realize data collection and joint motion monitoring in daily exercise environment. Through relevant calculations, high-precision monitoring results of human motion with physiological significance can be obtained, which has strong practicality.

[0077] Through the description of the above implementation modes, those skilled in the art can clearly understand that the method and / or device of the present disclosure can be implemented by means of software plus necessary general hardware, and of course can also be implemented by means of dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present disclosure, software program implementation is a better implementation mode in most cases.

[0078] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A human motion monitoring method based on multi-source information fusion, It is characterized in that The method comprises the following steps: S1. Acquire multi-source information of the test subject, wherein the multi-source information includes anthropometric information, plantar pressure distribution data, actual inertial measurement unit attachment position and collected data; S2. Based on multi-source information, the universal human skeleton model is adapted and adjusted to obtain a human skeleton model to be tested; S3, using the human skeleton model to be tested and the multi-source information, establishing an objective function with joint motion parameters as decision variables, and obtaining the joint motion parameters by minimizing the objective function; The objective function includes the sum of the first term and the second term; The first item is used to measure the position error between the human skeleton model to be tested and the person being tested with respect to the inertial measurement unit; The second item is used to measure all or part of the calculations related to energy metabolism during human joint movement; The objective function is as follows: Where: A is the joint motion parameter, q is the generalized position vector of the monitored joint, is the first-order derivative of q with respect to time t; is the second-order derivative of q with respect to time t; The elements of q include the displacement and angle of the joint movement, where the displacement ranges from [-0.02, 0.02] in meters, the angle ranges from [-π, π] in radians, and θ IMU and are the angle vectors of the inertial measurement unit obtained by simulation and actual measurement respectively; t 1 is the start time of the movement, t 2 is the end time of the movement; M is the generalized force vector composed of force component and torque component, G is the gravity vector, C is the Coriolis force and centrifugal force vector; F Act is the generalized active force vector driving the joint motion; F PE The generalized passive force vector generated by the joint stiffness set in the human skeleton model to be tested; F external is the generalized external force vector; f energy It is all or part of the calculation formula related to energy metabolism during human joint movement.

2. The method according to claim 1, It is characterized in that The human skeleton model to be tested is obtained by the following steps: S21, scaling the universal human skeleton model using the anthropometric information to obtain a human skeleton model to be tested; S22, using the plantar pressure distribution data, calculating the action center position and action concentration force of the ground reaction force, and associating the corresponding action center position and action concentration force as contact points and contact forces to the human skeleton model to be tested; S23. According to the actual inertial measurement unit attachment position, define a virtual inertial measurement unit and rigidly connect it to the human skeleton model to be measured.

3. The method according to claim 1, It is characterized in that The attachment positions of the inertial measurement unit include the following positions: head, neck, thoracic spine, lumbar spine, pelvis, upper arm, forearm, hand, thigh, calf, and foot.

4. The method according to claim 1, It is characterized in that The joints include shoulder joints, elbow joints, wrist joints, hip joints, knee joints, ankle joints, cervical vertebrae joints, and lumbar vertebrae joints.

5. The method according to claim 1, It is characterized in that The anthropometric information includes: geometric lengths of limbs and trunk, height, and weight.

6. The method according to claim 1, It is characterized in that The second item includes exercise physiological mechanical energy and metabolic rate.

7. The method according to claim 1, It is characterized in that The plantar pressure distribution data is obtained through a plantar pressure insole.

8. A human motion monitoring device based on multi-source information fusion, It is characterized in that The device includes an inertial measurement unit, a plantar pressure insole, a memory, and a processor; The memory stores data collected by the inertial measurement unit and the plantar pressure insole, and a computer program that can be loaded by the processor and execute any method according to claim 1 to 6; The inertial measurement unit is used to be attached to the body of the test subject and is used to record the motion trajectory of the inertial measurement unit; The plantar pressure insole is used to obtain plantar pressure distribution data.

9. The device according to claim 8, It is characterized in that The device also includes a data storage package for storing real-time data collected by the inertial measurement unit and the plantar pressure insole, and then transmitting and exporting the data in a wired or wireless manner.

Images