A multi-sensor based gait event prediction method and system

Abstract

The present application belongs to the technical field of pattern recognition, and discloses a gait event prediction method and system based on multiple sensors. The method comprises the following steps: S1, collecting electromyographic signals and IMU signals corresponding to different walking speeds to construct a database; S2, identifying the actual heel strike, foot flat and toe-off time, the predicted feature point of the heel strike and the predicted feature point of the toe-off; S3, calculating the time difference between the predicted feature point of the heel strike and the actual heel strike time, and the time difference between the predicted feature point of the toe-off and the actual toe-off time, and establishing a regression model of the time difference and the walking speed; S4, real-time estimation of the walking speed of the IMU signal to be predicted; S5, solving the time difference by using the real-time walking speed, determining the actual heel strike point and the actual toe-off feature point, and realizing the prediction process. Through the present application, the problem of delay caused by gait event recognition based on wearable sensors is solved.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, and more specifically, relates to a gait event prediction method and system based on multiple sensors. Background Technology

[0002] Lower limb exoskeletons can track the wearer's leg movements, thereby restoring or enhancing lower limb motor function. Accurately identifying human movement intentions can determine the future state of the exoskeleton, which is crucial for its control. The human gait cycle can be divided into multiple discrete phases based on gait events. Key gait events, such as heel strike and toe lift, mark the beginning of the stance and swing phases, respectively, and provide information such as gait cycle, stance phase time, and swing phase time. These timing parameters are important assessment parameters for lower limb movement.

[0003] Force plates and high-speed cameras are often used as the gold standard for determining gait events, but they are typically limited to laboratory environments. Pressure-sensitive switches are popular due to their ability to acquire on / off signals and their suitability for outdoor environments; however, they are prone to mechanical failure and can cause discomfort with prolonged use. Electromyography (EMG) signals hold great potential in detecting motor intentions because they detect activation prior to changes in muscle-tendon length. Current research using EMG signals largely employs machine learning algorithms, extracting EMG signal features to decode lower limb movements. However, threshold-based methods are actually simpler, more efficient, and better suited for real-time systems. Gait event recognition using inertial measurement units (IMUs) has been implemented. IMUs are inexpensive, portable, and wearable. Furthermore, the rich information provided by IMUs can be used to estimate spatiotemporal parameters such as stride length and gait speed. Clearly, combining multiple sensors can provide a wealth of information that can be used to evaluate relevant gait parameters.

[0004] Current gait recognition algorithms focus on improving detection accuracy and reducing detection latency, but they cannot completely eliminate latency. Furthermore, exoskeleton control requires response time, which can lead to a mismatch between exoskeleton assistance and human movement. If gait events could be predicted in advance, this problem could be effectively solved. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides a gait event prediction method and system based on multiple sensors, which solves the problem of gait recognition and detection delay in the prior art.

[0006] To achieve the above objectives, according to one aspect of the present invention, a gait event prediction method based on multiple sensors is provided, characterized in that the method includes the following steps:

[0007] S1 sets multiple different walking speeds for the test subject and collects the electromyographic signals of the gastrocnemius muscle and the IMU signals of the calf at different walking speeds.

[0008] S2 uses IMU signals at different walking speeds to identify the actual heel strike, foot flat, and toe lift-off times, as well as predicted feature points of heel strike; and uses electromyography signals at the same walking speeds to identify predicted feature points of toe lift-off.

[0009] S3 calculates the time difference t1 between the predicted feature point of heel strike and the actual time of heel strike, and the time difference t2 between the predicted feature point of toe lift-off and the actual time of toe lift-off. The corresponding t1 and t2 at different walking speeds are obtained, and a regression model of t1 and t2 with walking speed is established.

[0010] S4 uses the IMU signal to be predicted to estimate the corresponding walking speed.

[0011] S5 substitutes the estimated walking speed obtained in step S4 into the regression model to solve for t1 and t2 corresponding to the IMU signal to be predicted, identifies the predicted feature points of heel strike and toe lift-off corresponding to the IMU signal and electromyography signal to be predicted, and calculates the actual heel strike time and actual toe lift-off time in the IMU signal and electromyography signal to be predicted, thus realizing the prediction process.

[0012] More preferably, in step S1, the IMU signal includes the angular velocity, acceleration, and quaternion signal of the lower leg movement.

[0013] More preferably, in step S2, the times when the heel touches the ground, the foot is level, and the toes leave the ground are identified in the following manner:

[0014] S21 identifies the minimum negative peak value of the angular velocity ω along the sagittal plane of the object in the IMU signal. The time corresponding to the negative peak value is the predicted feature point of the heel strike event. The time corresponding to the first positive peak value after the minimum negative peak value is the time when the heel strike occurs. The time corresponding to the first valley point searched after the heel strike time is set as T0.

[0015] S22 In the acceleration signal, the time corresponding to the first point below the preset acceleration threshold after time T0 is the flat time T1;

[0016] S23 In the angular velocity ω of the object in the sagittal plane, the first positive peak point after time T1 corresponds to the moment when the toes leave the ground.

[0017] More preferably, in step S2, the predicted feature point of toe lift-off is set in the following manner: First, the electromyographic signal is processed by TKEO; then, an electromyographic threshold is set in the falling segment of the TKEO-processed signal, and the time corresponding to the time when the toe lift-off is less than the set electromyographic threshold is the predicted feature point of toe lift-off.

[0018] More preferably, the TEKO operation is performed as follows: first, the electromyographic signal is filtered using a bandpass filter; then, the TKEO operator is extracted from the bandpass-filtered signal; and finally, a low-pass filter is applied. The extraction of the TKEO operator is performed according to the following formula:

[0019] y(n)=x(n) 2 -x(n-1)x(n+1)

[0020] Where x(n) is the electromyographic signal, n is the number of samples, and y(n) is the obtained EMG energy value, which is used for subsequent threshold determination.

[0021] More preferably, the electromyography threshold is set according to the following formula:

[0022] y thr =μ+hσ

[0023] Where μ is the mean, σ is the standard deviation, h is a preset variable defining the threshold level, and y thr It is the electromyographic threshold.

[0024] More preferably, in step S3, the regression model is obtained by fitting using the least squares method.

[0025] More preferably, in step S3, the regression model is performed as follows:

[0026] t1 = av 2 +bv+c

[0027] t2 = mv + n

[0028] Where t1 is the time before the heel strike prediction feature point, t2 is the time before the toe-off prediction feature point, a, b, and c are quadratic regression curve coefficients, m and n are linear regression coefficients, and v is walking speed.

[0029] More preferably, in step S4, the walking speed is calculated according to the following formula:

[0030] At the moment of foot flatness: v0=ω0×r

[0031] Between two foot flat moments:

[0032]

[0033] Where FF(i) is the foot leveling moment in the i-th gait cycle, FF(i+1) is the foot leveling moment in the (i+1)-th gait cycle, and p L FF(i) is the foot trajectory during the i-th gait cycle, p L FF(i+1) is the foot trajectory during the (i+1)th gait cycle.

[0034] According to another aspect of the present invention, a system for predicting gait events according to the multi-sensor-based gait event prediction method described above is provided. This system includes a data acquisition module, a data analysis module, a model building module, and a calculation module, wherein...

[0035] The data acquisition module is used to acquire electromyographic signals and IMU signals;

[0036] The analysis module is used to determine the actual times of heel strike, foot flatness, and toe liftoff, as well as the predicted feature points of heel strike and toe liftoff.

[0037] The model building module is used to build regression models;

[0038] The calculation module is used to calculate the walking speed using IMU signals, the time difference t1 between the predicted feature point of heel strike and the actual heel strike time, and the time difference t2 between the predicted feature point of toe liftoff and the actual toe liftoff time, and to calculate the actual heel position and the actual toe liftoff feature point based on the predicted feature points.

[0039] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0040] 1. This invention can predict heel strike and toe lift events using only a single IMU and a single electromyography sensor, and can predict the actual heel strike time and toe lift time after the time difference. This reduces the delay caused by real-time or delayed monitoring of heel strike and toe lift time in the prior art, provides advance preparation time for the response of other machines, better matches human-machine movement, and solves the problem of delay caused by gait event recognition based on wearable sensors.

[0041] 2. This invention identifies the actual times of heel strike, foot flattening, and toe lift-off by performing signal recognition on IMU and electromyographic signals, based on the periodicity of the walking process and the characteristics of each moment. It also predicts the feature points of heel strike and toe lift-off. This method is simple, fast, and efficient.

[0042] 3. This invention employs a regression model, which shows a significant linear correlation between walking speed and gait cycle events. By linking gait event prediction with walking speed through the regression model, the prediction of gait events can be effectively achieved.

[0043] 4. The method provided by this invention has low computational complexity and can be applied to online systems in real time, which has practical value for realizing the coordinated matching of lower limb exoskeleton and human movement. Attached Figure Description

[0044] Figure 1 This is a flowchart of a multi-sensor-based gait event prediction method constructed according to a preferred embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a single inertial sensor (IMU) and a single electromyography sensor acquiring data according to a preferred embodiment of the present invention;

[0046] Figure 3 It is a regression model constructed according to a preferred embodiment of the present invention, which is a prediction feature point of heel strike and the time difference t1 of the actual heel strike event with respect to walking speed.

[0047] Figure 4 It is a regression model constructed according to a preferred embodiment of the present invention, which is a prediction feature point of toe lift-off and the time difference t2 of the actual heel strike event with respect to walking speed;

[0048] Figure 5 This is an example process for identifying gait events and predicting feature points using human sagittal plane angular velocity, resultant acceleration, and EMG energy values, constructed according to a preferred embodiment of the present invention, wherein (a) is the curve of angular velocity versus time, (b) is the curve of acceleration versus time, and (c) is the curve of EMG energy values ​​versus time. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0050] like Figure 1 As shown, a gait event prediction method and system based on multiple sensors is presented. The method includes the following steps:

[0051] S1: See Figure 2In this embodiment, a single inertial sensor (IMU) and a single electromyographic sensor are used. Electromyographic signals of the gastrocnemius muscle are collected by electromyographic electrodes attached to the gastrocnemius muscle, and angular velocities, accelerations and quaternion signals of the x, y and z axes are collected by an IMU installed on the lateral side of the lower leg. The z axis of the IMU is aligned with the long axis of the tibia.

[0052] In this embodiment, the IMU sampling frequency is set to 100Hz, and the electromyography sensor sampling frequency is set to 1kHz.

[0053] S2: Gait Events and Feature Point Detection

[0054] IMU signals are combined with finite state machines to perform threshold judgment to identify heel strike, foot flat, and toe lift events; and predict feature points for heel strike events; the collected electromyography signals are processed by TKEO, and the muscle cessation and activation time is determined by a pre-set threshold to determine the toe lift prediction feature point.

[0055] This invention divides a complete gait cycle into 3 gait phases, which are divided by 3 gait events, including heel strike event, foot flat event, and toe lift-off event.

[0056] IMU signals are combined with a finite state machine to perform threshold judgment to identify heel strike, foot flat, and toe lift events, as well as predicted feature points for heel strike events. The acquired electromyographic (EMG) signals are processed using the Teager-Kaiser Energy Operator (TKEO), and the muscle's deactivation point is determined by using a pre-set threshold to identify the predicted toe lift feature point. The rising phase of the curve represents the muscle activation phase, and the falling phase represents the muscle deactivation phase. EMG threshold judgment is used to determine the predicted toe lift feature point during the muscle deactivation phase.

[0057] The finite state machine settings for the IMU are as follows:

[0058] The IMU (Integrated Measurement Unit) acquires the angular velocity ω of the target object in the sagittal plane in real time. A time window containing three sample points is used. First, negative peaks are identified, corresponding to predicted feature points of heel strike events. Then, positive peaks of ω are identified; the first positive peak corresponds to a heel strike event. For flat-foot events, the algorithm searches for valleys in ω after a period of time (200ms) following the detection of a heel strike event. When a valley appears and the corresponding angular velocity is below 80 rad / s, the algorithm searches for the resultant acceleration, i.e., the root mean square value of the triaxial acceleration. When the resultant acceleration is less than the gravitational acceleration g for three consecutive sampling points, it corresponds to a flat-foot event. The second positive peak after this corresponds to a toe-off event.

[0059] The process of TKEO processing of electromyographic signals includes:

[0060] In an embodiment of the present invention, the electromyography signal is filtered using a fourth-order Butterworth bandpass filter with a bandpass frequency of 20-350Hz, then the TKEO operator is extracted from the signal, and finally filtered using a fourth-order Butterworth low-pass filter with a cutoff frequency of 10Hz. The calculation formula for the TKEO processing is as follows:

[0061] y(n)=x(n) 2 -x(n-1)x(n+1)

[0062] Where x(n) is the electromyographic signal and n is the number of samples;

[0063] Methods for determining the cessation of gastrocnemius muscle activation, with electromyographic thresholds differentiated individually based on different people, include:

[0064] Electromyographic signals were captured during the swing phase of a single walk, and the electromyographic threshold was set as follows:

[0065] y thr =μ+hσ;

[0066] Where μ is the mean, σ is the standard deviation, and h is a preset variable that defines the threshold level.

[0067] S3: Establish a regression model

[0068] Regression models were established by comparing the predicted feature points of heel strike and toe lift with the time difference and gait speed of the heel strike and toe lift events identified by the IMU.

[0069] Experiments revealed a significant correlation between the timing of feature points preceding gait events recognized by the IMU and gait speed. Therefore, a regression model was established using the least squares method to model the timing of these events and gait speed:

[0070] t1 = av 2 +bv+c

[0071] t2 = mv + n

[0072] Where t1 is the time before the heel strike prediction feature point, t2 is the time before the toe-off prediction feature point, a, b, c are the coefficients of the quadratic regression curve, m, n are the coefficients of the linear regression curve, and v is the walking speed.

[0073] S4: Walking speed estimation

[0074] Based on the inverted pendulum model, walking speed is estimated using an IMU located at the lower leg. The steps include:

[0075] At the moment of foot flatness, the motion of the lower leg in the sagittal plane is considered as circular motion. In this case, the angular velocity can be used to estimate the lower leg velocity, i.e.:

[0076] v0=ω0×r

[0077] Where ω0 is the angular velocity at the moment of foot flatness, and r is the distance from the IMU mounting point to the ankle.

[0078] Using this as the initial velocity value, before the next foot flat event occurs, the acceleration is integrated twice to estimate the displacement during this process, and finally divided by the gait period time to obtain the gait speed.

[0079] Two coordinate systems are used here: the IMU coordinate system (C) and the world coordinate system (L). Based on the inverted pendulum model, the velocity v at the moment of foot-flattening event can be estimated from the angular velocity. L :

[0080]

[0081] Where × represents the cross product, FF(i) is the foot flat event in the i-th gait cycle, r(k) is the magnitude of the distance from the IMU mounting location to the ankle, and its direction is the same as the acceleration, while ω L The three-axis angular velocities in the world coordinate system are calculated by the following formula:

[0082]

[0083] in Represents quaternion multiplication, q CL (k) represents the instantaneous rotation quaternion that transforms the vector from IMU coordinate system C to world coordinate system L, (q) CL (k)) * It is q CL (k) conjugate.

[0084] The velocity is calculated using the trapezoidal integral method with a sampling interval of Δt. The calculated velocity is:

[0085]

[0086] in Let t be the acceleration in the world coordinate system. i This represents the time elapsed in step i.

[0087] Next, the foot trajectory p within the gait cycle L Through the following calculations:

[0088]

[0089] Where p L (t i ) is t i The position of the foot trajectory at a given moment in time.

[0090] The walking speed during the i-th gait cycle is calculated as follows:

[0091]

[0092] S5: Gait Event Prediction

[0093] Based on the aforementioned prediction model and gait estimation algorithm, the current gait speed is input into the pre-regressed gait event prediction model at the predicted feature points of heel strike and toe lift-off, thus obtaining the prediction of the occurrence time of the two gait events.

[0094] like Figure 3 As shown, typical data collected from a subject, at different walking speeds, shows that the predicted heel strike point precedes the actual event by a time t1. A regression model is obtained through regression fitting, which can then be used to predict heel strike events.

[0095] like Figure 4 As shown, typical data collected from a subject, at different walking speeds, shows that the predicted toe-off point occurs before the actual event by a time t2. After regression fitting, a regression model () is obtained, which can then be used to predict heel-off events.

[0096] like Figure 5 As shown, an example of the identification process using a finite state machine is illustrated. By analyzing the angular velocity, resultant acceleration, and EMG energy signals using the methods described in the aforementioned finite state machine, the gait events of heel strike, toe lift, and flat gait, as well as the predicted feature points of heel strike and toe lift, are determined.

[0097] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A gait event prediction method based on multiple sensors, characterized in that, The method includes the following steps: S1 sets multiple different walking speeds for the test subject and collects the electromyographic signals of the gastrocnemius muscle and the IMU signals of the calf at different walking speeds. S2 uses the IMU signals at different walking speeds to identify the actual heel strike, foot flat, and toe lift-off times, as well as the predicted feature points of heel strike; and uses the electromyography signals at the same walking speeds to identify the predicted feature points of toe lift-off. S3 calculates the time difference t1 between the predicted feature point of heel strike and the actual time of heel strike, and the time difference t2 between the predicted feature point of toe lift-off and the actual time of toe lift-off. The corresponding t1 and t2 at different walking speeds are obtained, and a regression model of t1 and t2 with walking speed is established. S4 For the IMU signal to be predicted, estimate its corresponding walking speed using the IMU signal to be predicted; S5 Substitutes the estimated walking speed obtained in step S4 into the regression model to solve for t1 and t2 corresponding to the IMU signal to be predicted. Identifies the predicted feature points of heel strike and toe lift-off corresponding to the IMU signal and electromyography signal to be predicted. Combines t1 and t2 corresponding to the IMU signal to be predicted to calculate the actual heel strike time and actual toe lift-off time in the IMU signal and electromyography signal to be predicted, thus realizing the prediction process.

2. The gait event prediction method based on multiple sensors as described in claim 1, characterized in that, In step S1, the IMU signal includes the angular velocity, acceleration, and quaternion signal of the lower leg movement.

3. The gait event prediction method based on multiple sensors as described in claim 2, characterized in that, In step S2, the times when the heel touches the ground, the foot is level, and the toes leave the ground are identified in the following manner: S21 For the angular velocity ω along the sagittal plane of the acquired object in the IMU signal, identify the minimum negative peak value in the angular velocity. The time corresponding to the negative peak value is the predicted feature point of the heel strike event. The time corresponding to the first positive peak value after the minimum negative peak value is the time when the heel strike occurs. Set the time corresponding to the first valley point searched after the heel strike time as T0. S22 In the acceleration signal, the time corresponding to the first point below the preset acceleration threshold after time T0 is the flat time T1; S23 In the angular velocity ω of the object in the sagittal plane, the first positive peak point after time T1 corresponds to the moment when the toes leave the ground.

4. A gait event prediction method based on multiple sensors as described in claim 1 or 3, characterized in that, In step S2, the predicted feature point of toe lift-off is set in the following manner: First, the electromyographic signal is processed by TKEO; then, an electromyographic threshold is set in the falling segment of the TKEO-processed signal, and the time corresponding to the time when the toe lift-off is less than the set electromyographic threshold is the predicted feature point of toe lift-off.

5. The gait event prediction method based on multiple sensors as described in claim 4, characterized in that, The TEKO process is performed as follows: first, the electromyographic signal is filtered using a bandpass filter; then, the TKEO operator is extracted from the bandpass-filtered signal; and finally, a low-pass filter is applied. The extraction of the TKEO operator is performed according to the following formula: Where x(n) is the electromyographic signal, and n is the number of samples. This is the obtained EMG energy value, used for subsequent threshold determination.

6. The gait event prediction method based on multiple sensors as described in claim 5, characterized in that, The electromyography threshold is set according to the following formula: in, μ The mean, σ Standard deviation, h It is a preset variable that defines the threshold level. It is the electromyographic threshold.

7. The gait event prediction method based on multiple sensors as described in claim 1, characterized in that, In step S3, the regression model is obtained by fitting using the least squares method.

8. The gait event prediction method based on multiple sensors as described in claim 7, characterized in that, In step S3, the regression model is performed as follows: in, t 1 indicates that the predicted feature point for heel strike precedes the heel strike event. t 2. The predicted feature point for toe-off event precedes the toe-off event. a , b , c These are all coefficients of a quadratic regression curve. m , n These are linear regression coefficients. v That's walking speed.

9. A gait event prediction method based on multiple sensors as described in claim 1, characterized in that, In step S4, the walking speed is calculated according to the following formula: Foot flat moment: Between two foot flat moments: in, v ( i ) is the walking speed during the i-th gait cycle. FF ( i () represents the foot leveling moment in the i-th gait cycle. FF ( i+ 1 ) represents the foot leveling moment in the (i+1)th gait cycle, p L ( FF ( i )) is the foot trajectory during the i-th gait cycle, p L ( FF ( i+ 1 )) is the foot trajectory during the (i+1)th gait cycle.

10. A system for predicting gait events according to any one of claims 1-9 based on a multi-sensor method, characterized in that, The system includes a data acquisition module, a data analysis module, a model building module, and a computing module. The data acquisition module is used to acquire electromyographic signals and IMU signals; The data analysis module is used to determine the actual times of heel strike, foot flatness, and toe lift-off, as well as the predicted feature points of heel strike and toe lift-off. The model building module is used to build regression models; The calculation module is used to calculate the walking speed using IMU signals, the time difference t1 between the predicted feature point of heel strike and the actual heel strike time, and the time difference t2 between the predicted feature point of toe liftoff and the actual toe liftoff time, and to calculate the actual heel position and the actual toe liftoff feature point based on the predicted feature points.

Images