Gait event detection method based on built-in sensor and deep learning of smart phone

Abstract

According to the gait event detection method based on the built-in sensor of the smart phone and deep learning, diversified gait features can be accurately understood and recognized, high efficiency and convenience are achieved, interference on real gait information is effectively reduced, a more reliable gait event detection result is ensured, and the gait event detection efficiency is improved. And a widely used tool is provided for gait analysis in a non-laboratory environment. The method comprises the following steps: collecting real-time acceleration, angular velocity and equipment rotation angle data generated during walking through a built-in sensor of the smart phone; noise in the original data collected by the sensor is removed; excluding sensor data with small fluctuation amplitude; constructing an input and output data pair by using a sliding window mode; modernTCN is used as a basic model architecture, and a bidirectional gating cycle unit BiGRU and a cross attention mechanism are introduced; training a model by using the constructed data set; and inputting real-time data collected by a built-in sensor of the smart phone into the trained model, and outputting a corresponding gait event detection result by the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile health monitoring, and in particular relates to a gait event detection method based on built-in sensors of a smart phone and deep learning. Background Art

[0002] As a potential non-invasive biomarker, gait reflects the movement characteristics of an individual. Studies have found that the spatiotemporal parameters of gait can reflect an individual's health status, cognitive level, risk of falling, and quality of life, thus playing an important role in clinical diagnosis and health assessment. Gait analysis is an examination method for studying walking patterns. It aims to calculate, analyze, and interpret measurable gait parameters through biomechanical and kinematic means, thereby guiding rehabilitation assessment and treatment, and helping clinical diagnosis, efficacy evaluation, and mechanism research. Traditional gait analysis technology mainly relies on vision, sensors, and other combined / hybrid methods. Motion capture systems and pressure sensors are usually used as gold standard equipment. Although they can provide professional results, they usually require a lot of costs and auxiliary personnel, and many users have difficulty in accessing them.

[0003] The key to gait analysis is to detect gait events, such as heel strike and toe off. Common gait event detection methods are based on threshold and peak detection algorithms to identify gait phases. However, due to individual gait variations and physical defects, these algorithms have certain limitations when processing noisy data, such as sensitivity to noise interference, poor model robustness, and complex data acquisition and preprocessing. Summary of the invention

[0004] The present invention provides a gait event detection method based on built-in sensors of smartphones and deep learning. The method obtains real-time sensor data of human body movement through the built-in sensors of smartphones, processes the raw data and constructs a data set, and trains a deep learning model to detect gait events. The method can accurately understand and identify diverse gait features, is efficient and convenient, and can effectively reduce interference with real gait information within an allowable range, ensuring more reliable gait event detection results, thereby providing a widely used tool for gait analysis in non-laboratory environments.

[0005] The present invention provides a gait event detection method based on a smartphone built-in sensor and deep learning, the steps of which include:

[0006] Raw data collection: The real-time acceleration, angular velocity, and device rotation angle data generated during walking are collected through the built-in sensors of the smartphone as the basis for gait event detection;

[0007] Data processing: removing noise from the raw data collected by sensors;

[0008] Feature extraction: exclude sensor data with small fluctuations;

[0009] Constructing a data set: Use a sliding window approach to construct input and output data pairs as sample data for model training;

[0010] Model building: ModernTCN is used as the basic model architecture, and the bidirectional gated recurrent unit BiGRU and cross attention mechanism are introduced;

[0011] Model training: Use the constructed dataset to train the established model;

[0012] Gait event detection: The real-time data collected by the smartphone’s built-in sensors is input into the trained model, and the model outputs the corresponding gait event detection results.

[0013] Based on the above scheme, real-time gait data is collected using the built-in acceleration, angular velocity, and rotation angle sensors of smartphones, achieving efficient and convenient gait event detection. After removing the noise in the original data and excluding small fluctuations, a data set containing a sliding window is constructed to provide the model for training, ensuring that the model can accurately understand and identify diverse gait features. The model design based on ModernTCN combines the bidirectional gated recurrent unit BiGRU and the cross-attention mechanism, introducing time series continuity memory and multi-information fusion processing capabilities for gait event detection. The bidirectional information transmission characteristics of BiGRU give the model better scene understanding capabilities, while the cross-attention mechanism can accurately locate important information in multiple features, further improving the model's recognition accuracy of gait details, and can effectively reduce interference with real gait information within the allowable range to ensure more reliable gait event detection results.

[0014] According to one embodiment of the present invention, in the data processing, Kalman filter estimation and low-pass filter are used to remove noise in the original data.

[0015] According to one embodiment of the present invention, constructing a data set specifically includes:

[0016] a. Set the sliding window size;

[0017] b. Input sample construction: The data in each window is taken as an input sample, which contains multi-dimensional features of acceleration, angular velocity and device rotation angle after excluding data with smaller fluctuation amplitude;

[0018] c. Output label construction: Based on the reference standard gait events detected by the pressure insole, the gait event type of the corresponding sampling point of the smartphone is marked.

[0019] According to one embodiment of the present invention, the model building specifically includes:

[0020] The variable dimension is retained in the ModernTCN input layer, and a shard-based variable-independent embedding method is adopted;

[0021] Each module in the ModernTCN backbone network contains a depthwise separable convolution DWConv, two point-wise convolutional feedforward networks ConvFFN1 and ConvFFN2;

[0022] The bidirectional gated recurrent unit BiGRU is introduced after the ModernTCN backbone network;

[0023] Introduce a cross-attention mechanism after the bidirectional gated recurrent unit BiGRU;

[0024] The output of the cross-attention mechanism is used as the final time series representation.

[0025] According to one embodiment of the present invention, the steps further include:

[0026] Model evaluation: The time difference between the gait events detected by the model and the reference standard gait events is taken as the error to evaluate the effect of the model.

[0027] The present invention also provides a gait event detection system based on a smartphone built-in sensor and deep learning, comprising:

[0028] A data collection device is used to collect real-time walking acceleration, angular velocity, and equipment rotation angle data;

[0029] A data processing module to remove data noise;

[0030] Feature extraction and data set construction module, used to filter data features and construct input and output data pairs;

[0031] A deep learning model is used to detect gait events after training; and a display module is used to display gait event detection results.

[0032] According to one embodiment of the present invention, the data collection device comprises:

[0033] A smartphone device having a built-in acceleration sensor, angular velocity sensor, and orientation sensor;

[0034] A data acquisition module is used to collect three-axis acceleration data, three-axis angular velocity data, and rotation angle data of the mobile phone in three-dimensional space relative to the earth coordinate system in real time during walking through the built-in sensors of the smart phone device;

[0035] A wearable device used to fix a smartphone to the user's body.

[0036] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the gait event detection method of the above embodiment is implemented.

[0037] The present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the gait event detection method of the above embodiment is implemented.

[0038] The beneficial effects of the present invention include:

[0039] (1) In terms of data collection, the present invention utilizes the built-in sensors of smartphones to conveniently obtain real-time data of human motion. Compared with traditional motion capture systems and professional equipment such as pressure sensors, smartphones are more portable, easy to use, and low-cost;

[0040] (2) In terms of data processing, the present invention adopts a method that combines Kalman filtering and low-pass filtering to effectively remove noise interference in the original sensor data; at the same time, through feature extraction, sensor signals with small changes are eliminated, data redundancy is reduced, and the complexity of model calculation is reduced.

[0041] (3) In terms of model design, this paper takes ModernTCN as the basic architecture and introduces BiGRU and cross-attention mechanisms to further enhance the model's ability to capture temporal dependencies and its ability to adaptively aggregate key features.

[0042] (4) In terms of model evaluation, the present invention uses the time difference between the model-detected gait events and the reference standard gait events as an evaluation indicator, which intuitively reflects the accuracy of the detection. By comparing the performance of different deep learning models at different walking speeds, the superiority and robustness of the model of the present invention are fully verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the position where the smartphone is worn;

[0044] Figure 2 Schematic diagram of raw sensor data collected for a smartphone

[0045] Figure 3 Schematic diagram of the sliding window method

[0046] Figure 4 Schematic diagram of the deep learning model structure

[0047] Figure 5 Schematic diagram of the results of detecting gait events for the deep learning model DETAILED DESCRIPTION

[0048] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] Example 1

[0051] The present invention obtains real-time sensor data of human body movement through the built-in sensors of smart phones, processes the raw data and constructs a data set, trains a deep learning model to detect gait events, and evaluates their time errors.

[0052] The specific steps are as follows:

[0053] Set the smartphone sensor sampling rate to 50 Hz.

[0054] 1. Data collection: The smartphone is fixed to the user's thigh with a Velcro elastic band. The smartphone collects the three-axis acceleration data Ax, Ay, Az and three-axis angular velocity data Gx, Gy, Gz generated by walking through the built-in sensor, as well as the rotation angles Alpha, Beta, Gamma of the smartphone around the ZXY axis relative to the earth coordinate system in three-dimensional space; at the same time, a force platform based on plantar pressure insoles (M3232L, Roxifsr, China) is used as a reference standard for gait event detection.

[0055] 2. Data processing: During the data collection stage, the raw sensor data may be interfered by noise due to factors such as body shaking and equipment deviation. Kalman filtering and low-pass filtering are used to remove the noise.

[0056] The state of the dynamic system is estimated by Kalman filtering, which is based on a linear Gaussian model and includes two steps: prediction and update. Assuming the sensor data is X, the formula of Kalman filtering is as follows:

[0057] Predict current status:

[0058]

[0059] Forecast error covariance:

[0060]

[0061] Update the Kalman gain:

[0062]

[0063] Update the current state estimate:

[0064]

[0065] Update the error covariance:

[0066]

[0067] in, is the current state estimate, is the current state of the prediction, P K is the current state estimation error covariance, is the predicted current state error covariance, A is the state transfer matrix, B is the control input matrix, μ k is the control input, Q is the process noise covariance, R is the measurement noise covariance, K K is the Kalman gain, Z k are the measurements, H is the measurement matrix, and I is the identity matrix.

[0068] The low-pass filter is used to remove high-frequency noise signals and retain low-frequency components. The formula is as follows:

[0069] Y[n]=αX[n]+(1-α)Y[n-1]#(6)

[0070] Where Y[n] is the nth value of the output after filtering, X[n] is the nth value of the output after filtering, and α is the smoothing factor of the filter.

[0071] 3. Feature extraction: Since the overall change amplitude of acceleration Az, angular velocity Gx, and rotation angle Beta is small during walking, these three features are excluded to reduce the computational complexity of the model.

[0072] 4. Construct a data set: Use a sliding window to construct input and output data pairs. The specific steps are as follows:

[0073] (1) Sliding window size setting: Set the sliding window length to 40.

[0074] (2) Input sample construction: The data in each window is taken as an input sample, which contains 6-dimensional features (Ax, Ay, Gy, Gz, Alpha, Gamma) of acceleration, angular velocity and rotation angle.

[0075] (3) Output label construction: Based on the reference standard gait events detected by the pressure insole, the gait event type of the corresponding sampling point of the smartphone is marked (1 for the support phase and 0 for the swing phase).

[0076] 5. Model building: Considering the characteristics of time series data, ModernTCN is selected as the basic model architecture. ModernTCN modernizes the traditional convolutional network and optimizes time series. While maintaining the efficiency of the convolutional model, it can also capture the time, features and variable dependencies in the time series data. On this basis, the bidirectional gated recurrent unit (BiGRU) is further introduced to improve the model's modeling ability for time series data. At the same time, the cross-attention mechanism is also used to enhance the model's capture of key features. The specific steps are as follows:

[0077] (1) In the input layer, ModernTCN retains the variable dimension, does not directly compress the variable dimension, and adopts a shard-based variable independent embedding method. That is, each univariate time series is divided into several shards and then embedded separately, instead of compressing all variables into one embedding vector.

[0078] (2) In the backbone network, each module of ModernTCN contains a deep separable convolution (DWConv) and two point-by-point convolutional feedforward networks (ConvFFN1 and ConvFFN2). DWConv performs convolution in the time dimension independently on each feature and variable to learn temporal dependencies; ConvFFN1 learns feature representation within each variable through grouped convolution, and ConvFFN2 learns the dependencies between variables within each feature through grouped convolution.

[0079] (3) Based on ModernTCN, the bidirectional gated recurrent unit (BiGRU) is introduced to further enhance the model’s ability to model temporal dependencies. The input of BiGRU is the output representation of the ModernTCN backbone network, and its output serves as the key and value input of the subsequent cross-attention mechanism.

[0080] (4) After BiGRU, a cross-attention mechanism is used, the output representation of the ModernTCN backbone network is used as the query vector, and the output of BiGRU is used as the key and value vector. The importance weights are dynamically assigned to the representations of different time steps through the attention mechanism, realizing the adaptive screening and aggregation of key information. The output of the cross-attention will be used as the final time series representation.

[0081] 6. Model evaluation: The time difference between the gait events detected by the deep learning model and the reference standard gait events is used as an indicator to evaluate the model effect. The smaller the time difference, the higher the accuracy of the model in detecting gait events. Table 1 shows the mean absolute error (MAE, unit: milliseconds) of the ModernTCN-BiGRU-CrossAttention model used in the present invention and the other three models in detecting gait events. The results show that the MAE of the ModernTCN-BiGRU-CrossAttention model in detecting heel strike and toe-off at three different walking speeds is smaller than that of the other three models.

[0082] Table 1 Average absolute error of gait events detected by the proposed model and other three models

[0083]

[0084] Based on the scheme of the present invention, walking data is collected in real time through the built-in sensors of the smart phone, and the Kalman filter estimation and low-pass filter and other technologies are used to effectively remove noise, thereby improving the accuracy and reliability of the data and providing a high-quality data source for subsequent analysis. By screening data with small fluctuations and constructing input and output data pairs based on sliding windows, not only redundant information is reduced, but also the sensitivity of the model to changes in gait events is enhanced. At the same time, the accuracy and reliability of the output label are ensured by combining the reference standard gait events provided by external devices such as pressure insoles. ModernTCN is used as the basic model architecture, and the bidirectional gated recurrent unit BiGRU and the cross attention mechanism are introduced. This combination not only makes full use of the advantages of the temporal convolutional network in time series processing, but also enhances the model's ability to capture contextual information through the bidirectional GRU, and the cross attention mechanism further enhances the model's ability to recognize complex gait events. The model effect is evaluated by comparing the time difference between the model detection gait event and the reference standard gait event, which intuitively reflects the accuracy of the detection and provides a clear direction and basis for further optimization of the model.

[0085] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A gait event detection method based on smartphone built-in sensors and deep learning, its characteristic steps include: Raw data collection: The real-time acceleration, angular velocity, and device rotation angle data generated during walking are collected through the built-in sensors of the smartphone as the basis for gait event detection; Data processing: removing noise from the raw data collected by sensors; feature Extraction: exclude sensor data with small fluctuations; Constructing a data set: Use a sliding window approach to construct input and output data pairs as sample data for model training; Model building: ModernTCN is used as the basic model architecture, and the bidirectional gated recurrent unit BiGRU and cross attention mechanism are introduced; Model training: Use the constructed dataset to train the established model; Gait event detection: The real-time data collected by the smartphone’s built-in sensors is input into the trained model, and the model outputs the corresponding gait event detection results.

2. The gait event detection method based on smartphone built-in sensors and deep learning according to claim 1 is characterized in that: In the data processing, Kalman filter estimation and low-pass filter are used to remove noise in the original data.

3. The gait event detection method based on smartphone built-in sensors and deep learning according to claim 1 is characterized in that: The constructing of the data set specifically includes: a. Set the sliding window size; b. Input sample construction: The data in each window is taken as an input sample, which contains multi-dimensional features of acceleration, angular velocity and device rotation angle after excluding data with smaller fluctuation amplitude; c. Output label construction: Based on the reference standard gait events detected by the pressure insole, the gait event type of the corresponding sampling point of the smartphone is marked.

4. The gait event detection method based on smartphone built-in sensors and deep learning according to claim 1, characterized in that: The model building specifically includes: The variable dimension is retained in the ModernTCN input layer, and a shard-based variable-independent embedding method is adopted; Each module in the ModernTCN backbone network contains a depthwise separable convolution DWConv, two point-wise convolutional feedforward networks ConvFFN1 and ConvFFN2; The bidirectional gated recurrent unit BiGRU is introduced after the ModernTCN backbone network; Introduce a cross-attention mechanism after the bidirectional gated recurrent unit BiGRU; The output of the cross-attention mechanism is used as the final time series representation.

5. The gait event detection method based on smartphone built-in sensors and deep learning according to any one of claims 1 to 4, characterized in that: The steps also include: Model evaluation: The time difference between the gait events detected by the model and the reference standard gait events is taken as the error to evaluate the effect of the model.

6. A gait event detection system based on smartphone built-in sensors and deep learning, characterized in that: include: A data collection device is used to collect real-time walking acceleration, angular velocity, and equipment rotation angle data; A data processing module to remove data noise; Feature extraction and data set construction module, used to filter data features and construct input and output data pairs; A deep learning model to detect gait events after training; The display module is used to display the gait event detection results.

7. The gait event detection system based on smartphone built-in sensors and deep learning according to claim 6, characterized in that: The data collection device comprises: A smartphone device having a built-in acceleration sensor, angular velocity sensor, and orientation sensor; A data acquisition module is used to collect three-axis acceleration data, three-axis angular velocity data, and rotation angle data of the mobile phone in three-dimensional space relative to the earth coordinate system in real time during walking through the built-in sensors of the smart phone device; A wearable device used to fix a smartphone to the user's body.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the gait event detection method according to any one of claims 1 to 5 is implemented.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, the gait event detection method according to any one of claims 1 to 5 is implemented.

Images