Gait assessment method and system based on deep convolutional neural network of time series

Abstract

The disclosure provides a gait evaluation method and system based on a timing sequence deep convolutional neural network, and relates to the technical field of gait recognition, which comprises the following steps: acquiring timing data when a pedestrian walks; introducing a threshold step length segmentation method, extracting the start and end time of each step of the timing data through identification of an initial contact event, and filling the timing data in two channels to adapt to different walking speeds; inputting the processed gait timing data into a deep convolutional neural network, first extracting local features, synthesizing the local features to obtain a spatial feature map, compressing multiple units in the spatial feature map into a single target vector in a full connection layer, and outputting a step length or speed. The disclosure can more accurately extract the start time of each step and performs well under the walking habits and speeds of different movers.

Description

Technical Field

[0001] This disclosure relates to the field of gait recognition technology, specifically to a gait evaluation method and system based on time-series deep convolutional neural networks. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Gait assessment is an important research area in sports biomechanics, aiming to reveal information about movement, health, and disease by analyzing an individual's gait characteristics during walking. Traditional gait assessment methods mainly rely on manual observation, sensor devices, and motion capture systems. These methods have some challenges and limitations, as follows:

[0004] 1. Manual observation: Manual observation is one of the earliest methods used for gait assessment. Researchers visually observe and analyze the gait of subjects to obtain information about gait characteristics. However, this method is limited by subjectivity, subjective judgment differences, and reliance on experience, making it difficult to provide objective and accurate gait analysis.

[0005] 2. Sensor devices: Gait assessment using sensors such as inertial measurement units (IMUs) and pressure sensors is becoming increasingly mainstream. These sensors can capture data such as body movement and pressure distribution, but they are susceptible to drift, error accumulation, and sensor position calibration issues during long-term use, which reduces the accuracy and stability of gait assessment.

[0006] 3. Motion capture systems: These systems record the movement trajectories of the body's joints in real time by placing numerous optical or inertial sensors on the subject. While this method provides highly accurate motion data, its high cost, bulky equipment, and complex operation limit its widespread application in real life.

[0007] Faced with these challenges, deep learning-based gait assessment methods are gradually emerging. Deep learning models such as deep convolutional neural networks (DCNNs) possess powerful feature extraction and pattern recognition capabilities, enabling them to learn human behavioral patterns from complex time-series data and achieve gait analysis. However, existing deep learning methods typically use sliding windows to extract time-series data, which cannot adapt to the differences between individuals, thus failing to provide accurate real-time gait assessment. Furthermore, existing deep learning methods lack effective data preprocessing, so the final gait assessment accuracy is not superior to traditional data processing techniques. Summary of the Invention

[0008] To address the aforementioned issues, this disclosure proposes a gait evaluation method and system based on time-series deep convolutional neural networks. It introduces a thresholding method for real-time and effective extraction of the start time of each step and employs a dual-channel filling technique to overcome the blindness of traditional sliding window extraction of time-series data, thus adapting to the walking habits and speeds of different athletes.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions:

[0010] Gait evaluation methods based on time-series deep convolutional neural networks include:

[0011] Acquire temporal data of pedestrians walking;

[0012] A threshold stride segmentation method is introduced. By identifying the initial contact event, the start and end times of each step in the time series data are extracted, and the time series data is filled with dual channels to adapt to different travel speeds.

[0013] The processed gait time series data is input into a deep convolutional neural network. First, local features are extracted, and then the local features are synthesized to obtain a spatial feature map. In a fully connected layer, multiple units in the spatial feature map are compressed into a single target vector, and stride length or velocity is output.

[0014] According to some embodiments, the present disclosure adopts the following technical solutions:

[0015] A gait evaluation system based on time-series deep convolutional neural networks includes:

[0016] The data acquisition module is used to acquire time-series data of pedestrians walking.

[0017] The data processing module is used to introduce the threshold stride segmentation method. By identifying the initial contact event, it extracts the start and end times of each step of the time series data and performs dual-channel filling on the time series data to adapt to different travel speeds.

[0018] The data recognition module is used to input the processed gait time series data into a deep convolutional neural network. First, local features are extracted, and the local features are synthesized to obtain a spatial feature map. In the fully connected layer, multiple units in the spatial feature map are compressed into a single target vector, and stride length or velocity is output.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned gait evaluation method based on a time-series deep convolutional neural network.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the gait evaluation method based on time sequence deep convolutional neural networks.

[0023] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0024] This disclosure presents a gait evaluation method based on time-series deep convolutional neural networks, particularly for gait analysis using time-series data collected during pedestrian walking. The method includes time-series data acquisition, data cleaning and preprocessing, and training of a gait evaluation model based on a deep convolutional neural network. To further improve the system's real-time performance and adaptability, a thresholding method is introduced for real-time and effective extraction of the start time of each step, and a dual-channel imputation technique is employed to overcome the blindness inherent in traditional sliding window data extraction, adapting to the walking habits and speeds of different pedestrians. By integrating the thresholding method, dual-channel imputation technique, and the deep convolutional network-based gait evaluation model, a real-time, effective, and highly adaptable gait evaluation method is provided. Through these innovations, the system can more accurately extract the start time of each step and performs excellently under the different walking habits and speeds of pedestrians, bringing significant progress to the field of gait evaluation. Attached Figure Description

[0025] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0026] Figure 1 This is a flowchart of an embodiment of the present disclosure;

[0027] Figure 2 This is a diagram of the deep convolutional neural network structure according to an embodiment of the present disclosure;

[0028] Figure 3 This is a system connection diagram of an embodiment of the present disclosure. Detailed Implementation

[0029] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0032] Example 1

[0033] One embodiment of this disclosure provides a gait evaluation method based on a time-series deep convolutional neural network, including:

[0034] Step 1: Obtain time-series data of pedestrians walking;

[0035] Step 2: Introduce the threshold stride segmentation method. By identifying the initial contact event, extract the start and end times of each step of the time series data, and fill the time series data with dual channels to adapt to different travel speeds.

[0036] Step 3: Input the processed gait time series data into a deep convolutional neural network. First, extract local features, synthesize the local features to obtain a spatial feature map, and compress multiple units in the spatial feature map into a single target vector in a fully connected layer to output stride length or velocity.

[0037] As one embodiment, the specific implementation process of the gait evaluation method based on time-series deep convolutional neural networks disclosed herein is as follows:

[0038] 1. Gait time series data acquisition:

[0039] First, time-series data of pedestrians walking is collected. During the data collection process, an inertial measurement unit (IMU, which includes a 3D accelerometer and a 3D gyroscope) is used. This measurement unit can be embedded in the shoe / insole or placed on the outside of the shoe to collect inertial information; a plantar pressure sensor is used to obtain accurate body pressure information.

[0040] 2. Data cleaning and preprocessing:

[0041] The collected time-series data is cleaned and preprocessed to calculate the start and end points of specific gait patterns, and then filled in to adapt to different walking speeds. A threshold method is introduced to detect gait events through plantar pressure, particularly the accurate identification of the initial contact (IC) event (IC in healthy individuals is heel-first contact, but IC in gait disorder patients may not occur at the heel). The initial contact event refers to the initial movement during walking. Real-time and effective extraction of the start time of each step is achieved. Simultaneously, a dual-channel filling technique is employed to overcome the limitations of sliding window truncation, ensuring adaptability to different walking habits and speeds of different individuals and meeting the requirements of real-time gait analysis.

[0042] 3. Training of deep convolutional neural networks:

[0043] A gait evaluation model based on a deep convolutional neural network is designed. The model includes a convolutional encoder and an output layer: the first convolutional layer extracts local features from the input temporal sequence data; after max pooling to eliminate noise interference, a second convolutional layer extracts higher-dimensional features; after another max pooling to eliminate noise interference, tensor flattening transforms the multidimensional data into one-dimensional data; finally, a fully connected layer fits the high-dimensional features of the one-dimensional data to obtain the spatial feature information of the gait. This model architecture allows for effective extraction of local features, making gait evaluation more accurate.

[0044] Specifically, step S1: Collect time-series data of pedestrians walking;

[0045] Step S2: Clean and preprocess the collected time series data, calculate the start and end times of each step, and perform filling processing to adapt to different travel speeds.

[0046] Furthermore, data preprocessing includes stride segmentation and a data adaptation CNN dual-channel padding method. In step S2, a thresholding method is introduced to achieve real-time and effective extraction of the start time of each step by detecting gait events, especially accurately identifying the initial contact (IC) event.

[0047] Specifically, the threshold method is implemented as follows: when the detected pressure reaches 1% of the maximum pressure of the previous gait cycle (as the threshold), and the pressure value remains above this threshold for a continuous period of time, the system considers this the start of a new gait, i.e., an initial contact event. The default threshold setting is 10 (units) when the system is powered on.

[0048] Raw data from the inertial measurement unit (IMU) is segmented into individual strides. The IMU, embedded in the centerline of the instrument shoe, contains a 3D accelerometer (range ±8g) and a 3D gyroscope (range ±2000° / s). Stride segmentation relies on gait event detection. The initial contact (IC) event time is calculated based on plantar pressure sensors under the foot, ensuring high accuracy and stability.

[0049] Specifically, such as Figure 3 As shown, the data acquisition process includes: the logic controller acquires pressure information through a pressure acquisition array and obtains inertial information through an IMU. This information is transmitted to a microcontroller via a LoRa wireless communication module for data processing, visualization, and intervention control. In the microcontroller, the acquired pressure and inertial information are used to calculate whether intervention is needed and the appropriate intervention method. This information is then transmitted to the vibration controller via the LoRa wireless communication module, thereby controlling the vibration of the vibration intervention unit.

[0050] The dual-channel filling technology overcomes the shortcomings of low real-time performance and accuracy caused by blind truncation in sliding windows.

[0051] Further data preprocessing not only introduced stride segmentation but also employed dual-channel padding technology to adapt the data to CNN and ensure compatibility with different movement habits. In each gait cycle, zero-padding was used uniformly before and after the stride. To overcome the shortcomings of traditional sliding window methods that blindly extract time-series data, a single-stride padding technique was designed to effectively and in real-time process actual single steps and accommodate users with different walking speeds, thereby expanding the model's generalization ability. This technique ensures good adaptability to different movement habits and speeds by segmenting actual single steps and using zero-padding to a fixed number of samples. The segmented 6D-IMU data for a single stride was zero-paddinged to 2048 samples to ensure a fixed-size input for the DCNN.

[0052] Step S3: Train a gait evaluation model based on a deep convolutional neural network.

[0053] Furthermore, a gait evaluation model for deep convolutional networks is presented. This model consists of two parts: 1) a convolutional encoder with two convolutional layers and 2) a dense block composed of three fully connected layers. The architecture is as follows... Figure 2 As shown, this network allows for the extraction of local features from input gait sequence data, and then the synthesis of these local features at a higher level to obtain spatial information. It focuses only on local features and does not include long-range temporal information.

[0054] Below, convolutional layers are labeled Cx, max-pooling layers are labeled Sx, and fully connected layers are labeled Fx, where x is the layer index. Specifically, given raw stride length data with C channels, H is the 6 dimensions of the data, W is the stride sample, and X...in ∈R C×H×W Used as input to the network. Layer C1 is the first layer, with k = 6 kernels ω of size 32 × 3. c Then add a bias term to the trainable bias. The result is passed through the ReLU activation function, which activates the output of the generation layer C1.

[0055]

[0056] Where ReLU(z) = max(0,z), C1 contains 582 trainable parameters.

[0057] After layer C1, a max-pooling layer S1 is inserted. The pooling process is similar to the convolution process, as shown in the figure above, which means that the filter "scans" the values ​​in the neighborhood of the six feature maps and selects the maximum value to output to the next layer. The receptive fields do not overlap, so the number of feature maps H and W in S1 is half that in C1.

[0058] Layer C2 is a convolutional layer. Since the input data is 6-dimensional, it has six feature maps of size 1. Each unit in each feature map is connected to a max-pooling layer S2 in the corresponding feature map of C1. Similar to layer C1, the results are passed through the ReLU function. The 1×2 receptive fields of layer S2 overlap, so the number of rows and columns in the feature maps of layer S2 is half that of layer C1. Layer C2 has 2310 trainable parameters.

[0059] Layer F1 is a fully connected layer, which can be understood as having a depth of 6, a 1x1 feature map of size 6 points, and then these 6 points are fully connected to each other (this number is due to the design of the output layer). Similarly, F2 contains 4096 units and is fully connected to F1. It has 12,021,760 trainable parameters. As with classic neural networks, units in layer F2, up to the second fully connected layer, compute the dot product between their input vector and weight vector and add the bias.

[0060] Finally, the output layer compresses 4096 units into a single target variable (i.e., stride length or velocity).

[0061] Example 2

[0062] One embodiment of this disclosure provides a deep convolutional neural network gait evaluation system based on time-series sequences, including:

[0063] The data acquisition module is used to acquire time-series data of pedestrians walking.

[0064] The data processing module is used to introduce the threshold stride segmentation method. By identifying the initial contact event, it extracts the start and end times of each step of the time series data and performs dual-channel filling on the time series data to adapt to different travel speeds.

[0065] The data recognition module is used to input the processed gait time series data into a deep convolutional neural network. First, local features are extracted, and the local features are synthesized to obtain a spatial feature map. In the fully connected layer, multiple units in the spatial feature map are compressed into a single target vector, and stride length or velocity is output.

[0066] Example 3

[0067] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the aforementioned gait evaluation method based on a time-series deep convolutional neural network.

[0068] Example 4

[0069] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the gait evaluation method based on time sequence deep convolutional neural networks.

[0070] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A gait evaluation method based on time-series deep convolutional neural networks, characterized in that, include: Acquire temporal data of pedestrians walking; A threshold stride segmentation method is introduced, including: when the detected pressure reaches 1% of the maximum pressure of the previous stride cycle as a threshold, and the pressure value continues to be greater than the threshold for a continuous period of time, the system considers this to be the start of a new stride, i.e., an initial contact event; by identifying the initial contact event, the start and end times of each step in the time series data are extracted, and the time series data is filled in two channels to adapt to different walking speeds; the time series data is filled in two channels, including: for the actual single step length, the single step is segmented and filled with zeros evenly at the beginning and end, with the zero filling being a fixed size of sample data; The processed gait time series data is input into a deep convolutional neural network. First, local features are extracted, and the local features are synthesized to obtain a spatial feature map. In the fully connected layer, multiple units in the spatial feature map are compressed into a single target vector, and stride length or velocity is output. A deep convolutional neural network consists of a convolutional encoder with two convolutional layers and a dense block consisting of three fully connected layers. This deep convolutional neural network allows the extraction of local features from input gait time series data and then the synthesis of local features to obtain spatial information. It only focuses on local features and does not include long-distance temporal information.

2. The gait evaluation method based on time-series deep convolutional neural networks as described in claim 1, characterized in that, A threshold stride segmentation method is introduced, which identifies the initial contact event by segmenting the raw data from the inertial measurement unit (IMU) into individual strides. The IMU is embedded in the centerline of the instrument and contains a 3D accelerometer and a 3D gyroscope. The stride segmentation relies on gait event detection, and the initial contact event time is calculated based on the plantar pressure sensor under the foot.

3. The gait evaluation method based on time-series deep convolutional neural networks as described in claim 1, characterized in that, The segmented 6D-IMU data with a single step length is zero-padding to 2048 samples to ensure a fixed-size input for the deep convolutional neural network.

4. The gait evaluation method based on time-series deep convolutional neural networks as described in claim 1, characterized in that, Given raw stride length time series data with C channels, where H represents the 6 dimensions of the data and W represents the stride length samples, Used as input to the network, layer C1 is the first layer, with k=6 nodes of size [missing information]. kernel Then add a bias term to the trainable bias. The result is passed through the ReLU activation function, which activates the output of the generation layer C1.

5. The gait evaluation method based on time-series deep convolutional neural networks as described in claim 1, characterized in that, It also includes a pooling layer, which scans the values ​​in the neighborhood of the spatial feature map and selects the maximum value to output to the next layer. Each cell in each spatial feature map is connected to the max pooling layer in the corresponding feature map in C1.

6. A gait evaluation system based on time-series deep convolutional neural networks, characterized in that, A method for implementing the gait evaluation method based on temporal sequences of a deep convolutional neural network as described in any one of claims 1-5 includes: The data acquisition module is used to acquire time-series data of pedestrians walking. The data processing module is used to introduce the threshold stride segmentation method. By identifying the initial contact event, it extracts the start and end times of each step of the time series data and performs dual-channel filling on the time series data to adapt to different travel speeds. The data recognition module is used to input the processed gait time series data into a deep convolutional neural network. First, local features are extracted, and the local features are synthesized to obtain a spatial feature map. In the fully connected layer, multiple units in the spatial feature map are compressed into a single target vector, and stride length or velocity is output.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the gait evaluation method based on a time sequence deep convolutional neural network as described in any one of claims 1-5.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the gait evaluation method based on a time sequence deep convolutional neural network as described in any one of claims 1-5.

Images