Human motion recognition system based on LSTM (Long Short Term Memory)

Abstract

The invention belongs to the technical field of sensor measurement and recognition, and particularly relates to a human body motion recognition system based on LSTM (Long Short Term Memory), which comprises a data acquisition module, a data calculation module and a data application module, and utilizes a neural network model and a wearable electronic sensor to detect a human body walking state. A new long short term memory (LSTM) recurrent neural network model, namely an LSTM-STRM model, is established, the model combines multi-task learning, an attention mechanism and spatio-temporal feature fusion to accurately classify the walking state of the human body, and the walking state is compared with the original LSTM model. The result shows that the LSTM-STRM model can be used for classifying the time sequence data collected by the measuring unit, the states of the five targets can be classified with high precision, and the recognition accuracy is higher than that of an original LSTM model. The method improves the recognition precision, is high in adaptability, is suitable for the fields of medical treatment, human-computer interaction, exercise training and the like, and has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of sensor measurement and recognition technology, specifically to a human motion recognition system based on LSTM. Background Technology

[0002] Human motion recognition is a key technology in ubiquitous computing, human-computer interaction, and health monitoring. Traditional methods mainly rely on wearable sensors (such as accelerometers and gyroscopes) or computer vision-based technologies. While sensor-based solutions can provide direct motion data, they suffer from issues of intrusion, cost, and user compliance; vision-based solutions are susceptible to limitations imposed by lighting, occlusion, viewing angle, and privacy. With the advancement of deep learning, particularly its ability to process sequential data, automatic, end-to-end motion recognition using time-series signals (such as acceleration and angular velocity) acquired by sensors has become a research hotspot. This requires models that can effectively capture the complex temporal dynamics and long-term dependencies in motion data. Therefore, developing a new human motion recognition technology is particularly important. This technology should possess greater adaptability, be able to operate stably in different environments, and also be cost-effective, easy to promote, and easy to apply.

[0003] Recurrent Neural Networks (RNNs), due to their inherent recurrent connection structure, are naturally well-suited for processing time-series data. However, standard RNNs are prone to vanishing or exploding gradients during training, making it difficult for them to effectively learn and remember long-range temporal dependencies—which are crucial for distinguishing different human actions (such as the continuous gait differences between "walking" and "running"). Long Short-Term Memory (LSTM) networks were developed to address this issue, with their core innovation being the introduction of a sophisticated gating mechanism. Through input gates, forget gates, output gates, and cell states, LSTMs can selectively forget irrelevant historical information, update important new information, and output the current state, thus effectively solving the long-term dependency problem and enabling them to more accurately model the complex nonlinear dynamic characteristics and temporal context of human motion signals.

[0004] It is LSTM's superior ability to process long-term, complex-pattern evolution of sensor time-series data that makes it a core technology for building high-performance human motion recognition systems. Compared to traditional machine learning methods (such as SVM and HMM) or shallow networks, LSTM-based models can automatically learn more discriminative high-level spatiotemporal feature representations directly from raw or low-level preprocessed sensor data. This capability greatly improves the system's recognition accuracy, robustness, and adaptability for diverse motion patterns (such as daily activities, rehabilitation training movements, and sports-specific movements), laying a solid foundation for building more intelligent, natural, and non-invasive motion perception applications. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a human motion recognition system based on LSTM. This system uses a triaxial sensor to capture three types of information about human posture and uses an MTSA-LSTM model to train and match these three signals to achieve human posture recognition, thus solving the problems existing in the background technology.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0009] A human motion recognition system based on LSTM includes a data acquisition module, a data calculation module, and a data application module;

[0010] The data acquisition module includes an accelerometer, an angular velocity sensor, and a magnetometer;

[0011] Accelerometers, angular velocity sensors, and magnetometers were used to collect acceleration, angular acceleration, and location-related data from three body parts: the back, legs, and ankles; the data was then filtered and preprocessed.

[0012] Preprocessing includes noise filtering, missing value handling, and data normalization / standardization; and data is transmitted to the main control chip ZYNQ via Bluetooth module for subsequent model recognition.

[0013] The data computing module proposes an LSTM-STRM model based on the LSTM network model, which combines multi-task learning, attention mechanism and spatiotemporal feature fusion. The LSTM-STRM model adds two modules on the original basis: spatiotemporal fusion feature module and temporal attention mechanism. The main control chip ZYNQ chip uses the LSTM-STRM model to identify the data collected by the measurement unit and establish a database to identify the current motion mode.

[0014] The data application module transmits the data obtained from the ZYNQ chip to the host computer via Bluetooth and performs data visualization.

[0015] Furthermore, the accelerometer, angular velocity sensor, and magnetometer are a triaxial accelerometer, a triaxial gyroscope, and a triaxial magnetometer, which together constitute a triaxial sensor;

[0016] The three-axis sensors are fixed to the back, legs, and ankles of the human body, detecting changes in three values ​​at these three locations. The data is preprocessed through internal circuitry so that the data transmitted via Bluetooth can be used directly.

[0017] Furthermore, the main control chip ZYNQ chip: uses the digital signals transmitted from the triaxial sensor via Bluetooth module to train the model, establish the database, and complete the final recognition.

[0018] Furthermore, the LSTM network model, a special type of recurrent neural network, controls the flow of information by introducing memory units as well as input gates, forget gates, and output gates;

[0019] The LSTM-STRM model is an improved LSTM network model. The spatiotemporal fusion feature module is a fundamental and innovative module in the MTSA-LSTM architecture. Through a designed feature engineering strategy, it transforms the raw sensor data into an enhanced feature representation rich in spatial correlation information. This module simulates the way human kinematics experts analyze the coordinated movement of multiple parts of the body, explicitly encoding the movement relationships between different parts of the body.

[0020] The multi-task learning approach addresses the problem of collaborative optimization among related tasks. By sharing a low-level feature extractor between stair detection and fall prediction, orientation prediction is activated only upon a fall, avoiding interference from invalid labels and reducing computational overhead.

[0021] The temporal attention mechanism module is a key innovative module in the MTSA-LSTM architecture. It can dynamically identify and focus on the most diagnostically valuable critical moments in the sensor data stream. This layer is located between the bidirectional LSTM and the multi-task output layer. Through an intelligent weight allocation mechanism, it can extract the most representative features from long-term time-series data.

[0022] Furthermore, the host computer outputs real-time motion status and provides a fall alarm.

[0023] Furthermore, the main control chip ZYNQ chip performs weighted calculations on each unit in each sensor array according to the preset weights of the trained MTSA-LSTM model, thereby obtaining the weighted data of a single array, reducing the amount of computation required for recognition in the ZYNQ chip, and thus improving recognition efficiency.

[0024] (III) Beneficial Effects

[0025] Compared with existing technologies, this invention provides an LSTM-based human motion recognition system, which has the following advantages:

[0026] This invention uses a ZYNQ chip to build and train a deep learning model, constructing an improved LSTM-STRM model to enhance recognition accuracy.

[0027] This invention optimizes the existing recognition algorithms by addressing their high resource consumption and slow output rate; it also optimizes the existing computer vision-based methods by addressing their high light requirements, thus expanding the application scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the system workflow of an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the signal waveform acquired by the sensor array of the present invention;

[0030] Figure 3 The graph shows the accuracy and loss curves of the MTSA-LSTM model in this embodiment of the invention.

[0031] Figure 4 This is a confusion matrix diagram based on the MTSA-LSTM model in an embodiment of the present invention;

[0032] Figure 5 This is a t-sne plot based on the MTSA-LSTM model in an embodiment of the present invention;

[0033] Figure 6 This is a structural diagram of the MTSA-LSTM model according to an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example

[0036] like Figure 1-6 As shown in the figure, an embodiment of the present invention proposes a human motion recognition system based on LSTM. The system mainly consists of the following parts: a data acquisition module, a data calculation module, and a data application module.

[0037] 1. Data Acquisition Module

[0038] In the data acquisition module, the first step is to select a suitable motion sensor. The inertial measurement unit of this system includes a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer (electronic compass), which are used to collect acceleration, angular velocity, and location data, respectively.

[0039] A three-axis accelerometer, also known as a triaxial accelerometer, is used based on the principle of inertia to measure the acceleration (including gravity) of an object in motion along the x, y, and z axes. The acceleration measured by a three-axis accelerometer includes (1) the acceleration of the object's motion and (2) the acceleration due to gravity. When the sensor is stationary, the sensor output is only the acceleration due to gravity; when the sensor is moving, the output is both the acceleration due to gravity and the acceleration due to motion. When measuring human acceleration, its components are affected by the following factors: (1) the current type of human motion; (2) the specific position of the three-axis accelerometer worn on the body; and (3) the orientation of the coordinate system of the three-axis accelerometer relative to the human body.

[0040] A three-axis gyroscope measures acceleration by the angle between the vertical axis of an internal micro-gyroscope and the device in a three-axis coordinate system (the gyroscope's own rotation state). Unlike a three-axis accelerometer, which can only measure linear acceleration, a three-axis gyroscope is suitable for measuring the deflection and rotation of an object.

[0041] A three-axis magnetometer, also known as a triaxial magnetometer, is used to measure the strength and direction of a magnetic field. The Earth's magnetic field can be viewed as a vector, which can be decomposed into a vector perpendicular to the ground plane and two vectors parallel to the horizontal plane. Functionally similar to a compass, it can measure cardinal directions. Combined with a three-axis accelerometer, it can measure precise compass heading; and when used with navigation devices such as GPS, it can accurately pinpoint the user's location and orientation.

[0042] These three sensors are connected to the MCU. The acquired analog signals are preprocessed and then transmitted to the LSTM via Bluetooth for further data processing. Data preprocessing includes:

[0043] (1) Noise Filtering: In addition to human motion data, the signals collected by the sensor also contain a large amount of noise, including system noise and noise caused by minor external vibrations. Filtering is generally used to remove noise from the acquired signals. Common filtering methods include median filtering, Gaussian filtering, and Kalman filtering. In practice, a filter with better performance is often selected based on the type of noise. Median filtering is commonly used in signal processing fields such as image processing. It uses the median of all values ​​within a selected window as the new value to smooth the signal. Gaussian filtering is often used to handle signals containing Gaussian noise (a type of noise whose probability density follows a Gaussian distribution). The common method is to multiply the values ​​within the selected window by a Gaussian function kernel as weights. The main idea of ​​Kalman filtering is to predict the value of the next moment based on the optimal value of the previous moment, and to predict the new error based on the mean square error and noise of the previous error. After obtaining the measured value of the next moment, it is compared with the predicted value and corrected to obtain the optimal value of the next moment and update the error, iterating continuously. Kalman filtering is effective in practical signal processing but requires a large amount of computation.

[0044] (2) Missing Value Handling: In addition to filtering, it is also necessary to handle missing values ​​in the data. Due to the influence of sensor sampling frequency, communication problems, or other issues, some of the collected signal data may be missing. Common methods for handling missing values ​​include: deleting unnecessary missing values, ignoring them, mean interpolation, median interpolation, Lagrange interpolation, and using other variables without missing values ​​to build a model for prediction. Different methods need to be determined based on the importance of the missing variable and its relationship with other variables. When processing sensor signal data, especially time series data of human motion, where the data does not change drastically within a very short period, mean interpolation can be used to estimate the missing data using data before and after the missing values.

[0045] (3) Data normalization / standardization: When the human body performs vigorous movements with large amplitude, the data collected by the sensor will fluctuate greatly. Therefore, in order to avoid extreme values ​​having too much impact on the final model effect and to facilitate the comparison of data with different dimensions, the data should be normalized or standardized.

[0046] The data computing module mainly includes a data storage module, a data segmentation module, a feature extraction module, and a classifier module.

[0047] 2. Data Calculation Module:

[0048] The ZYNQ chip receives the preprocessed signal and builds an MSTA-LSTM model, which is then trained using a database. The following is the construction of the MSTA-LSTM model:

[0049] This system proposes an LSTM-STRM model that combines multi-task learning, attention mechanisms, and spatiotemporal feature fusion to improve model performance. This model adds two modules to the original design: a spatiotemporal feature fusion module and a temporal attention mechanism.

[0050] (1) Spatiotemporal fusion feature module

[0051] This paper proposes a spatiotemporal fusion feature module to explicitly calculate the spatial relationships between sensor data from different body parts, enhancing the expressive power of the original input. By placing three 3-axis sensors on the chest, waist, and ankle, a total of 27 dimensions (3*3*3) are obtained. This module arranges the acceleration, angular velocity, and magnetic field measured by the three sensors at different body parts in sequence. Subsequent difference calculations are performed on the chest and waist features to capture the motion coordination between the upper body and core, used to identify the difference between chest forward leaning and waist maintenance when going up and down stairs. Subsequent difference calculations are performed on the waist and ankle features to capture the motion correlation between the core and lower limbs, used to detect the correlation between waist twisting and foot misalignment during falls. After calculating the difference features, they are concatenated into a 45-dimensional feature set in the order of original chest features, original waist features, original ankle features, chest-waist difference, and waist-ankle difference, which serves as the input to the MTSA-LSTM model.

[0052] (2) Bidirectional LSTM core layer

[0053] Bidirectional LSTM comprehensively captures temporal dependencies by running two independent LSTM layers simultaneously: the forward LSTM layer processes the input sequence in forward chronological order (t = 1 → T), accumulating historical information and predicting future trends; the backward LSTM layer processes the sequence in reverse chronological order (t = T → 1), using the "future" context to backtrack to the current state. The output of each time step is formed by concatenating the forward and backward hidden states, creating a 256-dimensional feature vector (128 dimensions forward + 128 dimensions backward) containing complete contextual information. Structurally, at time step t, the hidden state of the forward LSTM layer comprehensively considers the current input x. t And the hidden state h from the previous moment t-1 By using structures such as forget gates, input gates, and output gates to filter, update, and pass information, the contextual information from the beginning of the sequence to the current moment is captured. Meanwhile, the hidden state h of the inverse LSTM layer... t Input x based on the current time t and the hidden state h in the next moment t+1 Similarly, context information from the end of the sequence to the current time step is obtained. Finally, the hidden states of the forward LSTM layer and the backward LSTM layer at the current time step are concatenated or fused to obtain an output that integrates the bidirectional context information of the sequence. t , that is ot = [h→t; h←t] or o t Combinations such as =h→t+h←t provide richer semantic information and sequence context features for subsequent task processing, such as sequence labeling and text classification.

[0054] This bidirectional structure allows the model to predict the current state based on past data (such as instability characteristics 0.5 seconds before a fall) and verify the current judgment based on subsequent information (such as vibration patterns 0.1 seconds after the impact). This enables high-precision real-time identification of complex movements, such as the transition from periodic motion of climbing stairs to a sudden fall, in human behavior analysis, and it is particularly adept at capturing key transitional features of motion state transitions. The forward and reverse sequences are defined as ht and ht, respectively, in the hidden layer. The hidden layer yt and the output layer can...

[0055] The definition is as follows:

[0056]

[0057] (3) Temporal attention layer

[0058] The temporal attention layer is a key innovative module in the MTSA-LSTM architecture, capable of dynamically identifying and focusing on the most diagnostically valuable critical moments in the sensor data stream. Located between the bidirectional LSTM and the multi-task output layer, this layer extracts the most representative features from long-term temporal data through an intelligent weight allocation mechanism. The core structure consists of an input feature interface, an attention scoring network, a dynamic weight generator, and a context vector synthesizer.

[0059] The input feature interface can receive 256-dimensional temporal features (time step × batch size) from the bidirectional LSTM output, which contain spatiotemporal patterns of whole-body motion (such as acceleration trends and angular velocity changes). The attention scoring network generates attention scores through linear transformation and tanh activation to evaluate the importance of each time step to the current prediction task. The dynamic weight generator normalizes the time dimension using the Softmax function, ensuring that the weights at each time step are ∈ [0,1], and the sum of all weights for a single sample is 1. The context synthesizer weights the original features based on the attention weights, generating a concise 256-dimensional context vector.

[0060] Attention score calculation, where h t ∈R 256 W a ∈R 64*256 b a ∈R 64 The 256-dimensional features are projected onto a 64-dimensional attention space to evaluate the "importance score" at each time step. The formula is:

[0061] scoret =tanh(W a ·h t +b a )

[0062] In attention weight normalization, Softmax transforms the scores into a probability distribution, where T is the total time step and α is the probability distribution. t This is a normalized weight for time step t, which can highlight key time steps and suppress less important time steps. The formula is:

[0063]

[0064] Context vector generation enables dynamic weighted fusion of temporal features. The formula is as follows:

[0065]

[0066] Multi-head attention extension, where H is the number of attention heads (usually 4). Let be the weight of the i-th attention head. Its formula is:

[0067]

[0068] The following is a detailed process for model training and hardware implementation. First, sensor signals are collected based on different movement modes, such as going up and down stairs, running, hiking, and falls. Next, the collected signals are preprocessed to obtain the required signals. Then, user information tables, movement type tables, and movement data tables are constructed based on the obtained data. Next, the data is segmented into data segments that are easy for the model to recognize. This requires processing using a sliding window technique. A sliding window with a width of 200 and a step size of 100 is designed, corresponding to body activity data within 2 seconds. This ensures that a single window can completely accommodate a complete movement without being too wide, which would reduce recognition accuracy. Next, an LSTM-STRM recurrent neural network algorithm module is constructed. This invention uses a ZYNQ chip for model construction and result comparison. In configuring the ZYNQ chip, the PL end of the ZYNQ chip is first configured using Vivado to generate a .xsa file. Based on the .xsa file, a custom Linux distribution is created using Petalinux. A platform project is created in Vitis, the DPU IP is integrated and compiled, and the final hardware bitstream is generated. Next, the model needs to be built and trained in Vitis. The input layer acts as a buffer in the entire network, used to add data to the network. X, Y, and Z represent the data collected by the sensor in the X, Y, and Z directions, respectively. The data from these three directions are combined into a 3D vector with a sliding window. The width of the sliding window is 200. Considering that the preprocessed data has a dimension of 40, the input to the LSTM layer is an n*200*40 3D matrix, where n is the sample size. Too many neural network nodes or hidden layers will significantly reduce the training speed and may lead to overfitting. Therefore, the hidden layer here is designed to contain only two LSTM layers, with 128 nodes in the first layer and 12 nodes in the second layer. To avoid overfitting, dropout with a 1 / 2 neuron disconnection ratio is added after the LSTM layer. The output layer is a fully connected layer with a softmax activation function, which achieves the final classification output. After 100 training iterations, a 32-bit floating-point weight file containing all model parameters is obtained. The trained model (including graph structure and weights) is then solidified into a single file. This process removes training-specific nodes (such as optimizers and loss functions) and converts variables to constants. The quantizer quantizes the model, converting the 32-bit floating-point (FP32) weights and activation values ​​to 8-bit fixed-point (INT8) numbers. After quantization, the model's parameters have changed from 32-bit floating-point to 8-bit fixed-point numbers. This parameter information is included in the quantized model and the final .xmodel file, and loaded into the bitstream of the DPU hardware accelerator by the PL. The DPU actually performs the neural network computations defined by xmodel on the PL.Ultimately, under the control of the PS, the application utilizes the DPU hardware accelerator on the PL side to efficiently perform the inference task of human motion recognition.

[0069] 3. Data Application Module:

[0070] The data application module outputs the results obtained by the ZYNQ chip through the data calculation module to the host computer. In the future, it is also possible to output the data by uploading it to the cloud.

[0071] Application areas:

[0072] The human posture recognition system of this invention has extremely broad application prospects. For example, it can be used for fall detection and early warning (especially for the elderly). This system can be used in the homes of elderly people living alone, nursing homes, and rehabilitation centers. The system monitors human posture and movements in real time, and once a sudden fall or prolonged lying down is detected, it immediately notifies caregivers or family members via alarms, SMS, telephone, etc., to gain crucial time for rescue.

[0073] Furthermore, the system has applications in rehabilitation training and assessment. For example, it can guide patients in proper movement, identifying their rehabilitation exercises (such as joint range of motion exercises and gait training), providing real-time feedback on whether the movements are standard and the range of motion is within acceptable limits, and offering visual or verbal guidance. It can also quantitatively assess rehabilitation progress, objectively recording and analyzing the recovery of patients' motor abilities (such as gait parameters, joint angles, and balance), providing data support for doctors to adjust rehabilitation plans. Finally, it enables remote rehabilitation monitoring, allowing patients to remotely transmit movement data and progress reports to therapists for assessment and guidance when performing rehabilitation training at home.

[0074] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human motion recognition system based on LSTM, characterized in that: It includes a data acquisition module, a data calculation module, and a data application module; The data acquisition module includes an accelerometer, an angular velocity sensor, and a magnetometer; Accelerometers, angular velocity sensors, and magnetometers were used to collect acceleration, angular acceleration, and location-related data from three body parts: the back, legs, and ankles. The data is then filtered and preprocessed. Preprocessing includes noise filtering, missing value handling, and data normalization / standardization; and data is transmitted to the main control chip ZYNQ via Bluetooth module for subsequent model recognition. The data computing module proposes an LSTM-STRM model based on the LSTM network model, which combines multi-task learning, attention mechanism and spatiotemporal feature fusion. The LSTM-STRM model adds two modules on the original basis: spatiotemporal fusion feature module and temporal attention mechanism. The main control chip ZYNQ chip uses the LSTM-STRM model to identify the data collected by the measurement unit and establish a database to identify the current motion mode. The data application module transmits the data obtained from the ZYNQ chip to the host computer via Bluetooth and performs data visualization.

2. The LSTM-based human motion recognition system according to claim 1, characterized in that: The accelerometer, angular velocity sensor, and magnetometer are a triaxial accelerometer, a triaxial gyroscope, and a triaxial magnetometer, which together constitute a triaxial sensor. The three-axis sensors are fixed to the back, legs, and ankles of the human body, detecting changes in three values ​​at these three locations. The data is preprocessed through internal circuitry so that the data transmitted via Bluetooth can be used directly.

3. The LSTM-based human motion recognition system according to claim 3, characterized in that: The main control chip, ZYNQ chip, is used to train the model and establish the database using the digital signals transmitted from the triaxial sensor via Bluetooth module, and to complete the final recognition.

4. The LSTM-based human motion recognition system according to claim 1, characterized in that: The LSTM network model, a special type of recurrent neural network, controls the flow of information by introducing memory units, input gates, forget gates, and output gates. The LSTM-STRM model is an improved LSTM network model. The spatiotemporal fusion feature module is a fundamental and innovative module in the MTSA-LSTM architecture. Through a designed feature engineering strategy, it transforms the raw sensor data into an enhanced feature representation rich in spatial correlation information. This module simulates the way human kinematics experts analyze the coordinated movement of multiple parts of the body, explicitly encoding the movement relationships between different parts of the body. The temporal attention mechanism module is a key innovative module in the MTSA-LSTM architecture. It can dynamically identify and focus on the most diagnostically valuable critical moments in the sensor data stream. This layer is located between the bidirectional LSTM and the multi-task output layer. Through an intelligent weight allocation mechanism, it can extract the most representative features from long-term time-series data. The core structure consists of an input feature interface, an attention scoring network, a dynamic weight generator, and a context vector synthesizer.

5. The LSTM-based human motion recognition system according to claim 1, characterized in that: The host computer outputs real-time motion status and provides fall alarms.

6. The LSTM-based human motion recognition system according to claim 3, characterized in that: The main control chip ZYNQ chip performs weighted calculations on each unit in each sensor array according to the preset weights of the trained MTSA-LSTM model, thereby obtaining the weighted data of a single array. This reduces the amount of computation required for recognition in the ZYNQ chip and improves recognition efficiency.

Images