Lower extremity gait motion recognition method

Abstract

The application relates to the technical field of gait recognition, and provides a lower limb gait action recognition method. The method comprises the following steps: collecting multi-source sensor data of a lower limb, and constructing a multi-modal feature sequence; inputting a bidirectional recurrent neural network for feature coding to obtain a hidden state sequence; generating a dynamic query vector for a preset action category through an action-specific attention mechanism, calculating attention weights in a local sliding window with the current time as the center, weighting and fusing the hidden state sequence based on the attention weights, and mapping the hidden state sequence to a preliminary action probability sequence through a classifier; taking the preliminary action probability sequence as an observation probability, combining a preset state transition probability matrix to construct a hidden Markov model, globally decoding the hidden Markov model to obtain a corrected action sequence that satisfies a biomechanical state transition constraint. While improving the recognition accuracy of sparse key actions, the method ensures that the output action sequence conforms to the biomechanical logic of a human body.

Description

Technical Field

[0001] This application relates to the field of gait recognition technology, specifically to a method for recognizing lower limb gait movements. Background Technology

[0002] Accurate recognition of lower limb gait movements is a crucial research topic in rehabilitation medicine and human-machine collaborative control. During routine rehabilitation training, subjects need to perform various movements such as standing, sitting, walking, squatting, and various posture transitions. Real-time and accurate recognition of these movements is essential for the safe control of assistive movement devices. However, existing lower limb movement recognition methods still face several technical challenges when dealing with long sequences of continuous movement data in daily rehabilitation scenarios.

[0003] First, the daily activity data of the subjects suffers from severe class imbalance. During long-term continuous monitoring, steady-state movements such as standing and walking occupy the vast majority of the time, while transitional movements such as sitting up, squatting and standing, and turning, although brief in duration and sparse in frequency, are precisely the critical moments with the highest risk of falls and the greatest need for assisted intervention. Existing motion recognition models, when processing such severely imbalanced long-sequence data, are easily dominated by the large number of steady-state motion samples during training, resulting in insufficient accuracy in recognizing sparse but crucial transitional movements, failing to meet the needs of rehabilitation safety monitoring.

[0004] Second, existing deep learning-based action recognition methods generally employ a purely data-driven strategy. The model learns only the statistical mapping between input and output from training data, lacking modeling of the biomechanical laws governing human movement. This results in a lack of logical consistency between frame-by-frame predictions, easily leading to abrupt changes in action states that violate physical laws during long-sequence inference—for example, a prediction that jumps directly from a sitting state to a walking state, skipping the necessary standing transition. If such unreasonable predictions are directly used to control motion assistive devices, they will cause dangerous torque mutations, seriously threatening the user's safety.

[0005] Therefore, how to simultaneously solve the two major problems of insufficient recognition accuracy of sparse key actions and lack of biomechanical logical constraints in the prediction sequence in the recognition of long sequence continuous actions is an important technical challenge in this field. Summary of the Invention

[0006] In view of this, embodiments of this application provide a lower limb gait action recognition method, which improves the recognition accuracy of sparse key actions while ensuring that the output action sequence conforms to human biomechanical logic through a collaborative strategy of action-specific attention mechanism and logical correction.

[0007] The first aspect of this application provides a method for recognizing lower limb gait movements, including: Collect and preprocess multi-source sensor data from the lower limbs to construct a multimodal feature sequence; The multimodal feature sequence is input into a bidirectional recurrent neural network for feature encoding to obtain a hidden state sequence. A dynamic query vector is generated for a preset action category through an action-specific attention mechanism. Attention weights are calculated within a local sliding window centered on the current time. The hidden state sequence is weighted and fused based on the attention weights, and the fusion result is mapped to a preliminary action probability sequence by a classifier. The initial action probability sequence is used as the observation probability, and a hidden Markov model is constructed by combining it with a preset state transition probability matrix. The hidden Markov model is then decoded globally based on the state transition probability matrix to obtain a corrected action sequence that satisfies the biomechanical state transition constraints.

[0008] In one embodiment, the acquisition and preprocessing of multi-source sensor data from the lower limbs to construct a multimodal feature sequence includes: Simultaneously acquire surface electromyography signals and inertial measurement unit data of the lower limbs; The surface electromyography signal is band-pass filtered, and the inertial measurement unit data is low-pass filtered. The data from the inertial measurement unit is upsampled to match the sampling rate of the surface electromyography signal using interpolation, thereby achieving temporal alignment of the multimodal data. The aligned channel data is standardized, and time-frequency domain features are extracted using a sliding window to construct the multimodal feature sequence.

[0009] In one embodiment, the surface electromyography (EMG) signal is acquired by a sensor disposed on the surface of a target muscle group in the lower limb, the target muscle group including at least one of the rectus femoris, vastus medialis, semitendinosus, tibialis anterior, medial head of gastrocnemius, and soleus; the inertial measurement unit (IMU) data is acquired by a sensor disposed at at least one location in the thigh, calf, and dorsum of the foot; the passband frequency range of the bandpass filter is 20-450 Hz, and the cutoff frequency of the low-pass filter is 10 Hz; the interpolation method is cubic spline interpolation; the normalization process is Z-score normalization; the time-frequency domain features include at least one of the following: mean absolute value, root mean square (RMS), waveform length, mean frequency, and median frequency.

[0010] In one embodiment, the bidirectional recurrent neural network is a bidirectional long short-term memory network; the step of generating dynamic query vectors for preset action categories through an action-specific attention mechanism, and calculating attention weights within a local sliding window centered on the current time, includes: For each of the preset action categories, the hidden state at the current moment is fused with the trainable query vector corresponding to that action to generate a dynamic query vector; Within a local sliding window centered at the current time and of size , the alignment score between the dynamic query vector and the hidden state at each time step within the window is calculated, and the attention weight is obtained by normalization. The hidden states within the window are weighted and summed according to the attention weights to generate a context vector. The context vector is then concatenated with the hidden state at the current moment and input into the classifier to obtain the preliminary action probability sequence.

[0011] In one embodiment, the formula for generating the dynamic query vector is: in, For dynamic query vectors, For the current moment t The hidden state, For each of the preset action categories, the first Trainable query vectors for action classes Hidden state The corresponding weight matrix For query vector The corresponding weight matrix, For bias terms, c Index for action categories; The formula for calculating the alignment score is: The formula for calculating the attention weight is: in, For the first in the window A time relative to time Action categories Alignment score; These are the corresponding normalized attention weights; For dynamic query vectors transpose; A trainable weight matrix for attention alignment; For the first in the window The hidden state vector at each time step; Indicates The local sliding window range centered on; Index of the time position within the window; This is the index of the time position in the normalized summation.

[0012] In one embodiment, the step of performing global optimal path decoding on the hidden Markov model based on the state transition probability matrix to obtain a corrected action sequence that satisfies biomechanical state transition constraints includes: The true intention of the action is taken as the hidden state, and the classification probability at each moment in the preliminary action probability sequence is taken as the observation probability. A state transition probability matrix is ​​constructed based on the legal transition relationships between human lower limb movements according to pre-statistics, and state transition probabilities that do not conform to biomechanical logic are set to zero. The Viterbi algorithm is used to recursively optimize the entire time series to find the optimal path with the highest posterior probability under the condition of satisfying the state transition constraints, which is then used as the corrected action sequence.

[0013] In one embodiment, the core recursive formula of the Viterbi algorithm is: in, To reach state at time 1 The maximum path probability, For at any time Arrival Status The maximum path probability, For state The initial probability; From state Transition to state The probability, For a moment Observed output And the state is The probability of observation, For a moment The observed values, and This is the state index in the Hidden Markov Model.

[0014] In one embodiment, the method further includes: generating control commands based on the modified motion sequence to drive the auxiliary motion device to perform auxiliary actions.

[0015] In one embodiment, the assistive motion device adopts a hierarchical control strategy. The upper layer outputs discrete motion intention tags according to the modified motion sequence, and the lower layer responds to motion switching through a finite state machine. When motion switching is detected, the joint angle reference curve of the corresponding motion is retrieved from the pre-stored reference trajectory library, and a control torque is generated accordingly.

[0016] In one embodiment, the preset action category includes steady-state actions and transition actions; the steady-state actions include at least one of standing, sitting, squatting, and walking; the transition actions include at least one of sitting to standing, standing to sitting, squatting to standing, standing to squatting, turning left, and turning right.

[0017] The lower limb gait recognition method provided in the first aspect of this application adopts a multi-source sensor fusion strategy. It utilizes the advanced characteristics of neuromuscular activation reflected by surface electromyography signals and the stability of limb kinematics reflected by inertial measurement unit signals. The two complement each other to improve recognition robustness. Through dynamic weighting of the action-specific attention mechanism, the model focuses on sparse but information-rich transition action segments, solving the class imbalance problem. Through Markov chain logic correction, the model uses the prior state transition probability matrix to perform global path optimization on the predicted sequence, eliminating unreasonable state jumps and ensuring the biomechanical rationality of the output sequence. The attention calculation strategy of local sliding window reduces the computational complexity compared to the global attention mechanism, meeting the requirements of real-time inference. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of a lower limb gait recognition method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the principle of an action-specific attention mechanism provided in an embodiment of this application; Figure 3 This is a block diagram of a hidden Markov logic correction framework provided in an embodiment of this application; Figure 4a This is a schematic diagram of the sensor deployment dissection location provided in an embodiment of this application; Figure 4b This is a schematic diagram of the actual wearing position of the sensor provided in one embodiment of this application; Figure 5 This is a schematic diagram of ten predefined actions provided in an embodiment of this application, wherein (a) standing, (b) sitting, (c) squatting, (d) from sitting to standing, (e) from standing to sitting, (f) walking, (g) turning left, (h) turning right, (i) from squatting to standing, and (j) from standing to squatting; Figure 6 is a radar chart of performance indicators for each action category provided in the embodiments of this application. The outer solid line represents the model of this application, and the inner solid line represents the baseline model; wherein... Figure 6a A radar chart showing the accuracy metrics for each action category. Figure 6b A radar chart showing the recall rate for each action category. Figure 6c A radar chart showing the F1 score for each action category; Figure 7 This is a schematic diagram of the confusion matrix during the inference stage provided in an embodiment of this application; Figure 8 This is a sequence prediction comparison waveform diagram provided in the embodiments of this application; Figure 9 This is a verification diagram of human-machine collaborative control and torque output of external limbs provided in the embodiments of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] An embodiment of this application provides a lower limb gait recognition method, including: Multi-source sensor data of the lower limbs were collected and preprocessed to construct a multimodal feature sequence.

[0026] The multimodal feature sequence is input into a bidirectional recurrent neural network for feature encoding to obtain a hidden state sequence. A dynamic query vector is generated for a preset action category through an action-specific attention mechanism. Attention weights are calculated within a local sliding window centered on the current time. The hidden state sequence is weighted and fused based on the attention weights, and the fusion result is mapped into a preliminary action probability sequence by a classifier.

[0027] The initial action probability sequence is used as the observation probability, and a hidden Markov model is constructed by combining it with a preset state transition probability matrix. The hidden Markov model is then decoded globally based on the state transition probability matrix to obtain a corrected action sequence that satisfies the biomechanical state transition constraints.

[0028] like Figure 1 The diagram illustrates the overall processing flow from multimodal data acquisition, feature encoding and preliminary classification, Markov chain logic correction to external limb control, as detailed below: Step S1: Collect multi-source sensor data of the lower limbs and preprocess it to construct a multimodal feature sequence.

[0029] In rehabilitation training scenarios, subjects wear a multi-sensor network consisting of surface electromyography (SEMG) sensors and inertial measurement units (IMUs). SEMG signals reflect the neural drive characteristics of muscle activation, and their signal changes can be detected before actual limb movement occurs, providing a proactive ability to perceive movement intentions. IMUs directly measure the acceleration and angular velocity of limb segments, providing stable kinematic information. The data from the two types of sensors complement each other at the information level: SEMG signals capture the body's active control intentions, while IMUs capture the actual movement state. After fusion, a multimodal feature sequence can be constructed, providing information-rich input for subsequent feature encoding.

[0030] Step S2: Input the multimodal feature sequence into a bidirectional recurrent neural network for feature encoding to obtain a hidden state sequence; generate a dynamic query vector for a preset action category through an action-specific attention mechanism, calculate attention weights within a local sliding window centered on the current time, perform weighted fusion of the hidden state sequence based on the attention weights, and map the fusion result to a preliminary action probability sequence through a classifier.

[0031] like Figure 2 The diagram illustrates the process by which a sliding window-based action-specific attention mechanism calculates attention weights within a local window and generates enhanced feature vectors through dynamic query vectors. The core of this step lies in addressing the class imbalance between steady-state and transitional movements in long sequences of continuous motion. In daily rehabilitation activities, steady-state movements such as standing and walking constitute the vast majority of the time frame, while transitional movements such as sitting up and squatting down, although short in duration, are often the moments with the highest risk of falls. Traditional deep learning models, when processing such imbalanced data, are easily dominated by the large number of steady-state action samples during training, resulting in insufficient ability to recognize sparse transitional movements.

[0032] To address this, this embodiment introduces an action-specific attention mechanism. This mechanism maintains a trainable query vector for each preset action category. By fusing the current hidden state with the query vector of the target action category, a dynamic query vector is generated. This dynamic query vector encodes the current state information and the prior features of the target action category, making the calculation of attention weights action-category specific. The attention weights are calculated and weighted within a local sliding window, and the resulting context vector is concatenated with the original hidden state and input into the classifier to obtain a preliminary action probability sequence.

[0033] By employing a local sliding window instead of a global attention mechanism, the computational complexity is reduced from the quadratic level of the sequence length to the linear level. This reduces computational overhead while preserving key local temporal features, thus meeting the requirements of real-time inference.

[0034] Step S3: Using the preliminary action probability sequence as the observation probability, construct a hidden Markov model by combining it with a preset state transition probability matrix, and perform global optimal path decoding on the hidden Markov model based on the state transition probability matrix to obtain a corrected action sequence that satisfies the biomechanical state transition constraints.

[0035] like Figure 3The diagram illustrates the process of using the initial predictions from a neural network as observation probabilities, combined with the state transition matrix, to decode the optimal action sequence using the Viterbi algorithm. This step addresses the problem of purely data-driven model outputs lacking biomechanical constraints. During long sequence inference, the frame-by-frame predictions of the neural network may produce action jumps that violate physical laws. For example, abruptly transitioning from a sitting state to a walking state, skipping the standing process. Directly applying such erroneous predictions to the control of extremity robots would lead to dangerous abrupt torques, threatening user safety.

[0036] This embodiment constructs a Hidden Markov Model framework, treating the true action intent as a hidden state and the initial probability output by the neural network as the observation probability. Using a pre-calculated state transition probability matrix as a hard constraint, state transition probabilities that do not conform to biomechanics are set to zero. The Viterbi algorithm is used to find the path with the highest posterior probability over the entire time series. This path is globally optimal while satisfying the state transition constraints, thus effectively eliminating unreasonable state jumps.

[0037] like Figure 8 As shown, the comparison of the effects before and after logic correction is illustrated. At time t=4.9s, the initial prediction result showed a jump signal that violated physical logic, from "sitting to squatting to standing". The Markov logic correction module detected that the transition probability was zero and corrected the error into a reasonable sequence of "sitting to squatting to standing" using the constraints of the transition matrix.

[0038] This embodiment uses attention feature encoding in series with Markov logic correction. The former solves the class imbalance problem at the data level, while the latter solves the logical rationality problem at the output level. The two work on different stages of the recognition process without conflicting with each other, and synergistically improve the accuracy and security of the recognition.

[0039] In one embodiment, the acquisition and preprocessing of multi-source sensor data from the lower limbs to construct a multimodal feature sequence includes: Simultaneously acquire surface electromyography signals and inertial measurement unit data of the lower limbs; The surface electromyography signal is band-pass filtered, and the inertial measurement unit data is low-pass filtered. The data from the inertial measurement unit is upsampled to match the sampling rate of the surface electromyography signal using interpolation, thereby achieving temporal alignment of the multimodal data. The aligned channel data is standardized, and time-frequency domain features are extracted using a sliding window to construct the multimodal feature sequence.

[0040] This embodiment employs a multi-sensor network composed of a surface electromyography (SEMG) signal sensor and an inertial measurement unit (IMU). The preprocessing procedure includes the following steps: First, a bandpass filter is applied to the SEMG signal to eliminate motion artifacts and high-frequency noise, retaining the effective frequency components related to muscle activation. Simultaneously, a low-pass filter is applied to the IMU data to remove high-frequency drift. Since the original sampling rates of the two types of sensors are usually inconsistent, with the IMU's sampling rate generally lower than that of the SEMG signal, interpolation is used to upsample the IMU data to match its sampling rate, achieving time-domain alignment. Subsequently, all channel data is standardized to eliminate dimensional differences, and time-frequency domain features such as mean absolute value, root mean square, and waveform length are extracted using a sliding window. Finally, a unified multimodal feature sequence is constructed as the input to the subsequent network.

[0041] The preprocessing workflow employs bandpass and lowpass filtering to specifically denoise the two types of signals, preserving effective information. Interpolation upsampling and standardization eliminate sampling rate and dimensional differences among multi-source data, enabling data from different modalities to be fused within a unified time scale and numerical range. Sliding window-based time-frequency domain feature extraction captures the local statistical characteristics and frequency distribution information of the signal, providing information-rich and format-consistent input for subsequent time-series feature encoding.

[0042] In one embodiment, the surface electromyography (EMG) signal is acquired by a sensor disposed on the surface of a target muscle group in the lower limb, the target muscle group including at least one of the rectus femoris, vastus medialis, semitendinosus, tibialis anterior, medial head of gastrocnemius, and soleus; the inertial measurement unit (IMU) data is acquired by a sensor disposed at at least one location in the thigh, calf, and dorsum of the foot; the passband frequency range of the bandpass filter is 20-450 Hz, and the cutoff frequency of the low-pass filter is 10 Hz; the interpolation method is cubic spline interpolation; the normalization process is Z-score normalization; the time-frequency domain features include at least one of the following: mean absolute value, root mean square (RMS), waveform length, mean frequency, and median frequency.

[0043] like Figure 4a The sensor deployment location, and Figure 4bAs shown in the schematic diagram of the actual wearing position, the surface electromyography (EMG) sensors are placed on the surface of six muscles in the right leg: the rectus femoris (EMG1) and vastus medialis (EMG2) are located on the front of the thigh and are mainly involved in knee extension; the semitendinosus (EMG3) is located on the back of the thigh and is mainly involved in knee flexion; the tibialis anterior (EMG4) is located on the front of the lower leg and is involved in ankle dorsiflexion; and the medial head of the gastrocnemius (EMG5) and soleus (EMG6) are located on the back of the lower leg and are involved in ankle plantarflexion. These six muscles cover the major extensor and flexor muscle groups of the lower limb, comprehensively reflecting the coordinated activation patterns of various muscle groups during gait movements. Inertial measurement units (IMUs) are placed in three locations: the thigh (IMU1), the lower leg (IMU2), and the dorsum of the foot (IMU3), acquiring kinematic information from the hip, knee, and ankle joints.

[0044] A fourth-order Butterworth bandpass filter (20-450Hz) was used to filter the surface electromyography (EMG) signals. The lower cutoff frequency of 20Hz effectively removed low-frequency motion artifacts caused by poor electrode-skin contact or limb movement, while the upper cutoff frequency of 450Hz filtered out high-frequency noise such as environmental electromagnetic interference. A low-pass filter with a cutoff frequency of 10Hz was used for the inertial measurement unit (IMU) data. Considering that the main frequency components of gait motion are typically below 6Hz, a 10Hz cutoff frequency effectively preserves useful kinematic signals while removing high-frequency interference caused by sensor drift and mechanical vibration.

[0045] Cubic spline interpolation constructs piecewise cubic polynomials to fit low-frequency inertial measurement unit (IMU) data, resulting in a smooth upsampled waveform that retains the original signal's trend. Z-score normalization transforms the data for each channel into a distribution with a mean of zero and a standard deviation of one, eliminating dimensional and numerical range differences between different sensor channels. In the time-frequency domain, the mean absolute value and root mean square (RMS) reflect the signal's amplitude characteristics, waveform length reflects the signal's complexity and drastic changes, and mean frequency and median frequency reflect the signal's spectral distribution characteristics.

[0046] In one embodiment, the bidirectional recurrent neural network is a bidirectional long short-term memory network; the step of generating dynamic query vectors for preset action categories through an action-specific attention mechanism, and calculating attention weights within a local sliding window centered on the current time, includes: For each of the preset action categories, the hidden state at the current moment is fused with the trainable query vector corresponding to that action to generate a dynamic query vector; Within a local sliding window centered at the current time and of size , the alignment score between the dynamic query vector and the hidden state at each time step within the window is calculated, and the attention weight is obtained by normalization. The hidden states within the window are weighted and summed according to the attention weights to generate a context vector. The context vector is then concatenated with the hidden state at the current moment and input into the classifier to obtain the preliminary action probability sequence.

[0047] Bidirectional Long Short-Term Memory (BiLSTM) networks are recurrent neural network architectures capable of simultaneously learning forward and backward dependencies in time series data. Compared to unidirectional SLTM networks, the bidirectional structure propagates information from both directions of the sequence simultaneously, allowing the hidden state at each time step to encode both past and future contextual information. In gait recognition tasks, such as recognizing the transition from sitting to standing, the forward path can capture changes in muscle activation patterns starting from the sitting posture, while the backward path can use the steady-state features after standing to help confirm the start time of the movement. Furthermore, the gating mechanism within the SLTM network effectively alleviates the gradient vanishing problem faced by traditional recurrent neural networks when processing long sequences of data, making it suitable for processing continuous gait data lasting several minutes or even longer.

[0048] like Figure 2 As shown, the action-specific attention mechanism works as follows: For each preset action category, the system maintains a trainable query vector, which learns the typical feature patterns of the corresponding action category during training. At each time step in the inference phase, this query vector is fused with the current hidden state through a weight matrix transformation to generate a dynamic query vector that simultaneously contains current state information and prior knowledge of the target action category.

[0049] A local sliding window is centered at the current time step, with a window size that is a preset constant. Within the window, the dynamic query vector is multiplied by the hidden state at each time step to obtain an alignment score, which is then normalized to attention weights using the Softmax function. These attention weights reflect the contribution of each time step within the window to the specific action category at the current time step. The hidden states within the window are then weighted and summed according to the attention weights to obtain a context vector. This context vector is concatenated with the original hidden state at the current time step and input into the Softmax classifier to output the probability distribution of each action category, forming a preliminary action probability sequence.

[0050] This mechanism assigns different query vectors to different action categories, making the calculation of attention weights category-specific. In steady-state action segments, it can reduce the response to transition action queries, while it can quickly increase the corresponding attention weights when early signals of transition actions are detected. Thus, it can effectively alleviate the class imbalance problem without increasing the number of training samples.

[0051] In one embodiment, the formula for generating the dynamic query vector is: in, For dynamic query vectors, For the current moment t The hidden state, For each of the preset action categories, the first Trainable query vectors for action classes Hidden state The corresponding weight matrix For query vector The corresponding weight matrix, For bias terms, c Index for action categories.

[0052] This formula fuses the hidden state at the current time step with the trainable query vector of the target action category through a bilinear transformation, and introduces a nonlinear transformation through the tanh activation function, so that the dynamic query vector can adaptively encode the current state information and the prior features of the target action category.

[0053] The formula for calculating the alignment score is: The formula for calculating the attention weight is: in, For the first in the window A time relative to time Action categories Alignment score; These are the corresponding normalized attention weights; For dynamic query vectors transpose; A trainable weight matrix for attention alignment; For the first in the window The hidden state vector at each time step; Indicates The local sliding window range centered on; Index of the time position within the window; This is the index of the time position in the normalized summation.

[0054] By parameterizing the similarity between the dynamic query vector and the hidden state using a trainable attention weight matrix, the alignment score calculation is no longer limited to a simple vector dot product, but can learn more flexible feature matching patterns. Softmax normalization ensures that the sum of all attention weights within the window is 1, making it probabilistically interpretable.

[0055] In one embodiment, the step of performing global optimal path decoding on the hidden Markov model based on the state transition probability matrix to obtain a corrected action sequence that satisfies biomechanical state transition constraints includes: The true intention of the action is taken as the hidden state, and the classification probability at each moment in the preliminary action probability sequence is taken as the observation probability. A state transition probability matrix is ​​constructed based on the legal transition relationships between human lower limb movements according to pre-statistics, and state transition probabilities that do not conform to biomechanical logic are set to zero. The Viterbi algorithm is used to recursively optimize the entire time series to find the optimal path with the highest posterior probability under the condition of satisfying the state transition constraints, which is then used as the corrected action sequence.

[0056] like Figure 3 As shown, a Hidden Markov Model (HMM) consists of three elements: a set of hidden states, observation probabilities, and state transition probabilities. In this embodiment, the set of hidden states corresponds to all preset action categories, the observation probabilities are directly taken from the Softmax probabilities output by the neural network, and the state transition probability matrix is ​​pre-constructed based on the biomechanical laws of human lower limb movement. Specifically, there are strict sequential logical relationships between human lower limb movements—for example, starting from a sitting state, one must go through a transition from sitting to standing to reach a standing state; it is impossible to directly jump to a walking state. Accordingly, all state transition probabilities that do not conform to biomechanical logic are set to zero, forming hard constraints.

[0057] The Viterbi algorithm recursively applies dynamic programming over the entire time series, selecting the predecessor state that maximizes the cumulative probability at each time step, and ultimately backtracking to obtain a globally optimal path. Since the transition probability that violates biomechanics is set to zero, these illegal paths are automatically eliminated during the recursive process, ensuring that the final output sequence of actions necessarily conforms to the preset biomechanical logic.

[0058] In one embodiment, the core recursive formula of the Viterbi algorithm is: in, To reach state at time 1 The maximum path probability, For at any time Arrival Status The maximum path probability, For state The initial probability; From state Transition to state The probability, For a moment Observed output And the state is The probability of observation, For a moment The observed values, and This is the state index in the Hidden Markov Model.

[0059] The recursive formula described above consists of two phases: initialization and recursion. At the initial time t=1, the maximum path probability equals the product of the initial probability and the observation probability, representing the joint probability that the sequence starts in a certain state and produces that observation. In subsequent time steps, the recursive formula decomposes the search for the globally optimal path into a step-by-step recursive process of selecting the path that maximizes the cumulative probability from all preceding states. When a transition probability in the state transition matrix is ​​zero, the cumulative probability of the corresponding path is also zero, and this illegal path is naturally eliminated during the recursive process.

[0060] In one embodiment, the method further includes: generating control commands based on the modified motion sequence to drive the auxiliary motion device to perform auxiliary actions.

[0061] In one embodiment, the assistive motion device adopts a hierarchical control strategy. The upper layer outputs discrete motion intention tags according to the modified motion sequence, and the lower layer responds to motion switching through a finite state machine. When motion switching is detected, the joint angle reference curve of the corresponding motion is retrieved from the pre-stored reference trajectory library, and a control torque is generated accordingly.

[0062] like Figure 9 As shown, this embodiment employs a hierarchical control architecture to map the recognition results into control commands for the extremity robot. The upper-level control outputs discrete motion intention labels based on the corrected motion sequence. When a change in motion label is detected—for example, from standing to sitting—the lower-level finite state machine responds to this event, retrieving the corresponding hip and knee joint angle reference curves from a pre-stored reference trajectory library using a lookup table method, and then controlling the control... Generate control torque to drive the joints of the exolimb robot to perform corresponding auxiliary actions.

[0063] Because the action sequence input from the upper layer has undergone Markov logic correction, eliminating unreasonable state transitions, the lower-level controller will not receive action switching instructions that violate physical logic, thus avoiding dangerous sudden torques caused by identification errors. Figure 9 As shown, the joint angle tracking curve and output torque curve are displayed. The control commands, after logic correction, can achieve smooth motion switching, ensuring the safety and robustness of the external limbs in the human-machine collaboration process.

[0064] In one embodiment, the preset action category includes steady-state actions and transition actions; the steady-state actions include at least one of standing, sitting, squatting, and walking; the transition actions include at least one of sitting to standing, standing to sitting, squatting to standing, standing to squatting, turning left, and turning right.

[0065] like Figure 5 As shown, this embodiment defines ten representative movement categories covering daily rehabilitation training scenarios. Among them, standing (ST), sitting (SI), squatting (SQ), and walking (WA) are steady-state movements, characterized by long duration and relatively stable movement patterns. From sitting to standing (SItoST), from standing to sitting (STtoSI), from squatting to standing (SQtoST), from standing to squatting (STtoSQ), turning left (TL), and turning right (TR) are transitional movements, characterized by short duration, drastic changes in muscle activation patterns, and the moment with the highest risk of falling.

[0066] As shown in Figure 6, the model in this embodiment achieves high accuracy ( Figure 6a ), recall rate ( Figure 6b ) and F1 score ( Figure 6c The model outperforms the baseline model in all three metrics. As shown in the radar chart, the performance difference between the two models is small for large-sample steady-state actions such as standing, sitting, and walking; however, for sparse key transition actions such as squatting to standing, turning left, and sitting to standing, the recognition performance of this embodiment's model is significantly better than the baseline model. Specifically, through the weighting of the action-specific attention mechanism, this embodiment improves the F1 score of the high-risk squatting to standing (SQtoST) action by approximately 11.74% compared to the baseline model, and improves it by approximately 12.75% for the left turn (TL) action, effectively overcoming the class imbalance problem in long sequence data.

[0067] like Figure 7 As shown, the vertical axis represents the true action label, and the horizontal axis represents the predicted action label. In the confusion matrix during the inference phase, darker colors represent a larger number of samples. The matrix shows that the main values ​​are highly concentrated on the diagonal, indicating that the model can correctly classify the vast majority of action categories, resulting in a high overall recognition accuracy. In particular, for easily confused similar actions (such as the action of standing up from different states), the model still maintains extremely high recognition accuracy, and there are very few misclassifications in the off-diagonal region. This verifies the effectiveness of the Markov logic correction module in eliminating unreasonable state transitions, ensuring the accuracy and logical consistency of the final output sequence.

[0068] This application adopts a multimodal sensor fusion strategy of sEMG and IMU, which utilizes the leading characteristics of neuromuscular activation reflected by sEMG signals and the stability of limb kinematics reflected by IMU signals. The complementarity of the two can enrich the perceived data and significantly improve the model's recognition robustness in complex dynamic environments.

[0069] This application employs a sliding window-based action-specific attention mechanism, which calculates attention weights within a local window using generated dynamic query vectors. This forces the model to focus on sparse but information-rich transition action segments, effectively solving the problem of low key action recognition rate in long-sequence continuous motion due to the imbalance between steady-state and transition action categories.

[0070] This application uses Markov chains as logic correctors and utilizes the state transition probability matrix obtained from prior statistics to perform global path optimization on the preliminary prediction sequence of the neural network. This effectively eliminates illegal state transitions that violate human biomechanics and ensures the logical rationality of the output action sequence.

[0071] This application adopts an attention computation strategy based on local sliding windows. Compared with the global attention mechanism, it reduces the computational complexity from the quadratic level of the sequence length to the linear level. While preserving local key temporal features, it significantly reduces computational and storage overhead, enabling high-precision real-time inference at a lower time cost.

[0072] This application adopts a hierarchical control mapping strategy, which drives the underlying controller based on the logically corrected action sequence. This eliminates dangerous sudden torque caused by recognition errors, and the external limb will not produce erroneous movements that violate the laws of physics. Thus, the external limb robot can be accurately and safely controlled to drive the subject to rehabilitation exercises.

[0073] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0074] This application also provides a system for implementing the above methods. The system framework mainly includes a multimodal data acquisition and preprocessing module, an action-specific feature encoding module, a Markov logic correction module, and an external limb control mapping module. The multimodal data acquisition and preprocessing module is used to simultaneously acquire surface electromyography (sEMG) signals and inertial measurement unit (IMU) data, and perform filtering, noise reduction, multi-source data alignment, and multi-domain feature extraction preprocessing on them. The action-specific feature encoding module addresses the problem of sparse key transition action samples in long, continuous action sequences, leading to recognition difficulties. It introduces a sliding window-based action-specific attention mechanism, generating dynamic query vectors to calculate attention weights within a local window, forcing the model to focus on sparse but information-rich transition action segments, thereby obtaining an enhanced preliminary action probability sequence. The Markov logic correction module addresses the problem of pure data-driven model outputs lacking biomechanical constraints and prone to logical jumps. It constructs a Hidden Markov Model framework, using the state transition probability matrix obtained from prior statistics as a hard constraint. It uses the Viterbi algorithm to perform global path optimization on the preliminary probability sequence, eliminating illegal action transitions that violate physiological logic, and outputting the optimal action sequence that conforms to physical laws. The extremity control mapping module, based on the corrected discrete action intent, calls in real-time a preset hip and knee joint reference trajectory library and torque curves to drive the extremity robot to assist the subject in completing smooth action transitions. This invention, through a recognition strategy based on action-specific attention and logical correction, addresses the class imbalance problem using dynamic weights while ensuring the biomechanical rationality of the output sequence through logical constraints, thereby improving the safety and accuracy of extremity human-machine collaborative control.

[0075] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0076] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recognizing lower limb gait movements, characterized in that, include: Collect and preprocess multi-source sensor data from the lower limbs to construct a multimodal feature sequence; The multimodal feature sequence is input into a bidirectional recurrent neural network for feature encoding to obtain a hidden state sequence. A dynamic query vector is generated for a preset action category through an action-specific attention mechanism. Attention weights are calculated within a local sliding window centered on the current time. The hidden state sequence is weighted and fused based on the attention weights, and the fusion result is mapped to a preliminary action probability sequence by a classifier. The initial action probability sequence is used as the observation probability, and a hidden Markov model is constructed by combining it with a preset state transition probability matrix. The hidden Markov model is then decoded globally based on the state transition probability matrix to obtain a corrected action sequence that satisfies the biomechanical state transition constraints.

2. The lower limb gait recognition method according to claim 1, characterized in that, The process of collecting and preprocessing multi-source sensor data from the lower limbs to construct a multimodal feature sequence includes: Simultaneously acquire surface electromyography signals and inertial measurement unit data of the lower limbs; The surface electromyography signal is band-pass filtered, and the inertial measurement unit data is low-pass filtered. The data from the inertial measurement unit is upsampled to match the sampling rate of the surface electromyography signal using interpolation, thereby achieving temporal alignment of the multimodal data. The aligned channel data is standardized, and time-frequency domain features are extracted using a sliding window to construct the multimodal feature sequence.

3. The lower limb gait recognition method according to claim 2, characterized in that, The surface electromyography (EMG) signals are acquired by sensors placed on the surface of target muscle groups in the lower limb, including at least one of the rectus femoris, vastus medialis, semitendinosus, tibialis anterior, medial head of gastrocnemius, and soleus. The inertial measurement unit (IMU) data are acquired by sensors placed at at least one location in the thigh, calf, and dorsum of the foot. The passband frequency range of the bandpass filter is 20-450 Hz, and the cutoff frequency of the low-pass filter is 10 Hz. The interpolation method is cubic spline interpolation. The normalization process is Z-score normalization. The time-frequency domain features include at least one of the following: mean absolute value, root mean square (RMS), waveform length, mean frequency, and median frequency.

4. The lower limb gait recognition method according to claim 1, characterized in that, The bidirectional recurrent neural network is a bidirectional long short-term memory network; the step of generating dynamic query vectors for preset action categories through an action-specific attention mechanism, and calculating attention weights within a local sliding window centered on the current time, includes: For each of the preset action categories, the hidden state at the current moment is fused with the trainable query vector corresponding to that action to generate a dynamic query vector; Within a local sliding window centered at the current time and of size , the alignment score between the dynamic query vector and the hidden state at each time step within the window is calculated, and the attention weight is obtained by normalization. The hidden states within the window are weighted and summed according to the attention weights to generate a context vector. The context vector is then concatenated with the hidden state at the current moment and input into the classifier to obtain the preliminary action probability sequence.

5. The lower limb gait recognition method according to claim 4, characterized in that, The formula for generating the dynamic query vector is: in, For dynamic query vectors, For the current moment t The hidden state, For each of the preset action categories, the first Trainable query vectors for action classes Hidden state The corresponding weight matrix query vector The corresponding weight matrix, For bias terms, c Index for action categories; The formula for calculating the alignment score is: The formula for calculating the attention weight is: in, For the first in the window A time relative to time Action categories Alignment score; These are the corresponding normalized attention weights; For dynamic query vectors transpose; A trainable weight matrix for attention alignment; For the first in the window The hidden state vector at each time step; Indicated by The local sliding window range centered on; Index of the time position within the window; This is the index of the time position in the normalized summation.

6. The lower limb gait recognition method according to claim 1, characterized in that, The global optimal path decoding of the Hidden Markov Model based on the state transition probability matrix yields a corrected action sequence that satisfies biomechanical state transition constraints, including: The true intention of the action is taken as the hidden state, and the classification probability at each moment in the preliminary action probability sequence is taken as the observation probability. A state transition probability matrix is ​​constructed based on the legal transition relationships between human lower limb movements according to pre-statistics, and state transition probabilities that do not conform to biomechanical logic are set to zero. The Viterbi algorithm is used to recursively optimize the entire time series to find the optimal path with the highest posterior probability under the condition of satisfying the state transition constraints, which is then used as the corrected action sequence.

7. The lower limb gait recognition method according to claim 6, characterized in that, The core recursive formula of the Viterbi algorithm is: in, To reach state at time 1 The maximum path probability, For at any time Arrival Status The maximum path probability, For state The initial probability; From state Transition to state The probability, For a moment Observed output And the state is The probability of observation, For a moment The observed values, and This is the state index in the Hidden Markov Model.

8. The lower limb gait recognition method according to claim 1, characterized in that, Also includes: Control commands are generated based on the modified action sequence to drive the auxiliary motion device to perform auxiliary actions.

9. The lower limb gait recognition method according to claim 8, characterized in that, The auxiliary motion device adopts a hierarchical control strategy. The upper layer outputs discrete motion intention tags according to the corrected motion sequence, and the lower layer responds to motion switching through a finite state machine. When motion switching is detected, the joint angle reference curve of the corresponding motion is retrieved from the pre-stored reference trajectory library, and control torque is generated accordingly.

10. The lower limb gait recognition method according to claim 1, characterized in that, The preset action categories include steady-state actions and transition actions; the steady-state actions include at least one of standing, sitting, squatting, and walking; the transition actions include at least one of sitting to standing, standing to sitting, squatting to standing, standing to squatting, turning left, and turning right.

Images