Lightweight multi-mode lower limb motion intention recognition method and system

Abstract

The invention discloses a lightweight multi-mode lower limb motion intention recognition method and system, and relates to the technical field of biomedicine. The method comprises the steps that multi-channel surface electromyogram signals sEMG of a target lower limb and joint angle signals and joint torque signals of lower limbs on the same side are synchronously collected; performing feature extraction by using a double-branch structure to obtain muscle-related deep time sequence features and joint-related high-level semantic features; performing feature fusion through a bidirectional cross attention mechanism to obtain cross attention fusion features; and performing flattening, nonlinear mapping, regularization and Softmax classification on the cross attention fusion features, and outputting the motion intention of the target lower limb. Through double-branch input, depth feature extraction and a bidirectional cross attention mechanism, on the premise of ensuring recognition precision, a lightweight attention module, a residual structure and a cross-modal interaction mechanism are introduced, and the parameter quantity and calculation overhead are reduced.

Description

Technical Field

[0001] This invention relates to the field of biomedical technology, and in particular to a lightweight multimodal method and system for recognizing lower limb movement intentions. Background Technology

[0002] With the development of intelligent rehabilitation and assistive technologies, lower limb motor function reconstruction systems based on functional electrical stimulation have become an important means to help paralyzed patients regain their walking ability. The core premise of precise and natural control is the rapid and accurate identification of the patient's movement intentions.

[0003] In existing technologies, the model structure is complex and computationally expensive: its dynamic graph construction and graph convolution operations involve complex matrix operations and iterative processes, resulting in high overall computational complexity (large MAC values) and slow inference speed, making it difficult to achieve low-latency real-time operation on resource-constrained embedded processors.

[0004] Hardware deployment difficulties: The huge computational load and memory consumption make it unsuitable for direct deployment in wearable FES control devices with limited computing power, storage space and battery capacity, thus limiting its potential for clinical application.

[0005] The fusion process has weak interpretability: the generation and updating of dynamic graphs is relatively complex, which is not conducive to engineers' understanding and debugging of the model, and poses certain obstacles in medical scenarios that require high reliability.

[0006] Therefore, existing technologies, especially the best-performing multimodal fusion methods, generally suffer from a contradiction between model computation efficiency and recognition accuracy when applied to practical embedded FES systems. There is a lack of lightweight and efficient multimodal fusion methods that can achieve ultra-low parameter count, ultra-low computational complexity, and ultra-low inference latency while maintaining extremely high recognition accuracy (>99%). This is a problem that needs to be solved by those in the field. Summary of the Invention

[0007] The purpose of this invention is to provide a lightweight multimodal lower limb movement intention recognition method and system, which aims to solve or improve at least one of the above-mentioned technical problems.

[0008] To achieve the above objectives, the present invention provides the following solution: A lightweight, multimodal lower limb movement intention recognition method includes: Simultaneously acquire multi-channel surface electromyography (sEMG) signals of the target lower limb, as well as joint angle and torque signals of the ipsilateral lower limb, and perform preprocessing to obtain sEMG time-series data and joint feature vectors; A dual-branch structure was used to extract features from sEMG time series data and joint feature vectors to obtain deep temporal features related to muscles and high-level semantic features related to joints. By using a bidirectional cross-attention mechanism, deep temporal features and high-level semantic features are fused to obtain cross-attention fused features; The cross-attention fusion features are flattened, nonlinearly mapped, regularized, and classified using Softmax to output the movement intention of the target lower limb.

[0009] Further preprocessing includes: The sEMG signal is filtered and rectified to suppress noise; The joint angle signals and joint torque signals are synchronized and time-windowed; after time alignment, they are stitched together.

[0010] Furthermore, the dual-branch structure is used for feature extraction from sEMG time-series data, including: After permuting the sEMG time series data, the first-level output features are generated by two-stage one-dimensional convolution, SE channel attention recalibration, and max pooling. The first-level output features are processed by two layers of one-dimensional convolution and an SE channel attention module to maintain the sequence length and enhance key features, thereby generating the second-level output features. The first-level output features and the second-level output features are concatenated along the channel dimension to obtain fused features containing multi-scale information; One-dimensional global average pooling is performed on the fused features to obtain the global feature vector; The global feature vector is input into the third fully connected layer with ReLU as the activation function to obtain deep temporal features.

[0011] Furthermore, after two levels of one-dimensional convolution, SE channel attention recalibration, and max pooling, the first-level output features are generated, including: First convolutional layer: Using a one-dimensional convolutional kernel with ReLU as the activation function, preliminary feature extraction is performed to obtain the first feature map; The second convolutional layer uses a one-dimensional convolutional kernel with ReLU as the activation function to further extract features from the first feature map, resulting in the second feature map. SE Channel Attention Module: For the second feature map, the Squeeze-and-Excitation module is introduced to recalibrate the channel features of the second feature map to obtain the first calibrated feature map; The max pooling layer downsamples the first calibration feature map using one-dimensional max pooling to obtain the first-level output features.

[0012] Furthermore, the Squeeze-and-Excitation module includes: A global average pooling layer is used to perform global average pooling on each channel of the second feature map to obtain the channel description vector. The first fully connected layer maps the channel description vector to a low-dimensional latent space and introduces non-linear expressive power through the ReLU activation function to obtain a low-dimensional representation. The second fully connected layer remaps the low-dimensional representation back to the original channel dimensions, generates attention weights for each channel, and obtains a weight vector. The reshaping layer multiplies the weight vector with the second feature map channel by channel to recalibrate the channel features and obtain the first calibrated feature map.

[0013] Furthermore, the dual-branch structure extracts features from the joint feature vectors, including: The joint feature vector is input into the fourth fully connected layer with ReLU as the activation function, and nonlinear transformation and feature dimensionality upscaling are performed to obtain the joint embedding vector. The joint embedding vectors are augmented using a ResMLP module with Dropout to obtain joint residual augmentation features; The joint residual enhancement features are input into the seventh fully connected layer with ReLU as the activation function to obtain high-level semantic features.

[0014] Furthermore, through a bidirectional cross-attention mechanism, deep temporal features and high-level semantic features are fused to obtain cross-attention fused features, including: The dimensions of deep temporal features and high-level semantic features are reshaped to obtain reshaped sEMG feature sequences and reshaped dynamic feature sequences; Through bidirectional cross-attention, the reconstructed sEMG feature sequence and the reconstructed dynamics feature sequence serve as mutual queries and keys. After multi-head attention calculation and fusion, the first attention feature and the second attention feature are generated. After adding the first attention sequence and the second attention sequence element by element, the ReLU activation function is used to introduce non-linearity, resulting in cross-attention fusion features.

[0015] Furthermore, through bidirectional cross-attention, the reconstructed sEMG feature sequence and the reconstructed dynamics feature sequence serve as mutual queries and keys. After multi-head attention computation and fusion, first attention features and second attention features are generated, including: The first attention layer uses the reconstructed sEMG feature sequence as the query and the reconstructed dynamic feature sequence as the key and value, respectively, through a learnable linear transformation matrix. , , After mapping, multi-head scaling dot product attention is calculated, the outputs of all heads are concatenated along the feature dimension, and fused through a learnable fully connected layer to obtain the first attention sequence; The second attention layer uses the reconstructed dynamics feature sequence as the query and the reconstructed sEMG feature sequence as the key and value, respectively, and performs linear transformations using matrices. , , After mapping, multi-head scaling dot product attention is calculated, the outputs of all heads are concatenated along the feature dimension, and then fused through a learnable fully connected layer to obtain the second attention sequence.

[0016] Furthermore, the cross-attention fusion features are flattened, nonlinearly mapped, regularized, and classified using Softmax to output the target lower limb's movement intention, including: The cross-attention fusion features are straightened to obtain a one-dimensional feature vector; One-dimensional feature vectors are input into a fully connected layer with ReLU as the activation function to obtain fused discriminative features; Regularization is applied to the fused discriminative features to obtain regularized discriminative features; Regularized discriminative features are input into a fully connected layer, and the Softmax activation function is used to output a probability distribution vector of the motion category. The motion category with the highest probability in the probability distribution vector is taken as the motion intention of the target lower limb.

[0017] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a lightweight multimodal lower limb movement intention recognition method and system. The method achieves efficient fusion of a lightweight neural network model through dual-branch input, deep feature extraction, and a bidirectional cross-attention mechanism. While ensuring recognition accuracy, the introduction of a lightweight attention module, residual structure, and cross-modal interaction mechanism reduces the number of parameters and computational overhead, thereby meeting the stringent resource and real-time requirements of embedded deployments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating the sEMG temporal feature extraction branch in this embodiment; Figure 3 This is a flowchart illustrating the joint motion feature extraction branch in this embodiment; Figure 4 These are MCC radar charts for different categories in this embodiment; Figure 5 This is the micro-average ROC curve in this embodiment; Figure 6 This is the confusion matrix diagram in this embodiment; Figure 7 This is a schematic diagram of the t-SNE visualization results of the real labels of different action categories in this embodiment; Figure 8 This is a schematic diagram of the t-SNE visualization results for different action category prediction labels in this embodiment. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The purpose of this invention is to provide a lightweight multimodal lower limb movement intention recognition method and system, which aims to solve or improve at least one of the above-mentioned technical problems.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Definitions: sEMG (Surface Electromyography): A technique that records electrical activity signals of muscles using electrodes on the skin surface.

[0024] SE Module (Squeeze-and-Excitation Module): A lightweight channel attention mechanism.

[0025] ResMLP (Residual Multi-Layer Perceptron): A fully connected network structure that includes skip connections (residual connections).

[0026] MACs (Multiply-Accumulate Operations): The number of multiply-accumulate operations, a common metric used to measure the computational complexity of a model.

[0027] Softmax: A function that maps any real vector to a probability distribution (where the sum of its elements is 1), often used in the output layer of multi-class classification tasks.

[0028] Example 1 like Figure 1 As shown, this invention provides a lightweight multimodal lower limb movement intention recognition method, comprising: Step 1: Simultaneously acquire multi-channel surface electromyography (sEMG) signals of the target lower limb, as well as joint angle and torque signals of the ipsilateral lower limb, and perform preprocessing to obtain sEMG time-series data and joint feature vectors, including: The sEMG signal is filtered and rectified to suppress noise, resulting in sEMG time series data with dimensions (C,T), where C is the number of channels and T is the time series length. The joint angle signals and joint torque signals are synchronized and segmented by time windows; after time alignment, they are spliced ​​to obtain a joint feature vector of dimension F, where F is the feature dimension.

[0029] The joint angle signal can be estimated by an inertial measurement unit (IMU); the joint torque signal can be obtained through inverse dynamics calculation or by using a specific sensor, such as a torque sensor.

[0030] Step 2: Use a dual-branch structure to extract features from sEMG time-series data and joint feature vectors to obtain deep temporal features related to muscles and high-level semantic features related to joints, including: like Figure 2 As shown, the sEMG temporal feature extraction branch extracts deep temporal features related to muscle activation patterns based on sEMG temporal data, including: S1, perform a permutation operation on the sEMG time series data; for example, adjust the input with dimension (9,256) to (256,9) through the permutation operation; S2, after the permutation operation, the sEMG time series data is subjected to two levels of one-dimensional convolution, SE channel attention recalibration and max pooling to generate the first level of output features; First convolutional layer: Uses 16 one-dimensional convolutional kernels of size 5, with ReLU as the activation function, to perform preliminary feature extraction and obtain the first feature map; , In the formula, For the first c One convolutional kernel; The first convolutional kernel under the current kernel coverage s One input; This represents the number of convolution kernels.

[0031] The second convolutional layer uses 32 one-dimensional convolutional kernels of size 5, with ReLU as the activation function, to further extract features from the first feature map and obtain the second feature map. SE Channel Attention Module: For the second feature map, the Squeeze-and-Excitation module is introduced to recalibrate the channel features of the second feature map to obtain the first calibrated feature map; , In the formula, This is the Squeeze operation function, which corresponds to Global Average Pooling. : Input a two-dimensional matrix of channel c in the feature map, with a size of H×W ; W : The width of the input feature map; H : Input the height of the feature map; ( i , j ): Channel c at position ( i , j The value of ); The SE channel attention module can be replaced by other lightweight attention mechanisms, such as convolutional block attention modules or efficient channel attention. These modules can also achieve adaptive adjustment of channel weights with similar computational overhead.

[0032] The Squeeze-and-Excitation module includes: The global average pooling layer performs global average pooling (Squeeze operation) on each channel of the second feature map to obtain the channel description vector; The first fully connected layer maps the channel description vector to a low-dimensional latent space and introduces non-linear expressive power through the ReLU activation function to obtain a low-dimensional representation. The second fully connected layer remaps the low-dimensional representation back to the original channel dimensions, generates attention weights for each channel, and obtains a weight vector. The reshaping layer multiplies the weight vector with the second feature map channel by channel to recalibrate the channel features and obtain the first calibrated feature map.

[0033] The aforementioned SE channel attention module dynamically enhances the feature responses of key muscle channels related to the current movement intention with extremely low computational cost, while suppressing the influence of irrelevant or noisy channels, thereby improving the discriminative power of the features.

[0034] The max pooling layer performs downsampling using a one-dimensional max pooling method with a pooling size of 2 to reduce the feature dimension and obtain the first-level output features. S3 processes the first-level output features through two layers of one-dimensional convolution and an SE channel attention module, preserving the sequence length and enhancing key features to generate the second-level output features, including: The third convolutional layer uses 64 one-dimensional convolutional kernels of size 3, maintains the sequence length by using same padding, and uses ReLU as the activation function to extract features from the first-level output features to obtain the third feature map. The fourth convolutional layer uses 64 one-dimensional convolutional kernels of size 3, maintains the sequence length by using same padding, and uses ReLU as the activation function to extract features from the third feature map to obtain the fourth feature map. The SE channel attention module introduces the Squeeze-and-Excitation module to recalibrate the channel features of the fourth feature map, resulting in a second calibrated feature map, which serves as the second-level output feature. S4. The first-level output features and the second-level output features are concatenated along the channel dimension to obtain fused features containing multi-scale information. S5, perform one-dimensional global average pooling on the fused features to obtain the global feature vector; S6 inputs the global feature vector into the third fully connected layer with 128 neurons and ReLU activation function to obtain 128-dimensional deep temporal features.

[0035] The aforementioned sEMG temporal feature extraction branch extracts the spatiotemporal patterns of sEMG signals through a convolutional network and selectively enhances key features using a lightweight SE attention mechanism, providing high-quality temporal semantic representations for subsequent fusion.

[0036] like Figure 3 As shown, the joint motion feature extraction branch extracts high-level semantic features that reflect the limb's motion state and dynamic characteristics based on the joint feature vector, including: S1, input the joint feature vector into the fourth fully connected layer with 64 neurons and ReLU as the activation function, perform preliminary nonlinear transformation and feature dimensionality upscaling to obtain the joint embedding vector; S2, joint residual enhancement features are obtained by performing residual enhancement on the joint embedding vector through the ResMLP module with Dropout; The ResMLP module includes: The fifth fully connected layer has 64 neurons and uses the ReLU activation function to process the joint embedding vector to obtain the first residual branch features. Dropout layer: Randomly discards a portion of the neuron outputs of the first residual branch features with a probability of 0.1 to prevent overfitting and obtain regularized features; The sixth fully connected layer has 64 neurons and uses linear activation to process regularized features, resulting in residual transformation features. Residual connection: The joint embedding vector is added element-wise to the residual transformation feature to obtain the joint residual enhancement feature.

[0037] The aforementioned ResMLP module further models the complex nonlinear relationship between angle and torque through a multilayer perceptron; it utilizes residual connections to alleviate the gradient vanishing problem that may occur in deep networks, ensuring the stability of network training and preserving important information in the input.

[0038] The ResMLP module mentioned above can be replaced by a standard multilayer perceptron (without residual connections), but the training stability may be slightly worse; it can also be replaced by a more complex small-scale temporal convolutional network if the dynamic data itself is in time series form.

[0039] S3 inputs the joint residual enhancement features into the seventh fully connected layer with 128 neurons and ReLU activation function, and maps them to obtain 128-dimensional high-level semantic features.

[0040] The aforementioned joint motion feature extraction branch transforms the original joint state data into high-level semantic features through a fully connected network and residual structure, providing stable kinematic and dynamic context information for fusion.

[0041] Step 3: Through a bidirectional cross-attention mechanism, deep temporal features and high-level semantic features are fused to obtain cross-attention fused features, including: Step 31: Reshape the deep temporal features and high-level semantic features to obtain a sequence with dimensions (batch_size, 1, 128), resulting in the reshaped sEMG feature sequence and the reshaped dynamic feature sequence. Step 32: Through bidirectional cross-attention, the reconstructed sEMG feature sequence and the reconstructed dynamics feature sequence are used as mutual queries and keys. After multi-head attention calculation and fusion, first attention features and second attention features are generated, including: Bidirectional cross-attention includes: The first attention layer uses the reconstructed sEMG feature sequence as the query and the reconstructed dynamic feature sequence as the key and value, respectively, through a learnable linear transformation matrix. , , After mapping, multi-head scaling dot product attention is calculated, the outputs of all heads are concatenated along the feature dimension, and fused through a learnable fully connected layer to obtain the first attention sequence; , In the formula, Q , K and V These represent the query, key, and value matrices, respectively. The dimension of the key vector; for K The transpose of the matrix; For activation functions; The first attention layer described above allows the sEMG features to actively "question" the dynamic features, looking for the kinematic / dynamic context information most relevant to the current muscle activity pattern, and then incorporating it.

[0042] The second attention layer uses the reconstructed dynamics feature sequence as the query and the reconstructed sEMG feature sequence as the key and value, respectively, and performs linear transformations using matrices. , , After mapping, multi-head scaling dot product attention is calculated, the outputs of all heads are concatenated along the feature dimension, and then fused through a learnable fully connected layer to obtain the second attention sequence; The aforementioned second attention layer allows the dynamic features to actively "question" the sEMG features, searching for muscle activation patterns that can explain the current limb movement state, thereby supplementing the information.

[0043] The role of the multi-head attention mechanism is to divide the 128-dimensional features into multiple subspaces along the head dimension (e.g., 4 heads, each subspace is 32-dimensional), and compute attention in parallel in each subspace.

[0044] This model introduces a dual-branch attention structure consisting of two multi-head attention modules to enhance the modeling capability of interactions between different features. Specifically, branch one and branch two output feature representations respectively. and .

[0045] In the first attention module, the output of branch one is... As a query matrix The output of branch two Simultaneously serving as a bond matrix Sum matrix Thus achieving To guide, We perform weighted modeling on the important features related to it to obtain the attention output of the first stage.

[0046] In the second attention module, the interaction between features is further swapped: the output of branch one is... Simultaneously serving as a query matrix Bond matrix The output of branch two As a value matrix The second-stage feature representation is obtained through attention calculation, which is used to characterize the selective aggregation relationship between the internal features of branch one and the information of branch two.

[0047] Finally, the outputs of the two attention modules are concatenated to form a fused feature representation, which serves as the input to subsequent networks. This dual-attention structure can model the relationship between two types of features from different interaction perspectives, thereby improving the sufficiency and discriminative power of feature representation.

[0048] Step 33: After adding the first attention sequence and the second attention sequence element by element, the ReLU activation function is used to introduce nonlinearity and obtain the cross-attention fusion feature.

[0049] Step 4: The cross-attention fusion features are flattened, nonlinearly mapped, regularized, and classified using Softmax to output the target lower limb's movement intention, including: Step 41: Straighten the cross-attention fusion features of dimension (batch_size, 1, 128) to obtain a one-dimensional feature vector of dimension (batch_size, 128); Step 42: Input the one-dimensional feature vector into a fully connected layer with 128 neurons and ReLU as the activation function to obtain the fused discriminative features; Step 43: Apply Dropout regularization to the fused discriminative features. In this embodiment, the dropout rate is 0.3, resulting in regularized discriminative features. By applying Dropout regularization, some neurons are randomly blocked, forcing the network to learn more robust features, effectively suppressing model overfitting and improving its ability to generalize to new subjects or new scenarios.

[0050] The Dropout regularization mentioned above can be partially replaced or combined with other regularization techniques, such as L1 / L2 weight regularization and batch normalization, to control model complexity and prevent overfitting.

[0051] Step 44: Input the regularized discriminative features into a fully connected layer with N neurons, and use the Softmax activation function to output a probability distribution vector. The movement category corresponding to the highest probability in the probability distribution vector is taken as the movement intention of the target lower limb.

[0052] The method of this invention significantly reduces computational complexity and inference latency without significantly sacrificing recognition accuracy, thereby achieving a synergistic optimization of high accuracy and high efficiency on embedded platforms.

[0053] Example 2 A lightweight, multimodal lower limb movement intention recognition system includes: The signal acquisition module acquires multi-channel surface electromyography (sEMG) signals, joint angle signals, and joint torque signals through a multi-channel surface electromyography sensor and a motion capture measurement system. The data processing module analyzes the movement intention of the target lower limb based on the surface electromyography (sEMG) signal, joint angle signal, and joint torque signal of the channel. The preferred device uses edge AI computing chips, such as NVIDIA Jetson Nano and Himax WiseEye, as embedded processors; it is electrically connected to the sEMG sensor via an analog-to-digital converter interface, and electrically connected to the motion capture and measurement system via a serial communication interface or data acquisition card to receive synchronous multimodal data.

[0054] The FES stimulation module is used to receive recognized motion intention commands; The specific device includes a microcontroller and an FES stimulator, whose control input interface is electrically connected to the output interface of the data processing and intent recognition module.

[0055] Step 1: Real-time synchronous acquisition of 9-channel electromyography signals from sEMG sensors (raw sampling rate, for example, 1000Hz), as well as angle (8-dimensional, including some coupled calculation dimensions) and torque (8-dimensional) signals from 6 joints of the left / right hip, knee, and ankle from the motion / kinetic system.

[0056] sEMG signal: Power frequency interference and baseline drift are filtered out by digital filtering algorithms (such as bandpass filtering 20-450Hz) on the embedded processor, and then full-wave rectification is performed.

[0057] Data alignment and window segmentation: To ensure time consistency, all signal streams are given a uniform timestamp. Continuous signals are segmented with a fixed window length (e.g., 256ms) and a sliding step size (e.g., 50ms) to form a series of synchronized data segments.

[0058] Data formatting: The sEMG data within each time window is organized into a 9×256 two-dimensional matrix. The joint angle and torque data within the same time window are concatenated to form a 16×1 vector.

[0059] Step 2: Input the data obtained in Step 1 into two independent feature extraction sub-networks simultaneously, including: The execution steps for sEMG temporal feature extraction channels include: Dimensional permutation: Permutation of the (9,256) matrix into (256,9) prioritizes the time dimension.

[0060] First-level convolution and SE attention: The first convolutional layer (static): 16 pre-trained one-dimensional convolutional kernels of size (5,9) are applied to perform convolution operations on the input, and the ReLU activation function is used to generate a feature map of size (252,16).

[0061] The second convolutional layer (static): 32 pre-trained one-dimensional convolutional kernels of size (5,16) are applied to the previous feature map to perform convolution operations, and the ReLU activation function is used to generate a feature map of size (248,32).

[0062] The SE module (dynamically active): For the (248, 32) feature map, it first performs global average pooling to obtain a one-dimensional channel description vector of length 32. Then, it passes through two fully connected layers (static weights). The first layer compresses the 32-dimensional vector to 4-dimensional, and the second layer restores it to 32-dimensional. A sigmoid function is then applied to generate a 32-dimensional channel weight vector. Finally, this weight vector is multiplied by the original (248, 32) feature map along the channel dimension to complete feature recalibration.

[0063] Downsampling: Perform one-dimensional max pooling with a window size of 2 on the feature map output by the SE module to obtain a feature map of (124,32).

[0064] Second-level convolution and SE attention: Third and fourth convolutional layers (static): Similarly, two convolutions are performed using pre-trained convolutional kernels (64 kernels, size 3) (both with 'same' padding to maintain length), resulting in feature maps of (124,64) and (124,64) respectively.

[0065] SE module (dynamic action): Perform the same SE channel attention operation as described above on the feature map of (124,64).

[0066] Feature fusion and global convergence: Concatenation: The output (124,32) after the first-level pooling is concatenated with the output (124,64) after the second-level SE module in the last dimension (channel dimension) to obtain the feature map (124,96).

[0067] Global average pooling: Perform global average pooling on the (124,96) feature map in the time dimension to obtain a one-dimensional global feature vector of length 96.

[0068] Final mapping: Input a 96-dimensional vector into a pre-trained fully connected layer with 128 neurons and ReLU activation, and output a 128-dimensional sEMG feature vector.

[0069] The execution process of the dynamic feature extraction channel includes: Input concatenation: The 16-dimensional joint angle / torque vector is directly used as input.

[0070] Basic transformation: Input vector into a pre-trained fully connected layer with 64 neurons and ReLU activation to obtain 64-dimensional features.

[0071] ResMLP module processing (dynamic function): Fully connected and activated: Input the 64-dimensional features into a pre-trained fully connected layer (64 neurons, ReLU activation) to obtain intermediate features.

[0072] Dropout: During the training phase, a portion of the intermediate features is randomly "turned off" with a probability of 0.1; during the inference / deployment phase, this operation is turned off, and all features are multiplied by a scaling factor of (1-0.1).

[0073] Fully connected and residual connections: The features after Dropout are input into another pre-trained fully connected layer (64 neurons, linear activation). The output of this layer is then added element-wise to the original input of the module (the output of step "Basic Transformation") to obtain the output of ResMLP.

[0074] Final mapping: The 64-dimensional features output by the ResMLP module are input into a pre-trained fully connected layer with 128 neurons and ReLU activation, which outputs a 128-dimensional dynamic feature vector.

[0075] Step 3, performing multimodal fusion based on bidirectional cross-attention, includes: Feature reshaping: The two 128-dimensional vectors obtained are reshaped into sequences of shape (1, 128).

[0076] Bidirectional cross-attention calculation (dynamic effect): Preparing queries, keys, and values: using a pre-trained linear transformation weight matrix. , , Map the sEMG feature sequences to the query. ,key ,value Similarly, using , , Map the dynamic feature sequence to , , .

[0077] Attention head partitioning: dividing each 128-dimensional head into its own parts. , , The features are divided into 4 groups (32 dimensions per group) on an average scale, forming 4 independent attention heads.

[0078] Calculate attention output: For the first sub-layer: calculate separately Each head and The dot product of the corresponding heads, divided by the scaling factor sqrt(32), is used to obtain the weights via Softmax, and then... The weighted sums of the corresponding heads yield the outputs of the four heads. These four outputs are then concatenated and passed through an output linear layer (with pre-trained weights) to obtain the final output O1 of the first sub-layer.

[0079] For the second sub-layer: and , Perform the same calculation as the previous step to obtain the output. Here , Originating from dynamic characteristics, This also stems from dynamic characteristics, which actually constitute a self-attention structure. Correction: According to the original scheme, the second sub-layer should be based on dynamic characteristics. sEMG features , Therefore , It should be passed by sEMG features , Mapped and executed. (Dynamics for query) and Subsequent operations such as dot product of (sEMG as key) are performed.

[0080] Fusion and activation: The outputs O1 and O2 of the two sub-layers are added element by element, and then the ReLU activation function is applied to the sum vector to generate the final 128-dimensional fused feature vector.

[0081] Step 4: Based on the fused feature vectors, perform motion category recognition and output, including: Fully Connected Layers and Dropout: A 128-dimensional fused feature vector is input into a pre-trained fully connected layer with 128 neurons and ReLU activation. During training, Dropout is performed on this layer with a probability of 0.3; during inference, Dropout is not performed, or scaling is applied.

[0082] Category prediction: The output of the previous layer is input into the final pre-trained fully connected layer (13 neurons) to obtain 13 scores (logits). The Softmax function is applied to transform these 13 scores into a probability distribution [p1,p2,...,p13].

[0083] Intent output: The category index with the highest probability, argmax([p1,p2,...,p13]), is taken as the motion intent identified in the current time window. This intent is encoded into a control command and sent to the FES stimulation module through the communication interface.

[0084] To verify the effectiveness of the method of the present invention, a rigorous five-fold cross-validation was performed on a public dataset (SIAT-LLMD) containing 13 types of lower limb movement patterns.

[0085] like Figure 4 As shown in the MCC radar charts for different categories, the method of the present invention demonstrates that the method has good classification performance.

[0086] As shown in Table 1, the method of the present invention achieves an average F1 score of 96.19% for each macro category; Table 1 Classification performance evaluation indicators for each type of action recognition

[0087] like Figure 5 As shown, the area under the micro-average ROC curve (AUC) is as high as 0.9997; like Figure 6 As shown, the confusion matrix is ​​concentrated on the diagonal, indicating that the method of the present invention has excellent overall classification performance and robustness.

[0088] As shown in Table 2, the total computational cost of the method of this invention is only 4.74 MMACs, and the average time for a single inference (processing data in a 256ms time window) is as low as 0.95 microseconds. Compared with the current best-performing baseline model (CNN-DGTG-WGIN, accuracy 99.71%), this invention reduces computational cost by approximately 63% and improves inference speed by approximately 108% while only reducing accuracy by 0.58 percentage points. Compared with other deep learning baseline models with large numbers of parameters and complex structures (such as CNN-LSTM, CNN-TCN, etc., whose computational cost is above 12 MMACs and inference time is between 1.33 and 2.55 microseconds), the efficiency advantage of this invention is more prominent.

[0089] Table 2. Comparison and analysis of MACs and average inference time for each model.

[0090] The above data demonstrates that the present invention can meet the stringent requirements of embedded devices for low power consumption, small storage, and fast response while maintaining classification performance comparable to state-of-the-art models, enabling real-time and accurate motion intent decoding on the microcontroller of a wearable FES system.

[0091] As shown in Table 3, on an independent test set of 10 different subjects, the recognition accuracy of the model of this invention remained stable with minimal fluctuations, ranking second among all comparative models in terms of average accuracy, and achieving the best performance on some subjects. This demonstrates that the model did not overfit to any specific individual and possesses good cross-subject generalization ability.

[0092] Table 3 Comparison of accuracy and average performance of each model on each subject test set.

[0093] like Figure 7 and Figure 8 As shown in the t-SNE visualization results, the high-dimensional features extracted and fused by the network of this invention exhibit clear inter-class separation and tight intra-class aggregation in the low-dimensional space. This indicates that the sEMG channel attention enhancement, dynamic residual feature extraction, and bidirectional cross-attention fusion mechanism designed in this invention can effectively learn action representations with strong discriminative power, ensuring the accuracy of classification at the feature level and improving the model's ability to distinguish similar actions (such as sitting down and standing up).

[0094] The above results demonstrate that the method of the present invention reduces the dependence on a single noisy signal and enhances robustness to individual differences and variations in action execution.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0096] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A lightweight multi-modal lower limb motion intention recognition method, characterized in that, The method comprises the steps of: synchronously collecting multi-channel surface electromyography (sEMG) signals of a target lower limb, joint angle signals and joint torque signals of a same-side lower limb, and performing preprocessing to obtain sEMG time series data and joint feature vectors; extracting features from the sEMG time series data and the joint feature vectors using a double-branch structure to obtain muscle-related deep time series features and joint-related high-level semantic features; performing feature fusion on the deep time series features and the high-level semantic features through a bidirectional cross-attention mechanism to obtain cross-attention fusion features; performing flattening, nonlinear mapping, regularization and Softmax classification on the cross-attention fusion features to output a movement intention of the target lower limb.

2. The lightweight multi-modal lower limb motion intention recognition method according to claim 1, characterized in that, The preprocessing comprises the steps of: performing filtering and rectification processing on the sEMG signals to suppress noise; performing synchronous alignment and time window segmentation on the joint angle signals and the joint torque signals; and performing splicing after time alignment.

3. The lightweight multi-modal lower limb motion intention recognition method according to claim 1, characterized in that, The double-branch structure comprises the steps of: performing permutation operation on the sEMG time series data, and then performing two-level one-dimensional convolution, SE channel attention recalibration and maximum pooling to generate first-level output features; performing two-layer one-dimensional convolution and SE channel attention module processing on the first-level output features to maintain the sequence length and strengthen the key features, and generating second-level output features; splicing the first-level output features and the second-level output features in the channel dimension to obtain fusion features containing multi-scale information; performing one-dimensional global average pooling on the fusion features to obtain a global feature vector; inputting the global feature vector into a third fully connected layer with a ReLU activation function to map and obtain deep time series features.

4. The lightweight multi-modal lower limb motion intention recognition method according to claim 3, characterized in that, The double-branch structure comprises the steps of: a first convolution layer: using a one-dimensional convolution kernel and a ReLU activation function to perform preliminary feature extraction to obtain a first feature map; a second convolution layer: using a one-dimensional convolution kernel and a ReLU activation function to further extract features from the first feature map to obtain a second feature map; an SE channel attention module: introducing a Squeeze-and-Excitation module to the second feature map to recalibrate the channel features of the second feature map to obtain a first calibrated feature map; a maximum pooling layer: performing one-dimensional maximum pooling on the first calibrated feature map to downsample the first calibrated feature map to obtain first-level output features.

5. The lightweight multi-modal lower limb motion intention recognition method according to claim 4, characterized in that, The Squeeze-and-Excitation module comprises: a global average pooling layer: performing global average pooling on each channel of the second feature map to obtain a channel description vector; a first fully connected layer: mapping the channel description vector to a low-dimensional hidden space and introducing a ReLU activation function to introduce nonlinear expression capability to obtain a low-dimensional representation; a second fully connected layer: remapping the low-dimensional representation to the original channel dimension to generate attention weights of each channel to obtain a weight vector; a reshaping layer: multiplying the weight vector and the second feature map channel by channel to recalibrate the channel features to obtain a first calibrated feature map.

6. The lightweight multi-modal lower limb motion intent recognition method according to claim 1, wherein, The double-branch structure comprises the steps of: The joint feature vector is input into a fourth fully connected layer with ReLU as the activation function for nonlinear transformation and feature dimensionality increase to obtain a joint embedding vector; The joint embedding vector is input into a ResMLP module with Dropout for residual enhancement to obtain a joint residual enhanced feature; The joint residual enhanced feature is input into a seventh fully connected layer with ReLU as the activation function for mapping to obtain a high-level semantic feature.

7. The lightweight multi-modal lower limb motion intent recognition method according to claim 1, wherein, The deep temporal feature and the high-level semantic feature are fused by a bidirectional cross-attention mechanism to obtain a cross-attention fusion feature, including: The dimensions of the deep temporal feature and the high-level semantic feature are reshaped to obtain a reshaped sEMG feature sequence and a reshaped kinetic feature sequence; The reshaped sEMG feature sequence and the reshaped kinetic feature sequence are taken as query and key values by the bidirectional cross-attention, and first attention features and second attention features are generated by multi-head attention calculation and fusion; The first attention sequence and the second attention sequence are added element by element, and then a ReLU activation function is used to introduce nonlinearity to obtain the cross-attention fusion feature.

8. The lightweight multi-modal lower limb motion intent recognition method according to claim 1, wherein, The reshaped sEMG feature sequence and the reshaped kinetic feature sequence are taken as query and key values by the bidirectional cross-attention, and first attention features and second attention features are generated by multi-head attention calculation and fusion, including: The first attention layer takes the reshaped sEMG feature sequence as a query, takes the reshaped kinetic feature sequence as a key and a value respectively, and respectively maps them through a learnable linear transformation matrix , , After the mapping, multi-head scaled dot-product attention calculation is performed, the outputs of all heads are spliced along the feature dimension, and a learnable fully connected layer is used for fusion to obtain the first attention sequence. The second attention layer takes the reshaped kinetic feature sequence as a query, takes the reshaped sEMG feature as a key and a value respectively, and respectively passes through a linear transformation matrix 、 、 After mapping, multi-head scaled dot-product attention calculation is performed, the outputs of all heads are spliced along the feature dimension, and fusion is performed through a learnable fully connected layer to obtain a second attention sequence.

9. The lightweight multi-modal lower limb motion intent recognition method according to claim 1, wherein, The cross-attention fusion feature is flattened, nonlinearly mapped, regularized, and classified by Softmax to output the movement intention of the target lower limb, including: The cross-attention fusion feature is flattened to obtain a one-dimensional feature vector; The one-dimensional feature vector is input into a fully connected layer with ReLU as the activation function to obtain a fusion discriminative feature; The fusion discriminative feature is regularized to obtain a regularized discriminative feature; The regularized discriminative feature is input into a fully connected layer, and a Softmax activation function is used to output a probability distribution vector of the movement class, and the movement class corresponding to the maximum probability in the probability distribution vector is taken as the movement intention of the target lower limb.

10. A lightweight multi-modal lower limb motion intention recognition system, characterized in that, including: A signal acquisition module acquires multi-channel surface electromyography signals sEMG, joint angle signals, and joint torque signals through a multi-channel surface electromyography sensor and a motion capture measurement system; A data processing module analyzes the movement intention of the target lower limb according to the channel surface electromyography signals sEMG, the joint angle signals, and the joint torque signals; An FES stimulation module is used to receive the recognized movement intention instruction.

Images