Dexterous hand motion decoding method based on collaborative base manifold

Abstract

The invention discloses a dexterous hand motion decoding method based on a collaborative base manifold, and relates to the technical field of man-machine interaction, and the method comprises the steps: collecting a multi-modal signal of a user, carrying out the noise reduction and feature extraction, and obtaining a high-dimensional motion feature matrix; learning from the high-dimensional motion feature matrix through a sparse coding dictionary learning method to obtain a collaborative base dictionary representing hand motion collaborative primitives, and projecting the high-dimensional motion feature matrix to a low-dimensional collaborative base manifold space by using a manifold embedding algorithm to form a general decoding reference; and in the use process of a user, acquiring a real-time feedback signal output by decoding, dynamically optimizing the collaborative base manifold topological structure of the general decoding reference and decoding matching parameters based on the feedback signal, and carrying out online personalized adaptation on the current user. According to the invention, intelligent evolution from a static and universal hand motion model to a dynamic and personalized decoder is realized.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method for dexterous hand motion decoding based on cooperative manifolds. Background Technology

[0002] With the rapid development of brain-computer interfaces and human-computer interaction technologies, precise motor control of highly dexterous hands has become a core challenge in fields such as rehabilitation engineering, intelligent prosthetics, and remote operation. Currently, motion decoding methods based on multimodal biosignals such as surface electromyography (sEMG) and inertial measurement units (IMUs) are gradually becoming a research hotspot. Traditional methods mainly rely on deep learning models (such as convolutional neural networks and recurrent neural networks) to perform end-to-end mapping of raw signals or manual features, achieving the recognition from signals to action intentions through extensive data training. In recent years, theories such as sparse coding and manifold learning have been introduced into the field of motion decoding, aiming to extract low-dimensional essential features from high-dimensional signals and improve the model's generalization ability. For example, some studies have attempted to extract muscle coordination patterns through independent component analysis or non-negative matrix factorization, or to construct a motion feature space using manifold dimensionality reduction methods such as isometric mapping and local linear embedding.

[0003] While existing technologies have achieved basic gesture classification and simple trajectory prediction to some extent, they face two major bottlenecks in complex scenarios. First, regarding cross-user generalization and dynamic adaptation, traditional methods heavily rely on training static models with large-scale labeled data, making it difficult to adapt to differences in electromyographic signal characteristics, wearing offsets, and fatigue states among different users. This necessitates retraining with new data for new users, resulting in a high practical threshold. Second, in balancing decoding accuracy and real-time performance, most solutions cannot simultaneously achieve high-precision continuous motion trajectory prediction and low latency: deep learning models, while highly accurate, have high computational complexity, making them difficult to run in real-time on embedded devices; while lightweight models, due to limited feature representation capabilities, struggle to capture the spatiotemporal dynamics of fine gestures. Furthermore, existing methods often lack explicit modeling of physiological constraints, potentially leading to decoding results that do not conform to hand biomechanics, affecting the naturalness and safety of control. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a dexterous hand motion decoding method based on cooperative manifolds to solve the problems of poor cross-user generalization ability and difficulty in achieving high-precision decoding in resource-constrained environments in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a dexterous hand motion decoding method based on cooperative basis manifolds. The method includes: acquiring multimodal signals from a user, performing noise reduction and feature extraction to obtain a high-dimensional motion feature matrix; learning a cooperative basis dictionary representing hand motion cooperative primitives from the high-dimensional motion feature matrix using a sparse coding dictionary learning method, and projecting the high-dimensional motion feature matrix onto a low-dimensional cooperative basis manifold space using a manifold embedding algorithm to form a universal decoding benchmark; acquiring real-time feedback signals from the decoding output during user interaction, dynamically optimizing the cooperative basis manifold topology and decoding matching parameters of the universal decoding benchmark based on the feedback signals, and performing online personalized adaptation for the current user; constructing a lightweight decoding model, and inputting the personalized low-dimensional manifold features into the lightweight decoding model to output a finger joint angle prediction sequence; performing short-term motion intent prediction on the finger joint angle prediction sequence to generate advanced control commands; converting the advanced control commands into a control protocol adapted to the target device, and driving the external device to complete the corresponding dexterous hand movements.

[0008] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifold described in this invention, the multimodal signals include electromyographic signals, inertial measurement unit data, and near-infrared spectral signals.

[0009] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the specific steps for obtaining the high-dimensional motion feature matrix are as follows:

[0010] Multimodal signals are acquired and time-aligned to obtain a synchronized multimodal signal stream;

[0011] Based on the synchronous multimodal signal stream, the quality index evaluation value of each modal signal is calculated, and the adaptive wavelet threshold algorithm is used for dynamic noise reduction to obtain the noise-reduced multimodal signal.

[0012] Time-domain, frequency-domain, and time-frequency-domain features are extracted from the denoised multimodal signal and weighted and fused through a collaborative attention mechanism to generate a high-dimensional motion feature matrix.

[0013] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative basis manifolds described in this invention, the specific steps for learning the cooperative basis dictionary representing hand motion cooperative primitives from the high-dimensional motion feature matrix using a sparse coding dictionary learning method are as follows:

[0014] Manifold structure initialization is performed on the high-dimensional motion feature matrix to obtain the atomic distribution of the cooperative basis dictionary;

[0015] Based on the atomic distribution of the collaborative base dictionary and combined with the context information of real-time motion tasks, a sparse coding objective function with dynamic topological constraints is constructed.

[0016] Perform iterative operations on the sparse coding objective function until the comprehensive convergence condition is met, and then obtain the cooperative base dictionary representing the cooperative primitives of hand movement.

[0017] In a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the steps for forming a universal decoding benchmark are as follows:

[0018] Based on the collaborative base dictionary, a continuous reference manifold is constructed in the feature space of the high-dimensional motion feature matrix;

[0019] Based on the high-dimensional motion feature matrix and the reference manifold, the coupling transmission relationship of motion semantic cost is defined;

[0020] The entropy-regularized optimal transmission algorithm is applied to the coupled transmission relationship to obtain the optimal coupling matrix between user data and the reference manifold.

[0021] Based on the optimal coupling matrix, the user's high-dimensional motion features are weighted and projected to generate a universal decoding benchmark in the low-dimensional cooperative basis manifold space.

[0022] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the following steps are taken: During user operation, real-time feedback signals of the decoding output are acquired; based on these feedback signals, the cooperative manifold topology of the general decoding benchmark and the decoding matching parameters are dynamically optimized; and online personalized adaptation is performed for the current user.

[0023] During user operation, the system acquires hand movements decoded from a universal decoding benchmark in real time, compares the hand movements with the expected movements, and generates real-time feedback signals.

[0024] Based on the feedback signal, a meta-reinforcement learning adapter with a general decoding benchmark as the initial strategy is constructed.

[0025] The collaborative basis manifold topology and decoding matching parameters of the general decoding benchmark are dynamically optimized through a meta-reinforcement learning adapter.

[0026] The dynamically optimized collaborative basis manifold topology is applied to the user's high-dimensional motion features to generate low-dimensional manifold features that are adapted to the current user.

[0027] When the low-dimensional manifold feature generation process reaches a stable state, online personalized adaptation is performed for the current user.

[0028] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the specific steps for constructing the lightweight decoding model are as follows:

[0029] Based on the convolutional neural network model, the low-dimensional manifold features are reduced in manifold dimension and optimized by algorithm pruning to obtain the pruned model architecture.

[0030] Based on the pruned model architecture, a neural architecture search space is constructed.

[0031] Define a multi-objective optimization function that integrates hardware constraints and knowledge distillation within the neural architecture search space;

[0032] The optimal lightweight model architecture is obtained by co-optimizing the multi-objective optimization function through differentiable neural architecture search.

[0033] Quantization-aware training is performed on the optimal lightweight model architecture to generate a lightweight decoding model.

[0034] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the specific steps for outputting the finger joint angle prediction sequence are as follows:

[0035] The personalized low-dimensional manifold features are input into the lightweight decoding model, which outputs the initial joint angle prediction.

[0036] A set of physical constraint relationships is established based on hand biomechanics and transformed into the corresponding potential energy function;

[0037] The initial joint angle prediction is used as the sampling starting point, and optimization iteration is performed in the constraint space defined by the potential energy function.

[0038] Posterior probability estimation and uncertainty quantification are performed on the joint angle samples obtained after optimization iteration to obtain the finger joint angle prediction sequence.

[0039] In a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the specific steps for generating advanced control commands are as follows:

[0040] Based on the finger joint angle prediction sequence, a temporal attention mechanism is used to calculate the contribution weight of historical time points and generate an attention-weighted historical feature vector.

[0041] The attention-weighted historical feature vector is input into the spatiotemporal graph neural network, and the hidden state sequence is output by combining the spatial coupling relationship between hand joints and temporal dynamics.

[0042] Short-term integral prediction of the hidden state sequence is performed to obtain the advanced joint angle prediction sequence.

[0043] The advanced joint angle prediction sequence is converted into advanced control commands by using inverse kinematic mapping.

[0044] As a preferred embodiment of the dexterous hand motion decoding method based on cooperative manifolds described in this invention, the specific steps of converting advanced control commands into a control protocol adapted to the target device and driving the external device to complete the corresponding dexterous hand movements are as follows.

[0045] Optimize the protocol parameters of the advanced control commands to generate optimized protocol parameters;

[0046] The optimized protocol parameters are converted into a control protocol adapted to the target device through a real-time performance awareness algorithm and sent to the external device driver interface.

[0047] The external device driver interface parses and executes the control protocol, driving the external device to complete the corresponding dexterous hand movements.

[0048] The beneficial effects of this invention are as follows: By constructing a dynamic optimization architecture that integrates a general decoding benchmark with online meta-learning adaptation, it realizes the intelligent evolution from a static, general hand motion model to a dynamic, personalized decoder, solving the core bottlenecks of poor cross-user generalization ability and high adaptation cost for new users caused by relying on static models; Through a precise control chain that combines Bayesian inference under physical constraints with advanced intent prediction, it achieves high-fidelity, low-latency decoding and control of dexterous hand movements on resource-constrained embedded platforms, solving the prominent contradiction that is difficult to balance between decoding accuracy, real-time performance and motion safety. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a dexterous hand motion decoding method based on cooperative manifolds.

[0051] Figure 2 A flowchart for establishing a universal decoding benchmark.

[0052] Figure 3 A flowchart for online personalized adaptation.

[0053] Figure 4 This is the optimal coupling matrix heatmap.

[0054] Figure 5 A graph showing the comparison of online adaptive errors for different decoding strategies. Detailed Implementation

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0056] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0058] Reference Figures 1-5 This is one embodiment of the present invention, which provides a dexterous hand motion decoding method based on cooperative manifolds, including the following steps:

[0059] S1. Collect the user's multimodal signals, and perform noise reduction and feature extraction to obtain a high-dimensional motion feature matrix.

[0060] Multimodal signals are acquired and time-aligned to obtain a synchronized multimodal signal stream.

[0061] It should be noted that electromyographic signals are acquired by an electrode array attached to the skin surface of the hand or forearm. Simultaneously, three-dimensional acceleration, angular velocity, and attitude information are recorded using an inertial measurement unit data acquisition device fixed to the same limb. Furthermore, changes in blood oxygen concentration and hemodynamic response are continuously monitored by a near-infrared spectral signal probe worn in the forearm muscle area. The above three types of multimodal signals are continuously acquired by their respective corresponding sensors within the same time period, and time axis calibration is performed according to a unified hardware clock or software timestamp mechanism. Linear interpolation or nearest neighbor alignment methods are used to eliminate timing deviations caused by differences in sampling frequency or transmission delays, forming a synchronous multimodal signal stream that is strictly corresponding in time.

[0062] Based on a synchronized multimodal signal stream, the quality index evaluation value of each modal signal is calculated. An adaptive wavelet threshold algorithm is used for dynamic noise reduction to obtain the denoised multimodal signal, expressed as:

[0063] ;

[0064] in, For the first Each signal mode at time point Quality indicator evaluation values Indices representing signal modes Indicates the current time point, The adjustment coefficient representing the signal-to-noise ratio impact factor. This represents the natural exponential function. The scaling parameter represents the signal-to-noise ratio. Indicates the first Each modality at time point signal-to-noise ratio, This represents the weighting coefficient of the signal energy term. Indicates the length of the integration time window. Indicates the first The modality at time... The signal amplitude, It represents a specific moment within a time window.

[0065] It should be noted that before calculating the quality index evaluation value, the modal signals in the synchronous multimodal signal stream have been dimensionally unified by the maximum value normalization method, so that the numerical range of all modal signals is mapped to the interval between zero and one, eliminating the influence of different physical dimensions and amplitude scales on subsequent fusion calculations.

[0066] Furthermore, The adjustment coefficient representing the signal-to-noise ratio (SNR) influence factor is used to adjust the impact of SNR on the quality index evaluation value. It is determined on the labeled data with noise reduction performance as the objective through grid search or gradient descent. The scaling parameter representing the signal-to-noise ratio controls the nonlinear sensitivity of the signal-to-noise ratio. It is obtained through offline tuning on synchronous multimodal signal stream labeled data with the goal of signal effectiveness or task performance, by means of grid search or Bayesian optimization. The weighting coefficients representing the signal energy term are used to balance the proportion of signal energy and are determined on labeled data with the goal of noise reduction effect or task performance through grid search or cross-validation.

[0067] The specific process includes: the quality index evaluation value is jointly determined by the signal-to-noise ratio and the local signal energy; the signal-to-noise ratio is weighted and combined with the square integral of the signal amplitude over a period of time after nonlinear transformation to form a comprehensive evaluation reflecting the reliability of the signal; the wavelet threshold is then dynamically adjusted based on the comprehensive evaluation to suppress noise components while retaining the effective components in the multimodal signal, and finally obtain the denoised multimodal signal.

[0068] It should be noted that the adaptive wavelet thresholding algorithm is a denoising method that dynamically adjusts the wavelet coefficient threshold based on the local features of the signal. By analyzing the statistical characteristics or signal quality indicators of the wavelet coefficients at each scale, it automatically determines the optimal threshold to suppress noise while preserving effective information (e.g., using subband energy, kurtosis, or signal-to-noise ratio to estimate the local noise level and adjust the threshold accordingly).

[0069] Time-domain, frequency-domain, and time-frequency-domain features are extracted from the denoised multimodal signal and weighted and fused through a collaborative attention mechanism to generate a high-dimensional motion feature matrix.

[0070] The specific process includes obtaining the time-domain, frequency-domain, and time-frequency-domain features of electromyography (EMG), inertial measurement unit (IMU) data, and near-infrared spectral signals from the denoised multimodal signals. The time-domain features include the root mean square (RMS) value and zero-crossing rate, the frequency-domain features include the dominant frequency component of the power spectral density, and the time-frequency-domain features include the energy distribution of the wavelet transform coefficients. The feature vectors of each mode in the three domains are input into a collaborative attention mechanism. Attention weights are generated through the interdependence between different modes and different feature domains. All features are then weighted and fused to generate a high-dimensional motion feature matrix.

[0071] S2. A collaborative basis dictionary representing hand motion collaborative primitives is learned from the high-dimensional motion feature matrix using the sparse coding dictionary learning method. The high-dimensional motion feature matrix is ​​then projected onto the low-dimensional collaborative basis manifold space using the manifold embedding algorithm to form a universal decoding benchmark.

[0072] The high-dimensional motion feature matrix is ​​initialized with a manifold structure to obtain the atomic distribution of the cooperative basis dictionary.

[0073] The specific process includes constructing a similarity graph based on the local neighborhood relationships between data points in the high-dimensional motion feature matrix, obtaining the low-dimensional feature vector of the Laplacian matrix of the similarity graph using the spectral embedding method, mapping the high-dimensional motion feature matrix to a low-dimensional manifold space that preserves the intrinsic geometric structure, and determining the position of the cooperative basis dictionary atoms in the low-dimensional manifold space through cluster centers or uniform sampling to obtain the distribution of cooperative basis dictionary atoms.

[0074] Based on the atomic distribution of the collaborative base dictionary and combined with real-time motion task context information, a sparse coding objective function with dynamic topological constraints is constructed.

[0075] The specific process includes: based on the geometric distribution of atoms in the cooperative basis dictionary in the low-dimensional cooperative basis manifold space, activating a subset of atoms related to the current motion task context information, and using the adjacency relationship of these atoms on the manifold to define a dynamically changing graph Laplacian regularization term. This graph Laplacian regularization term is combined with the reconstruction error term and the L1 sparsity penalty term to form an objective function that encourages sparse activation while maintaining the local connectivity of activated atoms on the manifold during the optimization process. This guides the high-dimensional motion features to be sparsely encoded under the premise of conforming to the task semantics and manifold structure.

[0076] It should be noted that real-time motion task context information refers to the category, target object, or operational intent of the current hand action, which is obtained through user instructions, visual recognition results, or task scheduling signals.

[0077] Perform iterative operations on the sparse coding objective function until the comprehensive convergence condition is met, and then obtain the cooperative base dictionary representing the cooperative primitives of hand movement.

[0078] The specific process includes distributing the cooperative base dictionary atoms as an initial base set, combining the sparse coding objective function with dynamic topological constraints, and repeatedly updating the sparse coefficients and cooperative base dictionary atoms using alternating optimization or gradient descent algorithms. In each iteration, the objective function value and the change in variables are obtained. When the decrease in the objective function value and the update magnitude of the cooperative base dictionary atoms are both lower than the convergence threshold, the comprehensive convergence condition is determined to be met. At this time, the output cooperative base dictionary is the cooperative base dictionary representing the cooperative primitives of hand movement.

[0079] It should be noted that the convergence threshold is preset based on the task's requirements for balancing decoding accuracy and computational efficiency, and is usually determined through cross-validation during the sparse coding algorithm development phase. An exemplary range is when the objective function value decreases by 10... -3 Up to 10 -5 Between these, the atomic update magnitude of the collaborative base dictionary is 10. -4 Up to 10 -6 Between; the comprehensive convergence condition refers to the joint stopping criterion that the decrease magnitude of the sparse coding objective function and the update magnitude of the cooperative base dictionary atoms are both less than the convergence judgment threshold; the sparse coefficients are the coefficient vectors that represent the linear combination of high-dimensional signals under an overcomplete dictionary during the sparse coding process, with most elements being zero or close to zero, and only a few non-zero elements corresponding to activated atoms; the knowledge surface of cooperative base dictionary atoms refers to the typical hand cooperative motion pattern corresponding to each atom in the cooperative base dictionary and its associated semantic labels, action categories, joint involvement, and manifold topological relationships, etc., and other structured prior information.

[0080] Based on the collaborative base dictionary, a continuous reference manifold is constructed in the feature space of the high-dimensional motion feature matrix.

[0081] The specific process involves using atoms in the cooperative basis dictionary as local basic units, connecting adjacent cooperative basic units in the feature space spanned by the high-dimensional motion feature matrix through interpolation or local linear embedding methods, forming a low-dimensional nonlinear subspace that preserves the cooperative structure of hand motion. The low-dimensional nonlinear subspace can continuously represent the transition relationship between different motion states, thereby constructing a continuous reference manifold.

[0082] Based on the high-dimensional motion feature matrix and the reference manifold, the coupling transmission relationship of motion semantic cost is defined.

[0083] The specific process includes treating each feature point in the high-dimensional motion feature matrix as a source distribution and the cooperative primitives on the reference manifold as a target distribution. An entropy regularization optimal transmission method is used to construct the transmission cost matrix between the two. The transmission cost is composed of the feature space distance and the motion semantic difference. The motion semantic difference is obtained by assigning semantic labels to different cooperative primitives through real-time motion task context information to obtain the dissimilarity between categories, thereby establishing a coupled transmission relationship of motion semantic cost that takes into account both geometric structure and semantic consistency.

[0084] The entropy-regularized optimal transport algorithm is applied to the coupled transport relationship to obtain the optimal coupling matrix between user data and the reference manifold.

[0085] The specific process includes using the high-dimensional motion feature matrix as the distribution representation of user data, taking the cooperative primitives on the reference manifold as the target distribution, using the motion semantic cost defined in the coupled transmission relationship as the transmission cost, and obtaining the joint probability distribution that minimizes the total transmission cost and satisfies the edge distribution constraint under the framework of the entropy regularization optimal transmission algorithm. The joint probability distribution is the optimal coupling matrix between user data and the reference manifold.

[0086] It should be noted that the entropy-regularized optimal transmission algorithm adds an entropy regularization term to the optimization objective of the classical optimal transmission problem. By combining it with a negative entropy term, the solution of the transmission plan is made smooth and unique, thereby obtaining a stable and computationally feasible coupling matrix while minimizing the total transmission cost.

[0087] like Figure 4 As shown, the optimal coupling matrix uses thermal colors to represent the coupling weights. The graph reveals significant color differences and structured high-weight clusters, indicating that the cross-user alignment relationship established in this invention is selective rather than random. High-weight regions represent a high degree of matching between the target anchor and the reference sample, while low-weight regions represent the suppression of irrelevant samples. This provides a reference support closer to the target distribution for online adaptation, further reducing generalization errors caused by static model mismatch.

[0088] Based on the optimal coupling matrix, the user's high-dimensional motion features are weighted and projected to generate a universal decoding benchmark in the low-dimensional cooperative basis manifold space.

[0089] The specific process includes using the correlation strength between the user's high-dimensional motion features represented by each element in the optimal coupling matrix and the cooperative primitives on the reference manifold, weighting the user's high-dimensional motion features according to this correlation strength, and mapping them to the low-dimensional cooperative basis manifold space spanned by the cooperative basis dictionary, thereby forming a universal decoding benchmark in the low-dimensional cooperative basis manifold space applicable to different users.

[0090] S3. During user operation, obtain real-time feedback signals of the decoding output, dynamically optimize the cooperative base manifold topology and decoding matching parameters of the general decoding benchmark based on the feedback signals, and perform online personalized adaptation for the current user.

[0091] During user interaction, the system acquires hand movements in real time from the decoded output of a universal decoding benchmark, compares the hand movements with the expected movements, and generates real-time feedback signals.

[0092] The specific process includes continuously receiving the angle sequence of each finger joint output by a general decoding benchmark as hand movements when the user performs operations such as grasping, pinching, or stretching. At the same time, the standard hand posture that should be completed at that moment is determined as the expected action based on the current task instruction or the semantic label of the target object. The absolute error or Euclidean distance between the two in the angle values ​​is obtained by obtaining them joint by joint, and the errors of all joints are weighted and summed to form a real-time feedback signal that characterizes the degree of overall action deviation.

[0093] Based on the feedback signal, a meta-reinforcement learning adapter is constructed with a general decoding benchmark as the initial strategy.

[0094] The specific process includes using real-time feedback signals as environmental rewards, using the hand action mapping relationship output by the general decoding benchmark as the initial policy, and rapidly adjusting the policy parameters through a small number of interaction samples under the meta-reinforcement learning framework. This enables the adapter to efficiently update the decoding policy based on historical experience in new user or new task scenarios, forming a meta-reinforcement learning adapter with the general decoding benchmark as the initial policy.

[0095] The meta-reinforcement learning adapter is used to dynamically optimize the cooperative basis manifold topology and decoding matching parameters of the general decoding benchmark.

[0096] The specific process involves using a meta-reinforcement learning adapter to simultaneously adjust the geometric connectivity of the low-dimensional cooperative basis manifold space in the general decoding benchmark and the mapping weights from high-dimensional motion features to hand movements after receiving real-time feedback signals. This allows the cooperative basis manifold topology to adapt to changes in the user's individual motion patterns, and the decoding matching parameters to accurately reflect the current user's neuromuscular control characteristics (such as electromyographic activation patterns, motor unit recruitment order, joint coordination strength, and individual movement preferences). The two evolve jointly during the policy gradient update process, achieving collaborative dynamic optimization of the general decoding benchmark.

[0097] The dynamically optimized collaborative base manifold topology is applied to the user's high-dimensional motion features to generate low-dimensional manifold features that are adapted to the current user.

[0098] The specific process includes using the local neighborhood relations and geometric constraints defined by the cooperative basis manifold topology structure adjusted by the econometric reinforcement learning adapter to perform nonlinear projection on the user's high-dimensional motion features. By maintaining the intrinsic structure of the user's high-dimensional motion features on the manifold and integrating personalized motion patterns, the high-dimensional motion features are mapped to a low-dimensional space, thereby generating low-dimensional manifold features that match the current user's neuromuscular characteristics.

[0099] When the low-dimensional manifold feature generation process reaches a stable state, online personalized adaptation is performed for the current user.

[0100] The specific process includes: after the update amplitude of the low-dimensional manifold features is lower than the convergence judgment threshold for several consecutive time steps and the deviation between the corresponding decoding output and the expected action tends to stabilize, the low-dimensional manifold features and their mapping relationship at this time are solidified into the current user's personalized decoding configuration for subsequent hand action decoding, thus completing the online personalized adaptation for the current user.

[0101] like Figure 5 As shown, during the online time-step progression, the joint angle error RMSE of different decoding strategies exhibits significant dynamic changes over time, indicating that the online decoding task suffers from continuous distributional fluctuations and individual difference perturbations. It can be seen that the overall errors of the "general decoding benchmark + meta-reinforcement learning adapter (rapid update)" and the "precise control chain (physically constrained Bayes + advanced intent prediction)" are lower and the fluctuations are more controllable. This demonstrates that the dynamic optimization architecture of this invention, which integrates a general decoding benchmark with online meta-learning adaptation, can achieve rapid evolution from a static general model to a personalized decoder, reducing the adaptation cost for new users and improving cross-user generalization ability. The red dashed box marks the local magnified window, and the green dashed line in the magnified image locates the point of maximum difference, labeled with a double-headed arrow indicating "local error difference," which visually demonstrates that this invention can continuously achieve better error suppression effects in the critical online interval.

[0102] S4. Construct a lightweight decoding model and input the personalized low-dimensional manifold features into the lightweight decoding model to output the finger joint angle prediction sequence.

[0103] Based on a convolutional neural network model, we perform manifold dimension reduction and algorithm pruning optimization on low-dimensional manifold features to obtain the pruned model architecture.

[0104] The specific process includes taking low-dimensional manifold features as input, using the basic model of convolutional neural networks to further compress the feature dimension through multi-layer convolution and pooling operations to achieve manifold dimension reduction, and simultaneously performing algorithmic pruning and optimization on redundant convolutional kernels and connections in the basic model of convolutional neural networks based on weight magnitude or gradient sensitivity. While maintaining decoding performance, unimportant structures are removed, ultimately forming a pruned model architecture with fewer parameters and lower computational complexity.

[0105] Based on the pruned model architecture, a neural architecture search space is constructed.

[0106] The specific process involves using the pruned model architecture as the basic skeleton, preserving the layer structure and connection methods, defining a set of optional operations in each layer based on dimensions such as the number of convolutional kernels, kernel size, activation function type, and whether residual connections are combined, and combining the optional operations of each layer to form discrete structural variables, thereby constructing a neural architecture search space that covers a variety of lightweight network variants.

[0107] Define a multi-objective optimization function that integrates hardware constraints and knowledge distillation within the neural architecture search space.

[0108] The specific process includes constructing a multi-objective optimization function in the neural architecture search space that simultaneously includes hardware constraints such as inference latency or memory usage on the target embedded device, as well as knowledge distillation terms representing the distribution differences between the output of the lightweight model and the output of the base model of the convolutional neural network. By weighted combination of these two types of objectives, a unified evaluation criterion is formed to guide the neural architecture search process to maintain high decoding accuracy while meeting deployment resource constraints.

[0109] By searching for differentiable neural architectures, the multi-objective optimization function is co-optimized to obtain the optimal lightweight model architecture.

[0110] The specific process includes relaxing each structure into continuous weights in the neural architecture search space, using gradient descent to simultaneously update the weight parameters and architecture parameters of the convolutional neural network, using a multi-objective optimization function as the optimization objective, and under the combined effect of hardware constraints limiting resource consumption and knowledge distillation maintaining decoding performance, the architecture parameters converge to the optimal configuration, and finally discretize the optimal configuration to obtain the optimal lightweight model architecture.

[0111] Quantization-aware training is performed on the optimal lightweight model architecture to generate a lightweight decoding model.

[0112] The specific process includes, based on the optimal lightweight model architecture, simulating quantization operations of low-bit weights and activation values ​​during training, converting floating-point parameters to fixed-point representations during forward propagation while retaining gradient information of quantization errors, updating floating-point parameters during backpropagation, adapting to the accuracy loss caused by quantization during the training phase, and solidifying the trained model parameters in a low-bit format to form a lightweight decoding model that can run efficiently on embedded devices.

[0113] The personalized low-dimensional manifold features are input into the lightweight decoding model, which outputs the initial joint angle prediction.

[0114] The specific process includes using the low-dimensional manifold features generated through online personalized adaptation as input to the lightweight decoding model. The lightweight decoding model performs forward inference through quantized convolutions and fully connected layers to map the angle values ​​of each finger joint corresponding to the current hand movement intention, thus forming an initial joint angle prediction.

[0115] A set of physical constraint relationships is established based on hand biomechanics and transformed into the corresponding potential energy function.

[0116] The specific process includes determining the range of thumb opposition, the range of interphalangeal joint flexion and extension, and the physiological limitations of coordinated movement of each finger based on the anatomical structure and kinematic characteristics of the hand. This forms a set of physical constraint relationships describing the feasible domain of joint angles (such as the range of interphalangeal joint flexion and extension, and the limit of thumb opposition angle) and coupling relationships (such as the proportion of multi-finger coordinated flexion and wrist-finger linkage constraints). Each physical constraint relationship is mapped to a potential energy function that increases rapidly at the constraint boundary, so that the combination of joint angles that violates the physiological limitations corresponds to a high potential energy value. Thus, the biomechanical laws of hand movement are explicitly embedded through the potential energy function.

[0117] It should be noted that the set of physical constraint relationships includes the thumb opposition range of 0°-45°, the interphalangeal joint flexion and extension range of 0°-90°, and the interphalangeal joint coupling relationship, wherein the flexion angle of the distal interphalangeal joint does not exceed 70% of the flexion angle of the proximal interphalangeal joint.

[0118] The initial joint angle prediction is used as the sampling starting point, and optimization iteration is performed in the constraint space defined by the potential energy function.

[0119] The specific process includes using the initial joint angle prediction as the starting point, using gradient descent or random sampling methods to search for lower energy joint angle combinations within the hand biomechanical constraint space characterized by the potential energy function, adjusting the joint angle according to the gradient of the potential energy function in each iteration, and gradually converging towards the region that conforms to physiological feasibility, while maintaining closeness to the original prediction intention.

[0120] Posterior probability estimation and uncertainty quantification are performed on the joint angle samples obtained after optimization iteration to obtain the finger joint angle prediction sequence.

[0121] The specific process includes: based on multiple joint angle samples that meet the potential energy function constraints generated during the optimization iteration process, using the Bayesian inference method to obtain the posterior probability distribution of each joint angle under given observation features, and determining the final predicted value through the mean or mode of the posterior probability distribution. At the same time, the variance or confidence interval of the posterior probability distribution is used to characterize the prediction uncertainty, thereby forming a finger joint angle prediction sequence.

[0122] S5. Perform short-term motion intention prediction on the finger joint angle prediction sequence to generate advanced control commands.

[0123] Based on the finger joint angle prediction sequence, a temporal attention mechanism is used to calculate the contribution weights of historical time points, generating an attention-weighted historical feature vector, expressed as follows:

[0124] ;

[0125] in, Indicates at a point in time At that time, historical point in time Attention weights Indicates the current time point The query vector, Indicates a point in history The key vector, This represents the dimensions of the query vector and the key vector. Represents the time decay intensity coefficient. This represents the time decay rate parameter. Indicates a historical point in time. The corresponding key vector, This represents the index of the loop variable that iterates through all historical time points. This represents the index of the specific historical time point where the weight to be calculated is currently being calculated. Represents the total length of the historical time window

[0126] It should be noted that, The time decay intensity coefficient is a hyperparameter used to control the decay rate of historical information over time intervals. It is determined on the validation set using grid search or Bayesian optimization methods with action prediction accuracy as the metric. The time decay rate parameter is a scalar that controls the exponential decay rate of historical information over time intervals. It takes a value between 0 and 1 and is determined on action sequence data with the goal of predicting stability through cross-validation or empirical parameter tuning.

[0127] The specific process includes scaling the similarity between the query vector at the current time point and the key vectors at each historical time point through a dot product, adjusting it by combining a time decay term that monotonically decreases with the increase of the time interval, normalizing the adjusted similarity scores of all historical time points to obtain the corresponding attention weights, and then multiplying the attention weights with the feature vectors of the corresponding historical time points and summing them to generate attention-weighted historical feature vectors.

[0128] The attention-weighted historical feature vector is input into the spatiotemporal graph neural network, and the hidden state sequence is output by combining the spatial coupling relationship between hand joints and temporal dynamics.

[0129] The specific process includes using attention-weighted historical feature vectors as initial features of nodes, constructing a graph structure with each joint of the hand as nodes and anatomical connections and motor coordination relationships as edges, aggregating the spatial information of adjacent joints through graph convolution operations in the spatiotemporal graph neural network, and using recurrent or temporal convolution mechanisms to model the evolution of joint states between consecutive time steps. After multi-layer spatiotemporal information fusion, a hidden state sequence corresponding to each joint at each time step is generated.

[0130] It should be noted that the spatial coupling relationship between hand joints is determined by the hand's anatomical structure and the laws of motion coordination, and is obtained through biomechanical modeling and statistical analysis of motion capture data. The temporal dynamics reflect the continuous evolution characteristics of joint angles over time, and are extracted from synchronously acquired multi-frame hand motion sequences through differential, filtering, or state-space modeling methods.

[0131] Furthermore, the training process of the spatiotemporal graph neural network uses the finger joint angle prediction sequence as input. First, a graph structure is constructed with hand joints as nodes and anatomical connections and kinematic coordination relationships as edges, and attention-weighted historical feature vectors are used as initial features of the nodes. At each training time step, spatial information of adjacent joints is aggregated through graph convolutional layers, and temporal dynamics are modeled by recurrent units or temporal convolutional layers to generate the hidden state sequence at the corresponding time. The hidden state sequence is mapped to joint angle estimates using fully connected layers, and the loss function is obtained by comparing it with the real joint angle labels, typically using mean squared error or smoothed L1 loss. The graph convolution weights, temporal modeling parameters, and attention mechanism-related parameters in the spatiotemporal graph neural network are jointly updated through backpropagation algorithm, and iterative optimization is performed until the loss converges, resulting in a spatiotemporal graph neural network that can effectively integrate the spatial coupling relationship and temporal dynamics of hand joints.

[0132] Short-term integral prediction is performed on the hidden state sequence to obtain the advanced joint angle prediction sequence.

[0133] The specific process includes using the hidden state sequence output by the spatiotemporal graph neural network as the initial condition input to the neural differential equation, using the neural network to parameterize the derivative function of the hidden state as it continuously evolves over time, performing numerical integration on the derivative function within a small time step to deduce the hidden state at several future moments, and mapping the future hidden state back to the joint angle space, thereby generating a joint angle prediction sequence that is ahead of the current moment.

[0134] The advanced joint angle prediction sequence is converted into advanced control commands by using inverse kinematic mapping.

[0135] The specific process includes using the inverse kinematics mapping relationship in the pre-established hand kinematics to map the angle values ​​of each finger joint in each time step of the advanced joint angle prediction sequence to the corresponding end effector pose or drive unit target parameters, forming advanced control commands that can be directly issued to the actuator.

[0136] It should be noted that the inverse kinematic mapping relationship is pre-established based on the hand skeletal structure and joint connection method through analytical geometry or numerical optimization methods.

[0137] S6. Convert advanced control commands into control protocols that are compatible with the target device, and drive external devices to complete corresponding dexterous hand movements.

[0138] The protocol parameters of the advanced control commands are optimized to generate optimized protocol parameters.

[0139] The specific process includes adjusting the protocol parameters such as timestamps, data frame formats, check bits, and transmission rates in the advanced control commands according to the communication protocol specifications of the target execution device, so that the advanced control commands can adapt to the specific hardware interface requirements while meeting the requirements of real-time performance and reliability, thereby generating optimized protocol parameters.

[0140] The optimized protocol parameters are converted into a control protocol adapted to the target device through a real-time performance awareness algorithm and sent to the external device driver interface.

[0141] The specific process includes using real-time performance awareness algorithms to monitor the target device's operating status, such as computing load, communication bandwidth, and response latency; dynamically adjusting the data packaging method, transmission cycle, and error retransmission strategy in the optimized protocol parameters; generating a control protocol that conforms to the target device's current operating conditions; and encapsulating the control protocol into a standard instruction format before sending it to the external device driver interface.

[0142] It should be noted that the real-time performance awareness algorithm is a method for dynamically adjusting control parameters based on indicators such as the computational load, communication bandwidth, and response latency of the target device during operation. The real-time performance awareness algorithm continuously collects device status data through a lightweight monitoring mechanism and generates configuration updates adapted to the current operating conditions.

[0143] The external device driver interface parses and executes the control protocol, driving the external device to complete the corresponding dexterous hand movements.

[0144] The specific process includes: after receiving the control protocol of the target device, the external device driver interface decodes the instruction content according to the frame structure, timing, and verification rules specified in the protocol, extracts the target position, speed, or torque parameters of each drive unit, and sends these parameters to the corresponding motor or actuator, thereby driving the external device to accurately reproduce the user's intention. Figure 1 Delicate hand movements.

[0145] In summary, this invention achieves intelligent evolution from a static, general hand motion model to a dynamic, personalized decoder by constructing a dynamic optimization architecture that integrates a universal decoding benchmark with online meta-learning adaptation. This solves the core bottlenecks of poor cross-user generalization ability and high adaptation costs for new users caused by reliance on static models. Furthermore, through a precise control chain combining Bayesian inference under physical constraints with advanced intent prediction, it achieves high-fidelity, low-latency decoding and control of dexterous hand movements on resource-constrained embedded platforms, resolving the prominent contradiction between decoding accuracy, real-time performance, and motion safety.

[0146] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for dexterous hand motion decoding based on cooperative manifolds, characterized in that: include, Collect users' multimodal signals, perform noise reduction and feature extraction to obtain a high-dimensional motion feature matrix; The coordinating basis dictionary representing hand motion coordinating primitives is learned from the high-dimensional motion feature matrix by the sparse coding dictionary learning method, and the high-dimensional motion feature matrix is ​​projected onto the low-dimensional coordinating basis manifold space by the manifold embedding algorithm to form a universal decoding benchmark. During user operation, real-time feedback signals of the decoding output are acquired. Based on the feedback signals, the cooperative base manifold topology and decoding matching parameters of the general decoding benchmark are dynamically optimized, and online personalized adaptation is performed for the current user. A lightweight decoding model is constructed, and the personalized low-dimensional manifold features are input into the lightweight decoding model to output the finger joint angle prediction sequence. Short-term motion intent prediction is performed on the finger joint angle prediction sequence to generate advanced control commands. The advanced control commands are converted into control protocols that are compatible with the target device, and the external device is driven to complete the corresponding dexterous hand movements.

2. The dexterous hand motion decoding method based on cooperative manifold as described in claim 1, characterized in that: The multimodal signals include electromyography signals, inertial measurement unit data, and near-infrared spectral signals.

3. The dexterous hand motion decoding method based on cooperative manifold as described in claim 2, characterized in that: The specific steps for obtaining the high-dimensional motion feature matrix are as follows: Multimodal signals are acquired and time-aligned to obtain a synchronized multimodal signal stream; Based on the synchronous multimodal signal stream, the quality index evaluation value of each modal signal is calculated, and the adaptive wavelet threshold algorithm is used for dynamic noise reduction to obtain the noise-reduced multimodal signal. Time-domain, frequency-domain, and time-frequency-domain features are extracted from the denoised multimodal signal and weighted and fused through a collaborative attention mechanism to generate a high-dimensional motion feature matrix.

4. The dexterous hand motion decoding method based on cooperative manifold as described in claim 3, characterized in that: The method of learning a cooperative base dictionary representing hand motion cooperative primitives from a high-dimensional motion feature matrix using a sparse coding dictionary learning approach involves the following specific steps. Manifold structure initialization is performed on the high-dimensional motion feature matrix to obtain the atomic distribution of the cooperative basis dictionary; Based on the atomic distribution of the collaborative base dictionary and combined with the context information of real-time motion tasks, a sparse coding objective function with dynamic topological constraints is constructed. Perform iterative operations on the sparse coding objective function until the comprehensive convergence condition is met, and then obtain the cooperative base dictionary representing the cooperative primitives of hand movement.

5. The dexterous hand motion decoding method based on cooperative manifold as described in claim 4, characterized in that: The specific steps for forming a universal decoding benchmark are as follows: Based on the collaborative base dictionary, a continuous reference manifold is constructed in the feature space of the high-dimensional motion feature matrix; Based on the high-dimensional motion feature matrix and the reference manifold, the coupling transmission relationship of motion semantic cost is defined; The entropy-regularized optimal transmission algorithm is applied to the coupled transmission relationship to obtain the optimal coupling matrix between user data and the reference manifold. Based on the optimal coupling matrix, the user's high-dimensional motion features are weighted and projected to generate a universal decoding benchmark in the low-dimensional cooperative basis manifold space.

6. The dexterous hand motion decoding method based on cooperative manifold as described in claim 5, characterized in that: During user operation, real-time feedback signals from the decoding output are acquired. Based on these feedback signals, the cooperative base manifold topology and decoding matching parameters of the general decoding benchmark are dynamically optimized, and online personalized adaptation is performed for the current user. The specific steps are as follows: During user operation, the system acquires hand movements decoded from a universal decoding benchmark in real time, compares the hand movements with the expected movements, and generates real-time feedback signals. Based on the feedback signal, a meta-reinforcement learning adapter with a general decoding benchmark as the initial strategy is constructed. The collaborative basis manifold topology and decoding matching parameters of the general decoding benchmark are dynamically optimized through a meta-reinforcement learning adapter. The dynamically optimized collaborative basis manifold topology is applied to the user's high-dimensional motion features to generate low-dimensional manifold features that are adapted to the current user. When the low-dimensional manifold feature generation process reaches a stable state, online personalized adaptation is performed for the current user.

7. The dexterous hand motion decoding method based on cooperative manifold as described in claim 6, characterized in that: The specific steps for constructing the lightweight decoding model are as follows. Based on the convolutional neural network model, the low-dimensional manifold features are reduced in manifold dimension and optimized by algorithm pruning to obtain the pruned model architecture. Based on the pruned model architecture, a neural architecture search space is constructed. Define a multi-objective optimization function that integrates hardware constraints and knowledge distillation within the neural architecture search space; The optimal lightweight model architecture is obtained by co-optimizing the multi-objective optimization function through differentiable neural architecture search. Quantization-aware training is performed on the optimal lightweight model architecture to generate a lightweight decoding model.

8. The dexterous hand motion decoding method based on cooperative manifold as described in claim 7, characterized in that: The specific steps for outputting the finger joint angle prediction sequence are as follows. The personalized low-dimensional manifold features are input into the lightweight decoding model, which outputs the initial joint angle prediction. A set of physical constraint relationships is established based on hand biomechanics and transformed into the corresponding potential energy function; The initial joint angle prediction is used as the sampling starting point, and optimization iteration is performed in the constraint space defined by the potential energy function. Posterior probability estimation and uncertainty quantification are performed on the joint angle samples obtained after optimization iteration to obtain the finger joint angle prediction sequence.

9. The dexterous hand motion decoding method based on cooperative manifold as described in claim 8, characterized in that: The specific steps for generating advanced control commands are as follows: Based on the finger joint angle prediction sequence, a temporal attention mechanism is used to calculate the contribution weight of historical time points and generate an attention-weighted historical feature vector. The attention-weighted historical feature vector is input into the spatiotemporal graph neural network, and the hidden state sequence is output by combining the spatial coupling relationship between hand joints and temporal dynamics. Short-term integral prediction of the hidden state sequence is performed to obtain the advanced joint angle prediction sequence. The advanced joint angle prediction sequence is converted into advanced control commands by using inverse kinematic mapping.

10. The dexterous hand motion decoding method based on cooperative manifold as described in claim 9, characterized in that: The specific steps for converting advanced control commands into a control protocol adapted to the target device and driving the external device to complete the corresponding dexterous hand movements are as follows. Optimize the protocol parameters of the advanced control commands to generate optimized protocol parameters; The optimized protocol parameters are converted into a control protocol adapted to the target device through a real-time performance awareness algorithm and sent to the external device driver interface. The external device driver interface parses and executes the control protocol, driving the external device to complete the corresponding dexterous hand movements.

Images