A residual reinforcement learning control method for a flexible bionic ankle exoskeleton robot

Abstract

The application discloses a residual error reinforcement learning control method for a flexible bionic ankle exoskeleton robot; first, the wearing equipment is bound to the waist of a to-be-tested rehabilitation personnel, a rope drawn through the binding of the lower leg reaches the back heel, and is clamped on the fixing anchor point on the shoe; along with the walking of the to-be-tested personnel, the input current of the physical model controller is calculated; then, a residual error reinforcement learning core module including a state space and an action space is set, and a compensation current value is calculated based on the residual error strategy output by an RL strategy network; the RL strategy network is iteratively updated, a gradient-based policy search is applied to gradually approximate the optimal control parameter, and the optimal is output, so that the total motor input current is input into the motor of the wearing equipment, the output tension is tracked, the tension is consistent with the ankle joint tension, and the auxiliary function is realized. The application improves the control precision, dynamic response speed and stable controllability of the ankle exoskeleton under a complex gait.

Description

Technical Field

[0001] This invention relates to the field of ankle exoskeleton control and rehabilitation engineering, and in particular to a residual reinforcement learning control method for flexible bionic ankle exoskeleton robots. Background Technology

[0002] With the aging population and the increasing number of patients with limb dysfunction caused by diseases such as stroke and spinal cord injury, improving the motor autonomy and quality of life of this population has become a key need in the fields of clinical rehabilitation and engineering.

[0003] As the core weight-bearing and driving joint of human gait movement, the recovery of the ankle joint directly determines gait stability and independent movement ability. Flexible ankle exoskeletons have become the mainstream development direction of rehabilitation walking aids due to their advantages such as lightweight, strong conformity and comfortable wearing. The core demand is to achieve precise and stable control under complex gait conditions, match the human movement intention and adapt to individual differences and environmental disturbances.

[0004] Achieving this goal requires overcoming three core challenges: 1) the strong nonlinearity of the system makes it difficult to accurately model the dynamics; 2) multi-source uncertainties easily lead to control deviations; and 3) complex gait switching places extremely high demands on the real-time performance and stability of the algorithm. Current mainstream control algorithms all have key flaws, making it difficult to balance performance and practicality.

[0005] Some model-based reinforcement learning algorithms rely on accurate dynamic models to achieve control through feedforward compensation and feedback correction. While these algorithms offer a certain degree of real-time performance, the flexibility of the ankle joint, cable transmission losses, and human-machine coupling effects make it difficult to accurately construct dynamic models. Model errors directly affect control quality and can even lead to system instability, making them unsuitable for adapting to the dynamic changes of complex pathological gait.

[0006] Model-free reinforcement learning algorithms, by interacting with the environment and iteratively optimizing strategies, do not rely on precise models and show potential in nonlinear systems. However, classic algorithms such as DDPG and TD3 suffer from problems such as low sample efficiency, slow training convergence, and Q-value estimation bias. They are prone to control instability in the initial training phase, and because they do not utilize prior knowledge of system dynamics, their generalization ability is insufficient and they are difficult to adapt to individual differences among patients. When directly applied to real robotic platforms, training efficiency and safety are difficult to guarantee.

[0007] In recent years, researchers have attempted to improve algorithm performance:

[0008] Reference 1: Sun Jiacheng, Research on ankle exoskeleton control algorithm based on deep reinforcement learning [D]. Suzhou University of Science and Technology, 2023. The Softmax deep bideterministic policy gradient (SD3) algorithm is proposed, which solves the Q-value estimation bias problem and verifies the effectiveness of ankle exoskeleton control on the simulation platform;

[0009] Reference 2: GU S, HOLLY E, LILLICRAP T, et al. Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates [EB / OL]. arXiv, 2016 [2024-11-15]. http: / / arxiv.org / abs / 1610.00633. This paper proposes the Asynchronous Normalized Advantage Function (NAF) algorithm, which improves efficiency through parallel training of multiple robots and introduces safety constraints to ensure training safety.

[0010] However, these improvements still do not solve the core problem: they do not fully combine the existing model information of the system with the advantages of reinforcement learning. They cannot avoid the dependence of model-based algorithms on model accuracy, nor can they completely improve the problems of low sample efficiency and insufficient security of model-free algorithms. Summary of the Invention

[0011] To address the problems of existing ankle exoskeleton control algorithms under complex gait conditions, this invention proposes a residual reinforcement learning control method for flexible bionic ankle exoskeleton robots. By constructing an accurate system dynamics model, a joint control framework of "physical model-driven basic control strategy + residual reinforcement learning compensation strategy" is designed to achieve real-time correction of model errors and nonlinear disturbances. This improves the control accuracy, dynamic response speed, and stability controllability of the ankle exoskeleton under complex gait conditions, providing an efficient and reliable control scheme for rehabilitation and motor assistance scenarios. It also solves the technical pain points of existing algorithms in practical applications, such as large trajectory tracking deviations and failures under extreme conditions.

[0012] The specific steps are as follows:

[0013] Step 1: Secure the wearable device to the waist of the person being tested for rehabilitation. The rope leading out of the wearable device passes through the bindings on the lower leg and reaches the heel, where it is secured to the fixed anchor point set in advance on the shoe.

[0014] The wearable device includes: a reel, a motor, a controller, a battery, and sensors, with a rope extending from the reel; the strap is positioned at the mid-calf.

[0015] Step 2: As the person undergoing rehabilitation walks, calculate the input current of the physical model controller. :

[0016] Step 201: Establish the motor dynamics model of the model controller and calculate the torque of the input motor. :

[0017]

[0018] in, The moment of inertia of the motor shaft. , and These are the angular acceleration, angular velocity, and angle of the motor shaft rotation, respectively. The coefficient of viscous friction of the motor shaft. For motor shaft output, For harmonic deceleration ratio, After deceleration, the output torque of the deceleration wheel is reduced. The rotation angle of the deceleration wheel;

[0019] Step 202: Establish a human-machine coupled dynamic model of the model controller and calculate the tension force output by the cable to the ankle joint. :

[0020]

[0021] in, To apply tension to the rope, Represents friction. The coefficient of friction, The angular velocity of the reel rotation;

[0022] Step 203: Based on the motor input torque and the cable outputs tension to the ankle joint The dynamic relationship between them is used to solve for the input current of the motor. :

[0023]

[0024] The torque constant; The moment of inertia of the output end of the reel. The angular acceleration of the rotating reel; The coefficient of viscous friction at the output end of the reel; The radius of the reel;

[0025] Step 204: Establish a motion model of the ankle joint and the reel for the person undergoing rehabilitation, and then obtain the updated input feedforward current. :

[0026] First, the relationship between the ankle joint angle and the thread sheave angle is calculated as follows:

[0027]

[0028] in The distance between the center point of the rope binding on the lower leg and the center of the ankle joint. The distance between the rope at the fixed anchor point and the center of the ankle joint. This represents the initial ankle angle; For the thread reel The perspective of time; This is the initial angle of the reel; for Ankle angle at any given moment;

[0029] Then, calculate the ankle joint angular velocity. and angular acceleration Relative to the angle and angular velocity of the reel and angular acceleration Relationship:

[0030]

[0031] Finally, adjust the angle of the thread reel. Replace with ankle joint angle The updated input feedforward current is obtained. ;

[0032] Step 205: Introduce a tension feedback controller into the physical model controller to calculate the actual tension force at the ankle joint. Feedback current :

[0033]

[0034] This is the proportionality coefficient. The target tension for the ankle joint. The differential coefficients are... The rate of change of the target tension. The rate of change of the actual tensile force. The rate of change of error, This is for tensile force error;

[0035] Step 206: Combine the feedforward current and the feedback current to form the input current that drives the physical model controller. :

[0036]

[0037] Step 3: Define the core modules of residual reinforcement learning, including state space and action space:

[0038] The state space s includes the actual tension. Target tension Tension error Ankle joint angular velocity and ankle joint angular acceleration .

[0039] Action space Including physical model controller .

[0040] Step 4: Calculate the compensation current value using the residual reinforcement learning core module. Its value is the output of the RL policy network. ;

[0041] ;

[0042] Step 5: Iteratively update the RL policy network, applying gradient-based policy search to gradually approximate the optimal control parameters. ;

[0043]

[0044] The action value function;

[0045] Step 6: Obtain the optimal control parameters Then, the RL policy network outputs the optimal residual policy. Thus, the total motor input current is obtained:

[0046]

[0047] Step 7: Set the total motor input current. The input is fed into the wearable device's motor, which then tracks and outputs the pulling force to achieve the same tension as the ankle joint. Consistency, thereby enabling auxiliary functions.

[0048] The advantages of this invention are:

[0049] 1. This invention establishes a complete dynamic model of the relationship between motor, human-machine coupling, and joint motion in a layered manner, accurately characterizing the physical characteristics of the ankle exoskeleton. Combined with the basic control strategy of "feedforward + PD feedback", it provides stable basic control performance and effectively reduces steady-state error under ideal working conditions.

[0050] 2. This invention addresses model uncertainty, nonlinear disturbances, and individual differences by using a residual reinforcement learning strategy to correct control deviations in real time. This solves the problem of insufficient adaptability of traditional algorithms and can maintain high-precision control even under complex gait and disturbance conditions. Its robustness is better than that of simple model-driven or reinforcement learning algorithms.

[0051] 3. This invention completes the proof of the bounded stability of the system based on Lyapunov stability theory, clarifies the error convergence range, avoids safety problems such as oscillation and instability during training and operation, and provides safety assurance for applications in clinical rehabilitation, sports assistance and other scenarios.

[0052] 4. This invention adopts a modular design, and the basic control strategy and residual strategy can be optimized independently. It supports integration with other control algorithms (such as fuzzy control and adaptive control). It can be adapted to ankle exoskeletons with different structures by updating the dynamic model parameters, or extended to multi-joint collaborative control scenarios, and has good compatibility and scalability. Attached Figure Description

[0053] Figure 1 This is a flowchart of a residual reinforcement learning control method for a flexible bionic ankle exoskeleton robot according to the present invention;

[0054] Figure 2 To establish the motion relationship diagram between the ankle joint angle and the wheel angle in the system dynamics model of this invention;

[0055] Figure 3 This is a control flowchart for a residual reinforcement learning algorithm designed for flexible ankle joints according to the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0057] This invention provides a residual reinforcement learning control method for flexible bionic ankle exoskeleton robots. It integrates reinforcement learning with available dynamic models, using the models to provide basic control guarantees and reinforcement learning to compensate for model errors and uncertainties. This reduces the dependence on model accuracy and significantly improves learning speed and control safety, thus solving the technical bottleneck of complex gait control for flexible ankle exoskeletons.

[0058] like Figure 1 As shown, the specific steps are as follows:

[0059] Step 1: Secure the wearable device to the waist of the person being tested for rehabilitation. The rope leading out of the wearable device passes through the bindings on the lower leg and reaches the heel, where it is secured to the fixed anchor point set in advance on the shoe.

[0060] The wearable device includes: a reel, a motor, a controller, a battery, and sensors, with a rope extending from the reel; the strap is positioned at the mid-calf.

[0061] Step 2: As the person undergoing rehabilitation walks, calculate the input current of the physical model controller. :

[0062] Step 201: Establish the motor dynamics model of the model controller and calculate the torque of the input motor. :

[0063]

[0064] in, The moment of inertia of the motor shaft. , and These are the angular acceleration, angular velocity, and angle of the motor shaft rotation, respectively. The coefficient of viscous friction of the motor shaft. For motor shaft output, For harmonic deceleration ratio, After deceleration, the output torque of the deceleration wheel is reduced. The rotation angle of the deceleration wheel;

[0065] The motor torque can also be expressed using the relevant variables of the reel, as follows:

[0066]

[0067] The moment of inertia of the output end of the reel. The angular acceleration of the rotating reel; The coefficient of viscous friction at the output end of the reel; The angular velocity of the reel rotation;

[0068] Step 202: Establish a human-machine coupled dynamic model of the model controller and calculate the tension force output by the cable to the ankle joint. ;

[0069] First, tension is applied to the rope. :

[0070]

[0071] The radius of the reel;

[0072] However, ropes suffer force loss due to their compliance and the nonlinear properties caused by friction.

[0073]

[0074] in, To apply tension to the rope, Represented as frictional force; The coefficient of friction;

[0075] Therefore, we can conclude that: ;

[0076] Step 203: Based on the motor input torque and the cable outputs tension to the ankle joint The dynamic relationship between them:

[0077]

[0078] The relationship between the motor input torque and current is as follows:

[0079]

[0080] Therefore, the input current is obtained. and the cable outputs tension to the ankle joint The derivable dynamic relationship between them is as follows:

[0081]

[0082] The torque constant;

[0083] Step 204: Establish a motion model of the ankle joint and the reel for the person undergoing rehabilitation, and then obtain the updated input feedforward current. :

[0084] First, such as Figure 2 As shown, calculate the ankle joint angle. Angle with reel The relationship between them is:

[0085]

[0086] in The distance between the center point of the rope binding on the lower leg and the center of the ankle joint. The distance between the rope at the fixed anchor point and the center of the ankle joint. This represents the initial ankle angle; For the spool itself in The perspective of time; This is the initial angle of the reel; for Ankle angle at any given moment;

[0087] Then, calculate the ankle joint angular velocity. and angular acceleration Relative to the angle and angular velocity of the reel and angular acceleration Relationship:

[0088]

[0089] in:

[0090]

[0091]

[0092]

[0093] Therefore, we can conclude that:

[0094]

[0095] Finally, adjust the angle of the thread reel. Replace with ankle joint angle The updated input feedforward current is obtained. :

[0096]

[0097] Step 205: Introduce a tension feedback controller into the physical model controller to calculate the actual tension force at the ankle joint. Feedback current ;

[0098] First, set up the overall control strategy structure:

[0099]

[0100] in, The total operating output refers to the reference current. ; For the model controller section, the current command is derived using a known model. ; It is the output of the RL policy network, used to correct errors, compensate for nonlinearities and disturbances online.

[0101] Then, based on the motor input torque and the cable outputs tension to the ankle joint The dynamic relationship between Data is obtained through experiments, and parameters are fitted using formulas. , and Then, use sensors to measure ankle joint angle parameters. , and According to the formula

[0102]

[0103] Obtain the output current of the feedforward section To improve system stability, a tension feedback controller, namely a PD controller, is introduced here:

[0104]

[0105] This is the proportionality coefficient. The target tension for the ankle joint. The differential coefficients are... The rate of change of the target tension. The rate of change of the actual tensile force. The rate of change of error, This is for tensile force error;

[0106] Step 206: Combine the feedforward current and the feedback current to form the input current that drives the physical model controller. :

[0107]

[0108] Step 3: Define the core modules of residual reinforcement learning, including state space and action space:

[0109] The state space s includes the actual tension. Target tension Tension error Ankle joint angular velocity and ankle joint angular acceleration .

[0110] Action space Including residual reinforcement learning output .

[0111] Step 4, in Based on this, the residual reinforcement learning core module is used to... The input current is corrected, and the compensation current value is calculated. Its value is the output of the RL policy network. ;

[0112] ;

[0113] Step 5: Iteratively update the RL policy network, applying gradient-based policy search to gradually approximate the optimal control parameters. ;

[0114] like Figure 3 As shown, firstly, the reward function is set, and a penalty for pulling error is set:

[0115]

[0116] In terms of algorithm structure, a reinforcement learning strategy based on the Actor-Critic (AC) architecture is adopted to correct the residuals of the traditional controller output. In this framework, the Critic network is mainly responsible for evaluating the expected reward of the current policy in a given state, which guides the direction of policy optimization.

[0117] according to From the function, we can know that: ;

[0118] structure network: ;

[0119] Construct the loss function: ;

[0120] renew Network weights: ;

[0121] Corresponding to the critic network, the actor network is responsible for learning the optimal policy from the current state and outputting the corresponding residual action compensation. Specifically, the actor network uses the current system state... Input, output residual action The policy network is constructed below:

[0122]

[0123] The update objective of the policy network is to maximize the output of the Critic network. value:

[0124]

[0125] The action value function;

[0126] Then update the policy network weights:

[0127]

[0128] After several iterations, the network parameters are updated via backpropagation. Gradient-based policy search is applied to progressively approximate the optimal control parameters. .

[0129] Step 6: Obtain the optimal control parameters Then, the RL policy network outputs the optimal residual policy. Thus, the total motor input current is obtained:

[0130]

[0131] Step 7: Set the total motor input current. The input is fed into the wearable device's motor, which then tracks and outputs the pulling force to achieve the same tension as the ankle joint. Consistency, thereby enabling auxiliary functions;

[0132] Finally, a Lyapunov stability analysis was performed on the system, transforming the current relationship into a torque relationship:

[0133]

[0134]

[0135] Combining actual dynamic equations:

[0136]

[0137] in For the unmodeled dynamic errors and disturbances, then take:

[0138]

[0139] Finally, we can obtain:

[0140]

[0141] The Lyapunov function for the error is defined as follows:

[0142]

[0143] After differentiating it and simplifying it:

[0144]

[0145]

[0146] RL policy network output It is used to correct errors online and compensate for nonlinearity and disturbances, therefore Will go Since it approaches, we assume that there exists a constant. , so that satisfaction and Therefore, the above Bounded, therefore we can define a bounded region where the error converges:

[0147] .

Claims

1. A residual reinforcement learning control method for a flexible bionic ankle exoskeleton robot, characterized in that, Includes the following steps: Step 1: Secure the wearable device to the waist of the person being tested for rehabilitation. The rope leading out of the wearable device passes through the bindings on the lower leg and reaches the heel, where it is secured to the fixed anchor point set in advance on the shoe. Step 2: As the person undergoing rehabilitation walks, calculate the input current of the physical model controller. : Step 201: Establish the motor dynamics model of the model controller and calculate the torque of the input motor. : ； in, The moment of inertia of the motor shaft. , and These are the angular acceleration, angular velocity, and angle of the motor shaft rotation, respectively. The coefficient of viscous friction of the motor shaft. For motor shaft output, For harmonic deceleration ratio, After deceleration, the output torque of the deceleration wheel is reduced. The rotation angle of the deceleration wheel; Step 202: Establish a human-machine coupled dynamic model of the model controller and calculate the tension force output by the cable to the ankle joint. : ； in, To apply tension to the rope, Represents friction. The coefficient of friction, The angular velocity of the reel rotation; Step 203: Based on the motor input torque and the cable outputs tension to the ankle joint The dynamic relationship between them is used to solve for the input current of the motor. : The torque constant; The moment of inertia of the output end of the reel. The angular acceleration of the rotating reel; The coefficient of viscous friction at the output end of the reel; The radius of the reel; Step 204: Establish a motion model of the ankle joint and the reel for the person undergoing rehabilitation, and then obtain the updated input feedforward current. : Step 205: Introduce a tension feedback controller into the physical model controller to calculate the actual tension force at the ankle joint. Feedback current : ； This is the proportionality coefficient. The target tension for the ankle joint. The differential coefficients are... The rate of change of the target tension. The rate of change of the actual tensile force. The rate of change of error, This is for tensile force error; Step 206: Combine the feedforward current and the feedback current to form the input current that drives the physical model controller. : ； Step 3: Define the core modules of residual reinforcement learning, including state space and action space: Step 4: Calculate the compensation current value using the residual reinforcement learning core module. Its value is the output of the RL policy network. ; Step 5: Iteratively update the RL policy network, applying gradient-based policy search to gradually approximate the optimal control parameters. ; ； The action value function; Step 6: Obtain the optimal control parameters Then, the RL policy network outputs the optimal residual policy. Thus, the total motor input current is obtained: ； Step 7: Set the total motor input current. The input is fed into the wearable device's motor, which then tracks and outputs the pulling force to achieve the same tension as the ankle joint. Consistency, thereby enabling auxiliary functions.

2. The method as described in claim 1, characterized in that, In step one, the wearable equipment includes: a reel, a motor, a controller, a battery, and a sensor, with a rope extending from the reel; the binding is located in the middle of the lower leg.

3. The method as described in claim 1, characterized in that, In step 201, the motor torque is expressed using relevant variables of the reel, as follows: ； The moment of inertia of the output end of the reel. The angular acceleration of the rotating reel; The coefficient of viscous friction at the output end of the reel is denoted as .

4. The method as described in claim 3, characterized in that, Step 204 specifically involves: First, the relationship between the ankle joint angle and the thread sheave angle is calculated as follows: ； in The distance between the center point of the rope binding on the lower leg and the center of the ankle joint. The distance between the rope at the fixed anchor point and the center of the ankle joint. This represents the initial ankle angle; For the thread reel The perspective of time; This is the initial angle of the reel; for Ankle angle at any given moment; Then, calculate the ankle joint angular velocity. and angular acceleration Relative to the angle and angular velocity of the reel and angular acceleration Relationship: ； ； Finally, adjust the angle of the thread reel. Replace with ankle joint angle The updated input feedforward current is obtained. : ； is the torque constant.

5. The method as described in claim 1, characterized in that, In step three, the state space s includes the actual tensile force. Target tension Tension error Ankle joint angular velocity and ankle joint angular acceleration ; Action space Including residual reinforcement learning output action current .

Images