A control method for lower limb rehabilitation robot based on TD3 algorithm

Abstract

The present invention discloses a control method for a lower limb rehabilitation robot based on the TD3 algorithm, which mainly solves the problem of individual difference adaptation between the robot and the patient and the adaptability challenge of the robot in different environments. The method comprises the following steps: S1, using the TD3 algorithm to plan the motion trajectory of the heel particle; S2, using the cubic spline interpolation method to smooth the motion trajectory of the heel particle; S3, deriving the angles of the knee joint and hip joint according to the motion trajectory of the heel particle after smoothing, and obtaining the joint motion target curve; S4, using the TD3 algorithm to control the lower limb exoskeleton joint motor, tracking the target angle curve, and realizing the control of the lower limb rehabilitation robot. Compared with the traditional method, the method of the present invention effectively copes with the problem of database bloat caused by frequent data collection, reduces the workload, and more accurately adapts to the differences between patients and the environment.

Description

Technical Field

[0001] The present invention relates to a method for controlling a lower limb rehabilitation robot based on a TD3 algorithm. Background Art

[0002] The number of elderly patients with brain injury and spinal cord injury is increasing, and the vast majority of patients will have lower limb motor disorders, which seriously affect their ability to take care of themselves. It has been clinically proven that patients can alleviate the further deterioration of lower limb motor dysfunction and even gradually restore lower limb motor function through rehabilitation training and reshaping of motor nerve pathways. The traditional rehabilitation treatment model has the disadvantages of heavy physical burden on doctors and high medical costs, resulting in a shortage of medical resources. In this context, rehabilitation robots have taken advantage of the times with their flexibility and stability, reduced workload of physiotherapists, and scientific and precise training methods.

[0003] At present, the representative achievements of lower limb exoskeleton rehabilitation robots in China include the AiLegs robot and Aiwalker robot of Beijing Da Ai Robot Technology Co., Ltd. The robot adopts a bipedal structure and can assist patients in restoring their natural gait for the personalized needs of patients with postoperative movement recovery of bone and joint surgery, spinal cord injury, brain injury, paralysis, stroke, muscle weakness, etc. Shanghai Fourier Intelligent Technology Co., Ltd. has launched the Fourier X2 wearable lower limb exoskeleton robot, which is mainly used for assisted walking, rehabilitation training, and strengthening motor function. It has multi-sensor fusion technology, motion control system and power unit, which can intelligently analyze the user's intention and dynamically adjust the power output according to the external mechanical environment such as motion trajectory and speed. Chengtian Technology's UGO rehabilitation exoskeleton is suitable for patients with lower limb motor dysfunction caused by spinal cord injury, stroke, lower limb muscle weakness or other neurological diseases. It is mainly used in hospital rehabilitation departments and rehabilitation medical institutions to help rehabilitation patients who can get out of bed to conduct walking training.

[0004] Most lower limb exoskeletons on the market have the following deficiencies: 1. They help patients to perform walking training on flat ground and restore lower limb motor function, but the patient's daily environment is not only flat, but also includes stairs, uphill and downhill environments; 2. The movement trajectory of the lower limb exoskeleton comes from the gait data of normal people. The gait trajectory of people is different due to factors such as height and weight, and the daily environment of patients is diverse. If the gait data of normal people are collected for different environments, the database will be bloated and cumbersome. This patent proposes a lower limb rehabilitation robot control method based on the TD3 algorithm to address the above deficiencies, which can better solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a lower limb rehabilitation robot control method based on the TD3 algorithm, which mainly solves the problem of individual difference adaptation between the robot and the patient and the adaptability challenge of the robot in different environments.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A lower limb rehabilitation robot control method based on TD3 algorithm comprises the following steps:

[0008] S1, using TD3 algorithm to plan the motion trajectory of the heel mass;

[0009] S2, using cubic spline interpolation to smooth the motion trajectory of the heel mass point;

[0010] S3, deriving the angles of the knee joint and the hip joint according to the motion trajectory of the smoothed heel mass point, and obtaining the joint motion target curve;

[0011] S4, uses the TD3 algorithm to control the lower limb exoskeleton joint motors, tracks the target angle curve, and realizes the control of the lower limb rehabilitation robot.

[0012] Furthermore, in step S1, the steps of planning the motion trajectory of the heel particle are as follows:

[0013] S11, establishing a grid map of heel particles;

[0014] S12, using the TD3 algorithm to plan the motion trajectory of the heel particle in the grid map;

[0015] S13, defines the prerequisites for the TD3 algorithm;

[0016] S14, training the value network and strategy network under defined prerequisites.

[0017] Furthermore, in step S13, the prerequisites include:

[0018] S131, defining a motion space of a heel particle at any position in the grid map;

[0019] S132, defining the action space set of the heel mass point;

[0020] S133, define the reward reward rule R for the single-step motion of the heel particle t (S t ,a t ):

[0021]

[0022] Among them, R tThe reward for a single step movement; S t is the coordinate of the heel particle at time t; a t It is the action of the heel mass point;

[0023] S134, define the cumulative reward reward rule Rn of the heel particle:

[0024]

[0025] Among them, Rn is the cumulative reward; γ is the discount coefficient;

[0026] S135, define 4 value neural networks: the first value network, the second value network, the third value network, and the fourth value network; there are 4 network layers in total; input layer: the input signal is the coordinates and actions of the heel mass point; the first hidden layer: a total of 128 neurons are taken, and the activation function is the Relu function; the second hidden layer: 128 neurons are taken, and the activation function is the Relu function; the output layer: the output value is the evaluation value of the coordinates and actions of the heel mass point;

[0027] S136, define two strategy neural networks: the first strategy network and the second strategy network; there are 4 network layers in total; input layer: the input signal is the coordinates of the heel mass point; the first hidden layer: take 128 neurons, the activation function is the Relu function; the second hidden layer: take 128 neurons, the activation function is the Relu function; output layer: the output value is the action of the heel mass point;

[0028] S137, define a data pool.

[0029] Furthermore, in step S14, the steps of training the value network and the strategy network are as follows:

[0030] S141, execute the first strategy network in the motion space to generate N pieces of data, each piece of data contains 4 parts: the current coordinate S t 、Current action a t , single-step reward R t , Next step coordinate S t+1 , and update the data pool;

[0031] S142, randomly sampling k pieces of data from the data pool;

[0032] S143, training value network: all the information of k data, that is, the current coordinate S t 、Current action a t , single-step reward R t , Next step coordinate S t+1 Input the first value network and the third value network respectively, and record the results as v1 and v2; record the next step coordinate S of k data t+1Input the second strategy network to get the next action a t+1 ; The next step coordinate S in k data t+1 Lose and next action a t+1 Input the second value network and the fourth value network respectively, and the results are recorded as target1 and target2; TD target y t for:

[0033] y t =f1·target1+(1-f1)·target2

[0034] Among them, f1 is the weighting coefficient;

[0035] According to the formula:

[0036] δ it =v i (S t ,a t ,w)-y t

[0037] =v i (S t ,a t ,w)-[R t +γv i (S t+1 ,a t ,w)](i=1,2)

[0038] Calculate the TD error; where δ it is the TD error, y t TD target; v i (S t+1 ,a t ,w) is the value network calculation value, w is the value network parameter;

[0039] According to the formula:

[0040]

[0041] Complete the training of the first value network and the third value network; where α is the learning rate;

[0042] S144, take the smaller TD error in step 143 as δ t , according to the formula:

[0043]

[0044] Complete the training of the first policy network; where λ is the parameter of the policy network, π is the policy network, and β is the learning rate;

[0045] S145, jump to step 142, and execute the loop 64 times;

[0046] S146, updating the parameters of the second value network, the fourth value network, and the second strategy network using a soft update method;

[0047] S147, jump to step 141, and execute the loop 64 times;

[0048] S148, executing the first strategy network in the motion space to generate a final motion planning trajectory.

[0049] Furthermore, in step S2, the motion trajectory of the heel particle after smoothing is expressed as:

[0050]

[0051] Furthermore, in step S3, the joint motion target curve is obtained by the smoothed motion trajectory and formula:

[0052]

[0053] and the formula:

[0054]

[0055] We get: i Indicates the angles of the knee and hip joints.

[0056] Furthermore, in step S4, the method of tracking the target angle curve is as follows:

[0057] S41, defines the prerequisites required for the TD3 algorithm;

[0058] S42, training the value network and the strategy network in the manner of step S14;

[0059] S43, executing the first strategy network in the motion space to generate actual motion curves of the lower limb exoskeleton joints.

[0060] Furthermore, in step S41, the specific steps of defining the prerequisites are:

[0061] S411, define the motion space set of 4 joints: {-90°~90°};

[0062] S412, define 4 joint driving torque space sets: {-200 N·m~200 N·m};

[0063] S413, define the single-step motion of 4 joints to obtain the reward return R′:

[0064]

[0065] Among them, θ aim is the desired angle of the joint;

[0066] S414, define the cumulative reward return R n ′:

[0067]

[0068] S415, define 4 value neural networks, the first value network, the second value network, the third value network, and the fourth value network; there are 4 network layers in total, input layer: the input signal is the joint angle and the joint driving torque; the first hidden layer: take 128 neurons, the activation function is the Relu function; the second hidden layer: take 128 neurons, the activation function is the Relu function; the output layer: the output value is the evaluation value of the joint angle and the joint driving torque;

[0069] S416, define two strategy neural networks, the first strategy network and the second strategy network; the neural network has four network layers, the input layer: the input signal is the joint angle; the first hidden layer: take 128 neurons, the activation function is the Relu function; the second hidden layer: take 128 neurons, the activation function is the Relu function; the output layer, the activation function uses the 200×tanh function, and the output value is the joint driving torque;

[0070] S417, define a data pool.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] (1) The present invention adopts a motion trajectory planning method to obtain the target tracking curve of the lower limb exoskeleton rehabilitation robot. Compared with the traditional method, it effectively addresses the problem of database bloat caused by frequent data collection, reduces the workload, and more accurately adapts to the differences between patients and the environment.

[0073] (2) The present invention adopts the deep reinforcement learning algorithm TD3 as the lower limb exoskeleton robot control algorithm, which can better track the upper target curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a block diagram of a lower limb rehabilitation robot control method in an embodiment of the present invention.

[0075] Figure 2 A schematic diagram of a grid map established in an embodiment of the present invention.

[0076] Figure 3 Schematic diagram of the value (strategy) neural network in an embodiment of the present invention.

[0077] Figure 4This is a schematic diagram of the final motion planning trajectory generated by executing the first strategy network in an embodiment of the present invention.

[0078] Figure 5 The effect diagram of the smooth motion trajectory in the embodiment of the present invention.

[0079] Figure 6 Schematic diagram of joint angle definition in an embodiment of the present invention.

[0080] Figure 7 Graph of joint motion target angles in an embodiment of the present invention.

[0081] Figure 8 Actual joint motion curve diagram in the embodiment of the present invention. DETAILED DESCRIPTION

[0082] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The embodiments of the present invention include but are not limited to the following embodiments.

[0083] Example

[0084] like Figure 1 As shown, the present invention discloses a lower limb rehabilitation robot control method based on the TD3 algorithm, which can plan a gait trajectory that matches the patient's body shape and environment in a stair environment, and at the same time complete the robot's tracking of the target trajectory, gradually restore the patient's lower limb muscle strength, and achieve a rehabilitation effect.

[0085] The method uses the TD3 algorithm to plan the motion trajectory of the heel mass point. In this embodiment, the right leg heel is regarded as mass point B, and a grid map is established, such as Figure 2 As shown in the figure, the TD3 algorithm is used to plan the motion trajectory of the heel particle in the grid map.

[0086] Define the prerequisites for the TD3 algorithm, including:

[0087] Define the motion space of the heel mass point at any position in the grid map; define the action space set of the heel mass point: {up, upper right, right, lower right, down, lower left, left, upper left};

[0088] Define the reward reward rule R for the single-step motion of the heel particle t (S t ,a t ):

[0089]

[0090] Among them, R t The reward for a single step movement; S t is the coordinate of the heel particle at time t; a t It is the action of the heel mass point;

[0091] Define the cumulative reward rule Rn for the heel particle:

[0092]

[0093] Among them, Rn is the cumulative reward; γ is the discount coefficient;

[0094] Define 4 value neural networks: the first value network, the second value network, the third value network, and the fourth value network. Their network structures are as follows: Figure 3 As shown; there are 4 network layers in total; input layer: the input signal is the coordinates and action of the heel particle; the first hidden layer: a total of 128 neurons are taken, and the activation function is the Relu function; the second hidden layer: 128 neurons are taken, and the activation function is the Relu function; the output layer: the output value is the evaluation value of the coordinates and action of the heel particle.

[0095] Define two strategy neural networks: the first strategy network and the second strategy network; there are four network layers in total; input layer: the input signal is the coordinates of the heel particle; the first hidden layer: take 128 neurons, and the activation function is the Relu function; the second hidden layer: take 128 neurons, and the activation function is the Relu function; output layer: the output value is the action of the heel particle.

[0096] Finally, define the data pool with a capacity of 8192 records.

[0097] Then, the value network and strategy network training are performed under the above-defined prerequisites:

[0098] (1) Execute the first strategy network in the motion space to generate 256 data, each of which contains 4 parts: the current coordinate S t 、Current action a t , single-step reward R t , Next step coordinate S t+1 , and update the data pool;

[0099] (2) Randomly sample 32 data from the data pool;

[0100] (3) Training value network: All information of 32 data, that is, the current coordinate S t 、Current action a t , single-step reward R t , Next step coordinate S t+1 Input the first value network and the third value network respectively, and record the results as v1 and v2; record the next step coordinates S of the 32 data t+1 Input the second strategy network to get the next action a t+1 ; The next step coordinate S in the 32 data t+1 Lose and next action at+1 Input the second value network and the fourth value network respectively, and the results are recorded as target1 and target2; TD target y t for:

[0101] y t =f1·target1+(1-f1)·target2

[0102] Among them, f1 is the weighting coefficient;

[0103] According to the formula:

[0104] δ it =v i (S t ,a t ,w)-y t

[0105] =v i (S t ,a t ,w)-[R t +γv i (S t+1 ,a t ,w)](i=1,2)

[0106] Calculate the TD error; where δ it is the TD error, y t TD target; v i (S t+1 ,a t ,w) is the value network calculation value, w is the value network parameter;

[0107] According to the formula:

[0108]

[0109] Complete the training of the first value network and the third value network; where α is the learning rate;

[0110] (4) Take the smaller TD error in step (3) as δ t , according to the formula:

[0111]

[0112] Complete the training of the first policy network; where λ is the parameter of the policy network, π is the policy network, and β is the learning rate;

[0113] (5) Jump to step (2) and execute the loop 64 times;

[0114] (6) Using a soft update method to update the parameters of the second value network, the fourth value network, and the second strategy network;

[0115] (7) Jump to step (1) and execute the loop 64 times;

[0116] (8) Execute the first strategy network in the motion space to generate the final motion planning trajectory, such as Figure 4 shown.

[0117] S2, the motion trajectory of the heel mass point is smoothed by using the cubic spline interpolation method; the effect diagram is shown in Figure 4 , the function expression is shown in the formula:

[0118]

[0119] The motion trajectory and formula after smoothing:

[0120]

[0121] and the formula:

[0122]

[0123] Get the joint motion target angle curve, see Figure 7 ; where θ i Indicates the angles of the knee and hip joints. θ1 represents the left knee joint angle, θ2 represents the left hip joint angle, θ3 represents the right hip joint angle, and θ4 represents the right knee joint angle. See the definition of joint angles for Figure 6 .

[0124] The TD3 algorithm is used to control the lower limb exoskeleton joint motors, track the target angle curve, and realize the control of the lower limb rehabilitation robot.

[0125] First, define the prerequisites required for the TD3 algorithm:

[0126] (1) Define the motion space set of four joints: {-90°~90°};

[0127] (2) Define four joint driving torque space sets: {-200 N·m~200 N·m};

[0128] (3) Define the reward R′ obtained by the single-step motion of the four joints:

[0129]

[0130] Among them, θ aim is the desired angle of the joint;

[0131] (4) Define the cumulative reward return R n ′:

[0132]

[0133] (5) Define four value neural networks, the first value network, the second value network, the third value network, and the fourth value network; there are four network layers in total: input layer: the input signal is the joint angle and the joint driving torque; the first hidden layer: 128 neurons are taken, and the activation function is the Relu function; the second hidden layer: 128 neurons are taken, and the activation function is the Relu function; the output layer: the output value is the evaluation value of the joint angle and the joint driving torque;

[0134] (6) Define two strategy neural networks, the first strategy network and the second strategy network; the neural network has four network layers: input layer: the input signal is the joint angle; the first hidden layer: 128 neurons, the activation function is the Relu function; the second hidden layer: 128 neurons, the activation function is the Relu function; the output layer, the activation function uses the 200×tanh function, and the output value is the joint driving torque;

[0135] (7) Define the data pool.

[0136] Then, train the value network and strategy network according to the previous training method.

[0137] Finally, the first policy network is executed in the motion space to generate the actual motion curves of the lower limb exoskeleton joints. Figure 8 As shown in the figure, the actual motion curve of the lower limb exoskeleton controlled by the TD3 algorithm can track the upper target curve well.

[0138] The above embodiment is only one of the preferred implementation modes of the present invention and should not be used to limit the protection scope of the present invention. Any changes or modifications that are made to the main design concept and spirit of the present invention and have no substantive significance, and the technical problems they solve are still consistent with the present invention, should be included in the protection scope of the present invention.

Claims

1. A lower limb rehabilitation robot control method based on TD3 algorithm, characterized in that: The following steps are involved: S1, use TD3 algorithm to plan the motion trajectory of the heel mass point; the specific steps are as follows: S11, establishing a grid map of heel particles; S12, using the TD3 algorithm to plan the motion trajectory of the heel particle in the grid map; S13, defining the prerequisites of the TD3 algorithm; the prerequisites include: S131, defining a motion space of a heel particle at any position in the grid map; S132, defining the action space set of the heel mass point; S133, define the reward reward rule for the single-step motion of the heel particle: Among them, R t The reward for a single step movement; S t is the coordinate of the heel particle at time t; a t It is the action of the heel mass point; S134, define the cumulative reward reward rule for the heel particles: Among them, Rn is the cumulative reward; γ is the discount coefficient; S135, define 4 value neural networks: the first value network, the second value network, the third value network, and the fourth value network; each value neural network has 4 network layers; input layer: the input signal is the coordinates and actions of the heel mass point; the first hidden layer: a total of 128 neurons are taken, and the activation function is the Relu function; the second hidden layer: 128 neurons are taken, and the activation function is the Relu function; the output layer: the output value is the evaluation value of the coordinates and actions of the heel mass point; S136, define two policy neural networks: the first policy network and the second policy network; each policy neural network has four network layers; input layer: the input signal is the coordinates of the heel mass point; the first hidden layer: take 128 neurons, the activation function is the Relu function; the second hidden layer: take 128 neurons, the activation function is the Relu function; the output layer: the output value is the action of the heel mass point; S137, define data pool; S14, training the value network and the strategy network under the defined prerequisites; wherein the steps of training the value network and the strategy network are as follows: S141, execute the first strategy network in the motion space to generate N pieces of data, each piece of data includes 4 parts: S t 、a t , R t , Next step coordinate S t+1 , and update the data pool; S142, randomly sampling k pieces of data from the data pool; S143, training value network: all the information of k data, that is, S t 、a t , R t , Next step coordinate S t+1 Input the first value network and the third value network respectively, and record the results as v1 and v2; record the next step coordinate S of k data t+1 Input the second strategy network to get the next action a t+1 ; The next step coordinate S in k data t+1 and the next action a t+1 Input the second value network and the fourth value network respectively, and the results are recorded as target1 and target2, and the TD target y t for: y t =f1·target1+(1-f1)·target2 Among them, f1 is the weighting coefficient; According to the formula: δ it =v i (S t ,a t ,w)-y t =v i (S t ,a t ,w)-[R t +γv i (S t+1 ,a t ,w)] i=1,2 Calculate the TD error; where δ it is the TD error, y t TD target; v i (S t+1 ,a t ,w) is the value network calculation value, w is the value network parameter; According to the formula: Complete the training of the first value network and the third value network; where α is the learning rate; S144, taking the smaller TD error in step S143 as δ t , according to the formula: Complete the training of the first policy network; where λ is the parameter of the policy network, π is the policy network, and β is the learning rate; S145, jump to step S142, and execute the loop 64 times; S146, updating the parameters of the second value network, the fourth value network, and the second strategy network using a soft update method; S147, jump to step S141, and execute the loop 64 times; S148, executing the first strategy network in the motion space to generate a final motion planning trajectory; S2, using cubic spline interpolation to smooth the motion trajectory of the heel mass point; S3, deriving the angles of the knee joint and the hip joint according to the motion trajectory of the smoothed heel mass point, and obtaining the joint motion target curve; S4, uses the TD3 algorithm to control the lower limb exoskeleton joint motors, tracks the target angle curve, and realizes the control of the lower limb rehabilitation robot.

2. A lower limb rehabilitation robot control method based on TD3 algorithm according to claim 1, characterized in that: In step S2, the motion trajectory of the heel particle after smoothing is expressed as:

3. A lower limb rehabilitation robot control method based on TD3 algorithm according to claim 2, characterized in that: In step S3, the joint motion target curve is a motion trajectory after smoothing and the formula: and the formula: We get: i Indicates the angles of the knee and hip joints.

4. The control method of a lower limb rehabilitation robot based on the TD3 algorithm according to claim 3 is characterized in that: In step S4, the method of tracking the target angle curve is as follows: S41, defines the prerequisites required for the TD3 algorithm; S42, training the value network and the strategy network in the manner of step S14; S43, executing the first strategy network in the motion space to generate actual motion curves of the lower limb exoskeleton joints.

5. The lower limb rehabilitation robot control method based on TD3 algorithm according to claim 4, characterized in that: In step S41, the specific steps of defining the prerequisites are: S411, define the motion space set of 4 joints: {-90°~90°}; S412, define 4 joint driving torque space sets: {-200 N·m~200 N·m}; S413, define the single-step motion of 4 joints to obtain the reward return R′: Among them, θ aim is the desired angle of the joint; S414, define the cumulative reward return R n ′: S415, define 4 value neural networks, the first value network, the second value network, the third value network, and the fourth value network; each value neural network has 4 network layers, input layer: the input signal is the joint angle and the joint driving torque; the first hidden layer: take 128 neurons, the activation function is the Relu function; the second hidden layer: take 128 neurons, the activation function is the Relu function; the output layer: the output value is the evaluation value of the joint angle and the joint driving torque; S416, define two policy neural networks, the first policy network and the second policy network; each policy neural network has four network layers, the input layer: the input signal is the joint angle; the first hidden layer: take 128 neurons, the activation function is the Relu function; the second hidden layer: take 128 neurons, the activation function is the Relu function; the output layer, the activation function uses the 200×tanh function, and the output value is the joint driving torque; S417, define a data pool.

Images