A Model-Free Reinforcement Learning-Based End-Factory Position Control Method for Floating Base Space Robots

Abstract

This invention relates to a model-free reinforcement learning-based method for end-effector position control of a floating-based space robot, belonging to the field of space robot control. This invention addresses the lack of robustness and stability in controllers obtained from existing model-free reinforcement learning algorithms. By introducing the Lyapunov method into reinforcement learning, the algorithm obtained through reinforcement learning gains stability guarantees, improving its robustness to changes in environmental parameters and external disturbances. Based on sampled Lyapunov stability conditions, system stability can be determined using the Lyapunov method without a system model, adding constraints on policy stability during model-free optimization. The invention employs Lyapunov functions and Lyapunov cost functions independent of the value function and reward function, avoiding limitations on action space exploration and improving update efficiency. This method can be applied to end-effector position control of space robots.

Description

Technical Field

[0001] This invention belongs to the field of space robotic arm control, specifically relating to a method for controlling the end position of a floating-based space robotic arm based on model-free reinforcement learning. Background Technology

[0002] Space robotic arms are increasingly being used in space missions. Floating-base space robotic arms, in particular, possess unique advantages due to their fuel efficiency, maneuverability, and low collision losses. Unlike stationary-based robotic arms, floating-base robotic arms exhibit strong dynamic coupling between the base and the arm. The arm's movement causes changes in the base's position and attitude, resulting in different kinematic and dynamic characteristics compared to stationary robotic arms. This makes traditional robotic arm modeling and control methods unsuitable for floating-base robotic arms. Furthermore, establishing accurate dynamic models for floating-base robotic arms is challenging due to fuel consumption and errors in dynamic component parameters. Model-free reinforcement learning, a data-driven algorithm, employs an agent to interact with the environment to acquire data, which is then optimized to create a controller. Using model-free reinforcement learning for space robotic arm control avoids complex kinematic and dynamic modeling processes, offering unique advantages over traditional control methods. However, existing model-free reinforcement learning is highly sensitive to training parameters and the training environment, resulting in controllers derived from these algorithms lacking stability and generalization performance. Consequently, algorithm performance typically degrades under environmental changes (such as variations in system parameters or external disturbances). Furthermore, algorithms trained using model-free reinforcement learning cannot be theoretically proven to be stable, posing security risks to systems deployed on them.

[0003] In summary, controllers obtained based on existing model-free reinforcement learning algorithms lack robustness and stability, and the stability of existing model-free reinforcement learning algorithms cannot be theoretically proven. Therefore, this greatly limits the application of model-free reinforcement learning in the field of space robotic arm control. Summary of the Invention

[0004] The purpose of this invention is to address the lack of robustness and stability of controllers obtained based on existing model-free reinforcement learning algorithms, and to propose a model-free reinforcement learning-based end-effector position control method.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a method for end-effector position control of a floating-based space manipulator based on model-free reinforcement learning, the method specifically including the following steps:

[0006] Step 1: Determine the input state variables and control output types of the floating base space robotic arm system;

[0007] Step 2: Determine the control strategy function π based on the motion space dimension of the robotic arm. θ Value function Q φ and Lyapunov function L ψ The network structure is as follows: θ is the network parameter of the control policy function, φ is the network parameter of the value function, and ψ is the network parameter of the Lyapunov function.

[0008] The control strategy function, value function, and Lyapunov function are all represented by an MLP neural network;

[0009] Step 3: Design the reward function and Lyapunov cost function;

[0010] Step 4: Set the total number of training steps, the length of each episode (an episode refers to the process of the task from start to finish; the robotic arm moves from the initial position to the target position and ends when the maximum number of steps in an episode is reached; the length refers to the number of steps contained in each episode), the batch size during policy updates, the buffer capacity, and the policy update frequency.

[0011] Step 5: Initialize the control strategy function π θ Value function Q φ and Lyapunov function L ψ The network parameters are set, and the target value function is defined. target value function The network parameters are initialized to the value function Q. φ Network parameters;

[0012] Step 6: Reset the environment, set the space robotic arm system to its initial state, and set the target position;

[0013] Step 7: Obtain the current state s from the environment. t Then, the current state s will be obtained. t Input control strategy function π θ The space robotic arm executes the control strategy function π θ Output action a t And enter the next state s t+1 ;

[0014] tuple(s) t ,a t ,r t ,c t ,s t+1 Stored in the buffer, where r t The space robotic arm is in its current state s t Perform action a tThe reward received, c t The space robotic arm is in its current state s t Perform action a t The Lyapunov cost obtained; each time the space robotic arm performs an action and enters the next state is defined as a step;

[0015] Step 8: Determine if the task for the current scene has ended, i.e., determine if the number of steps has reached the length of the current scene;

[0016] If the task of the current scene has not ended, then use state s t+1 Return to step seven;

[0017] Otherwise, once the current task ends, determine whether the policy update frequency has been reached;

[0018] If the policy update frequency is reached, then sample a batch of tuples from the buffer to update the control policy function π. θ Value function Q φ and Lyapunov function L ψ The network parameters are then updated, and the value function Q is updated. φ Network parameters copied to the target value function Return to step six;

[0019] Otherwise, if the policy update frequency has not been reached, proceed to step nine;

[0020] Step 9: Determine if the set total number of training steps has been reached. If the set total number of training steps has been reached, end the training and proceed to Step 10. Otherwise, if the set total number of training steps has not been reached, return to Step 6.

[0021] Step 10: Use the network parameters of the control strategy function obtained at the end of training as the trained network parameters, and perform real-time control of the floating base space robotic arm system based on the trained network parameters.

[0022] Furthermore, the input state variables of the floating base space robotic arm system include information such as the base position p. b Base posture b Space robotic arm joint position q, joint angular velocity and target position p t .

[0023] For the position control task of a floating-base space robotic arm system, the input state variables must contain the above information. In order to improve the algorithm performance, information such as the position of the robotic arm end effector and the current end position error can also be added.

[0024] Furthermore, the control output type of the floating base space robotic arm system is joint angular velocity or joint torque.

[0025] Furthermore, the reward function is designed as follows:

[0026]

[0027] Where r represents the reward function, p t p represents the target position vector. e Let ||·||2 represent the position vector of the end effector of the space robot arm, and let ||·||2 represent the 2-norm.

[0028] Furthermore, the Lyapunov cost function is:

[0029] c = ||p t -p e ||2 (2)

[0030] Where c represents the Lyapunov cost function.

[0031] Furthermore, the update control strategy function π θ Value function Q φ and Lyapunov function L ψ The network parameters, and their specific update process are as follows:

[0032] For the control strategy function π θ The parameter update is an optimization problem that completes equation (3):

[0033]

[0034] Where, π θ (s t ) indicates that the state s t Input control strategy function π θ The output action, Q φ (s t ,π θ (s t )) represents the control strategy function π θ The state value function under condition p(s) t+1 |s t ,π θ (s t )) indicates that in state s t Execute action π θ (s t After that, it enters state s. t+1 The probability, L ψ (·) represents the Lyapunov function value, L ψ (s t ) represents state s t The Lyapunov function value at f, where α is the scaling factor, α > 0, fc It is a positive definite Lyapunov loss function;

[0035] Using the Lagrange method, the dual problem of the optimization problem in equation (3) is obtained:

[0036]

[0037] Where λ is the Lagrange multiplier, L ψ (s t+1 ) represents state s t+1 The Lyapunov function value at point J is the objective function to be optimized.

[0038] Equation (4) is optimized using stochastic gradient descent, and its gradient is:

[0039]

[0040] in, This represents the gradient of J. Represents the control strategy function π θ The gradient, π θ (a t |s t ) represents the control strategy function π θ In state s t Output action a t , Represents the control strategy function π θ The expected value of the following mathematical expression Represents the control strategy function π θ and state space The expected value of the following;

[0041] Control policy function π θ The parameters are updated using equation (6):

[0042]

[0043] Where δ is the learning rate when the network parameters of the control policy function are updated;

[0044] After updating the parameters of the control policy function, fit the value function Q. φ The fitting of the value function is equivalent to:

[0045]

[0046] Among them, y t Q represents the objective value of the value function. φ (s t ,a t ) indicates that in state s tPerform action a t value, Representing the state space The expected value of J Q The loss function used to fit the value function;

[0047] y t The objective value function is derived via bootstrapping:

[0048]

[0049] Where, π θ (a t+1 |s t+1 ) represents strategy π θ In state s t+1 Output action a t+1 γ represents the discount rate for cumulative rewards. The objective value function in state s t+1 and action a t+1 The output below;

[0050] Value function Q φ The network parameters are updated using equation (9):

[0051]

[0052] in, J represents Q The gradient of δ′ is the learning rate of the value function when updating the network parameters.

[0053] Lyapunov function L ψ The network parameter update is equivalent to equation (10):

[0054]

[0055] Among them, L ψ (s t ) indicates that in state s t The Lyapunov function value, The state s is obtained through sampling t The unbiased estimator of the Lyapunov function, J L It is the loss function used for fitting with the Lyapunov function;

[0056]

[0057] Among them, f c (s t ) represents s t The state is a positive definite Lyapunov loss function, where T represents the length of each scene;

[0058] The network parameters of the Lyapunov function are updated via equation (12):

[0059]

[0060] in, J represents L The gradient.

[0061] Furthermore, the Lagrange multiplier is updated as follows:

[0062]

[0063] Among them, J λ Represents the loss function during Lagrange multiplier updates;

[0064] Lagrange multipliers updated via expression (14):

[0065]

[0066] in, J represents λ The gradient.

[0067] Furthermore, the positive definite Lyapunov loss function f c for:

[0068]

[0069] Among them, s e This is the target state of the floating-based space robotic arm system. Represents the control strategy function π θ The expected value of the following mathematical expression.

[0070] Furthermore, the learning rate of both the control policy function and the Lyapunov function is 0.0003, and the learning rate of the value function is 0.001.

[0071] Furthermore, the capacity of the buffer is no less than three times the batch size during policy updates. If the buffer capacity limit is reached, the oldest tuple stored in the current buffer is deleted, and the new tuple is saved into the buffer.

[0072] The beneficial effects of this invention are:

[0073] 1. This invention employs model-free reinforcement learning for the positioning control of a floating-based space manipulator. It directly generates the control strategy through the input of system state variables, avoiding the complex modeling and model error issues inherent in traditional floating-based space manipulators. Furthermore, this method is more universal than traditional control schemes and can be directly applied to various configurations of space manipulators without requiring repeated controller design.

[0074] 2. This invention introduces the Lyapunov method into reinforcement learning, ensuring the stability of the algorithm obtained through reinforcement learning and improving its robustness to changes in environmental parameters and external disturbances. Experiments show that the method of this invention can still achieve high-precision position control when facing changes in system dynamic parameters and external torque disturbances, and its robustness surpasses that of current mainstream model-free reinforcement learning algorithms.

[0075] 3. This invention proposes a sampling-based Lyapunov stability condition, which allows for the determination of system stability using the Lyapunov method without the need for a system model, thereby adding constraints on policy stability during model-free optimization.

[0076] 4. This invention employs a Lyapunov function and a Lyapunov cost function that are independent of the value function and reward function. Compared with existing methods that use the Lyapunov function as the value function in secure reinforcement learning, this invention avoids the problems of limited exploration of the action space and decreased update efficiency caused by the positive definiteness of the Lyapunov function. Attached Figure Description

[0077] Figure 1 This is a flowchart of the method of the present invention;

[0078] Figure 2 This is a diagram showing the position tracking error of the present invention under external disturbances;

[0079] Figure 3(a) is a diagram of the position tracking error of the method of the present invention under changes in the weight of the base;

[0080] Where, Δm b It is the change in the weight of the base;

[0081] Figure 3(b) shows the position tracking error of the PPO algorithm under changes in base weight;

[0082] Figure 3(c) shows the position tracking error of the SAC algorithm under changes in base weight;

[0083] Figure 4(a) is a diagram of the position tracking error of the method of the present invention under the change of the end effector length;

[0084] Where, Δl eIt is the change in the length of the end effector;

[0085] Figure 4(b) shows the position tracking error of the PPO algorithm under varying end effector length.

[0086] Figure 4(c) shows the position tracking error of the SAC algorithm under the change of the end effector length. Detailed Implementation

[0087] Specific Implementation Method 1: Combination Figure 1 This embodiment describes an end-to-end floating space manipulator position control method with stability guarantees based on model-free reinforcement learning. This method, based on model-free reinforcement learning, can be used for space manipulator systems of arbitrary structures. The method specifically includes the following steps:

[0088] Step 1: After setting up the simulation environment in CoppeliaSim, analyze the floating six-DOF space manipulator system (this implementation method is applicable to space manipulators of any structure. For manipulators of other structures and degrees of freedom, it is only necessary to modify the network output of the neural network according to its motion output dimension) to determine the input state variables and control output type.

[0089] The input state variables contain information such as the base position p. b Base posture b Space robotic arm joint position q, joint angular velocity and target position p t The control output type is either joint angular velocity or joint torque.

[0090] For the position control task of a floating-base space robotic arm system, the input state variables must contain the above information. In order to improve the algorithm performance, information such as the position of the robotic arm end effector and the current end position error can also be added.

[0091] In this embodiment, the state variable s is determined by the position p of the base. b The posture of the base b Robotic arm joint position q, joint angular velocity robotic arm end position p e and target position p t Composition, denoted as The control command is the joint angular velocity of the robotic arm.

[0092] Step 2: Determine the control strategy function π based on the motion space dimension of the robotic arm. θ Value function Q φ and Lyapunov function L ψThe network structure is as follows: θ is the network parameter of the control policy function, θ is the network parameter of the value function, and ψ is the network parameter of the Lyapunov function.

[0093] The control strategy function, value function, and Lyapunov function are all represented by an MLP neural network, with subscripts indicating neural network parameters; in this embodiment, the control strategy function π... θ The network structure is (24, 64, 128, 64, 6), where each number represents the number of neurons in that layer. The network input is the state variable s, 24 is the dimension of the state variable, (64, 128, 64) is the hidden layer structure, and the network output is the angular velocity of the robotic arm joints, where 6 represents the angular velocity of the six joints. Similarly, the value function Q... φ and Lyapunov function L ψ The network structures are (30, 64, 128, 64, 1) and (24, 64, 128, 64, 1) respectively. To ensure the positive definiteness of the Lyapunov function, the Lyapunov function network is configured to output L... ψ (s) was processed, and the final output is

[0094] Step 3: Design the reward function and Lyapunov cost function;

[0095] The reward function is designed as follows:

[0096]

[0097] Where r represents the reward function, p t p represents the target position vector. e Let ||·||2 represent the position vector of the end effector of the space robot arm, and let ||·||2 represent the 2-norm.

[0098] The Lyapunov cost function is:

[0099] c = ||p t -p e ||2 (2)

[0100] Where c represents the Lyapunov cost function.

[0101] Step 4: Set the total training steps to 300,000, the length of each episode (episode refers to the process of the task from start to finish; the robotic arm moves from the initial position to the target position, and the episode ends after reaching the maximum number of steps; length refers to the number of steps contained in each episode) to 100, the batch size during policy updates, the buffer capacity (to ensure a balanced sample distribution, the buffer size should be no less than 3 times the batch size; if the buffer capacity limit is reached, delete the oldest tuple stored in the current buffer and save the new tuple into the buffer), and the policy update frequency to once every 5,000 steps; control the learning rate of the policy function and Lyapunov function to be 0.0003, and the learning rate of the value function to be 0.001;

[0102] The strategy is updated using 80 iterations, and the discount rate λ of the reward function is 0.99.

[0103] Step 5: Initialize the control strategy function π θ Value function Q φ and Lyapunov function L ψ To ensure stability during training, the network parameters were configured with a target value function. The target value function With value function Q φ Having the same network structure, the objective value function The network parameters are initialized to the value function Q. φ Network parameters;

[0104] Q is changed every certain number of steps during training. φ Copy the parameters Only used for the value function Q φ The fitting itself does not participate in the policy iteration;

[0105] Step 6: Reset the environment, set the space robotic arm system to its initial state, and set the target position;

[0106] Step 7: Obtain the current state s from the environment. t Then, the current state s will be obtained. t Input control strategy function π θ Space robotic arm execution control strategy network π θ Output action a t And enter the next state s t+1 ; tuple (s t ,a t ,r t ,c t ,s t+1 Stored in the buffer, where r tThe space robotic arm is in its current state s t Perform action a t The reward received, c t The space robotic arm is in its current state s t Perform action a t If the Lyapunov cost obtained reaches the buffer capacity limit, the oldest stored tuple is deleted.

[0107] Each time the space robotic arm performs an action and enters the next state, it is defined as a step.

[0108] Step 8: Determine if the task for the current scene has ended;

[0109] If the task of the current scene has not ended, then use state s t+1 Return to step 7;

[0110] Otherwise, once the current task ends, determine whether the policy update frequency has been reached;

[0111] If the policy update frequency is reached, then sample a batch of tuples from the buffer to update the control policy function π. θ Value function Q φ and Lyapunov function L ψ After checking the network parameters, return to step six;

[0112] Otherwise, if the policy update frequency has not been reached, proceed to step nine;

[0113] The update control strategy function π θ Value function Q φ and Lyapunov function L ψ The network parameters, and their specific update process are as follows:

[0114] For the control strategy function π θ The parameter update is an optimization problem that completes equation (3):

[0115]

[0116] Where, π θ (s t ) indicates that the state s t Input control strategy function π θ The output action, Q φ (s t ,π θ (s t )) represents the control strategy function π θ The state value function under condition p(s) t+1 |s t ,π θ (s t)) indicates that in state s t Execute action π θ (s t After that, it enters state s. t+1 The probability, L ψ (·) represents the Lyapunov function value, L ψ (s t ) represents state s t The Lyapunov function value at f, where α is the scaling factor, α > 0, f c It is a positive definite Lyapunov loss function;

[0117] Using the Lagrange method, the dual problem of the optimization problem in equation (3) is obtained:

[0118]

[0119] Where λ is the Lagrange multiplier, L ψ (s t+1 ) represents state s t+1 The Lyapunov function value at point J is the objective function to be optimized.

[0120] Equation (4) is optimized using stochastic gradient descent, and its gradient is:

[0121]

[0122] in, This represents the gradient of J. Represents the control strategy function π θ The gradient, π θ (a t |s t ) represents the control strategy function π θ In state s t Output action a t , Represents the control strategy function π θ The expected value of the following mathematical expression Represents the control strategy function π θ and state space The expected value of the following;

[0123] Control policy function π θ The parameters are updated using equation (6):

[0124]

[0125] Where δ is the learning rate when the network parameters of the control policy function are updated;

[0126] After updating the parameters of the control policy function, fit the value function Q.φ The fitting of the value function is equivalent to:

[0127]

[0128] Among them, y t Q represents the target value fitted to the value function. φ (s t ,a t ) indicates that in state s t Perform action a t value, Representing the state space The expected value of J Q The loss function used to fit the value function;

[0129] y t The objective value function is derived via bootstrapping:

[0130]

[0131] Where, π θ (a t+1 |s t+1 ) represents strategy π θ In state s t+1 Output action a t+1 γ represents the discount rate for cumulative rewards. The objective value function in state s t+1 and action a t+1 The output below;

[0132] Value function Q φ The network parameters are updated using equation (9):

[0133]

[0134] in, J represents Q The gradient of δ′ is the learning rate of the value function when updating the network parameters.

[0135] Lyapunov function L ψ The network parameter update is equivalent to equation (10):

[0136]

[0137] Among them, L ψ (s t ) indicates that in state s t The Lyapunov function value, The state s is obtained through sampling t The unbiased estimator of the Lyapunov function, JL It is the loss function used for fitting with the Lyapunov function;

[0138]

[0139] Among them, f c (s t ) represents s t The state is a positive definite Lyapunov loss function, where T represents the length of each scene;

[0140] The network parameters of the Lyapunov function are updated via equation (12):

[0141]

[0142] in, J represents L The gradient.

[0143] The Lagrange multiplier is updated as follows:

[0144]

[0145] Among them, J λ Represents the loss function during Lagrange multiplier updates;

[0146] Lagrange multipliers updated via expression (14):

[0147]

[0148] in, J represents λ The gradient.

[0149] Step 9: Determine if the set total number of training steps has been reached. If the set total number of training steps has been reached, end the training and proceed to Step 10. Otherwise, if the set total number of training steps has not been reached, return to Step 6.

[0150] Step 10: Use the network parameters of the control strategy function obtained at the end of training as the trained network parameters, and perform real-time control of the floating base space robotic arm system based on the trained network parameters.

[0151] The positive definite Lyapunov loss function f c for:

[0152]

[0153] Among them, s e This is the target state of the floating-based space robotic arm system. Represents the control strategy function π θ The expected value of the following mathematical expression.

[0154] Experimental Section

[0155] According to Specific Implementation Method 1, the algorithm of the present invention was trained and tested in a simulation environment. To verify the robustness of the algorithm, a position tracking task was performed under different system parameters and with the addition of external perturbations, and the results were compared with the current mainstream model-free reinforcement learning algorithms Proximal Policy Optimization (PPO) and Flexible Action-Evaluation (SAC). Figure 2 To account for position tracking errors caused by external disturbances, the disturbances were uniformly sampled within the range of (-0.1, 0.1)N. Each algorithm was tested with ten random seeds. The graph represents the mean, and the shaded area represents the 1-σ confidence interval. As can be seen from the graph, the algorithm of this invention maintains high position tracking accuracy even under disturbances, while PPO and SAC fail to converge. Figures 3(a), 3(b), 3(c), 4(a), 4(b), and 4(c) verify the robustness of this invention to changes in system parameters. Figures 3(a), 3(b), and 3(c) show the tracking errors under changes in the mass of the floating base, and Figures 4(a), 4(b), and 4(c) show the tracking errors under changes in the length of the end effector of the floating base. As can be seen from Figures 3(a) and 4(a), the present invention can still achieve accurate position tracking even when faced with changes in base mass and end effector, while PPO (as shown in Figures 3(b) and 4(b)) and SAC (as shown in Figures 3(c) and 4(c)) have diverged. The present invention effectively improves the stability and generalization of model-free reinforcement learning algorithms while maintaining position control accuracy, and has high practical application value.

[0156] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for end-effector position control of a floating-based spatial robotic arm based on model-free reinforcement learning, characterized in that, The method specifically includes the following steps: Step 1: Determine the input state variables and control output types of the floating base space robotic arm system; Step 2: Determine the control strategy function π based on the motion space dimension of the robotic arm. θ Value function Q φ and Lyapunov function L ψ The network structure is as follows: θ is the network parameter of the control policy function, φ is the network parameter of the value function, and ψ is the network parameter of the Lyapunov function. Step 3: Design the reward function and Lyapunov cost function; Step 4: Set the total number of training steps, the length of each scene, the batch size during policy updates, the buffer capacity, and the policy update frequency; Step 5: Initialize the control strategy function π θ Value function Q φ and Lyapunov function L ψ The network parameters are set, and the target value function is defined. target value function The network parameters are initialized to the value function Q. φ Network parameters; Step 6: Reset the environment, set the space robotic arm system to its initial state, and set the target position; Step 7: Obtain the current state s from the environment. t Then, the current state s will be obtained. t Input control strategy function π θ The space robotic arm executes the control strategy function π θ Output action a t And enter the next state s t+1 ; tuple(s) t ,a t ,r t ,c t ,s t+1 Stored in the buffer, where r t The space robotic arm is in its current state s t Perform action a t The reward received, c t The space robotic arm is in its current state s t Perform action a t The Lyapunov cost obtained; each time the space robotic arm performs an action and enters the next state is defined as a step; Step 8: Determine if the task for the current scene has ended, i.e., determine if the number of steps has reached the length of the current scene; If the task of the current scene has not ended, then use state s t+1 Return to step seven; Otherwise, once the current task ends, determine whether the policy update frequency has been reached; If the policy update frequency is reached, then sample a batch of tuples from the buffer to update the control policy function π. θ Value function Q φ and Lyapunov function L ψ The network parameters are then updated, and the value function Q is updated. φ Network parameters copied to the target value function Return to step six; Otherwise, if the policy update frequency has not been reached, proceed to step nine; Step 9: Determine if the set total number of training steps has been reached. If the set total number of training steps has been reached, end the training and proceed to Step 10. Otherwise, if the set total number of training steps has not been reached, return to Step 6. Step 10: Use the network parameters of the control strategy function obtained at the end of training as the trained network parameters, and perform real-time control of the floating base space robotic arm system based on the trained network parameters.

2. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 1, characterized in that, The input state variables of the floating base space robotic arm system include information such as the base position p. b Base posture b Space robotic arm joint position q, joint angular velocity and target position p t .

3. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 2, characterized in that, The control output type of the floating base space robotic arm system is joint angular velocity or joint torque.

4. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 3, characterized in that, The reward function is designed as follows: Where r represents the reward function, p t p represents the target position vector. e Let ||·||2 represent the position vector of the end effector of the space robot arm, and let ||·||2 represent the 2-norm.

5. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 4, characterized in that, The Lyapunov cost function is: c=||p t -p e ||2 (2) Where c represents the Lyapunov cost function.

6. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 5, characterized in that, The update control strategy function π θ Value function Q φ and Lyapunov function L ψ The network parameters, and their specific update process are as follows: For the control strategy function π θ The parameter update is an optimization problem that completes equation (3): Where, π θ (s t ) indicates that the state s t Input control strategy function π θ The output action, Q φ (s t ,π θ (s t )) represents the control strategy function π θ The state value function under condition p(s) t+1 |s t ,π θ (s t )) indicates that in state s t Execute action π θ (s t After that, it enters state s. t+1 The probability, L ψ (·) represents the Lyapunov function value, L ψ (s t ) represents state s t The Lyapunov function value at f, where α is the scaling factor, α > 0, f c It is a positive definite Lyapunov loss function; Using the Lagrange method, the dual problem of the optimization problem in equation (3) is obtained: Where λ is the Lagrange multiplier, L ψ (s t+1 ) represents state s t+1 The Lyapunov function value at point J is the objective function to be optimized. Equation (4) is optimized using stochastic gradient descent, and its gradient is: in, Denotes the gradient of J. This represents the gradient of the control policy function πθ. θ (a t |s t ) represents the control strategy function π θ In state s t Output action a t , Represents the control strategy function π θ The expected value of the following mathematical expression Represents the control strategy function π θ and state space The expected value of the following; Control policy function π θ The parameters are updated using equation (6): Where δ is the learning rate when the network parameters of the control policy function are updated; After updating the parameters of the control policy function, fit the value function Q. φ The fitting of the value function is equivalent to: Among them, y t Q represents the objective value of the value function. φ (s t ,a t ) indicates that in state s t Perform action a t value, Representing the state space The expected value of J Q The loss function used to fit the value function; y t The objective value function is derived via bootstrapping: Where, π θ (a t+1 |s t+1 ) represents strategy π θ In state s t+1 Output action a t+1 γ represents the discount rate for cumulative rewards. The objective value function in state s t+1 and action a t+1 The output below; Value function Q φ The network parameters are updated using equation (9): in, J represents Q The gradient of δ′ is the learning rate of the value function when updating the network parameters. Lyapunov function L ψ The network parameter update is equivalent to equation (10): Among them, L ψ (s t ) indicates that in state s t The Lyapunov function value, The state s is obtained through sampling t The unbiased estimator of the Lyapunov function, J L It is the loss function used for fitting with the Lyapunov function; Among them, f c (s t ) represents s t The state is a positive definite Lyapunov loss function, where T represents the length of each scene; The network parameters of the Lyapunov function are updated via equation (12): in, J represents L The gradient.

7. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 6, characterized in that, The Lagrange multiplier is updated as follows: Among them, J λ This represents the loss function during Lagrange multiplier updates; Lagrange multipliers updated via expression (14): in, J represents λ The gradient.

8. The method for end-effector position control of a floating-based space manipulator based on model-free reinforcement learning according to claim 7, characterized in that, The positive definite Lyapunov loss function f c for: Among them, s e This is the target state of the floating-based space robotic arm system. Represents the control strategy function π θ The expected value of the following.

9. The method for end-effector position control of a floating-based space manipulator based on model-free reinforcement learning according to claim 8, characterized in that, The learning rates for the control policy function and the Lyapunov function are both 0.0003, and the learning rate for the value function is 0.

001.

10. The method for end-effector position control of a floating-based space robotic arm based on model-free reinforcement learning according to claim 9, characterized in that, The capacity of the buffer is no less than three times the batch size during policy updates. If the buffer capacity limit is reached, the oldest tuple stored in the current buffer is deleted, and the new tuple is saved into the buffer.

Images