Reinforcement learning parameter self-tuning leaky adaptive spatial robot control method

Abstract

This invention provides a leakage-type adaptive space robot control method with self-tuning of reinforcement learning parameters, belonging to the field of space robot control technology. The control method includes: acquiring the current operating state of the space robot system, the control input from the previous moment, and the control parameters from the previous moment, as the current observation state; inputting the observation state into a policy network to acquire the current control parameters; outputting the current control parameters to a leakage-type adaptive robust controller to acquire the current control input; applying the current control input to the space robot system to drive it to perform a task; acquiring the current driving state of the space robot system; evaluating the driving state and acquiring the evaluation result; and adjusting the parameters of the policy network based on the evaluation result. This invention achieves adaptive parameter optimization by online tuning of key parameters in the leakage-type adaptive robust control structure.

Description

Technical Field

[0001] This invention relates to the field of space robot control technology, and more specifically to a leakage-type adaptive space robot control method with self-tuning of reinforcement learning parameters. Background Technology

[0002] As key equipment in spacecraft on-orbit servicing, space assembly, and deep space exploration missions, the control accuracy and stability of space robots directly affect the safety and reliability of mission execution. Compared with terrestrial robots, space robots typically operate in microgravity, strongly coupled, and highly uncertain environments, making it difficult to accurately establish system dynamics models. Furthermore, mission execution is often accompanied by uncertainties such as load changes, external disturbances, and measurement noise. When space robots perform tasks such as end-effector grasping, path tracking, and assembly, specific servo constraints must be met, meaning the system state must strictly follow a predetermined time function or mission trajectory. Existing space robot control methods are mostly based on precise models or robust control strategies with fixed parameters. Their control parameters are usually tuned offline based on human experience, making it difficult to maintain ideal control performance when the mission environment changes or system uncertainties increase.

[0003] In recent years, reinforcement learning methods have been introduced into the field of robot control. However, existing technologies mostly use reinforcement learning to directly generate control inputs or only for error prediction, resulting in complex control structures, difficulty in stability analysis, or inability to effectively integrate with traditional robust control theory. Furthermore, traditional adaptive control methods are prone to parameter drift during long-term operation, affecting system stability.

[0004] Therefore, there is an urgent need for a space robot control method that can suppress parameter drift and achieve online self-tuning of control parameters while ensuring strict satisfaction of servo constraints, so as to improve the robustness and adaptability of the system in complex and uncertain environments. Summary of the Invention

[0005] The purpose of this invention is to provide a leakage-type adaptive space robot control method with self-tuning of reinforcement learning parameters. By combining reinforcement learning with a leakage-type adaptive robust control structure, online self-tuning of control parameters is achieved, which suppresses parameter drift and improves the servo constraint tracking accuracy and system stability of the space robot system in uncertain environments.

[0006] To achieve the above objectives, embodiments of the present invention provide a leakage-type adaptive space robot control method with self-tuning reinforcement learning parameters, comprising: The current operating state of the space robot system, the control input and control parameters of the previous moment are obtained as the current observation state. The observed state is input into the policy network to obtain the control parameters at the current moment; The control parameters at the current moment are output to the leakage-type adaptive robust controller to obtain the control input at the current moment; The control input at the current moment is applied to the space robot system to drive the space robot system to perform the task; Obtain the current driving state of the space robot system; The driving state is evaluated, and the evaluation result is obtained; The steps involve adjusting the parameters of the policy network based on the evaluation results, and then returning the current operating state of the space robot system, the control input and control parameters of the previous moment as the current observation state.

[0007] Optionally, the current operating state of the space robot system, the control input from the previous moment, and the control parameters from the previous moment are acquired as the current observation state, including: The observation state is determined according to formula (1). (1) in, for The state of observation at any given moment. For the observation state mapping function, for Time-based control input, for Control parameters at time, for The servo constraint error vector at any given time. for The first derivative of the servo constraint error vector at any given time. for The unknown parameter vector is estimated online at any time using a leaky adaptive law.

[0008] Optionally, the observed state is input into the policy network to obtain the control parameters at the current time, including: The motion space of the space robot system is constructed according to formula (2). (2) in, for The space of action at any moment for Damping adjustment parameters at any time for Time uncertainty compensation gain parameter; The observed state is input into the policy network, and the probability distribution parameters of the action space are output according to formulas (3) and (4). (3) (4) in, For sampling variables, The mean of a Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution. For mapping functions; The control parameters at the current moment are obtained according to formulas (5) and (6). (5) (6) in, These are the preset upper and lower limits for the damping adjustment parameters. , For sampling variables The components of the subdimension, , These are the preset upper and lower limits for the uncertainty compensation gain parameters.

[0009] Optionally, the control parameters at the current moment are output to the leakage-type adaptive robust controller to obtain the control input at the current moment, including: According to formula (7), the leakage-type adaptive law is obtained. (7) in, Update rate for unknown parameter vectors. For an unknown parameter vector, , All are servo constraint error vectors. 、 For adaptive gain, It is a non-negative constant. It is a positive integer. For the vector of unknown parameters estimated online, This is the function of uncertainty.

[0010] Optionally, obtaining the control input at the current moment also includes: The nominal constraint force compensation term is obtained according to formula (8). (8) in, For nominal constraint compensation items, For the joint position vectors of the space robot, For joint velocity vectors, For time variables, The nominal inertia matrix, For the servo constraint matrix, To constrain the relevant acceleration vector, The nominal pseudo-acceleration vector is represented by the "+" sign, which is the Moore–Penrose generalized inverse. The error feedback stabilization term is obtained according to formula (9). (9) in, For error feedback stabilization, To calm and enhance, It is a positive definite matrix. To constrain the relevant velocity vector, This is the servo constraint error vector; The uncertainty adaptive compensation term is obtained according to formula (10). (10) in, For the uncertainty adaptive compensation term, For damping adjustment function, This is an error-uncertainty coupling term.

[0011] Optionally, obtaining the control input at the current moment also includes: The control input at the current moment is obtained according to formula (11). (11) in, for Time-based control input, To control the input mapping function, For nominal constraint compensation items, For error feedback stabilization, This is an adaptive compensation term for uncertainty.

[0012] Optionally, the driving state is evaluated to obtain the evaluation result, including: The evaluation result at the current moment is obtained according to formula (12). (12) in, For the multi-objective fusion reward function value, To track the error term, For error penalty weights, To normalize the error, For Huber's robust penalty function, For the rate of change of error, The penalty weight is the rate of change of error. To control energy consumption, As energy consumption penalty weight, For normalized control input, For parameter smoothing term, Smoothing penalty weights for parameters, For parameter changes, This represents the normalized parameter change. This is a saturation hard penalty term. For saturation penalty weights, This is an indicator function.

[0013] Optionally, the parameters of the policy network are adjusted based on the evaluation results, including: At the beginning of training, the policy network and value network are initialized; Obtain the policy network parameters and value network parameters at the current moment; Based on the policy network at the current moment, obtain the trajectory data of the space robot system; Obtain the advantage function value and target reward value at the current moment based on the trajectory data; The policy network and value network are trained based on trajectory data, advantage function value, and target reward value, and the parameters of the policy network and value network are updated accordingly. Determine whether the strategy has converged at the current moment based on the evaluation results; If convergence fails, return to the steps of obtaining the policy network parameters and value network parameters at the current moment; If convergence is achieved, output the updated policy network parameters and value network parameters.

[0014] Optionally, the advantage function value and target reward value at the current moment are obtained based on the trajectory data, including: The dominant function value is obtained according to formulas (13) and (14). (13) (14) in, The advantage function value reflects the advantage of the current action relative to the average action. The trajectory length represents the total number of time steps for a complete control trajectory. For TD error, For index variables, For value network (parameters are) ), used to approximate state values, This is a discount factor used to weigh immediate rewards against future rewards. This is a decay factor used to balance variance and bias; Obtain the target return value according to formula (15). (15) in, The target return value.

[0015] Optionally, the policy network and value network are trained based on trajectory data, advantage function values, and target reward values, and the parameters of the policy network and value network are updated, including: The acquired trajectory data is randomly shuffled and divided into multiple mini-batches; For each mini-batch, the policy objective function is obtained according to formula (16), and the policy network parameters are updated through stochastic gradient ascent. (16) in, Let the policy objective function be... The importance sampling ratio, The cropping threshold, For the clipping function, For expectation operators; The loss function of the value network is determined according to formula (17), and the parameters of the value network are updated by stochastic gradient descent. (17) in, This is the loss function for the value network.

[0016] Through the above technical solution, the present invention provides a leakage-type adaptive space robot control method with self-tuning of reinforcement learning parameters. This method obtains the current operating state of the space robot system, the control input from the previous moment, and the control parameters from the previous moment as the current observation state. The observation state is then input into a policy network to obtain the current control parameters. These parameters are output to a leakage-type adaptive robust controller to obtain the current control input. This control input is then applied to the space robot system to drive it to perform tasks. The current driving state of the space robot system is then obtained. This driving state is evaluated, and the evaluation result is obtained. Based on the evaluation result, the parameters of the policy network are adjusted, and the process returns to the step of obtaining the current observation state. This invention effectively suppresses the parameter drift problem of traditional adaptive control through parameter decomposition and leakage-type adaptive law design, enhancing the system's robustness to uncertainties such as load changes and measurement errors. Through state observation, action generation, multi-objective reward design, and iterative policy network updates, online dynamic optimization of control parameters is achieved, avoiding the limitations of manual experience-based tuning. It can dynamically balance "accuracy, energy consumption, robustness, and saturation risk" in complex time-varying environments, achieving globally optimal control. Furthermore, the reinforcement learning algorithm does not directly participate in the generation of control inputs; it is only used for online tuning of key parameters in the leakage-type adaptive robust control structure. Thus, while maintaining the analytical and provable stability of the control law structure, adaptive parameter optimization is achieved. This method is applicable to various high-precision task scenarios in space robot systems, such as on-orbit grasping, assembly, and path tracking, and can cope with the complexity and uncertainty of the space environment.

[0017] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a leakage adaptive space robot control method with self-tuning reinforcement learning parameters according to one embodiment of the present invention. Figure 2 This is a flowchart illustrating the process of obtaining control parameters at the current moment according to one embodiment of the present invention; Figure 3 This is a flowchart illustrating the process of obtaining the control input at the current moment according to one embodiment of the present invention. Figure 4 This is a parameter flowchart of an adjustment strategy network according to one embodiment of the present invention; Figure 5 This is a flowchart illustrating an embodiment of the present invention for obtaining the advantage function value and the target return value; Figure 6 This is a flowchart illustrating the update strategy network parameters and value network parameters according to one embodiment of the present invention. Detailed Implementation

[0019] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0020] In the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0021] like Figure 1 The diagram shows a flowchart of a leakage-type adaptive space robot control method with self-tuning reinforcement learning parameters, according to one embodiment of the present invention. Figure 1 In this context, the control method may include: In step S1, the current operating state of the space robot system, the control input of the previous moment, and the control parameters of the previous moment are obtained as the current observation state; In step S2, the observation state is input into the policy network to obtain the control parameters at the current time. In step S3, the control parameters at the current moment are output to the leakage-type adaptive robust controller to obtain the control input at the current moment; In step S4, the control input at the current moment is applied to the space robot system to drive the space robot system to perform the task; In step S5, the driving state of the space robot system at the current moment is obtained; In step S6, the driving state is evaluated, and the evaluation result is obtained; In step S7, the parameters of the policy network are adjusted according to the evaluation results, and the current operating state of the space robot system, the control input and control parameters of the previous moment are returned as the observation state of the current moment.

[0022] In Figure 1In the method shown, step S1 can be used to obtain the current operating state of the space robot system, the control input of the previous moment, and the control parameters of the previous moment, as the current observation state. The specific method for obtaining the current observation state can be any of the forms known to those skilled in the art. In one example of this invention, based on Lagrange mechanics theory, considering the inertia, Coriolis force / centrifugal force, gravity, friction, and other dynamic characteristics of the space robot system, a dynamic model is constructed, and a complete dynamic equation for the space robot system is established to accurately describe the dynamic response law of the space robot system in the space environment. The dynamic equation of the space robot system in the space environment can be expressed by formula (18). (18) in, For the runtime of the space robot system, Joint position state vector Let V be the velocity state vector of the joints in the space robot system. Let be the acceleration state vector of the joints in the space robot system. Let be the inertia matrix of the space robot system, which has positive definite properties and reflects the inertial distribution of the system. Uncertain parameters in a space robot system include payload mass, inertia, and time-varying disturbances. This is the Coriolis force / centrifugal force, which is related to joint position and velocity. For the gravity of the space robot system, For the friction force of space robot systems, It serves as a control input, used to drive the system to meet servo constraints and trajectory tracking requirements.

[0023] To address the on-orbit grasping and assembly requirements of space robots, the motion or task constraints that the end effector must satisfy are clearly defined. Servo constraint equations are then established at three levels: position, velocity, and acceleration. Through the mapping relationship between joint space and maneuver space, the state of the space robot system is ensured to meet the task requirements. Specifically, based on the task requirements that the joint position states must satisfy, position-level constraint equations are established, yielding the original form: The matrix form is ,in, For the servo constraint matrix, For constraint vectors, To constrain the number, For joint degrees of freedom, For row index, For column indexes.

[0024] The position-level constraint equations with respect to time Differentiating the equations, introducing the joint velocity term, and establishing the velocity-level constraint equations, we obtain the matrix form: ,in, For the servo constraint matrix, To constrain the relevant velocity vector, , , Servo constraint matrix The line, number Column elements, To constrain the relevant velocity vector The Each component.

[0025] The constraint equations for velocity with respect to time Differentiating the equations and introducing the joint acceleration term, we establish the acceleration-level constraint equations, resulting in matrix form: ,in, To constrain the relevant acceleration vector, , To constrain the relevant acceleration vector The Each component.

[0026] Due to uncertainties such as load variations and measurement errors in space robot systems, the core parameters are decomposed into nominal and uncertain components: ; ; ; ; in, For the parameter nominal part, This represents the uncertain part of the parameters. To simplify the derivation of the control strategy, the following auxiliary function is defined: ; ; ; From the above definition, it can be deduced that and This provides a simplified form for the design of subsequent control items.

[0027] In step S1, the observation state at the current moment can be determined according to formula (1). (1) Considering the real-time requirements of engineering implementation, discrete-time control is adopted, and the discrete-time step is defined. T The sampling period is (Set according to the response requirements of the space robot mission to ensure that the system status can be observed and controlled in a timely manner, usually 1ms~10ms). for The state of observation at any given moment. For the observation state mapping function, for The control input at any given time is used to avoid system instability caused by sudden changes in control input. for Control parameters at all times to ensure smooth parameter adjustments. for The servo constraint error vector at any given time. for exist The general formula for calculation at any given time directly reflects the system's tracking accuracy of servo constraints. for The first derivative of the servo constraint error vector at any given time. It reflects the trend of error changes and is used to suppress system oscillations. for The unknown parameter vector estimated online at each moment using a leaky adaptive law reflects the current estimation level of system uncertainty. By incorporating the control input and control parameters from the previous moment into the current observation state, the temporal continuity of control behavior can be explicitly perceived, thereby avoiding drastic changes in parameters between adjacent moments and effectively suppressing parameter chattering and control oscillations.

[0028] Step S2 can be used to input the observed state into the policy network to obtain the control parameters at the current time. For the initial parameters... , Empirical values ​​can be used to set the parameters to accelerate the convergence speed of reinforcement learning. The specific method for obtaining the control parameters at the current moment can be of various forms known to those skilled in the art. In one example of the present invention, the method for obtaining the control parameters at the current moment may include, for example... Figure 2 The steps are shown. Specifically: In step S11, the motion space of the space robot system is constructed according to formula (2). (2) in, for The space of action at any moment for Damping adjustment parameters at any time for Time uncertainty compensation gain parameter; In step S12, the observation state is input into the policy network. The policy network adopts a deep neural network structure, and the input is the observation state. The output is the action space. The probability distribution parameters. To ensure the continuity and controllability of the action output, the policy network output has a Gaussian distribution mean. and standard deviation The probability distribution parameters of the action space are output according to formulas (3) and (4). (3) (4) in, For sampling variables, The mean of a Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution. A mapping function used to map unbounded sampled variables. Convert to bounded control parameters to avoid system oscillation or control saturation caused by improper parameter values; In step S13, the following is adopted: The function converts unbounded variables into bounded parameters, and obtains the control parameters at the current moment according to formulas (5) and (6). (5) (6) in, These are the preset upper and lower limits for the damping adjustment parameters. , For sampling variables The components of the subdimension, , This mapping ensures that the preset upper and lower limits of the uncertainty compensation gain parameters are used to ensure that... , .

[0029] Step S3 can be used to output the control parameters at the current moment to the leakage-type adaptive robust controller to obtain the control input at the current moment. The specific method for obtaining the control input at the current moment can be of various forms known to those skilled in the art. In one example of the present invention, the method for obtaining the control input at the current moment may include, for example... Figure 3 The steps are shown. Specifically: In step S21, to suppress the parameter drift problem of the traditional adaptive law, the leakage-type adaptive law is obtained according to formula (7). (7) in, Update rate for unknown parameter vectors. For an unknown parameter vector, , All are servo constraint error vectors. 、 For adaptive gain, It is a non-negative constant. It is a positive integer. For the vector of unknown parameters estimated online, The uncertainty effect function has a first term that is an adaptive update term, which makes the parameter estimates... Adjust in the direction of offsetting the error; the second term is the leakage term, when When the norm is too large, its growth is suppressed through negative feedback to ensure... Boundedness is ensured to avoid parameter drift. By introducing a leakage term related to the norm of the parameter estimate into the adaptive law, the leakage term inhibits parameter updates when the system error is small or the parameter estimate tends to increase. This avoids the problem of unbounded parameter growth in traditional adaptive control, ensures the boundedness of parameter estimates and control inputs, and further improves the stability of the system in long-term operation and under highly uncertain environments.

[0030] Under ideal conditions where system uncertainties are known or negligible, to ensure that the space robot system strictly satisfies servo constraints at the dynamic level, ideal constraint forces are first constructed based on the system's nominal dynamic model. From the space robot's dynamic equations and servo constraint equations, it can be obtained that when the constraint conditions... and Given that, under the constraint consistency assumption, the ideal constraint force required for the system to satisfy the servo constraints can be expressed as:

[0031] in, "+" indicates Moore–Penrose generalized inverse.

[0032] The aforementioned ideal constraint force is used to compensate for the nominal dynamic behavior of the system under uncertainty-free conditions, ensuring that the system strictly meets servo constraint requirements. However, in actual space missions, system parameters are generally subject to uncertainty and external disturbances, making it difficult to guarantee control performance by relying solely on the ideal constraint force. Therefore, in this invention, the ideal constraint force is used as the nominal constraint force compensation term, and an error feedback stabilization term and an uncertainty adaptive compensation term are introduced on this basis to form a complete leakage-type adaptive robust control input. The nominal constraint force compensation term, error feedback stabilization term, and uncertainty adaptive compensation term are determined according to steps S22 to S24, respectively.

[0033] In step S22, the nominal constraint force compensation term is obtained according to formula (8). (8) in, For nominal constraint compensation items, For the joint position vectors of the space robot, For joint velocity vectors, For time variables, The nominal inertia matrix, For the servo constraint matrix, To constrain the relevant acceleration vector, , The nominal pseudo-acceleration vector is represented by the "+" sign, which is the Moore–Penrose generalized inverse. In step S23, the error feedback stabilization term is obtained according to formula (9). (9) in, For error feedback stabilization, To calm and enhance, To constrain the relevant velocity vector, For the servo constraint error vector, It is a positive definite matrix and satisfies: , for There exists a constant Make , Through the error vector The negative feedback enables local stabilization of the system, accelerates error convergence, and improves the dynamic response performance of the system.

[0034] In step S24, the uncertainty adaptive compensation term is obtained according to formula (10). (10) in, For the uncertainty adaptive compensation term, The compensation strength is determined by the uncertainty compensation gain parameter. Decide, Here is the damping adjustment function, and the damping adjustment parameters are... Directly applied to functions Balancing the sensitivity of uncertainty compensation with system stability For error-uncertainty coupling terms, Let uncertainty affect the function. , , , The function is concave and is used to quantify the range of uncertainty. This part updates the estimated value of the unknown parameter vector based on the leakage adaptive law of formula (7), and then uses the updated parameter estimate to calculate the uncertainty adaptive compensation term, adjust the compensation strength online, and offset the impact of uncertainty on system performance.

[0035] In step S25, the control input at the current moment is obtained according to formula (11). (11) in, for Time-based control input, To control the input mapping function, For nominal constraint compensation items, For error feedback stabilization, This is an adaptive compensation term for uncertainty.

[0036] In step S25, the bounded control parameters output by the policy network are... , Substituting the leakage-type adaptive robust controller, the expression in formula (11) is generated. The control input at any time, among which, and Depends solely on the observed state ， Depend on 、 and Jointly decided. The space robot system receives control input. Under the influence of, from state of time Evolved to state of time The state transition process follows the space robot's dynamic equations, servo constraints, and leakage-type adaptive laws, and can be abstractly represented as: in, The state transition function implicitly contains the system's dynamic characteristics, servo constraints, and sources of uncertainty.

[0037] Step S4 can be used to apply the control input at the current moment to the space robot system, driving the space robot system to perform tasks. Step S5 can be used to obtain the driving state of the space robot system at the current moment. Step S6 can be used to evaluate the driving state and obtain the evaluation result. The specific method for obtaining the evaluation result can be of various forms known to those skilled in the art. In one example of the present invention, the evaluation result at the current moment can be obtained according to formula (12). (12) in, For the multi-objective fusion reward function value, This is a tracking error term used to penalize servo constraint tracking errors. For error penalty weights, For normalized error ( (the maximum allowable error of the system). The Huber robust penalty function avoids excessive penalty during sudden error changes. This is the error rate of change term, used to penalize drastic changes in error and suppress system oscillations. The penalty weight is the rate of change of error. To control energy consumption, this is used to penalize excessive control inputs and reduce system energy consumption. As energy consumption penalty weight, For normalized control input ( To control the upper limit of input hardware). This is a parameter smoothing term used to penalize drastic changes in control parameters and ensure smooth parameter adjustment. Smoothing penalty weights for parameters, For parameter changes, This represents the normalized parameter change. This is a saturation hard penalty term, used to impose a severe penalty when the control input becomes saturated, ensuring system safety. For saturation penalty weights, This is an indicator function (it takes the value 1 if the condition is met, otherwise it takes the value 0).

[0038] The weighting coefficients must meet the following requirements: The priority is to ensure the safety of the space robot system (avoiding control saturation), followed by ensuring the accuracy of collaborative control, while also considering oscillation suppression, energy consumption reduction, and parameter smoothness, which aligns with the priority requirements of collaborative tasks for space station robots. A multi-objective fusion reward function quantifies the actual effect of the control input generated by the current control parameters, awarding positive rewards if the space robot's trajectory tracking is accurate, control energy consumption is low, and no saturation oscillations occur; conversely, negative penalties are imposed if tracking errors are large or control quantities exceed limits. Evaluation results are obtained based on the multi-objective fusion reward function value. These evaluation results, considering control accuracy, system oscillations, energy consumption, parameter smoothness, and prioritizing the avoidance of control saturation, provide feedback for subsequent policy network updates.

[0039] Step S7 can be used to adjust the parameters of the policy network based on the evaluation results, and return the current operating state of the space robot system, the control input at the previous moment, and the control parameters at the previous moment as the current observation state. The specific method for adjusting the parameters of the policy network can be of various forms known to those skilled in the art. In one example of the present invention, the method for adjusting the parameters of the policy network may include, for example... Figure 4 The steps are shown. Specifically: In step S31, at the beginning of training, the policy network and the value network are initialized; In step S32, the policy network parameters and value network parameters at the current moment are obtained; In step S33, the trajectory data of the space robot system is obtained based on the policy network at the current moment; In step S34, the advantage function value and target reward value at the current moment are obtained based on the trajectory data; In step S35, the policy network and value network are trained based on trajectory data, advantage function value and target reward value, and the parameters of the policy network and value network are updated. In step S36, it is determined whether the strategy at the current moment has converged based on the evaluation results; In step S37, if convergence is not achieved, return to the step of obtaining the policy network parameters and value network parameters at the current time. In step S38, if convergence is achieved, the updated policy network parameters and value network parameters are output.

[0040] In Figure 4 In the method shown, step S31 can be used to initialize the policy network and value network at the beginning of training. The policy network is used to generate actions and acquire trajectory data, while the value network is used to calculate the advantage function value or target reward value. The policy network is the sole generator of control parameters, and its core function is to dynamically generate damping adjustment parameters adapted to the current working conditions based on the real-time observation state of the space robot. and uncertainty compensation gain parameters The policy network provides key parameter support for generating control inputs for the subsequent leaky adaptive robust controller, while the value network only provides evaluation basis for the training and updating of the policy network. The two have a clear division of labor and work together to complete the training and updating of the policy network. Step S32 can be used to obtain the policy network parameters and value network parameters at the current moment. Step S33 can be used to interact with the space robot system based on the policy network at the current moment to obtain the trajectory data of the space robot system. Step S34 can be used to obtain the advantage function value and target reward value at the current moment based on the trajectory data. The specific methods for obtaining the advantage function value and target reward value can be of various forms known to those skilled in the art. In one example of the present invention, the method for obtaining the advantage function value and target reward value may include, for example... Figure 5 The steps are shown. Specifically: In step S41, the dominant function value is obtained according to formulas (13) and (14). (13) (14) in, The advantage function value reflects the advantage of the current action relative to the average action. The trajectory length represents the total number of time steps for a complete control trajectory. For TD error, For index variables, For value network (parameters are) ), used to approximate state values, This is a discount factor used to weigh immediate rewards against future rewards. The attenuation factor is used to balance variance and bias. In one example of this invention, the discount factor is set to 0.99 and the attenuation factor is set to 0.95. In step S42, the target return value is obtained according to formula (15). (15) in, The target return value.

[0041] Step S35 can be used to train the policy network and value network based on trajectory data, advantage function values, and target reward values, and update the policy network parameters and value network parameters. The specific method for updating the policy network parameters and value network parameters can be of various forms known to those skilled in the art. In one example of the present invention, the method for updating the policy network parameters and value network parameters may include, for example... Figure 6 The steps are shown. Specifically: In step S51, the acquired trajectory data is randomly shuffled and divided into multiple mini-batches.

[0042] In step S52, for each mini-batch, the policy objective function is obtained according to formula (16), and the policy network parameters are updated through stochastic gradient ascent. (16) in, Let the policy objective function be... The importance sampling ratio, The old strategy before the update. For the clipping function, For expectation operator, The pruning threshold, in one example of this invention, is set to 0.2 to limit the difference between the new and old strategies. When (the current action is better than the average action), if Cut into To avoid instability caused by the new strategy network excessively increasing the probability of this action, when When (the current action is worse than the average action), if Cut into This avoids instability caused by the new strategy network excessively reducing the probability of this action.

[0043] In step S53, the value network loss function is determined according to formula (17), and the value network parameters are updated by stochastic gradient descent. (17) in, This is the loss function for the value network.

[0044] In Figure 6 In the steps shown, the acquired trajectory data of the space robot system is randomly shuffled and divided into multiple mini-batches. For each mini-batch, the policy objective function is calculated according to formula (16). The policy network parameters are updated through stochastic gradient ascent, and the value network loss function is calculated according to formula (17). The value network parameters are updated using stochastic gradient descent.

[0045] Step S36 can be used to determine whether the policy has converged at the current moment based on the evaluation results. Policy convergence means that the policy (i.e., action selection) no longer changes significantly, or the policy's performance (such as average reward) tends to stabilize; this is the core objective of reinforcement learning training. Step S37 can be used to return to the step of obtaining the policy network parameters and value network parameters at the current moment if convergence has not occurred. If the change in average reward between adjacent rounds is greater than or equal to a preset threshold, then the policy at the current moment has not converged, and the step of obtaining the policy network parameters and value network parameters at the current moment is returned. Step S38 can be used to output the updated policy network parameters and value network parameters if convergence has occurred. If the change in average reward between adjacent rounds is less than a preset threshold, then the policy at the current moment has converged, and the updated policy network parameters and value network parameters are output. The average reward is obtained by averaging the multi-objective fusion reward function values ​​at each step in the multi-round trajectory. By observing the change in this average value, it is possible to intuitively determine whether the update of the policy network parameters improves the control effect (the closer the average reward is to 0, the smaller the total penalty, and the better the control effect).

[0046] Through the above technical solution, this invention provides a leakage-type adaptive space robot control method with self-tuning of reinforcement learning parameters. It obtains the current operating state of the space robot system, the control input from the previous moment, and the control parameters from the previous moment as the current observation state. This observation state is input into a policy network to obtain the current control parameters. The current control parameters are then output to a leakage-type adaptive robust controller to obtain the current control input. This current control input is applied to the space robot system to drive it to perform tasks. The current driving state of the space robot system is then obtained, evaluated, and the evaluation results are obtained. Based on the evaluation results, the parameters of the policy network are adjusted, and the process returns to the step of obtaining the current observation state. This invention, based on a Lagrange dynamics model combined with multi-layered servo constraints, can accurately describe the dynamic characteristics and task requirements of the space robot, providing a solid theoretical foundation for high-precision control. By employing parameter decomposition and leakage-type adaptive law design, the parameter drift problem of traditional adaptive control is effectively suppressed, enhancing the system's robustness to uncertainties such as load changes and measurement errors. Through state observation, action generation, multi-objective reward design, and iterative updates of the policy network, online dynamic optimization of control parameters is achieved, avoiding the limitations of manual experience-based tuning. This method can dynamically balance "accuracy, energy consumption, robustness, and saturation risk" in complex time-varying environments, achieving globally optimal control. Furthermore, the reinforcement learning algorithm does not directly participate in the generation of control inputs but is only used for online tuning of key parameters in the leakage-type adaptive robust control structure. Thus, while maintaining the analytical and provable stability of the control law structure, adaptive optimization of parameters is achieved. This method is applicable to various high-precision task scenarios such as on-orbit grasping, assembly, and path tracking in space robot systems, and can cope with the complexity and uncertainty of the space environment.

[0047] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0048] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0049] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0050] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0051] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0052] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0053] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0054] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0055] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for controlling a leaky adaptive space robot with self-tuning of reinforcement learning parameters, characterized in that, The control method includes: The current operating state of the space robot system, the control input and control parameters of the previous moment are obtained as the current observation state. The observed state is input into the policy network to obtain the control parameters at the current moment; The control parameters at the current moment are output to the leakage-type adaptive robust controller to obtain the control input at the current moment; The control input at the current moment is applied to the space robot system to drive the space robot system to perform the task; Obtain the current driving state of the space robot system; The driving state is evaluated, and the evaluation result is obtained; The steps involve adjusting the parameters of the policy network based on the evaluation results, and then returning the current operating state of the space robot system, the control input and control parameters of the previous moment as the current observation state.

2. The control method according to claim 1, characterized in that, The current operating state of the space robot system, the control input from the previous moment, and the control parameters from the previous moment are acquired as the current observation state, including: The observation state is determined according to formula (1). ，（1） in, for The state of observation at any given moment. For the observation state mapping function, for Time-based control input, for Control parameters at time, for The servo constraint error vector at any given time. for The first derivative of the servo constraint error vector at any given time. for The unknown parameter vector is estimated online at any time using a leaky adaptive law.

3. The control method according to claim 2, characterized in that, The observed state is input into the policy network to obtain the control parameters for the current time, including: The motion space of the space robot system is constructed according to formula (2). ，（2） in, for The space of action at any moment for Damping adjustment parameters at any time for Time uncertainty compensation gain parameter; The observed state is input into the policy network, and the probability distribution parameters of the action space are output according to formulas (3) and (4). ，（3） ，（4） in, For sampling variables, The mean of a Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution. For mapping functions; The control parameters at the current moment are obtained according to formulas (5) and (6). ，（5） ，（6） in, These are the preset upper and lower limits for the damping adjustment parameters. , For sampling variables The components of the subdimension, , These are the preset upper and lower limits for the uncertainty compensation gain parameters.

4. The control method according to claim 3, characterized in that, The control parameters at the current moment are output to the leakage-type adaptive robust controller to obtain the control input at the current moment, including: According to formula (7), the leakage-type adaptive law is obtained. ，（7） in, Update rate for unknown parameter vectors. For an unknown parameter vector, , All are servo constraint error vectors. 、 For adaptive gain, It is a non-negative constant. It is a positive integer. For the vector of unknown parameters estimated online, This is the function of uncertainty.

5. The control method according to claim 4, characterized in that, Obtaining the control input at the current moment also includes: The nominal constraint force compensation term is obtained according to formula (8). ，（8） in, For nominal constraint compensation items, For the joint position vectors of the space robot, For joint velocity vectors, For time variables, The nominal inertia matrix, For the servo constraint matrix, To constrain the relevant acceleration vector, The nominal pseudo-acceleration vector is represented by the "+" sign, which is the Moore–Penrose generalized inverse. The error feedback stabilization term is obtained according to formula (9). ，（9） in, For error feedback stabilization, To calm and enhance, It is a positive definite matrix. To constrain the relevant velocity vector, This is the servo constraint error vector; The uncertainty adaptive compensation term is obtained according to formula (10). ，（10） in, For the uncertainty adaptive compensation term, For damping adjustment function, This is an error-uncertainty coupling term.

6. The control method according to claim 5, characterized in that, Obtaining the control input at the current moment also includes: The control input at the current moment is obtained according to formula (11). ，（11） in, for Time-based control input, To control the input mapping function, For nominal constraint compensation items, For error feedback stabilization, This is an adaptive compensation term for uncertainty.

7. The control method according to claim 1, characterized in that, The driving state is evaluated, and the evaluation result is obtained, including: The evaluation result at the current moment is obtained according to formula (12). ，（12） in, For the multi-objective fusion reward function value, To track the error term, For error penalty weights, To normalize the error, For Huber's robust penalty function, For the rate of change of error, The penalty weight is the rate of change of error. To control energy consumption, As energy consumption penalty weight, For normalized control input, For parameter smoothing term, Smoothing penalty weights for parameters, For parameter changes, This represents the normalized parameter change. This is a saturation hard penalty term. For saturation penalty weights, This is an indicator function.

8. The control method according to claim 7, characterized in that, Adjusting the parameters of the policy network based on the evaluation results includes: At the beginning of training, the policy network and value network are initialized; Obtain the policy network parameters and value network parameters at the current moment; Based on the policy network at the current moment, obtain the trajectory data of the space robot system; Obtain the advantage function value and target reward value at the current moment based on the trajectory data; The policy network and value network are trained based on trajectory data, advantage function value, and target reward value, and the parameters of the policy network and value network are updated accordingly. Determine whether the strategy has converged at the current moment based on the evaluation results; If convergence fails, return to the steps of obtaining the policy network parameters and value network parameters at the current moment; If convergence is achieved, output the updated policy network parameters and value network parameters.

9. The control method according to claim 8, characterized in that, Based on the trajectory data, obtain the current advantage function value and target reward value, including: The dominant function value is obtained according to formulas (13) and (14). ，（13） ，（14） in, The advantage function value reflects the advantage of the current action relative to the average action. The trajectory length represents the total number of time steps for a complete control trajectory. For TD error, For index variables, For value network (parameters are) ), used to approximate state values, This is a discount factor used to weigh immediate rewards against future rewards. This is a decay factor used to balance variance and bias; Obtain the target return value according to formula (15). ，（15） in, The target return value.

10. The control method according to claim 9, characterized in that, The policy network and value network are trained based on trajectory data, advantage function values, and target reward values, and their parameters are updated, including: The acquired trajectory data is randomly shuffled and divided into multiple mini-batches; For each mini-batch, the policy objective function is obtained according to formula (16), and the policy network parameters are updated through stochastic gradient ascent. ，（16） in, Let the policy objective function be... The importance sampling ratio, The cropping threshold, For the clipping function, For expectation operators; The loss function of the value network is obtained according to formula (17), and the parameters of the value network are updated by stochastic gradient descent. ，（17） in, This is the loss function for the value network.

Images