Method and system for controlling track of mechanical arm based on reinforcement learning full-state constraint

Abstract

The invention provides a method and system for controlling a mechanical arm track based on reinforcement learning full-state constraint. The method comprises the steps that a rigid mechanical arm dynamic model of an n-degree-of-freedom rotary joint is established; designing an error conversion function according to the dynamic model and the state constraint, and converting a constrained error e into an unconstrained error xi; according to the updating rate of the unconstrained error xi, a cost function of the mechanical arm system is designed; constructing a Jacobian-Bellman-Hamiltonian equation through a cost function, and solving to obtain an optimal controller; using a critic neural network to approximate the optimal cost function to obtain an approximate optimal controller; and according to the approximate optimal controller, an approximate Hamiltonian-Jacobian-Bellman equation is derived, difference between the approximate Hamiltonian-Jacobian-Bellman equation and the original Hamiltonian-Jacobian-Bellman equation is obtained, and gradient descent is carried out to obtain a critic network weight matrix update rate. The technical problems that under the uncertainty and dynamic change scene and under the multi-constraint condition, mechanical arm trajectory tracking is difficult to achieve, and the mechanical arm trajectory tracking precision is low are solved.

Description

Technical Field

[0001] The present invention relates to the field of automation control, and in particular to a method and system for controlling the trajectory of a robotic arm based on full-state constraints of reinforcement learning. Background Art

[0002] Smart manufacturing and automation have become the main trends in the development of the global manufacturing industry. Traditional robotic arm control systems have limitations in dealing with complex tasks, especially in dealing with rapidly changing production environments, and cannot be flexibly adjusted. The control method based on reinforcement learning can enable the robotic arm to learn autonomously and adapt to different production needs, greatly improving production efficiency and flexibility, and reducing dependence on human intervention.

[0003] Modern manufacturing, service, and medical fields have put forward increasingly complex operational requirements for robotic arms. For example, in production lines, warehousing, medical surgery, and other environments, robotic arms must be able to operate in complex and changing environments while ensuring accuracy and safety. Traditional control methods often show instability in these applications due to their limited ability to respond to environmental changes. Reinforcement learning technology can continuously optimize trajectory control through continuous interactive learning with the environment, achieving higher flexibility and adaptability.

[0004] The movement of the robot arm involves multiple physical states, which need to be constrained to ensure the safety of task completion and the feasibility of trajectory planning. Full-state constraint control can monitor and adjust these parameters in real time to prevent the robot arm from exceeding its physical limitations, reduce the risk of failure, and improve operational accuracy, especially in scenarios involving dangerous and precise operations.

[0005] In practical applications, the physical model of the robot may not be accurate enough, or the environmental information may be incomplete, which will bring difficulties to traditional model-based control methods. Reinforcement learning does not rely on accurate environmental or system models, but learns the best strategy through a large amount of interaction data, which enables it to perform well in environments with incomplete information or uncertainty.

[0006] The full-state constraint control robot arm system based on reinforcement learning has broad application prospects in many fields such as industry, medical treatment, service robots, and aerospace. For example, in the medical field, the robot arm can autonomously perform complex surgical operations; in aerospace, the robot arm can handle delicate tasks in the space station or repair equipment in harsh environments. This system can continuously improve the task execution capability of the robot arm through reinforcement learning, achieving a wider range of application scenarios.

[0007] There are three main difficulties in robot arm trajectory tracking control:

[0008] (1) Realize trajectory tracking of the robotic arm under scenarios of uncertainty and dynamic environmental changes.

[0009] (2) The movement of the robotic arm involves many physical and operational constraints. How to achieve robotic arm trajectory tracking under multiple constraints?

[0010] (3) How to improve the robot arm trajectory tracking accuracy.

[0011] In the existing robot trajectory tracking control scheme based on model prediction, for example, in the existing public document "Robot trajectory tracking algorithm based on model predictive control", a prediction model is constructed based on the dynamic model of the robot, and the state information q(t) at the current time t is used. Predict the system state q(t+k) and output in the next N time steps. The optimization goal is to minimize the cost function of the trajectory tracking error and control input at the future moment. The common objective function form is:

[0012]

[0013] The first term represents the trajectory tracking error, that is, the difference between the predicted joint angle q(t+k) and the reference joint angle q ref The matrix Q determines the weight of the error. The second term is the cost of the control input, which is used to limit the size of the joint torque τ(t+k) to avoid excessive control actions. The matrix R is the weight of the control input. The above-mentioned prior art has the following disadvantages:

[0014] (1) Standard model predictive control methods usually assume that the system is linear or can be handled through linearization. If the robot arm has significant nonlinear characteristics, such as nonlinear friction of the joints, inertia changes, etc., the traditional linear model predictive control method cannot effectively handle these problems.

[0015] (2) Model predictive control relies on the prediction of future system states. As the prediction time increases, the uncertainty of the system model and the impact of environmental disturbances will increase, leading to inaccurate prediction results. Even in short-term predictions, model errors or external disturbances may lead to control deviations, and long-term predictions make the problem more complicated.

[0016] In the existing robot trajectory tracking solutions based on sliding mode control, such as the existing public document "Six-degree-of-freedom robot trajectory tracking control based on fuzzy sliding mode switching gain", the core of sliding mode control is to design a suitable sliding surface, on which the state of the system moves and eventually converges to the expected value. The sliding surface is usually defined as a linear combination of state variables, so that the system meets the expected performance when moving on the sliding surface. Common sliding surface designs are:

[0017]

[0018] Where q is the current state of the system, is the speed of the system, q ref and is the desired trajectory and velocity, and λ is a positive number that controls the response speed of the system.

[0019] The above-mentioned prior art has the following disadvantages:

[0020] (1) One of the most significant drawbacks of sliding mode control is the chattering phenomenon. When the system state is close to the sliding surface, the control input switches frequently, causing the system to oscillate at high frequencies. This phenomenon is called chattering. Especially in practical applications, due to the limited switching rate and sensor noise, chattering may affect the stability and control accuracy of the system.

[0021] (2) The switching control law in sliding mode control is discontinuous, which causes the control input to change frequently when approaching the sliding surface. This discontinuity not only causes chattering, but also causes problems in actual actuators, especially in precision control scenarios, where frequent control changes may not be executed by actual hardware.

[0022] In summary, the existing technology has technical problems such as difficulty in achieving robot arm trajectory tracking and low robot arm trajectory tracking accuracy under scenarios of uncertainty and dynamic environmental changes and multiple constraints. Summary of the invention

[0023] The technical problem to be solved by the present invention is: how to solve the technical problem that the existing technology is difficult to achieve robot arm trajectory tracking and the robot arm trajectory tracking accuracy is low under scenarios of uncertainty and dynamic environmental changes and multiple constraints.

[0024] The present invention adopts the following technical solutions to solve the above technical problems: A method for controlling the trajectory of a robotic arm based on full-state constraints of reinforcement learning includes:

[0025] S1. Establish a dynamic model of a rigid manipulator with n-DOF rotation joints;

[0026] S2. According to the rigid manipulator dynamics model, the state constraints are obtained according to the actual control environment requirements information, and the error conversion function is designed in combination with the state constraints to convert the constrained error e into the unconstrained error ξ;

[0027] S3, according to the error conversion function, processing and according to the update rate of the unconstrained error ξ, designing the cost function of the manipulator system;

[0028] S4, constructing the Jacobi-Bellman-Hamilton equation according to the cost function, and solving it to obtain the optimal controller and the optimal cost function;

[0029] S5, using the critic neural network to approximate the optimal cost function and obtain an approximate optimal controller;

[0030] S6. Based on the approximate optimal controller, the approximate Hamilton-Jacobi-Bellman equation is derived, the approximate Hamilton-Jacobi-Bellman equation is subtracted from the Hamilton-Jacobi-Bellman equation, and the weight matrix update rate of the critic network is obtained using the gradient descent method.

[0031] The present invention uses a reinforcement learning method and a deep neural network as a policy network or a value function network. The neural network used in the present invention is used as a tool to approximate complex nonlinear mappings. Therefore, through deep reinforcement learning, the system can learn the complex nonlinear relationship from the state space to the action space, optimize the control output, and achieve good control of the nonlinear system.

[0032] The present invention can effectively handle all state constraints and ensure that each state variable of the system meets the predetermined constraint conditions during the control strategy optimization process to avoid system instability or default behavior.

[0033] The reinforcement learning control method adopted by the present invention has strong robustness and can maintain trajectory tracking accuracy and system stability in the presence of external interference and internal parameter uncertainty.

[0034] In a more specific technical solution, in S1, the following logic is used to establish a rigid manipulator dynamics model:

[0035]

[0036] In the formula, Represent the position vector and velocity vector respectively, M(q)∈R n×n represents the inertia matrix, represents the Coriolis matrix and the centrifugal matrix, G(q) represents the gravity vector, and τ represents the input of the control system.

[0037] In a more specific technical solution, S2 includes:

[0038] S21. Use the following logic to define the system error:

[0039]

[0040] In the formula, q d Indicates that the desired target point is a constant vector;

[0041] S22. Define the constrained error e using the following logic:

[0042]

[0043] S23. Use the following formula to find the derivative of the constrained error e:

[0044]

[0045] S24, given the system state constraint, and according to the system error, processing to obtain the error constraint;

[0046] S25. Using the following logic, define the error conversion function to implement the system state constraint:

[0047]

[0048] The error update rate is defined using the following formula:

[0049]

[0050] in,

[0051] S26. Combining the derivative of the constrained error e and the error update rate, the following derivation formula is derived:

[0052]

[0053] Among them, α(s)=Γa(e), β(s)=Γb(e).

[0054] In a more specific technical solution, in S24, the following logic is used to give the system state constraint:

[0055] x imin <x i <x imax

[0056] In the formula, x imin and x imax They represent the lower and upper bounds of the system state constraints respectively;

[0057] According to formula (2), the error constraint is obtained:

[0058] e imin <e i <e imax

[0059] In the formula, e imin and e imax denote the lower and upper bounds of the error constraints respectively.

[0060] In a more specific technical solution, in S3, the cost function of the error conversion function is defined using the following logic:

[0061]

[0062] Where, Q(s,τ)=s T As+τ T Bτ represents the local cost function, A∈R 2n×2n , B∈R n×n represents a positive symmetric matrix; according to formula (7), the original error cost function V is derived:

[0063]

[0064] The converted cost function J is used as the upper bound of the original error cost function V.

[0065] In a more specific technical solution, S4 includes:

[0066] S41. Use the following logic to construct the Comparable-Bellman-Hamilton equation:

[0067]

[0068] S42. According to the optimal control theory and the Comparable-Bellman-Hamilton equation, the following formula is obtained:

[0069]

[0070] Where, represents the optimal cost function;

[0071] S43, using the following logic, define the optimal cost function J * :

[0072]

[0073] S44. Use the following logic to solve the optimal cost function and obtain the optimal controller:

[0074]

[0075] In the learning process of the present invention, the reinforcement learning algorithm will balance multiple objectives such as energy consumption and control accuracy, and reduce unnecessary high-frequency switching operations by continuously optimizing the control strategy, thereby reducing the overall energy consumption of the system. In addition, reinforcement learning can encourage the system to learn more energy-saving control methods through a reward and punishment mechanism.

[0076] In a more specific technical solution, S5 includes:

[0077] S51. Use the following logic to approximate the optimal cost function using the critic network:

[0078]

[0079] Where W c ∈Rm represents the weight matrix, m represents the number of hidden nodes in the critic network, represents the activation function, ε(s) represents the reconstruction error;

[0080] S52, combining the optimal controller and formula (13), the optimal controller is obtained:

[0081]

[0082] S53. Substitute equation (13) into the comparable-Bellman-Hamilton equation and process to obtain the optimal comparable-Bellman-Hamilton equation:

[0083]

[0084] Where Z = β(s)B -1 β T (s),

[0085] S54. Use the critic network to approximate the optimal cost function and process it to obtain the value function:

[0086]

[0087] in, Represents the estimated weight matrix of the critic network;

[0088] S55. According to the value function, the approximate optimal controller is obtained:

[0089]

[0090] The reinforcement learning used in this invention has the ability of online learning and can continuously adjust the control strategy as the environment and system change. Even in the face of uncertainty in long-term predictions, the system can continuously learn and adjust so that the control strategy can dynamically adapt to changes in the external environment and changes in the internal state of the robot arm, thereby improving the accuracy and stability of trajectory tracking.

[0091] In a more specific technical solution, S6 includes:

[0092] S61. According to the following logic, the critic network is used to process and obtain the approximate Hamilton-Jacobi-Bellman equation:

[0093]

[0094] S62. Use the following logic to define the approximate error equation of the Hamilton-Jacobi-Bellman equation:

[0095]

[0096] S63. Using the following logic, define the error function of the Hamilton-Jacobi-Bellman equation:

[0097]

[0098] S64. Use the following logic to minimize the error function of the Hamilton-Jacobi-Bellman equation and solve the approximate weight matrix update rate.

[0099] S65 introduces auxiliary items The weight matrix update rate of the critic network is obtained through processing.

[0100] The reinforcement learning of the present invention can avoid high-frequency switching behavior by adaptively learning the control strategy. Through strategy optimization, reinforcement learning can generate continuous and smooth control inputs, thereby reducing or even completely eliminating the chattering phenomenon. And the control strategy can be adjusted in real time according to feedback, so that the controller can avoid high-frequency oscillations while maintaining robustness. Reinforcement learning can minimize energy consumption by optimizing strategies.

[0101] In a more specific technical solution, in S65, the following logic is used to process and obtain the weight matrix update rate of the critic network:

[0102]

[0103] In the formula, η1 and η2 represent design normal constants. V(s) represents the Lyapunov function,

[0104] In a more specific technical solution, the full-state constraint control robot trajectory system based on reinforcement learning includes:

[0105] Dynamic model building module, used to build the dynamic model of rigid manipulator with n-DOF rotation joint;

[0106] The error conversion module is used to obtain state constraints according to the rigid manipulator dynamics model and the actual control environment requirement information, and to design an error conversion function in combination with the state constraints to convert the constrained error e into the unconstrained error ξ. The error conversion module is connected to the dynamics model construction module;

[0107] A cost function design module is used to process and obtain the cost function of the robotic arm system according to the error conversion function and the update rate of the unconstrained error ξ, and the cost function design module is connected with the error conversion module;

[0108] An optimal controller acquisition module is used to construct the Jacobi-Bellman-Hamilton equation according to the cost function, and solve it to obtain the optimal controller and the optimal cost function. The optimal controller acquisition module is connected with the cost function design module;

[0109] The approximate optimal controller acquisition module uses a critic neural network to approximate the optimal cost function to obtain an approximate optimal controller. The approximate optimal controller acquisition module is connected to the optimal controller acquisition module.

[0110] The weight matrix update rate acquisition module is used to derive the approximate Hamilton-Jacobi-Bellman equation according to the approximate optimal controller, to make a difference between the approximate Hamilton-Jacobi-Bellman equation and the Hamilton-Jacobi-Bellman equation, and to obtain the weight matrix update rate of the critic network using the gradient descent method. The weight matrix update rate acquisition module is connected to the approximate optimal controller acquisition module and the optimal controller acquisition module.

[0111] Compared with the prior art, the present invention has the following advantages:

[0112] The present invention uses a reinforcement learning method and a deep neural network as a policy network or a value function network. The neural network used in the present invention is used as a tool to approximate complex nonlinear mappings. Therefore, through deep reinforcement learning, the system can learn the complex nonlinear relationship from the state space to the action space, optimize the control output, and achieve good control of the nonlinear system.

[0113] The present invention can effectively handle all state constraints and ensure that each state variable of the system meets the predetermined constraint conditions during the control strategy optimization process to avoid system instability or default behavior.

[0114] The reinforcement learning control method adopted by the present invention has strong robustness and can maintain trajectory tracking accuracy and system stability in the presence of external interference and internal parameter uncertainty.

[0115] In the learning process of the present invention, the reinforcement learning algorithm will balance multiple objectives such as energy consumption and control accuracy, and reduce unnecessary high-frequency switching operations by continuously optimizing the control strategy, thereby reducing the overall energy consumption of the system. In addition, reinforcement learning can encourage the system to learn more energy-saving control methods through a reward and punishment mechanism.

[0116] The reinforcement learning used in this invention has the ability of online learning and can continuously adjust the control strategy as the environment and system change. Even in the face of uncertainty in long-term predictions, the system can continuously learn and adjust so that the control strategy can dynamically adapt to changes in the external environment and changes in the internal state of the robot arm, thereby improving the accuracy and stability of trajectory tracking.

[0117] The reinforcement learning of the present invention can avoid high-frequency switching behavior by adaptively learning the control strategy. Through strategy optimization, reinforcement learning can generate continuous and smooth control inputs, thereby reducing or even completely eliminating the chattering phenomenon. And the control strategy can be adjusted in real time according to feedback, so that the controller can avoid high-frequency oscillations while maintaining robustness. Reinforcement learning can minimize energy consumption by optimizing strategies.

[0118] The present invention solves the technical problems in the prior art that it is difficult to achieve robot arm trajectory tracking and the robot arm trajectory tracking accuracy is low under scenarios of uncertainty and dynamic environmental changes and multiple constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 This is an algorithm structure diagram of a method for controlling a robot arm trajectory based on full-state constraints based on reinforcement learning according to Embodiment 1 of the present invention;

[0120] Figure 2 Schematic diagram of the basic steps of the method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning in Example 1 of the present invention. DETAILED DESCRIPTION

[0121] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0122] Example 1

[0123] like Figure 1 and Figure 2 As shown, the method for controlling the trajectory of a robotic arm based on full-state constraints of reinforcement learning provided by the present invention includes the following basic steps:

[0124] S1. Establish a dynamic model of a rigid manipulator with n-DOF rotation joints;

[0125] In this embodiment, a dynamics model of a rigid mechanical arm with n-degree-of-freedom rotating joints is established:

[0126]

[0127] in, Represent the position vector and velocity vector respectively, M(q)∈R n×n represents the inertia matrix, represents the Coriolis matrix and the centrifugal matrix, G(q) represents the gravity vector, and τ represents the input of the control system.

[0128] S2, according to the dynamic model established in the above step S1, combined with the state constraints required by the actual control environment, design an error conversion function to convert the constrained error e into the unconstrained error ξ;

[0129] In this embodiment, the error conversion function is designed using the following formula:

[0130] Specifically, the systematic error is defined using the following formula:

[0131]

[0132] Among them, q d Indicates that the desired target point is a constant vector. Definition

[0133] The derivative of the constrained error e is calculated using the following formula:

[0134]

[0135] In this embodiment, given the system state constraints:

[0136] x imin <x i <x imax

[0137] x imin and x imax They represent the lower and upper bounds of the state constraints respectively. According to the above formula (2), the corresponding error constraint can be obtained: imin <e i <e imax , e imin and e imax denote the lower and upper bounds of the error constraints respectively.

[0138] In this embodiment, to implement the state constraint, the following error conversion function is defined:

[0139]

[0140] The error update rate is defined using the following formula:

[0141]

[0142] in,

[0143] In this embodiment, in order to facilitate the subsequent formula derivation, the above formula (3) and formula (5) are combined to obtain the following formula:

[0144]

[0145] Among them, α(s)=Γa(e), β(s)=Γb(e).

[0146] S3, designing a cost function of the robotic arm system according to the update rate of the unconstrained error ξ obtained by the error conversion function designed in the aforementioned step S2;

[0147] In this embodiment, the cost function of the aforementioned error conversion function is designed using the following formula; specifically, the cost function is defined as follows:

[0148]

[0149] Where Q(s,τ)=s T As+τ T Bτ represents the local cost function, A∈R 2n×2n , B∈R n×n represents a positive definite symmetric matrix. From this we can see that:

[0150]

[0151] That is, the converted error cost function J is the upper bound of the original error cost function V.

[0152] S4, constructing the Jacobi-Bellman-Hamilton equation through the cost function designed in the above step S3, and solving it to obtain the optimal controller;

[0153] In this embodiment, the Hamilton-Jacobi-Bellman equation is constructed to solve the aforementioned optimal controller; specifically, the Hamilton-Jacobi-Bellman equation is as follows:

[0154]

[0155] According to the optimal control theory, the following formula can be obtained:

[0156]

[0157] Among them, J * Represents the optimal cost function, and the optimal cost function is defined as follows:

[0158]

[0159] In this embodiment, the optimal controller is obtained by solving the optimal cost function:

[0160]

[0161] S5, using the critic neural network to approximate the optimal cost function, thereby obtaining an approximate optimal controller;

[0162] In this embodiment, the following formula is used to approximate the optimal cost function using the aforementioned critic network:

[0163]

[0164] Among them, W c ∈R m represents the weight matrix, m represents the number of hidden nodes in the critic network, represents the activation function, and ε(s) represents the reconstruction error.

[0165] Combining the above equations (12) and (13), the optimal controller can be obtained as follows:

[0166]

[0167] In this embodiment, by substituting equation (13) into equation (9), the optimal Hamilton-Jacobi-Bellman equation can be obtained as follows:

[0168]

[0169] Where Z = β(s)B -1 β T (s),

[0170] In this embodiment, the value function can be approximated using the critic network to obtain the following value function:

[0171]

[0172] in, represents the estimated weight matrix of the critic network. It can be seen that the approximate optimal controller is:

[0173]

[0174] S6. According to the approximate optimal controller obtained in the above step S5, an approximate Hamilton-Jacobi-Bellman equation is derived, and the approximate Hamilton-Jacobi-Bellman equation is subtracted from the original Hamilton-Jacobi-Bellman equation, and the update rate of the critic network weight matrix is ​​obtained using the gradient descent method.

[0175] In this embodiment, the critic network designed in the aforementioned step S5 uses the gradient descent method to obtain the update rate of the critic network weight matrix; specifically, the critic network is used to approximate the Hamilton-Jacobi-Bellman equation as follows:

[0176]

[0177] Define the approximate error equation for the Hamilton-Jacobi-Bellman equation:

[0178]

[0179] The error function of the Hamilton-Jacobi-Bellman equation is defined as follows:

[0180]

[0181] In this embodiment, by minimizing The update rate of the approximate weight matrix obtained by solving The stability of the online learning of the robotic arm system cannot be guaranteed, so an auxiliary item is introduced Ensure the stability of the online learning of the robotic arm system. Therefore, the update rate of the critic network weight matrix is ​​as follows:

[0182]

[0183] Among them, η1 and η2 represent design normal constants. V(s) represents the Lyapunov function, In summary, the present invention uses reinforcement learning to enhance the adaptive automatic optimization control strategy of the robot arm. Reinforcement learning can automatically adapt to the uncertainty and external disturbances of the system through continuous interaction with the environment. Compared with traditional control methods, reinforcement learning can quickly adjust the control strategy in a real-time changing environment. Therefore, when dealing with uncertainty, changing loads or external disturbances, the robot arm can maintain a high control accuracy and avoid manual adjustment of system parameters. In addition, the need for manual intervention is reduced, and more accurate trajectory tracking and full-state constraint satisfaction can be achieved. Especially in complex tasks, reinforcement learning can find the optimal strategy that is difficult to design with traditional methods.

[0184] The present invention reduces the robot's reliance on precise models, improves the accuracy of trajectory tracking, and reduces the difficulty of controller parameter adjustment. Reinforcement learning is controlled in a data-driven manner and does not need to rely on the precise dynamic model of the robot system. This means that reinforcement learning can work properly even in systems where the model is imprecise or difficult to model. Through repeated iterations, the optimal control strategy is learned to achieve high-precision trajectory tracking. It can optimize multi-dimensional states such as joint speed, position, and acceleration to meet complex task requirements. It also simplifies the complexity of controller design and maintenance, reduces the workload of engineers during debugging and maintenance, and improves the development efficiency of the system.

[0185] The present invention uses a reinforcement learning method and a deep neural network as a policy network or a value function network. The neural network used in the present invention is used as a tool to approximate complex nonlinear mappings. Therefore, through deep reinforcement learning, the system can learn the complex nonlinear relationship from the state space to the action space, optimize the control output, and achieve good control of the nonlinear system.

[0186] The present invention can effectively handle all state constraints and ensure that each state variable of the system meets the predetermined constraint conditions during the control strategy optimization process to avoid system instability or default behavior.

[0187] The reinforcement learning control method adopted by the present invention has strong robustness and can maintain trajectory tracking accuracy and system stability in the presence of external interference and internal parameter uncertainty.

[0188] In the learning process of the present invention, the reinforcement learning algorithm will balance multiple objectives such as energy consumption and control accuracy, and reduce unnecessary high-frequency switching operations by continuously optimizing the control strategy, thereby reducing the overall energy consumption of the system. In addition, reinforcement learning can encourage the system to learn more energy-saving control methods through a reward and punishment mechanism.

[0189] The reinforcement learning used in this invention has the ability of online learning and can continuously adjust the control strategy as the environment and system change. Even in the face of uncertainty in long-term predictions, the system can continuously learn and adjust so that the control strategy can dynamically adapt to changes in the external environment and changes in the internal state of the robot arm, thereby improving the accuracy and stability of trajectory tracking.

[0190] The reinforcement learning of the present invention can avoid high-frequency switching behavior by adaptively learning the control strategy. Through strategy optimization, reinforcement learning can generate continuous and smooth control inputs, thereby reducing or even completely eliminating the chattering phenomenon. And the control strategy can be adjusted in real time according to feedback, so that the controller can avoid high-frequency oscillations while maintaining robustness. Reinforcement learning can minimize energy consumption by optimizing strategies.

[0191] The present invention solves the technical problems in the prior art that it is difficult to achieve robot arm trajectory tracking and the robot arm trajectory tracking accuracy is low under scenarios of uncertainty and dynamic environmental changes and multiple constraints.

[0192] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning, characterized in that: The method comprises: S1. Establish a dynamic model of a rigid manipulator with n-DOF rotation joints; S2, according to the rigid manipulator dynamics model, according to the actual control environment requirement information to obtain state constraints, combined with the state constraints, design an error conversion function, and convert the constrained error e into an unconstrained error ξ; S3, according to the error conversion function, processing and according to the update rate of the unconstrained error ξ, designing the cost function of the manipulator system; S4, constructing the Jacobi-Bellman-Hamilton equation according to the cost function, and solving the equation to obtain an optimal controller and an optimal cost function; S5, using a critic neural network to approximate the optimal cost function to obtain an approximate optimal controller; S6. According to the approximate optimal controller, an approximate Hamilton-Jacobi-Bellman equation is derived, the approximate Hamilton-Jacobi-Bellman equation is subtracted from the Hamilton-Jacobi-Bellman equation, and the weight matrix update rate of the critic network is obtained by using the gradient descent method.

2. The method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning according to claim 1 is characterized in that: In S1, the following logic is used to establish the dynamic model of the rigid manipulator: In the formula, Represent the position vector and velocity vector respectively, M(q)∈R n×n represents the inertia matrix, represents the Coriolis matrix and the centrifugal matrix, G(q) represents the gravity vector, and τ represents the input of the control system.

3. The method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning according to claim 1 is characterized in that: The S2 includes: S21. Use the following logic to define the system error: In the formula, q d Indicates that the desired target point is a constant vector; S22. Define the constrained error e using the following logic: S23, using the following formula, to find the derivative of the constrained error e: S24, given a system state constraint, and according to the system error, processing to obtain an error constraint; S25. Using the following logic, define the error conversion function to implement the system state constraint: The error update rate is defined using the following formula: in, S26. Combining the derivative of the constrained error e and the error update rate, the following derivation formula is derived: Among them, α(s)=Γa(e), β(s)=Γb(e).

4. The method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning according to claim 3 is characterized in that: In S24, the system state constraint is given using the following logic: x imin <x i <x imax In the formula, x imin and x imax denote the lower bound and upper bound of the system state constraint respectively; According to the formula (2), the error constraint is obtained: and imin <e i <e imax In the formula, e imin and e imax denote the lower and upper bounds of the error constraint respectively.

5. The method for controlling the trajectory of a robotic arm based on reinforcement learning full-state constraints according to claim 1 is characterized in that: In S3, the cost function of the error conversion function is defined using the following logic: Where, Q(s,τ)=s T As+τ T Bτ represents the local cost function, A∈R 2n×2n , B∈R n×n represents a positive symmetric matrix; according to the formula (7), the original error cost function V is derived: The converted cost function J is used as the upper bound of the original error cost function V.

6. The method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning according to claim 1 is characterized in that: The S4 includes: S41. Use the following logic to construct the Comparable-Bellman-Hamilton equation: S42. According to the optimal control theory and the Comparable-Bellman-Hamilton equation, the following formula is obtained: Where, represents the optimal cost function; S43, using the following logic, define the optimal cost function J * : S44. Solve the optimal cost function using the following logic to obtain the optimal controller:

7. The method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning according to claim 1 is characterized in that: The S5 includes: S51, using the following logic, using the critic network to approximate the optimal cost function: Where W c ∈R m represents the weight matrix, m represents the number of hidden nodes in the critic network, represents the activation function, ε(s) represents the reconstruction error; S52, combining the optimal controller and the formula (13), processing to obtain the optimal controller: S53, substituting the formula (13) into the comparable-Bellman-Hamilton equation, and processing to obtain the optimal comparable-Bellman-Hamilton equation: Where, Z=β(s)B -1 b T (s), S54, using the critic network to approximate the optimal cost function, and processing to obtain a value function: in, Represents the estimated weight matrix of the critic network; S55. According to the value function, the approximate optimal controller is obtained by processing:

8. The method for controlling the trajectory of a robotic arm based on full-state constraints based on reinforcement learning according to claim 1 is characterized in that: The S6 includes: S61. According to the following logic, the critic network is used to process and obtain the approximate Hamilton-Jacobi-Bellman equation: S62. Using the following logic, define the approximate error equation of the Hamilton-Jacobi-Bellman equation: S63. Using the following logic, define the error function of the Hamilton-Jacobi-Bellman equation: S64, using the following logic, the error function of the Hamilton-Jacobi-Bellman equation is minimized to obtain the approximate weight matrix update rate S65 introduces auxiliary items The weight matrix update rate of the critic network is obtained by processing.

9. The method for controlling the trajectory of a robotic arm based on reinforcement learning full-state constraints according to claim 8 is characterized in that: In S65, the weight matrix update rate of the critic network is obtained by processing using the following logic: In the formula, η1 and η2 represent design normal constants. represents the Lyapunov function, 10. A full-state constraint control robot trajectory system based on reinforcement learning, characterized in that: The system comprises: Dynamic model building module, used to build the dynamic model of rigid manipulator with n-DOF rotation joint; An error conversion module is used to obtain state constraints according to the rigid manipulator dynamics model and the actual control environment requirement information, and to design an error conversion function in combination with the state constraints to convert the constrained error e into an unconstrained error ξ. The error conversion module is connected to the dynamics model construction module; A cost function design module, used to process and obtain a cost function of the robotic arm system according to the error conversion function and according to the update rate of the unconstrained error ξ, wherein the cost function design module is connected to the error conversion module; An optimal controller acquisition module, used to construct the Jacobi-Bellman-Hamilton equation according to the cost function, and solve it to obtain an optimal controller and an optimal cost function, wherein the optimal controller acquisition module is connected to the cost function design module; An approximate optimal controller acquisition module, which uses a critic neural network to approximate the optimal cost function to obtain an approximate optimal controller, and the approximate optimal controller acquisition module is connected to the optimal controller acquisition module; A weight matrix update rate acquisition module is used to derive an approximate Hamilton-Jacobi-Bellman equation according to the approximate optimal controller, to make a difference between the approximate Hamilton-Jacobi-Bellman equation and the Hamilton-Jacobi-Bellman equation, and to obtain the weight matrix update rate of the critic network by using a gradient descent method. The weight matrix update rate acquisition module is connected to the approximate optimal controller acquisition module and the optimal controller acquisition module.

Images