Fixed time safety tracking control method for bolt operation robot under multi-learning tunnel performance

Abstract

The invention relates to a bolt operation robot fixed time safety tracking control method under multi-learning tunnel performance, and belongs to the field of robot self-adaption. The method comprises the following steps: firstly, establishing a bolt operation robot system model, designing a series of important transfer functions, and introducing a tunnel type preset performance function into tracking error constraint control to realize a global control effect; a continuous excitation condition is eliminated by utilizing composite learning control, a weak excitation condition of interval excitation is realized, a novel nonlinear fast integral terminal sliding mode fixed time convergence controller is designed, fixed time convergence of tracking errors is ensured, and meanwhile, the problem of singularity is avoided. And finally, the control scheme is combined with an Actor-Critic architecture of reinforcement learning to form a novel safe reinforcement learning method, and optimization of the control scheme is realized while the tracking performance safety of the robot is ensured. According to the method, the precision and efficiency of the bolt operation robot are improved, and the operation safety is ensured.

Description

Technical Field

[0001] The invention belongs to the field of robot adaptation and relates to a fixed time safety tracking control method for a bolt operation robot under multi-learning tunnel performance. Background Art

[0002] In the process of bolt tightening, the tightening effect of the bolt will directly affect the quality of production. However, in some current manufacturing, the bolt tightening method is still the traditional manual operation. It is very easy for workers to "over-tighten" and "under-tighten" when tightening bolts based on experience, resulting in potential hidden dangers such as fastener loosening and breakage, and more serious consequences that are difficult to estimate. Applying intelligent assembly technology to production and manufacturing, improving the level of production and manufacturing, and saving human resources is of great social value. At present, the precision performance and intelligent operation of some bolt robots in operation are not high and need to be improved. Therefore, it is a general trend to study a high-precision, high-efficiency, and high-reliability automated bolt tightening control algorithm.

[0003] In order to obtain satisfactory working performance, many robot control methods have been studied in the past few decades, such as sliding mode control, model predictive control, and adaptive control. Among these methods, adaptive control has been well applied due to its online learning ability to update control parameters and handle the uncertainty of robot models.

[0004] In general, the parameter estimation methods in adaptive control include two different schemes, namely, the indirect scheme that uses the prediction error of the filtered control torque to generate parameter estimation and the direct scheme that uses the tracking error to drive parameter adaptation. However, these methods can only guarantee the asymptotic control and parameter estimation of the system under continuous excitation conditions, which is very strong and often not feasible in practice. Compound adaptive control is a comprehensive control strategy that directly combines adaptation and indirect adaptation, and its parameter adaptation law is driven by tracking error and prediction error. By using compound adaptive control, smaller tracking error and faster parameter estimation can be achieved through higher adaptive gain without exciting high-frequency unmodeled dynamics. However, the disadvantage of compound adaptive control is that continuous excitation conditions are still required to ensure accurate parameter estimation. In this regard, some scholars have proposed an emerging technology called compound learning control to achieve parameter convergence under more achievable conditions, called interval excitation conditions, rather than strict continuous excitation conditions. In compound learning, the time interval integral of the filtered regressor is used to construct the prediction error, and the prediction error and tracking error are used to update the parameter estimate, thereby obtaining accurate and smooth parameter estimates. Some researchers have proposed compound learning robot control for robot systems to ensure the convergence of parameters. Other researchers have proposed a least squares modulated composite learning control based on Moore-Penrose pseudo-inverse to achieve a balanced and easily adjustable parameter convergence rate. Composite learning control is also applied to further improve parameter convergence in hydraulic systems. Combining composite learning control with neural networks, a neural network-based robot control system is proposed, which can be used to handle unknown disturbances.

[0005] Although the above composite learning control scheme can guarantee accurate parameter estimation and system control under weak interval excitation conditions, in most cases, the convergence is asymptotic, which in principle means an infinite convergence time. Obviously, both in theory and in practical applications, finite / fixed time convergence is preferable to exponential convergence because of faster convergence performance. Terminal sliding mode control is considered to be an effective method to achieve finite time convergence. However, when terminal sliding mode control is combined with adaptive control to estimate unknown system parameters, potential singularity problems may arise. In this regard, a two-phase robust control strategy can be used to deal with this problem, further making the control gain conservative. Some scholars have proposed an adaptive optimal finite time parameter estimation and tracking control method for a benchmark servo system. However, these methods cannot guarantee finite time parameter estimation in the absence of strict continuous excitation conditions. In response to this, some scholars have proposed a finite time composite learning control method for a robot manipulator based on interval excitation, in which the parameter estimation of this method converges to a small tight set of its real value in a finite time, while the tracking error converges exponentially to a small tight set. Currently, there is no feasible composite control scheme based on interval excitation to ensure accurate parameter estimation and robot control within a finite time, and fixed-time schemes are even more difficult.

[0006] In robot tracking control with tracking error constraints, the idea of ​​preset performance function is widely used for analysis, that is, the robot's error is constrained to a preset performance function to ensure the safety of the robot. However, most preset performance controls have an implicit assumption: the performance constraint is met from the beginning of the system operation, indicating that the knowledge of the initial conditions is completely known. If there is an initial condition conflict, these methods are invalid because they are not defined outside the allowed set. From a global perspective, it produces local tracking results. However, in actual scenarios, delayed error constraint problems are often encountered. The tracking error may initially violate the performance constraint and then be adjusted to the allowed set within a reasonable user-specified time. A typical example is: the robot first passes through an open area without any obstacles. After a certain period of time, the robot enters a space surrounded by obstacles. At this time, it is necessary to make certain constraints on the robot's behavior to regulate the robot's travel path, avoid obstacles, and ensure the safety of the robot system. Most preset performance control schemes require that the initial error of the robot is constrained within the preset performance function, and many performance functions, such as the funnel-shaped performance function, are symmetrically distributed on both sides of the error. This results in the performance function being only applicable to symmetrical performance constraints and having strict restrictions on initial tracking conditions. In addition, the funnel-shaped performance leads to loose control performance. As the actual requirements of engineering projects continue to increase, more stringent requirements are often placed on its transient performance.

[0007] Ensuring safety and good performance in various situations remains a technical key and practical challenge for the widespread deployment of robotic systems. Considering both the adaptability and safety of robots has always been a research hotspot. When there is uncertainty in the system, reinforcement learning as a learning-based sequential control method has attracted strong interest and attention. However, there are various limitations in practical applications, especially for robotic systems with safety requirements. One of the main reasons for these limitations is the lack of safety assurance. When there is inevitable system uncertainty in the robotic system, it will lead to unpredictable and harmful situations. In order to optimize the control scheme, the most control problem must be considered. The traditional solution is to calculate the optimal control law by solving the Hamilton-Jacobi-Bellman equation, which adopts the Bellman optimality principle. However, the HJB equation contains inherent nonlinearity and is usually quite complex, which makes it impossible to directly calculate its unique solution in most cases.

[0008] Reinforcement learning is a widely discussed and studied method that uses neural networks to iteratively approximate the optimal solution of the HJB function, avoiding the direct solution of the HJB equation. Some scholars have proposed an optimized backstepping control for the virtual control of each subsystem of the associated nonlinear system learning for the strict feedback form of the system. This method uses the optimized backstepping technology under the Actor-Critic framework to optimize the overall control of the system, and is achieved by optimizing the virtual control of each inversion subsystem. The reinforcement learning method is used to independently approximate the critic and the actor; and a program that satisfies the Bellman optimality principle of the continuous-time HJB equation is developed through iterative learning. Although this method and structure are quite simple, they do not take into account the constraints of the system. For most reinforcement learning-based systems that are not designed with safety guarantees, even if the final control performance can be obtained after learning, the state variables may escape from the safe area during the learning process or when the system is disturbed, resulting in failure.

[0009] In this regard, some scholars have combined the obstacle Lyapunov method with reinforcement learning and called it an adaptive control method for safety reinforcement learning. Safety reinforcement learning is a promising method for safety-critical systems that optimizes control schemes while satisfying safety constraints. There are two main categories of research on safety reinforcement learning, namely model-free methods and model-based methods. For model-free safety reinforcement learning methods, safety performance can only be achieved after sufficient learning sets; this may be violated during learning interactions, especially in the early stages of learning. Therefore, this characteristic also means that this method is difficult to respond safely to changing environments in a timely manner. Model-based safety reinforcement methods directly determine actions during the learning process or supervise actions through backup controllers to ensure safety, that is, safety assurance and action exploration are determined separately in the design. Model predictive control is a typical model-based method that uses accurate models to explore safe actions in reinforcement learning, and mostly attempts to pre-calculate in advance to ensure safety performance.

[0010] The preset performance function has better constraint performance than the obstacle Lyapunov function, and the constraint performance is set in advance, which can effectively ensure the safety of the robot system during the movement. However, there is currently no robot adaptive safety tracking control method that combines the preset performance function with reinforcement learning. Summary of the invention

[0011] In view of this, the object of the present invention is to provide a fixed-time safety tracking control method for a bolt operation robot under multi-learning tunnel performance.

[0012] The intelligent bolting robot considered in the present invention is a robot that aims to achieve bolt tightening in the intelligent manufacturing process. Considering the typical scenarios that bolting robots often encounter in the process of tightening bolts, that is, due to the heavy weight and many threads of bolts in large equipment, the bolting robot is not subject to any error constraints in the initial stage of tightening bolts. However, after a certain period of time, in order to achieve the goal of tightening the bolts but not over-tightening them, it is necessary to impose certain constraints on the force and time of the bolting robot to tighten the bolts.

[0013] The present invention patent takes into account the performance and speed of the convergence of the tracking error of the bolt operation robot at the same time, and designs a series of tunnel-type preset performance functions for the error convergence performance. The function can ensure that the bolt-tightening tracking error of the bolt operation robot has a tighter convergence set and a better control effect. In the process of designing the function, the globality and delay of the error are also considered, that is, no matter how much the initial error is, no matter when the constraint condition starts, the constraint of the tracking error can be guaranteed, and the function can meet the control requirements. For the convergence speed of the error, the composite learning control is used to design a composite learning fixed-time sliding mode controller, which eliminates the continuous excitation conditions required by the traditional controller, making the feasibility of the scheme stronger. At the same time, a nonlinear fast integral terminal sliding mode control block is designed. This module not only ensures the fixed-time convergence speed, but also eliminates the singularity problem of the traditional sliding mode control block. For the safety and optimization problems of the robot, the present invention combines preset performance control with reinforcement learning, and uses the Actor-Critic framework to obtain the solution of the HJB equation, which ensures the optimality of the scheme while ensuring the safety of the bolt operation robot, improves the robust safety of the robot system, and saves computing resources.

[0014] In order to achieve the above object, the present invention provides the following technical solutions:

[0015] The fixed time safety tracking control method of the bolt operation robot under multi-learning tunnel performance includes the following steps:

[0016] Step 1: Establish a typical n-link bolt operation robot system model based on the Euler-Lagrange equation, select appropriate parameters as state variables, and obtain the state space equation of the bolt operation robot system;

[0017] Step 2: Design a series of important conversion functions and introduce the tunnel-type preset performance function into the tracking error constraint control to achieve a global control effect that is independent of the initial value of the tracking error;

[0018] Step 3: Use compound learning control to eliminate the strong continuous excitation conditions required in the controller design process and realize the weak excitation conditions of interval excitation;

[0019] Step 4: Combining composite learning and sliding mode control methods, a new type of nonlinear fast integral terminal sliding mode fixed time convergence controller is designed;

[0020] Step 5: Combine the above control scheme with the Actor-Critic architecture of reinforcement learning to form a new safe reinforcement learning method based on a preset performance function to achieve the safety of the robot's tracking performance and the optimality of the control scheme.

[0021] Furthermore, in step 2, important conversion functions include a scaling function and an error conversion function.

[0022] Furthermore, the scaling function is used to realize the unconstrainedness of the initial value of the error, and the error transfer function is used to introduce the global control effect that is independent of the initial value of the tracking error into the tracking error constraint control.

[0023] Furthermore, in step three, the composite learning control uses the time interval integral of the filtered regressor to construct a prediction error, and uses the prediction error and the tracking error to update the parameter estimation, thereby obtaining an accurate and smooth parameter estimation.

[0024] Furthermore, in the step 4, the nonlinear fast integral terminal sliding mode controller includes a sliding module and an auxiliary variable, and the design of the sliding module and the auxiliary variable can avoid the singularity problem and achieve fixed time convergence of the tracking error.

[0025] Furthermore, in step five, the safety reinforcement learning method includes the steps of defining a performance indicator function, constructing a Hamiltonian function, establishing an HJB equation, using a neural network to approximate the optimal solution of the HJB equation, and designing an update law of an Actor-Critic network.

[0026] Further, the defining of the performance indicator function includes defining a cost function and a Hamiltonian function.

[0027] Further, constructing the Hamiltonian function includes adding the cost function and the Hamiltonian function.

[0028] Furthermore, the establishment of the HJB equation includes bringing the optimal control strategy and the optimal performance index function into the Hamiltonian function, and obtaining the partial derivative of the optimal control strategy.

[0029] Furthermore, the update law of the actor-critic network is designed by constructing a positive definite function using the square of the Bellman residual, and obtaining the update law by calculating the negative gradient of the positive definite function.

[0030] The beneficial effects of the present invention are:

[0031] (1) This design considers the delayed tracking error constraint problem of neither "over-tightening" nor "under-tightening" faced by the bolt-tightening robot during the bolt-tightening process. Combined with the tunnel-type preset performance function, a tighter convergence set is provided for the error constraint. Regardless of the initial error and when it starts, parameter convergence performance can be achieved.

[0032] (2) Aiming at the continuous excitation conditions required by traditional controller design, a composite learning method is used to construct a tracking error and prediction error to jointly drive the parameter adaptive law update, thereby realizing the weak excitation conditions of interval excitation, which provides help for the implementation of the controller;

[0033] (3) Aiming at the singularity problem existing in the traditional terminal sliding mode control, the present invention designs a novel nonlinear fast integral terminal sliding mode control block. The controller designed based on the sliding module can achieve the fixed time convergence of the tracking error and eliminate the singularity problem encountered during the execution of the controller.

[0034] (4) The present invention innovatively combines the preset performance function with the optimal backstepping method to design a new type of safe reinforcement learning algorithm. While ensuring the constraint performance of the robot system tracking error, the control scheme is optimized.

[0035] (5) In order to solve the problem that the update law of the traditional reinforcement learning Actor-Critic network is relatively complicated, the present invention uses the square of the Bellman residual to construct a new positive definite function, which is equivalent to the HJB equation. The update law designed based on the positive definite function has simpler calculation, which is conducive to the real-time control of the control scheme and relaxes the continuous excitation conditions.

[0036] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 It is the technical roadmap of the present invention;

[0039] Figure 2 Preset performance fixed-time safety reinforcement learning sliding mode controller for tunnels. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0042] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0043] The present invention considers the performance and speed of the bolt operation robot tracking error convergence at the same time. Under the premise that the bolt operation robot system has the error delay control requirement, a tunnel-type preset performance fixed time composite learning sliding mode control method that is independent of the initial error condition is designed to achieve the tracking error constraint within a limited time. The introduction of the composite learning method eliminates the continuous excitation condition required by the traditional control method, and only requires interval excitation to realize the control method. The introduction of the tunnel-type preset function eliminates the requirement for the initial error and also brings more compact constraint performance.

[0044] After achieving convergence within a limited time for the bolting robot system, the reinforcement learning Actor-Critic learning network is introduced to ensure the safety of the robot system and optimize the control scheme of the bolting robot. The preset performance control method is combined with reinforcement learning optimization, and the HJB equation of the robot system is obtained using the cost function and the Hamiltonian equation. Due to the inherent strong coupling nonlinearity of the HJB equation, the optimal control scheme cannot be directly obtained. The Actor-Critic learning network is used to continuously approximate the optimal solution of the HJB equation, thereby optimizing the control strategy of the system while ensuring the tracking safety of the robot system.

[0045] See also Figure 1 and Figure 2 The present invention is designed to consist of the following steps:

[0046] (1) Based on the Euler-Lagrange equation, a typical n-link bolt operation robot system model is established, and appropriate parameters are selected as state variables to obtain the state space equation of the bolt operation robot system. From the state space equation, it can be seen that the robot system is a nonlinear strict feedback system. Therefore, in the subsequent controller design process, the backstepping method and other classical controller design methods can be used to design the control law, laying the foundation for the subsequent controller design, the research and establishment of preset performance control.

[0047] (2) Design a series of important conversion functions, introduce the tunnel-type preset performance function into the tracking error constraint control, and achieve a global control effect that is independent of the initial value of the tracking error. Use compound learning control to eliminate the strong continuous excitation conditions required in the controller design process, realize the weak excitation conditions of interval excitation, and combine compound learning and sliding mode control methods to design a new type of nonlinear fast integral terminal sliding mode fixed time convergence controller.

[0048] (3) By introducing the Actor-Critic architecture of reinforcement learning and combining it with preset performance control, a new type of safe reinforcement learning optimization control method is designed to achieve the safety of the robot system and the optimality of the control scheme.

[0049] To achieve the above object, the technical solution of the present invention is as follows:

[0050] The bolt operation robot preset safety fixed time tracking error convergence scheme designed by the present invention mainly consists of the following parts: composite learning utilizes the time interval integration of the filtered regressor to construct the prediction error, and utilizes the prediction error and the tracking error to update the parameter estimation, so as to obtain accurate and smooth parameter estimation, eliminates the influence of the continuous excitation condition on the controller implementation, and only requires the interval excitation condition to achieve the convergence of the tracking error; the tunnel type preset performance function provides a new performance function for the constraint of the tracking error, which has better convergence performance for the error; the nonlinear fast integral terminal sliding mode controller realizes the fixed time convergence of the parameters, and avoids the singularity problem at the same time; the above control scheme is combined with the Actor-Critic architecture of reinforcement learning to form a new safety reinforcement learning method based on the preset performance function, which realizes the safety of the robot tracking performance and the optimality of the control scheme.

[0051] Part I Model establishment and system description

[0052] AModel establishment

[0053] Consider a type of n-link bolt handling robot system, which can be expressed by the Euler-Lagrange equation as follows:

[0054]

[0055] Where q(t)=[q1(t),…q n (t)], is the angular position of the joint, M(q)∈R n×n , represents the inertia matrix, represents the centripetal torque and the Lilio torque, is the viscous friction torque, g(q)R n represents the gravitational torque, τ∈R n Represents the input control torque of the robot system.

[0056] Select appropriate state variables, let x1(t) = q, The state space equation of the bolt handling robot system obtained by the Euler-Lagrange equation is as follows:

[0057]

[0058] BSystem Description

[0059] Both models can be used to design the control law of the robot system, and each model has its own advantages. The above two model equations will be used flexibly in the design of the subsequent control scheme. To facilitate the subsequent design, some important properties of the dynamic equations of the robot system expressed by the Euler-Lagrange equation are listed below:

[0060] Property 1: M(q) is a symmetric positive definite matrix. There exist two constants m1 and m2 such that m1I<<M(q)< <Im2.

[0061] Nature 2 is a skew symmetric matrix. That is, the following equation holds:

[0062]

[0063] Property 2 shows that internal forces do not do work and is applicable to any type of arm robot system, that is, to bolt operation robot systems.

[0064] Property 3 The robot's dynamic system can be used with an unknown constant parameter vector θ∈R m , linearize, that is:

[0065]

[0066] where ξ is an arbitrary vector, represents the first-order derivative of ξ, Represents the dynamic regression matrix.

[0067] For the convenience of expression, the above formula can be rewritten as:

[0068]

[0069] Use a linear filter to remove The filtered expression is:

[0070]

[0071] The continuous excitation and interval excitation of the signal are defined as follows:

[0072] Definition 1 If (σ is the excitation length), in the interval t∈[T e -t d ,T e ] If it holds, then the signal Φ∈R m×n , is called an interval-excitable signal.

[0073] Definition 2 If At any t>0, we have:

[0074] If it holds, then the signal Φ∈R m×n , is called a bounded continuous excitation signal.

[0075] Lemma 1 For a general dynamical system x(0)=x0, where x∈Rn , assuming that the origin is an equilibrium point of the system, if there exists a continuously differentiable, positive definite, fundamentally unbounded function V(x) that satisfies:

[0076]

[0077] Among them, α, β>0, 0<γ1<1, γ2>1. Then the stability of the system is global fixed-time stability, and the stable time T can be estimated by equation (8):

[0078]

[0079] Lemma 2 For any x i ≥0, and there exists a real number 0<γ1<1,γ2>1, the following inequality holds:

[0080]

[0081] Assumption 1: The expected tracking trajectory q d (t) and its first-order derivative are bounded.

[0082] The tracking error of the robot system is defined as follows:

[0083] e(t)=x1(t)-q d (t) (10)

[0085] Among them, q d (t) represents the expected target signal.

[0086] The parameter update error of the robot system is defined as follows:

[0087]

[0088] The control objective of the present invention is to develop a safe preset performance finite time sliding mode control algorithm to achieve the following control objectives:

[0089] All error signals e(t) and Under the premise of satisfying interval excitation and non-singularity problems, it converges to 0 in a fixed time;

[0090] P2 For any initial tracking condition (including initial constraint violation), the tracking error e(t) can be adjusted to a pre-defined set of tunnel performance within the user-specified stabilization time, with globally safe control performance.

[0091] Part II Description of Important Technical Means

[0092] ACompound Learning Control

[0093] In adaptive control, parameter convergence is ideal because it improves the overall stability and robustness of the closed-loop system. However, in traditional adaptive control, in order to ensure parameter convergence, a strict condition must be met, namely continuous excitation. However, the strict continuous excitation condition is very difficult to achieve in real control, making it difficult to truly implement the control scheme in actual control. Composite learning control uses the time interval integral of the filter regressor to construct the prediction error, thereby eliminating the need for time derivation of the state, and uses the prediction error and filter tracking error to update the parameter estimate. The global stability of the closed-loop system can be achieved under the condition of interval excitation.

[0094] In order to relax the continuous incentive conditions, the compound learning control constructs a variable as follows:

[0095]

[0096] where τ d is a continuous integration time, multiply both sides of the above formula by get:

[0097]

[0098] Where Θ(t) is the excitation matrix. If the interval excitation condition is satisfied, according to Definition 1, we have the existence of a positive constant So that Θ(T e )≥σI, then, the prediction error ε is defined as:

[0099]

[0100] The modified incentive matrix is ​​further defined as:

[0101]

[0102] The following relationship can be easily found:

[0103]

[0104] Define a projection operator as follows:

[0105]

[0106] Based on the above theorems and contents, we can design the parameter update law.

[0107] B tunnel preset performance

[0108] Compared with other performance functions such as funnel type, tunnel preset performance provides a tighter allowed set, so that better transient performance can be obtained. The constraint problem considered by the present invention is that the system tracking error is not subject to any constraints in the initial stage, and after a certain period of time, it is constrained in the prescribed tunnel preset performance:

[0109]

[0110] e l (t), e u (t) gives the upper and lower limits of the tunnel's preset performance, and τ is the given time. The upper and lower bounds are defined as follows:

[0111]

[0112] Where 0<δ<1, e0=e(0), represents the initial value of the tracking error, ρ(t)=(ρ0-ρ ∞ ) -lt +ρ ∞ , and 0<ρ ∞ <ρ0,l>0.

[0113] The inherent mechanism of tunnel preset performance is to unify different initial conditions into a concise performance description by using the characteristics of symbolic functions. Most previous preset performance controls are symmetrically distributed on both sides, and cannot impose tighter constraints due to their loose performance boundaries. Tunnel preset performance has better finite time convergence and flexible performance due to its asymmetric performance boundaries and concise expression.

[0114] C Nonlinear Fast Integral Terminal Sliding Mode Control

[0115] Terminal sliding mode control is regarded as an important technology to achieve fixed-time convergence of tracking error in nonlinear strict feedback systems. In traditional terminal sliding mode control, the sliding module is designed as follows:

[0116] s=ζ+λsig γ (ζ),0<γ<1 (20)

[0118] When using the traditional terminal sliding module for regression matrix adaptive robot control, the corresponding auxiliary variables should be designed as:

[0119]

[0120] However, the design of the auxiliary variables will inevitably bring about singularity problems. To solve this problem, the present invention proposes a nonlinear fast integral terminal sliding mode control block. The sliding module and the auxiliary variables are designed as follows:

[0121]

[0122] Among them, λ1, λ2, λ3, λ4 are positive constants, 0<α<1, -1<β<0,

[0123]

[0124] The above sliding mode control block can not only ensure fixed-time tracking control, but also avoid the singularity problem. The above characteristics make the proposed control strategy suitable for fixed-time parameter estimation and tracking control tasks.

[0125] Part III Design of a fixed-time preset performance sliding mode controller

[0126] AGlobal tunnel type preset performance function

[0127] In order to achieve the globality of tracking error constraints and good tracking performance, several important conversion functions are firstly introduced so as to better introduce the tunnel-type preset performance function into the controller design of error constraints.

[0128] ① Scaling function

[0129] In order to achieve no constraints on the initial value of the error, a scaling function is defined.

[0130] Definition 1: Scaling function b f (t) has the following unique properties:

[0131] 1) When t ≥ 0, b f (t) Any C n+1 The order is bounded;

[0132] 2) In the interval t∈[0,τ), b f (t) Strictly monotonically increasing;

[0133] 3) b f (0) = 0, and when t ≥ τ, b f (t) = 1, for any time t, b f (t)∈[0,1], when t→τ, b f The first-order derivative of (t) approaches 0.

[0134] Selected b f (t) The function is as follows:

[0135]

[0136] Where l is a defined positive definite constant.

[0137] In practical applications, the robot system is subject to a wide range of initial condition constraints. If the system violates the performance constraints, it may cause singularity problems, which may lead to control design failure. In order to deal with this limitation, the present invention proposes a scaling function and its properties. From Definition 1, we can see that b f (t) is multiplied by e(t), due to b f (0) = 0, so that the initial value can always be guaranteed to be 0, so there is a unified solution for any initial condition. Since the tracking error must impose performance constraints immediately after a finite time τ, for t ≥ τ, b f (t)e(t)=e(t). For the time 0≤t<τ, the scaling function is required to increase smoothly from 0 to 1 and remain unchanged thereafter. Such a design mechanism can lead to extensive research, combining various scaling functions with alternative control methods.

[0138] ② Error transfer function

[0139] For any initial tracking condition, the tracking error e(t) is scaled using the scaling function b f (t), construct the following error transfer function:

[0140]

[0141] Among them, e u (t), e l (t) are the upper and lower bounds of the tunnel preset performance.

[0142] According to the definitions of the above functions, we can see

[0143]

[0144] Therefore, no matter how much the given initial error is (including initial constraint violations), the error constraints can be guaranteed.

[0145] In order to ensure that the tracking error is always constrained within the preset tunnel performance boundary, an auxiliary variable is introduced based on the previous functions:

[0146]

[0147] The tunnel-type preset performance function that is independent of the initial value is established as:

[0148]

[0149] The zeta function has an important property, that is, to ensure that the zeta function is bounded, it is possible to ensure that the tracking error is always within the preset performance of the tunnel. And this tracking error function provides a unified solution for handling symmetric constraints and asymmetric constraints, and there is no need to redevelop the conversion function.

[0150] Another outstanding feature of the zeta function is that no matter what the initial value and initial direction of the robot system are, the initial value of the zeta function is 0, which makes the initial control signal applicable to any initial conditions.

[0151] The proof is as follows:

[0152] For any initial error value e(0), according to the scaling function b f (t) and the definition of the auxiliary variable χ, we can get χ(0) = 0, which leads to b f (0) = 0; According to the definition of the tunnel error transfer function, the following inequality always holds true:

[0153]

[0154] That is, no matter what the initial tracking error of the robot system is and when the error constraint is started, it can be ensured that the initial error is always limited within the preset tunnel-type performance function.

[0155] Next, as long as the zeta function is bounded, it can be achieved:

[0156]

[0157] Right now:

[0158] -e l <e s (t) <e u (32)

[0160] The proof is complete.

[0161] The initial value-independent tunnel-type preset performance function describes an explicit relationship between the ζ function and the auxiliary variable χ, which plays a vital role in the control design process for solving delay-dependent error constraint problems. However, it is difficult to obtain such an explicit relationship due to the inherent complexity and coupled expressions of the commonly used logarithmic and hyperbolic tangent transformation functions.

[0162] Taking the first-order derivative of the ζ function and sorting out the related equations, we have:

[0163]

[0164] in,

[0165] Substituting the above formula into the dynamic expression of the robot system is:

[0166]

[0167] Multiply both sides by get:

[0168]

[0169] in:

[0170]

[0171] Through the above transformation, the robot system (1) subject to tracking error constraints is transformed into an equivalent "unconstrained" robot system (35). In the next section, a composite learning fixed-time controller will be designed. One of the tasks of the designed controller is to ensure the boundedness of ζ.

[0172] B. Design of compound learning fixed-time controller

[0173] Using the composite learning method, we construct an incentive matrix for interval incentives:

[0174]

[0175] According to the sliding module auxiliary variables and nonlinear fast integral terminal sliding module designed previously, the update law and control law of the designed system are as follows:

[0176]

[0177] Substituting the above control law and update law into the dynamic equation, the dynamic closed-loop system of the bolt operation robot is obtained as follows:

[0178]

[0179] Part IV System Stability and Convergence Performance Analysis

[0180] Consider the following Lyapunov-like positive definite function:

[0181]

[0182] The function V is derived with respect to time, and the closed-loop expression of the dynamic system of the bolt handling robot is brought in. According to the properties of the robot's Euler-Lagrange equation, we can obtain:

[0183]

[0184] According to the inequality:

[0185]

[0186] Substituting the result of the inequality into:

[0187]

[0188] According to the modified activation matrix and combined with Lemma 2, we have:

[0189]

[0190] in,

[0191]

[0192] Secondly, considering the stability on t∈[0,∞], ignoring the last two terms of inequality (45), we obtain:

[0193]

[0194] According to the above results, we can get s and are bounded, so we can deduce that ζ and are also bounded. Therefore, we conclude that the sliding module converges asymptotically to zero under interval excitation.

[0195] According to the above analysis, we can say that the system always converges before the interval excitation condition is met. When the interval excitation condition is met, we have:

[0196]

[0197] Combining the above results and using Lemma 2, we have:

[0198]

[0199] According to Lemma 1, we can conclude that s and It converges to 0 in a fixed time. It can be concluded that the tracking error converges to 0 in a fixed time. According to the boundedness of ζ, it can be inferred that the tracking error is always constrained within the preset tunnel-type preset performance function, that is, the tracking error constraint is guaranteed.

[0200] Part V: Optimizing Secure Reinforcement Learning Controllers

[0201] In the above controller design process, we obtained a sliding mode controller with global fixed-time convergence preset performance, but the optimization problem of the controller was not considered in the controller design process. The traditional safety reinforcement learning controller design is to introduce the obstacle Lyapunov function into the controller design of the optimal backstepping method, which not only ensures the constraint of the tracking error but also achieves the optimality of the control scheme.

[0202] However, this error constraint method using the barrier Lyapunov function has a hidden prerequisite, which is that the initial tracking error must be constrained by the barrier Lyapunov function from the beginning, and the controller design is real-time. This design method will fail when there is no tracking error constraint at the beginning, and the tracking error constraint is only applied after a certain period of time. This is because the definition of the barrier Lyapunov function has been violated at this time.

[0203] We have proposed a tunnel-type preset performance control method that is independent of the initial error value and when the error constraint begins. If the preset performance control is combined with the optimal backstepping method, a new error-constrained optimization method will be created.

[0204] In the implementation of the optimal backstepping method, it is necessary to use the Actor-Critic network of reinforcement learning to approximate the HJB equation to obtain the optimal controller. The traditional update law solution is obtained by descenting the square of the Bellman residual equation, which is equivalent to estimating the HJB equation and inevitably brings many nonlinear terms. In order to simplify the solution of the update law, the present invention designs a simple positive definite function, and the update law obtained by calculating the negative gradient is simple in form. At the same time, the continuous excitation conditions can also be relaxed, making it easier to use in real-time controllers.

[0205] Define the performance indicator function:

[0206]

[0207] Among them, r(ζ(τ),u(ζ))=ζ(t)Q(x)ζ(t)+u T u is the cost function, Q(x) is a positive definite matrix, and u is the system input. The Hamiltonian function is as follows:

[0208]

[0209] Let u * is the optimal control strategy, and the optimal performance index function is:

[0210]

[0211] Where Ψ(Ω) is a compact set of feasible control strategies. * Substituting (51) into the HJB equation, we obtain:

[0212]

[0213] Find u for (52) * The partial derivative of can get the optimal control strategy:

[0214]

[0215] Substituting (53) back into (52), we have:

[0216]

[0217] By solving equation (54), we can obtain Then we can obtain u * .

[0218] However, this equation has strong nonlinearity and is difficult to solve through numerical calculations. Therefore, it is necessary to introduce the reinforcement learning Actor-critic architecture approximated by neural networks to solve this problem.

[0219] V * (z) is divided into two parts:

[0220] V * (ζ)=β||ζ(t)|| 2 -β||ζ(t)|| 2 +V * (ζ)

[0221] =β||ζ(t)|| 2 +V o (ζ) (55)

[0222] Where β is a design constant, V o (ζ)=V * (ζ)-β||ζ(t)|| 2

[0223] According to the universal approximation theorem of neural networks, the continuous function V o (ζ) can be approximated by a neural network as:

[0224] V o (ζ)=W *T S(ζ)+ε(ζ) (56)

[0225] W * is the ideal m-dimensional weight vector, S(ζ) is the basis function vector, and m is the number of neurons.

[0226] From (55), we can get u approximated by the neural network: * ,

[0227]

[0228] However, the ideal m-dimensional weight vector W * is an unknown value, and we need to learn its estimated value through the Actor-Critic architecture of reinforcement learning:

[0229]

[0230] in, and is the estimated value obtained by reinforcement learning. Substituting (58) into the HJB equation estimated by reinforcement learning

[0231]

[0232] The Bellman residual φ(t) is defined as the difference between equation (52) and equation (59):

[0233]

[0234] Definition: Φ(t)=φ(t) 2 , is a positive definite function.

[0235] To simplify the design of the update law, a new positive definite function is considered. beg The partial derivative of can be obtained:

[0236]

[0237] The above equation is equivalent to the HJB equation, that is, Equivalent to Therefore, the function after partial derivative can be used to design the update law, which will greatly simplify the calculation amount of the update law and improve the real-time performance of the control scheme.

[0238] Define a new positive definite function:

[0239]

[0240] It can be clearly seen that P(t) = 0, and it can be inferred that Thus, it is concluded Therefore, the excellent properties of the P(t) function provide great convenience for the design of the update law.

[0241] The update law of the Actor-Critic network is designed as follows:

[0242]

[0243]

[0244] Among them, γ c and γ a are the learning rates of the Critic network and the Actor network respectively.

[0245] So far obtained and According to the definition, Φ(t)>0, Therefore, it is asymptotically stable, that is, and will approach the final true value. In this way, the reinforcement learning Actor-Critic architecture solves the problem of W * Unknown problems, so as to solve the optimal control strategy u * , to achieve the purpose of safe tracking and control.

[0246] Compared with the traditional update law, the update law designed in this way makes the optimized control algorithm significantly simpler, and at the same time, it can also release the strict conditions of persistent incentives. Usually, the reinforcement learning of optimal control is an iterative process, and the critics and actors are trained at the same time. Therefore, the high complexity of the control design mainly comes from the construction of the critic update method and the actor update method. In the published optimization methods, both update laws are designed based on the square of the Bellman residual. Since this equation is a complex nonlinear equation, it inevitably increases the complexity of the control design. In the present invention, the reinforcement learning algorithm is designed based on a simple positive function, which is equivalent to the HJB equation, which is very meaningful for reducing the complexity of the control design.

[0247] The safety reinforcement learning control method proposed in this invention appropriately stitches the tunnel-type preset performance function into the optimization backstepping method to keep the state variables within the designed safety range, and greatly reduces the variance of the control performance under system uncertainty during the learning process. This technology strictly considers the important factors of safety assurance and exploration, without the need for additional safety assurance, such as uncertainty estimation, auxiliary safety controller or a priori safety set.

[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A fixed time safety tracking control method for a bolt operation robot under multi-learning tunnel performance, characterized by: The following steps are involved: Step 1: Establish a typical n-link bolt operation robot system model based on the Euler-Lagrange equation, select appropriate parameters as state variables, and obtain the state space equation of the bolt operation robot system; Step 2: Design a series of important conversion functions and introduce the tunnel-type preset performance function into the tracking error constraint control to achieve a global control effect that is independent of the initial value of the tracking error; Step 3: Use compound learning control to eliminate the strong continuous excitation conditions required in the controller design process and realize the weak excitation conditions of interval excitation; Step 4: Combining composite learning and sliding mode control methods, a new type of nonlinear fast integral terminal sliding mode fixed time convergence controller is designed; Step 5: Combine the above control scheme with the Actor-Critic architecture of reinforcement learning to form a new safe reinforcement learning method based on a preset performance function to achieve the safety of the robot's tracking performance and the optimality of the control scheme.

2. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 1 is characterized in that: In the step 2, important conversion functions include a scale function and an error conversion function.

3. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 2 is characterized in that: The scaling function is used to realize the unconstrainedness of the initial value of the error, and the error transfer function is used to introduce the global control effect that is independent of the initial value of the tracking error into the tracking error constraint control.

4. The method for controlling the fixed time safety tracking of a bolt operation robot under multiple learning tunnel performances according to claim 1 is characterized in that: In the step three, the composite learning control constructs a prediction error by using the time interval integral of the filtered regressor, and updates the parameter estimation by using the prediction error and the tracking error, thereby obtaining an accurate and smooth parameter estimation.

5. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 1 is characterized in that: In the step 4, the nonlinear fast integral terminal sliding mode controller includes a sliding module and an auxiliary variable. The design of the sliding module and the auxiliary variable can avoid the singularity problem and achieve fixed time convergence of the tracking error.

6. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 1 is characterized in that: In step five, the safety reinforcement learning method includes defining a performance indicator function, constructing a Hamiltonian function, establishing an HJB equation, using a neural network to approximate the optimal solution of the HJB equation, and designing an update law of an Actor-Critic network.

7. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 6 is characterized in that: Defining the performance indicator function includes defining a cost function and a Hamiltonian function.

8. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 6 is characterized in that: The constructing of the Hamiltonian function includes adding the cost function and the Hamiltonian function.

9. The method for controlling the fixed time safety tracking of a bolt operation robot under multiple learning tunnel performances according to claim 6 is characterized in that: The establishment of the HJB equation includes bringing the optimal control strategy and the optimal performance index function into the Hamiltonian function, and obtaining the partial derivative of the optimal control strategy.

10. The method for secure tracking control of a bolt operation robot during fixed time under multiple learning tunnel performances according to claim 6 is characterized in that: The update law of the actor-critic network is designed by using the square of the Bellman residual to construct a positive definite function, and the negative gradient of the positive definite function is calculated to obtain the update law.

Images