Data-driven reinforcement learning robust control method for nonlinear industrial systems
By designing robust term and data sample processing methods in nonlinear industrial systems, combined with experience playback technology, adaptive robust control under uncertain data is achieved, and the problem of the impact of traditional methods' dependence models and data correlation is solved. Improves control accuracy and stability.
Patent Information
- Application Number
- CN202210765784.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-01
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-07-01
AI Technical Summary
The existing reinforcement learning robust control technology is difficult to achieve adaptive adjustment and effective convergence when facing uncertain data and continuous time nonlinear industrial systems. Especially when the model is unknown and perturbed, traditional methods rely on model information. And the data sample correlation affects the stability of the algorithm.
A nonlinear industrial system data-driven reinforcement learning robust control method is designed. By designing the cost function and data sample calculation model of robust terms, combined with experience playback technology, incomplete data sets are processed, complete data sets are formed, and adaptive Iteration and stability improvement of control strategies.
Adaptive and robust control of nonlinear industrial systems is realized under uncertain data, which alleviates the impact of data sample correlation on algorithm convergence, improves control accuracy and stability, and is suitable for robotic arms, servo motors, inverted pendulums, etc. system.
Smart Images

Figure CN115016286B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data-driven control of nonlinear industrial systems, and in particular to a data-driven reinforcement learning robust control method for nonlinear industrial systems. Specific nonlinear industrial systems include but are not limited to robotic arms, servo motors, inverted pendulum systems, etc. Background Art
[0002] With the advancement of modern industry, nonlinear industrial systems are increasingly demanding production precision. Consequently, the control of nonlinear industrial systems has attracted considerable attention. In practical applications, common nonlinear industrial systems, such as industrial robotic systems, servo motor systems, and inverted pendulum systems, can introduce various disturbance signals due to factors such as assembly errors, calculation errors, friction coefficient fluctuations, electromagnetic fluctuations, and load fluctuations. These disturbances can affect the actual control output and, in turn, the quality of the final industrial product. For example, load fluctuations in robotic systems can cause changes in the overall moment of inertia, affecting the output speed of the drive system and reducing control accuracy. Calculation errors and electromagnetic fluctuations in servo motor systems can directly affect the control voltage, which in turn affects the motor's output torque, leading to unstable output speed. Assembly errors and friction coefficient fluctuations in inverted pendulum systems can also affect the output torque of the drive motor, thereby affecting the stability of the swing arm at its equilibrium position. These are unavoidable problems in modern industrial production and are typical issues that need to be addressed in robust control. Traditional robust control methods for nonlinear industrial systems, such as variable structure control, backstepping control, and model predictive control, rely on precise mathematical models for control strategy design. However, as modern industrial systems grow in scale and complexity, accurate mathematical models of these systems are often difficult to obtain. If traditional model-dependent control methods are used when disturbances occur, system control accuracy will deteriorate. Therefore, the development of data-driven, model-free control methods is crucial.
[0003] In recent years, reinforcement learning (RL) has been used as a data-driven control method in the field of nonlinear industrial system control. RL uses system operating data to find the optimal control policy, enabling model-free control. However, existing RL robust control techniques still face two unresolved issues: First, when the collected industrial system data is uncertain, the algorithm's stability and convergence are affected. Different-policy algorithms are a typical data-driven RL algorithm. In the presence of interference, different-policy algorithms derive their control policies from limited historical industrial data and are therefore unable to adaptively adjust their outputs. To mitigate the impact of uncertain data on algorithm performance, online iterative same-policy algorithms are often designed to improve their adaptability to industrial environments. However, these methods often rely on model information about the industrial system. Second, when faced with uncertain industrial system scenarios, control policies must be derived quickly using a small amount of system data. In data-driven RL algorithms, linear correlation between data samples can affect algorithm convergence. With limited industrial system data, especially for continuous-time control problems, correlation between adjacent data samples is even more severe, significantly impacting algorithm stability. In recent years, some RL solutions for natural language processing have utilized experience replay techniques to mitigate this data correlation issue. However, in the field of reinforcement learning-based control, relevant technologies are still underdeveloped.
[0004] Therefore, there is an urgent need to develop a data-driven, highly adaptive robust control strategy for nonlinear industrial systems. This strategy can achieve adaptive improvement of robust control strategies for nonlinear industrial systems such as robotic arms, servo motors, and inverted pendulums under the influence of uncertain data. There is also a pressing need to design a data sample processing method to mitigate the impact of correlation between adjacent data samples on the convergence of data-driven algorithms in continuous-time nonlinear industrial system control problems. Summary of the Invention
[0005] The present invention provides a data-driven reinforcement learning robust control method for nonlinear industrial systems. This method solves the robust control problem of nonlinear industrial systems with unknown models under uncertain data, implements adaptive improvements to the data-driven robust control strategy for nonlinear industrial systems, and alleviates the impact of the correlation between adjacent data samples on the convergence of the data-driven algorithm. Details are described below:
[0006] A reinforcement learning data-driven robust control method for a nonlinear industrial system, the method comprising:
[0007] For nonlinear industrial systems with control channel disturbances, a cost function with robust terms is designed and the optimal control strategy is given.
[0008] Through mathematical equivalent transformation, the optimal control strategy is converted into a data-driven control strategy. The original data of the system operation is collected, and a data sample calculation model is designed. The data samples are collected into the experience pool until a complete data set is formed, and a data-driven control strategy iteration model is constructed.
[0009] The data-driven control strategy is calculated in the inner loop iteration, and the obtained data-driven control strategy is applied to the controlled system in the outer loop iteration;
[0010] A new data sample storage experience pool is designed. Before the outer-loop iterative control strategy is used for the controlled system, data samples are extracted and processed to form an incomplete data set, which is stored in the experience pool. The outer-loop iterative control strategy is applied to the controlled system, and new data samples are collected to supplement the experience pool until a complete data set is formed, thereby re-entering the inner-loop iteration in the double closed-loop framework.
[0011] Among them, the cost function containing the robust term is:
[0012]
[0013]
[0014] Among them, J(x(t)) is the cost function, is the integrated utility function, is the integrand time variable, U(x(t),u(t)) is the utility function; Q and M are positive definite matrices of corresponding dimensions, γ is a positive real number, is the robust term of the cost function, x(t) is the system state, u(t) is the control signal, is a function related to the disturbance signal.
[0015] Furthermore, the data sample calculation model is:
[0016]
[0017]
[0018] in, and is the data sample. c (·) is the evaluation network activation function, φ a (·) is the behavioral network activation function, is the behavior network weight parameter matrix; is the identity matrix. d0 (x)=u0(x)+d u (x) is the initial stable control strategy for disturbance, u0(x) is the initial stable control strategy, d u (x) is the system disturbance signal; is the matrix Kronecker product operation;
[0019] The data-driven control strategy iteration model is:
[0020] W (i,j+1) =[(Π (i,j) ) T Π (i,j) ] -1 (Π (i,j) ) T Λ (i,j)
[0021] in, is the neural network weight matrix, and vec(·) represents the column vector transformation of the matrix. is the evaluation network weight parameter matrix. (i,j) and Λ (i,j) is a set of data samples.
[0022] The data-driven control strategy is calculated in the inner loop iteration, and the data-driven control strategy obtained is applied to the controlled system in the outer loop iteration as follows:
[0023] a: Initialization: Set the initial inner and outer loop iteration steps i=j=0, the initial stable control u (0) =u d0 , the dimension of the collected samples W dim , the maximum number of iterations of the inner loop is M m , sampling time T, cumulative maximum sample dimension N m , inner loop iteration convergence termination condition β;
[0024] b: Data Collection: Application Control Strategy The first time you loop through the frames, use u (0) ) for the controlled system, collecting original system data including: system status and disturbance control signals;
[0025] c: Strategy evaluation: calculate and store data samples and In the experience pool, until a complete data set π is formed (i,j) and Λ (i,j) ;
[0026] Strategy improvement: Based on the data-driven control strategy iteration model, calculate the neural network weight parameter W (i,j+1) , update the weight parameters of the evaluation network and behavior network and
[0027] d: Judgment condition: When ‖W (i,j+1) -W (i,j) ‖ 2 >=β, if the maximum number of iterations Mm If not satisfied, enter process c, and j←j+1; if ‖W (i,j+1) -W (i,j) ‖ 2 <β and the maximum training time is reached, stop the algorithm iteration and obtain the control strategy; otherwise, process the data samples based on the designed data processing method and enter process b, with i←i+1.
[0028] The method further includes: extracting all sample data in the experience pool "set 1", dim The order of dimensional data samples is randomly disrupted, each data sample is numbered, and q is generated by a random number algorithm. m1 A random number, q m1 And each random number is less than W dim A positive number; q in the experience pool m The random numbers are matched with the data samples in the complete data set and marked, and the remaining unlabeled data samples are eliminated. The labeled data samples are arranged in a shuffled order to form an incomplete data set and stored in the experience pool.
[0029] Furthermore, the method is applied to a robotic arm, a servo motor, and an inverted pendulum system.
[0030] The beneficial effects of the technical solution provided by the present invention are:
[0031] 1) This paper designs a data-driven robust control method based on reinforcement learning for nonlinear industrial systems with unknown models, taking into account the uncertain data caused by control channel disturbances. This method can achieve adaptive robust control of nonlinear industrial systems with unknown models under uncertain data, and has stronger adaptability than classic reinforcement learning control methods.
[0032] 2) In the problem of robust control of continuous-time nonlinear industrial systems, a new data sample processing method based on experience replay technology is designed, which can alleviate the problem of data correlation between adjacent data samples and improve the convergence of the data-driven reinforcement learning algorithm; in the outer loop iteration of data-driven robust control, historical data samples are randomly retained and used to construct a new complete data set. While introducing new data samples, the system's historical data information is retained, thereby improving the stability of the control method and data utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is the structural diagram of the data-driven robust control framework;
[0034] Figure 2 is a flow chart of the data sample processing method;
[0035] Figure 3A schematic diagram for evaluating the convergence process of network weight parameters;
[0036] Figure 4 Schematic diagram of the convergence process of behavior network weight parameters;
[0037] Figure 5 It is the curve diagram of the angular position trajectory of the inverted pendulum system. DETAILED DESCRIPTION
[0038] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.
[0039] Example 1
[0040] A reinforcement learning data-driven robust control method for a nonlinear industrial system, the method comprising:
[0041] Step 101: For a nonlinear industrial system with control channel disturbances, a special cost function with a robust term is designed to provide an optimal control strategy.
[0042] Step 102: Convert the optimal control strategy into a data-driven control strategy through mathematical equivalent transformation, collect raw data of system operation, design a data sample calculation model, collect data samples into an experience pool, until a complete data set is formed, and construct a data-driven control strategy iteration model;
[0043] Step 103: Design a data-driven robust control dual closed-loop framework based on reinforcement learning. In the inner loop iteration, the data-driven control strategy is calculated, and in the outer loop iteration, the obtained data-driven control strategy is applied to the controlled system.
[0044] Step 104: Design a new data sample storage experience pool. Before the outer-loop iterative control strategy is used for the controlled system, extract and process the data samples to form an incomplete data set, store the incomplete data set in the experience pool, apply the outer-loop iterative control strategy to the controlled system, collect new data samples to supplement the experience pool until a complete data set is formed, and then enter the inner-loop iteration in the double closed-loop framework again.
[0045] For ease of reading, here are the following formula variable declarations:
[0046] The form is variable a(t), represents the first-order derivative of the variable a(t), represents the second-order derivative of the variable a(t); Representation function Partial derivative with respect to variable a; a(t)∈R means a(t) belongs to the set of real numbers R;
[0047] The shape is like matrix A, A TRepresents the transpose of matrix A, A∈R n Indicates that A is an n-dimensional column vector, A∈R m×n Indicates that A is an m×n dimensional matrix; Represents the integral of the function F(t) from t to ∞.
[0048] In the above technical solution, step 101 further includes the following steps:
[0049] The object of (1-1) is an uncertain nonlinear industrial system with control channel disturbances. This system can be a common industrial system such as a servo system or a robotic arm system. Without loss of generality, the original mathematical model of the system can be described as:
[0050]
[0051] Where x(t)∈R n is the system operating state. For example, in a robotic arm system, it can be the robotic arm angle, robotic arm speed, etc.; in a servo system, it can be the servo motor speed, motor current, etc.; in an inverted pendulum system, it can be the swing arm angle, swing arm angular velocity, etc. f(x(t)) represents the unknown nonlinear industrial system dynamic model. g(x(t)) is the nonlinear industrial system control gain matrix. u(t)∈R m It is a control signal. In specific industrial systems, such as robotic arm systems and inverted pendulum systems, it can be output torque, etc., and in servo systems, it can be control voltage. u (x) is the bounded control channel disturbance related to the system state, satisfying in is a known bounded function related to the disturbance signal, which is a common definition in the art.
[0052] For the same industrial system mathematical model, the descriptions of system states and control signals described above may differ. The system state for a robotic arm system can also include the arm's angular position, the speed error for a servo system, and the arm angle error for an inverted pendulum system. The control signal for a robotic arm system and an inverted pendulum system can also include the drive circuit voltage signal, while for a servo system it can include current signals or excitation signals. These differences arise from different approaches to building nonlinear industrial system models.
[0053] (1-2) The nominal mathematical model of the controlled system can be expressed as:
[0054]
[0055] For the nominal mathematical model, a special cost function with robust terms is designed:
[0056]
[0057]
[0058] Among them, J(x(t)) is the cost function, is the integrated utility function, is the integrand time variable, U(x(t),u(t)) is the utility function, and is equal to zero at the system equilibrium point. Q and M are positive definite matrices of corresponding dimensions, γ is a positive real number, is the robust term of the cost function.
[0059] (1-3) Based on the cost function, the Lyapunov equation of the nonlinear industrial system is constructed:
[0060]
[0061] Then, the optimal cost function can be expressed as:
[0062]
[0063] Among them, “*” represents the optimal value of the function, such as J * (x(t)) represents the optimal cost function, Ω u is the set of admission control strategies. Indicates that under the condition u(t)∈Ω u Down, The minimum value can be obtained. For a continuously differentiable cost function, the system Hamiltonian equation can be obtained:
[0064]
[0065] in, represents the Hamiltonian function; represents the partial derivative of J(x(t)) with respect to x(t).
[0066] Furthermore, based on the Bellman optimality principle, the optimal solution to the control problem can be obtained by solving the HJB (Hamilton-Jacobi-Bellman) equation, where the HJB equation is as follows:
[0067]
[0068] The optimal control strategy is derived:
[0069]
[0070] Among them, u * (t)∈Ω u To satisfy the optimal solution of the HJB equation, J * (x(t)) is the optimal cost function.
[0071] We can get a policy iteration algorithm based on reinforcement learning, including policy evaluation:
[0072]
[0073] Among them, u (i) (x(t)) is the control strategy of the i-th iteration, U(x(t),u (i) (x(t))) is the utility function of the i-th iteration.
[0074] Improve with strategies:
[0075]
[0076] Here, i refers to the number of iteration steps.
[0077] By repeating the cycle of strategy evaluation and strategy improvement, until the deviation of the cost function before and after meets the threshold condition ‖J (i+1) (x(t))-J (i) (x(t))‖<ξ1, ξ1 is a positive number, stop the algorithm iteration and output the control strategy.
[0078] In the above technical solution, step 102 further includes the following steps:
[0079] (2-1) For nonlinear industrial systems with unknown models, the control gain matrix g(x(t)) is unknown. However, the optimal control strategy (Equation (8)) still requires this system model information. Therefore, it is necessary to convert the above model-based control strategy into a data-driven control strategy.
[0080] Combining the Lyapunov equation constructed by the nominal model and the optimal control strategy, specifically, the corresponding strategy evaluation formula (9) and strategy improvement formula (10) in the algorithm, based on the integral reinforcement learning method, can be obtained:
[0081]
[0082] Among them, u0(x) is the initial stable control strategy of the specific industrial system. Under the initial stable control strategy, the state of the industrial system only needs to remain non-divergent. d0 (x)=u0(x)+d u (x) represents the initial stable control strategy u0(x) and the disturbance signal d u (x) Disturbance initial stability control strategy under joint action.
[0083] Furthermore, by integrating the above formula over the time interval [t, t+δt], we can obtain:
[0084]
[0085] By collecting system operation data in the time interval [t, t+δt], we can solve Equation (12) and obtain a data-driven robust control strategy. Where t is the integrated time variable.
[0086] Among them, the model information of the nonlinear industrial system is no longer needed, and the solution of the above formula is mathematically consistent with the optimal control strategy, which ensures the optimality of the method.
[0087] (2-2) Based on the neural network, the evaluation network and behavior network are constructed to approximate the cost function and control strategy:
[0088]
[0089] in, and are the weight matrices from the hidden layer to the output layer of the evaluation network and the behavior network, k c With k a are the number of neurons in the hidden layer of the evaluation network and the behavior network, respectively, c (·) and φ a (·) are the activation functions of the evaluation network and the behavior network, respectively. Their specific forms include but are not limited to polynomial activation function, hyperbolic tangent activation function, etc.
[0090] In formula (13), the subscript i in the upper right corner of the symbol is the number of cycles of the outer loop iteration of the data-driven robust control double closed-loop framework, and j is the number of cycles of the inner loop iteration, as shown in represents the cost function of the j+1th inner loop iteration in the i-th outer loop iteration, and similarly and They correspond to the control strategy, evaluation network weight matrix and behavior network weight matrix of the j+1th inner loop iteration in the i-th outer loop iteration respectively.
[0091] Then, define a time series {t k},k∈{0,...,W dim}, W dim is the dimension of the data samples that need to be collected. At each moment t k Collect system status x k and the disturbance control signal u d0 , the disturbance control signal is the sum of the current control strategy and the control channel disturbance. To simplify the description, we use replace Combined with formula (12), we get the neural network residual term ε including the control channel perturbation k (t):
[0092]
[0093] (2-3) Collect the original data of system operation including system status and disturbance control signals, and derive the data sample calculation model as follows:
[0094]
[0095]
[0096] in, is the identity matrix.
[0097] Data Sample and The original data of system operation including system status and disturbance control signals can be calculated and stored.
[0098] Furthermore, the residual term of the neural network can be expressed as:
[0099]
[0100] in, It is composed of the weight parameter matrices of the evaluation network and the behavior network, and vec(X) represents the column vector transformation of the matrix X.
[0101] In the time series {t k},k∈{0,...,W dim}Calculate and store data samples and In the experience pool, a complete data sample set is formed:
[0102]
[0103]
[0104] Then, the data-driven control strategy iteration model can be obtained:
[0105] W (i,j+1) =[(Π (i,j) ) T Π (i,j) ] -1 (Π (i,j) ) T Λ (i,j) (20)
[0106] Combined with formula (20), the neural network weight parameters can be solved based on the least squares iteration technique. dim is the dimension of the data samples that need to be collected, and must satisfy W dim ≥rank(Π (i,j) ), namely W dim ≥k c +k a m.
[0107] In the above technical solution, step 103 further includes the following steps:
[0108] Design a data-driven robust control double closed-loop framework based on reinforcement learning, including inner loop iteration and outer loop iteration. The structure diagram of the data-driven robust control framework is as follows: Figure 1 As shown, the framework process is shown below ad:
[0109] a: Initialization: Set the initial inner and outer loop iteration steps i=j=0, the initial stable control u (0) =u d0 , the sample dimension W that needs to be collected dim , the maximum number of iterations of the inner loop is M m , sampling time T, cumulative maximum sample dimension N m , the inner loop iteration convergence termination condition β.
[0110] b: Data Collection: Application Control Strategy (When looping through the first frame, use u (0) ) for the controlled system, collecting original system data including: system status and disturbance control signals.
[0111] c: Strategy evaluation: Based on the data sample calculation model (15) and (16), calculate and store the data sample and In the experience pool, based on formulas (18) and (19), until the complete data set Π is formed (i,j) and Λ (i,j) .
[0112] Strategy improvement: Based on the data-driven control strategy iteration model (20), the neural network weight parameter W is calculated. (i ,j+1) , update the weight parameters of the evaluation network and behavior network and
[0113] d: Judgment condition: When ‖W (i,j+1) -W (i,j) ‖ 2 >=β, if the maximum number of iterations M m If not satisfied, enter process c, and j←j+1; if ‖W (i,j+1) -W (i,j) ‖ 2 <β and the maximum training time is reached, stop the algorithm iteration and obtain the control strategy; otherwise, process the data samples based on the designed data processing method and enter process b, with i←i+1.
[0114] Among them, in the first iteration cycle, it is necessary to set the admissible control strategy ud0 (x)=u0(x)+d u (x), ensuring that the system does not diverge under the influence of interference signals. Control channel disturbance d u (x) will interfere with the control strategy and reduce the system stability. More importantly, the control channel disturbance will affect the accuracy of the collected data set and is not conducive to the convergence of the data-driven algorithm.
[0115] In the embodiment of the present invention, a data-driven robust control dual closed-loop framework based on reinforcement learning is designed, combined with the adaptability of the same-strategy algorithm to uncertain environments. The control strategy obtained by inner loop iteration is fed back for system control, and some new system data is collected again to calculate new data samples, so that the control strategy can be autonomously adjusted to the disturbance problem, thereby improving the adaptability of the data-driven robust control method.
[0116] In the above process b, the collected system raw data includes: system status and disturbance control signal.
[0117] The above process c corresponds to the inner loop iteration part of the data-driven robust control double closed-loop framework. Combining the raw data collected in process b, based on the data sample calculation model equations (15) and (16), the data samples are calculated and stored in the experience pool. Based on equations (18) and (19), the samples are stored until the sample dimension in the experience pool reaches the expected value, forming a complete data set. Based on the data-driven control strategy iteration model equation (20), the neural network parameter matrix can be obtained.
[0118] In process d, determine whether the inner loop iteration termination condition is met. If not, enter the next cycle of the inner loop iteration, bring the neural network parameter matrix of the first step into the data sample calculation model, update the complete data set, and obtain the neural network parameter matrix of the next step based on the data-driven control strategy iteration model; if it is met, enter the outer loop iteration process;
[0119] Determine whether the outer loop iteration termination conditions are met. If not, the current control strategy is calculated based on the neural network parameter matrix, and the next outer loop iteration cycle is entered, where this strategy is applied to the controlled system. If satisfied, the iteration process of the data-driven robust control dual closed-loop framework is terminated.
[0120] Furthermore, the above step 104 specifically includes:
[0121] (4-1) In data-driven reinforcement learning methods, the correlation between adjacent data in continuous-time control increases significantly. When samples are generated through sequential exploration of an environment, it is difficult to assume that the samples are independent and identically distributed. This can lead to data correlation issues, reduced computational efficiency, and full-rank conditions in data-driven reinforcement learning, which in turn affect the convergence and stability of the framework iteration.
[0122] The present invention, based on experience replay technology, alleviates this problem by designing an experience pool to store and release data samples. By randomly extracting datasets of specific dimensions from the experience pool and mixing the latest samples, the data sample correlation in the continuous-time control problem is broken. During the framework iteration, the original sequence of adjacent data is broken, thereby reducing data correlation, making it easier to meet the full rank condition and improving the stability of the algorithm. Furthermore, the reuse of experience data improves data utilization, reduces the time required to recollect data, and improves the efficiency of the algorithm.
[0123] Design a new data sample processing method. Before the outer loop iterative control strategy is applied to the controlled system, extract and process data samples to form an incomplete data set, store the incomplete data set in the experience pool, and after the outer loop iterative control strategy is applied to the controlled system, collect data samples to supplement the experience pool until a complete data set is formed, thus re-entering the inner loop iteration in the double closed loop framework. For the specific data sample processing method flow chart, see Figure 2 .
[0124] Here, “set i”, i = 1, 2, 3, ..., corresponds to the complete data set of the i-th outer loop iteration in the data-driven robust control double closed-loop framework. The collection process of the complete data set can refer to process c in the framework process. The difference is that the collection of “set 1” is based on the initial stable control u d0 The collection of "set i", i = 2, 3, 4..., is based on the control strategy of inner loop iteration i=2,3,4..., carried out, j i is the number of inner loop iterations corresponding to the i-th outer loop iteration.
[0125] The following is an example of how to obtain q in the experience pool "set 1". m1 Dimensional data samples are the key to data processing. First, extract all sample data in the experience pool "set 1" and convert W dim The order of dimensional data samples is randomly disrupted and each data sample is numbered. Generate q by random number algorithm m1 random numbers, where q m1 And each random number is less than W dim A positive number. m The random numbers are matched with the data samples in the complete data set and marked, and the remaining unlabeled data samples are eliminated. The labeled data samples are arranged in a shuffled order to form an incomplete data set and stored in the experience pool.
[0126] Through data processing, the amount of data required for the next outer loop iteration is reduced, further improving data utilization efficiency. At the same time, the random selection of data can effectively reduce the correlation of data samples, thereby improving the convergence performance of the framework iteration.
[0127] (4-2) After data processing, since the dimension of the data samples in the experience pool is less than the dimensionality condition for entering the inner loop iteration, it is necessary to continue to add data samples to the experience pool until a complete data set is formed, thereby entering the inner loop iteration again.
[0128] Apply the outer loop iterative control strategy to the controlled system and collect the original system data containing the new control channel disturbance, including system state and disturbance control signal. Then, based on the data sample model, calculate new data samples and store them in the experience pool until the data set data sample dimension reaches W dim dimension, that is, forming a complete data set, thus entering the inner loop iterative process in the double closed-loop framework again.
[0129] Example 2
[0130] In order to enable those skilled in the art to better understand the present invention, the data-driven reinforcement learning robust control method for nonlinear industrial systems is described in detail below in conjunction with specific embodiments.
[0131] Considering the control channel disturbance, for the following second-order inverted pendulum system:
[0132]
[0133] Where, the mass of the pendulum is m l = 1 / 3kg, the length of the pendulum l = 3 / 2m, the acceleration due to gravity θ l is the angular position, ν l is the angular velocity, and F is the control signal corresponding to the angular acceleration. F is the control channel disturbance, specifically:
[0134]
[0135] Among them, [e -0.01t ·(sin(5πt)+sin(15πt)+sin(17πt)+sin(34πt)+sin(50πt))·10 -1 ]<0.18. Therefore, in the cost function, the robust term is defined d F1 (ν l ,θ l ) is [ν l (t)·θ l (t)],d F2 (t) is [e-0.01t ·(sin(5πt)+sin(15πt)+sin(17πt)+sin(34πt)+sin(50πt))·10 -1 ].
[0136] The control purpose of this embodiment is to move the pendulum from the lowest position (θ l =π) to the highest position (θ l =0). When the pendulum swings to its highest position, the system is easily affected by disturbances and then oscillates. Therefore, this is a challenging robust control problem for nonlinear industrial systems.
[0137] During initialization, the initial state is set to The matrix and parameters in the cost function are set to Q = diag[1,1], M = 1, and γ = 1, respectively, where diag[] represents a diagonal matrix; the activation functions of the evaluation network and the behavior network are defined as:
[0138] and The initial neural network parameters are set as and The initial stable control strategy is combined with φ a (ν l ,θ l )and Get; the sample dimension W that needs to be collected dim =50; Maximum number of iterations of the inner loop M m =100; sampling time T = 0.1s; cumulative maximum sample dimension N m =125; inner loop iteration convergence termination condition β=10e-6.
[0139] Under the influence of control channel disturbance, the convergence process of the evaluation network weight parameters of the data-driven robust control method is as follows: Figure 3 As shown, the convergence results are The convergence process of behavior network weight parameters is as follows Figure 4 As shown, the convergence results are as follows:
[0140] It can be seen that the neural network parameters are continuously updated under the influence of disturbances, and the stability of the system is guaranteed by timely adjustment of the network parameters. Furthermore, combined with the behavioral network function representation in formula (13), a specific data-driven robust control strategy can be obtained for the control of the inverted pendulum system.
[0141] The angular position trajectory curve of the inverted pendulum system is as follows: Figure 5As shown. In order to better demonstrate the effectiveness of this method, this method is compared with the classic same-strategy method and the different-strategy method. Obviously, the classic same-strategy method and the different-strategy method cannot provide good control, while the data-driven robust control method can obtain better control effect and expected response. It can be seen that when the pendulum is stable at the highest point, the control channel disturbance d F (ν l ,θ l ) will gradually decrease. On the contrary, the unstable dynamic response [ν l ,θ l ] T Will disturb d through the control channel F (ν l ,θ l ) has a worse impact on the stability of the system. Therefore, Example 2 effectively verifies the stability and robustness of the data-driven robust control method.
[0142] Unless otherwise specified, the embodiments of the present invention do not limit the models of the components. Any component that can perform the above functions may be used.
[0143] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A reinforcement learning data-driven robust control method for nonlinear industrial systems, characterized in that: The method comprises: For nonlinear industrial systems with control channel disturbances, a cost function with robust terms is designed and the optimal control strategy is given. Through mathematical equivalent transformation, the optimal control strategy is converted into a data-driven control strategy, the original data of system operation is collected, the data sample calculation model is designed, and the data samples are collected into the experience pool until a complete data set is formed, and a data-driven control strategy iteration model is constructed; The data-driven control strategy is calculated in the inner loop iteration, and the obtained data-driven control strategy is applied to the controlled system in the outer loop iteration; Design a new data sample storage experience pool. Before the outer loop iterative control strategy is used in the controlled system, extract and process data samples to form an incomplete data set, store the incomplete data set in the experience pool, apply the outer loop iterative control strategy to the controlled system, collect new data samples to supplement the experience pool until a complete data set is formed, and then enter the inner loop iteration in the double closed loop framework again. Among them, the cost function containing the robust term is: Among them, J(x(t)) is the cost function, is the integrated utility function, is the integrand time variable, U(x(t),u(t)) is the utility function; Q and M are positive definite matrices of corresponding dimensions, γ is a positive real number, is the robust term of the cost function, x(t) is the system state, u(t) is the control signal, is a function related to the disturbance signal; Wherein, the data sample calculation model is: in, and is the data sample, φ c (·) is the evaluation network activation function, φ a (·) is the behavioral network activation function, is the behavior network weight parameter matrix; is the unit matrix, u d0 (x)=u0(x)+d u (x) is the disturbed initial stable control strategy, u0(x) is the initial stable control strategy, d u (x) is the system disturbance signal; is the matrix Kronecker product operation; The data-driven control strategy iteration model is: W (i,j+1) =[(Π (i,j) ) T P (i,j) ] -1 (P (i,j) ) T L (i,j) in, is the neural network weight matrix, vec(·) represents the column vector transformation of the matrix, is the evaluation network weight parameter matrix, Π (i,j) and Λ (i,j) is a data sample set.
2. The method for robust control of a nonlinear industrial system by reinforcement learning data-driven according to claim 1, characterized in that: The data-driven control strategy is calculated in the inner loop iteration, and the obtained data-driven control strategy is applied to the controlled system in the outer loop iteration as follows: a: Initialization: Set the initial inner and outer loop iteration steps i=j=0, initial stable control u (0) =u d0 , the dimension of the collected samples is W dim , the maximum number of iterations of the inner loop is M m , sampling time T, cumulative maximum sample dimension N m , the inner loop iteration convergence termination condition β; b: Data Collection: Application Control Strategy During the first frame loop, use u (0) For the controlled system, the original system data collected includes: system status and disturbance control signal; c: Strategy evaluation: calculate and store data samples and In the experience pool, until a complete data set Π is formed (i,j) and Λ (i,j) ; Strategy improvement: Based on the data-driven control strategy iteration model, calculate the neural network weight parameter W (i,j+1) , update the weight parameters of the evaluation network and behavior network and d: Judgment condition: When ‖W (i,j+1) -W (i,j) ‖ 2 >=β, if the maximum number of iterations M m If not satisfied, enter process c, and j←j+1; if ‖W (i,j+1) -W (i,j) ‖ 2 <β and the maximum training time is reached, stop the algorithm iteration and get the control strategy; otherwise, process the data samples based on the designed data processing method and enter process b, and i←i+1.
3. The method for robust control of a nonlinear industrial system based on reinforcement learning data driven according to claim 1, characterized in that: Extract all sample data in the experience pool "set 1" and convert W dim The order of the dimensional data samples is randomly disrupted, each data sample is numbered, and q is generated by a random number algorithm. m1 A random number, q m1 And each random number is less than W dim A positive number; q in the experience pool m The random numbers are matched with the data samples in the complete data set and marked, and the remaining unlabeled data samples are eliminated. The labeled data samples are arranged in a shuffled order to form an incomplete data set and stored in the experience pool.
4. The method for robust control of a nonlinear industrial system based on reinforcement learning data driven according to claim 1, characterized in that: The method is applied to a robotic arm, a servo motor, and an inverted pendulum system.
