Self-adaptive dynamic planning method and system based on Chebyshev polynomial
By using Chebyshev polynomial fitting models and evaluation functions in adaptive dynamic programming, the problem of poor algorithm implementation caused by neural networks is solved, achieving fast convergence and high-precision optimal control.
Patent Information
- Application Number
- CN202310598691.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-05-24
AI Technical Summary
In existing adaptive dynamic programming methods, the use of neural networks leads to poor algorithm implementation, slow iteration and convergence speeds, and high requirements for debugging skills, making it difficult to meet the application needs of industrial sites.
By replacing the neural network with Chebyshev polynomials, the convergence speed and control accuracy of the algorithm are improved by fitting the model function, action function and evaluation function, and utilizing the zero point of Chebyshev polynomials and Newton's iteration method, thus achieving optimal control.
This improves the execution efficiency and computational accuracy of the algorithm, reduces the implementation difficulty, and enables the adaptive dynamic programming algorithm to quickly respond to dynamic changes in the system in the digital control unit, achieving online implementation.
Smart Images

Figure CN119024686A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of digital control technology, and in particular, relates to an adaptive dynamic programming method and system based on Chebyshev polynomials. Background Art
[0002] In the field of digital control technology in industrial production, aerospace, and power grid regulation, it is often necessary to consider the optimal control problem, that is, to use the minimum control resources to make the controlled quantity reach the control target as quickly as possible.
[0003] In the prior art, dynamic programming is essentially a nonlinear programming method, the core of which is the Bellman optimality principle. It is relatively easy to solve discrete time optimal control problems using dynamic programming, but due to the "curse of dimensionality" problem when this method deals with complex problems, its application has certain limitations. Subsequently, adaptive dynamic programming (Adaptive Dynamic Programming, ADP) was proposed, the principle of which is to use the function approximation structure to obtain the solution of optimal control. There are three basic types of ADP methods, namely heuristic dynamic programming (HDP), dual heuristic dynamic programming (DHP) and globalized dual heuristic dynamic programming (GDHP). Each type of ADP method contains three modules, namely, evaluation module, model module and action module, and each module is implemented in the form of neural network. However, neural network needs to use a large number of parameters in the implementation process, such as network topology, weights and initial values of thresholds; the learning process has a certain randomness, the output result is difficult to analyze, the algorithm has a "black box" nature, debugging is inconvenient, and seriously affects the implementation effect of the algorithm. Neural networks need to initialize weights and states. Unreasonable initialization parameters will directly affect the convergence of the ADP algorithm. The iteration and convergence speed of the neural network algorithm are slow. The iterative convergence depends on the selection of initial values. It requires high debugging skills for algorithm users, which is not conducive to on-site use in engineering projects. Summary of the invention
[0004] In order to solve the deficiencies in the prior art, the present invention provides an adaptive dynamic programming method and system based on Chebyshev polynomials, which approximates the performance index function in dynamic programming based on Chebyshev polynomials, thereby obtaining the optimal performance index function and optimal control to satisfy the Bellman optimality principle, improve the convergence speed of the algorithm, effectively shorten the control time of the digital control unit, and improve the control accuracy of the digital control unit.
[0005] The present invention adopts the following technical solution.
[0006] An adaptive dynamic programming method based on Chebyshev polynomials is used to fit and control a controlled object model in a digital control unit, wherein the controlled object model includes a model function, an action function and an evaluation function; including:
[0007] Step 1, obtaining the fitting parameters of the Chebyshev polynomial of each function, including: fitting order, value range of state quantity, value range of control quantity;
[0008] Step 2: within the range of the state quantity and the control quantity, obtain the zero points of the Chebyshev polynomials of the model function, the action function and the evaluation function as the first interpolation point, the second interpolation point and the third interpolation point respectively; fit the model function using the Chebyshev polynomial at the first interpolation point;
[0009] Step 3, determine whether the initial values of the iteration of the action function and the evaluation function have been obtained. If the initial values of the iteration have been obtained, proceed to step 4; if the initial values of the iteration have not been obtained, when the iteration state is the initial state, select the second interpolation point within the value range of the control amount as the initial value of the iteration;
[0010] Step 4: Based on the initial value of the iteration, an inner iteration of the Chebyshev polynomial of the evaluation function is performed at the third interpolation point, and the latest evaluation function is obtained through the inner iteration. In the outer iteration, the control amount corresponding to the latest evaluation function under the convergence criterion is used to update the action function to obtain the updated action function;
[0011] Step 5: Obtain the control quantity according to the state quantity using the updated action function, and output the control quantity to the controlled object model.
[0012] Preferably, in step 1, the Chebyshev polynomials include one-dimensional Chebyshev polynomials and multi-dimensional Chebyshev polynomials.
[0013] Preferably, in step 2, for the one-dimensional Chebyshev polynomial, the zero point of the Chebyshev polynomial is determined according to the fitting order;
[0014] For multidimensional Chebyshev polynomials, the multidimensional Chebyshev polynomials are expressed as tensor product functions of multiple one-dimensional Chebyshev polynomials, wherein the tensor product function is the product of trigonometric functions of multidimensional angular variables, the values of the multidimensional angular variables are determined according to the zero values of the trigonometric functions, and the values of the multidimensional angular variables are converted into values of the multidimensional variables as the zero points of the multidimensional Chebyshev polynomials.
[0015] Preferably, step 2 also includes: sampling at the first interpolation point to obtain a test data set, using the test data set to calculate the error of the Chebyshev polynomial of the model function, and when the absolute value of the error is greater than a limit value, increasing the fitting order of the model function, wherein the limit value is not greater than 0.01.
[0016] Preferably, in step 3, if the iteration initial value is not obtained and the iteration state is a non-initial state, the following steps are included:
[0017] Step 3.1, within the value range of the state quantity and the value range of the control quantity, construct an artificial evaluation function using a continuous positive definite function and a model function about the state quantity and the control quantity;
[0018] Step 3.2, fitting the action function using Chebyshev polynomial at the second interpolation point;
[0019] Step 3.3, using the control amount corresponding to the action function obtained in step 3.2 to update the evaluation function;
[0020] Step 3.4, selecting multiple artificial iteration initial values within the range of the control variable; based on different artificial iteration initial values, using Newton iteration method to obtain the minimum value of the artificial evaluation function and the corresponding control variable;
[0021] Step 3.5, comparing the minimum values of different artificial evaluation functions, and taking the control variable corresponding to the minimum value among the minimum values of the artificial evaluation function as the minimum control variable;
[0022] Step 3.6, based on the minimum control variable, update the positive definite function. When the updated positive definite function is not greater than the positive definite function before the update, the minimum control variable and its corresponding evaluation function are used as the initial value of the iteration.
[0023] Preferably, the manual evaluation function g(x,u) is expressed as:
[0024]
[0025] In the formula,
[0026] U(x k ,u k ) represents the k-th dimension state quantity x k and the control variable u k The corresponding evaluation function is,
[0027] F(x k ,u k ) represents the k-th dimension state x k , control variable u k The corresponding model function is,
[0028] is the model function F(x k ,u k ) corresponds to a sufficiently large continuous positive definite function, and the positive definite function exists for the Chebyshev zeros of both the action function and the evaluation function, as follows:
[0029]
[0030] In the formula, is the k-th dimension state x k is a continuous positive definite function of .
[0031] Preferably, based on the Newton iteration method, the control variable when the artificial evaluation function takes a minimum value is obtained by the following relationship:
[0032]
[0033] In the formula,
[0034] u l and u l+1 are the control variables at the lth and l+1th iterations, respectively.
[0035] g′(x,u l ) and g″(x,u l ) are the first-order derivative and second-order derivative of the artificial evaluation function g(x,u), respectively.
[0036] Preferably, after obtaining the minimum control amount, the positive definite function is updated:
[0037]
[0038] In the formula, are positive definite functions before and after updating respectively.
[0039] Preferably, step 4 comprises: performing inner layer iteration of the Chebyshev polynomial of the evaluation function at the zero point of the Chebyshev polynomial of the evaluation function based on the initial value of the iteration; when the number of inner layer iterations reaches a preset number, obtaining the latest internal evaluation function;
[0040] Based on the latest internal evaluation function, the Newton iteration method is used to perform the outer iteration of the control quantity; before the number of outer iterations reaches the preset number, for all the zeros of the Chebyshev polynomials of the internal evaluation function, there are two times when the absolute value of the difference between the internal evaluation functions is less than the set value, then the outer iteration ends;
[0041] When both inner and outer layer iterations are completed, the latest evaluation function and the optimal control function are obtained.
[0042] Preferably, the inner iteration of the evaluation function is performed according to the following relationship:
[0043]
[0044] In the formula,
[0045] i is the number of outer layer iterations,
[0046] ji is the number of inner layer iterations corresponding to the i-th outer layer iteration,
[0047] x k is the k-th dimension state quantity,
[0048] v i (x k ) is the k-th dimension state x at the i-th outer iteration k The corresponding control quantity,
[0049] F(x k , v i (x k )) is the k-th dimension state quantity x k , control quantity v i (x k ) and serves as the k+1th dimensional state quantity x k+1 ,
[0050] is the jth value of the i-th outer iteration i +1 inner iteration k-th dimension state x k The corresponding internal evaluation function is,
[0051] is the jth value of the i-th outer iteration i The k+1th dimension state x at the outer iteration k+1 The corresponding internal evaluation function is,
[0052] U(x k , v i (x k )) is the state quantity x k , control quantity v i (x k ) is the corresponding evaluation function.
[0053] Preferably, based on the latest internal evaluation function, the Newton iteration method is used for outer iteration to fit the control quantity with the following relationship:
[0054]
[0055] In the formula,
[0056] U(x k ,u k ) is the state quantity x k , control variable u k The corresponding evaluation function is,
[0057] F(x k ,u k ) is the k-th dimension state quantity x k , control variable uk The corresponding model function is,
[0058] V i-1 (F(x k ,u k )) is the model function F(x) at the i-1th outer layer iteration k ,u k ) corresponds to the internal evaluation function.
[0059] Preferably, the preset number of times is not less than 1, and the set value ε is not greater than 0.01.
[0060] An adaptive dynamic programming system based on Chebyshev polynomials includes: an input module, an interpolation point module, a model function fitting module, an iteration initial value module, an action function and evaluation function fitting module, an output module,
[0061] An input module is used to obtain the fitting parameters of the Chebyshev polynomials of each function, including: fitting order, value range of state quantity, and value range of control quantity;
[0062] An interpolation point module is used to obtain the zero points of the Chebyshev polynomials of the model function, the action function and the evaluation function as the first interpolation point, the second interpolation point and the third interpolation point within the value range of the state quantity and the control quantity;
[0063] A model function fitting module, used for fitting the model function using Chebyshev polynomials at the first interpolation point;
[0064] The iteration initial value module is used to determine whether the iteration initial values of the action function and the evaluation function have been obtained. If the iteration initial values have been obtained, the evaluation function module is called; if the iteration initial value has not been obtained, when the iteration state is the initial state, the second interpolation point is selected within the value range of the control amount as the iteration initial value;
[0065] The action function and evaluation function fitting module is used to perform inner iteration of the Chebyshev polynomial of the evaluation function at the third interpolation point based on the initial value of the iteration, obtain the latest evaluation function through the inner iteration, and update the action function with the control amount corresponding to the latest evaluation function under the convergence criterion in the outer iteration to obtain the updated action function;
[0066] The output module obtains the control quantity according to the state quantity using the updated action function, and outputs the control quantity to the controlled object model.
[0067] A terminal comprises a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method.
[0068] A computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.
[0069] The beneficial effect of the present invention is that, compared with the prior art, the adaptive dynamic programming method based on Chebyshev polynomials proposed in the present invention realizes the optimal control of the controlled object. The Chebyshev polynomials are used to replace the original neural network to realize the fitting of the model object, the evaluation function, and the action function, which improves the execution efficiency of the algorithm, avoids multiple forward and backward iteration processes when training the neural network, reduces the difficulty of implementing the algorithm, and makes the evaluation function and the action function have a clear polynomial structure, which is convenient to implement using a digital control unit. In the calculation process of the algorithm, the Newton iteration method is used to obtain the extreme value of the algorithm, which further improves the implementation accuracy and calculation efficiency of the algorithm, and avoids obtaining the extreme value by searching for a large number of sampled values. And the convergence speed of the algorithm is improved by the method of iterating the outer control strategy and iterating the inner evaluation value, making the online implementation of the adaptive dynamic programming algorithm possible. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 It is a flow chart of an adaptive dynamic programming method based on Chebyshev polynomials proposed by the present invention;
[0071] Figure 2 is a Chebyshev polynomial fitting flow chart in an embodiment of the present invention;
[0072] Figure 3 It is an algorithm architecture diagram of the heuristic adaptive dynamic programming (HDP) in the embodiment of the present invention;
[0073] Figure 4 It is an execution effect diagram of the adaptive dynamic programming method based on Chebyshev function in an embodiment of the present invention. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.
[0075] The present invention proposes to solve the dynamic programming problem by using Chebyshev polynomials to replace function approximation based on the Bellman optimality principle, and realize time-forward dynamic programming to overcome the "dimensionality curse" problem.
[0076] The adaptive dynamic programming method based on Chebyshev polynomials includes the fitting of execution function, evaluation function and model function, which belongs to the optimal control solution method. ADP does not need to establish an accurate model for the nonlinear system, but only needs to perform adaptive learning through external evaluation information.
[0077] The present invention proposes an adaptive dynamic programming method based on Chebyshev polynomials for fitting and controlling a controlled object model, wherein the controlled object model includes a model function, an action function and an evaluation function; Figure 1 As shown, the specific steps of the method are as follows:
[0078] Step 1, obtaining the fitting parameters of the Chebyshev polynomial of each function, including: fitting order, value range of state quantity, and value range of control quantity.
[0079] Specifically, the model function, action function and evaluation function set corresponding parameters such as fitting order, value range of state quantity, value range of control quantity and convergence criterion, which are used to provide necessary parameter information for the Chebyshev polynomial fitting function and provide a parameter adjustment interface to avoid affecting the versatility of the dynamic programming method due to parameter solidification.
[0080] The fitting order is used to determine the order of the Chebyshev polynomial. The Chebyshev polynomials of each function may be different, so the fitting order determined according to the fitting conditions of each function is also different. A larger fitting order can improve the accuracy of the fitting function, but a larger fitting order leads to an increase in calculation time and causes overfitting of the algorithm.
[0081] State quantity is a physical quantity that describes the state of a material system, such as position, velocity, momentum, kinetic energy, angular velocity, angular momentum, pressure, temperature, volume, potential energy, etc. The state of a material system will change over time under the influence of the outside world, and the state quantity that describes the system will also change over time. The integral value of any closed loop state quantity is zero, or in other words, the change of the state quantity after any process is only related to the initial and final states, and has nothing to do with the process path.
[0082] Furthermore, the value range of the state quantity and the value range of the control quantity are set according to the actual on-site use requirements of the digital control unit, and the state quantity and the control quantity are normalized to facilitate the fitting of the Chebyshev polynomial.
[0083] Furthermore, the convergence criterion is used to determine whether the adaptive dynamic programming method has converged, and the value is a given reasonable small value.
[0084] Step 2: Within the range of state variables and control variables, obtain the zero points of the Chebyshev polynomials of the model function, action function, and evaluation function as the first interpolation point, the second interpolation point, and the third interpolation point, respectively; and fit the model function using the Chebyshev polynomial at the first interpolation point.
[0085] Each function needs to obtain the characteristics of the function when fitting, and the interpolation point is a point selected on the function to reflect the characteristic information of the function. After obtaining the interpolation point, fitting can be performed only when the characteristic information of the fitted function is mastered. The present invention proposes to use Chebyshev polynomials to characterize the functions involved in the adaptive dynamic programming algorithm. Therefore, the fitting of Chebyshev polynomials can obtain a better fitting effect through a specific method of selecting interpolation points. Selecting the zero point of the Chebyshev polynomial as the interpolation point can reduce the interpolation error and effectively avoid the Runge phenomenon.
[0086] Specifically, the system state equation is regarded as a black box, the zero point of the Chebyshev polynomial is selected as the interpolation point according to the fitting order obtained in step 1, sampling is performed at the interpolation point, and the model function is initially fitted using the Chebyshev polynomial, and the three functions are fitted in the subsequent iterative process.
[0087] Specifically, Chebyshev polynomials include one-dimensional Chebyshev polynomials and multidimensional Chebyshev polynomials. Among them, multidimensional Chebyshev polynomials are more widely used. Multidimensional Chebyshev polynomials are used to fit model functions, action functions and evaluation functions, which has a wider range of applications.
[0088] Specifically, for a one-dimensional Chebyshev polynomial, the zero point of the Chebyshev polynomial is determined according to the fitting order.
[0089] Specifically, for multidimensional Chebyshev polynomials, the multidimensional Chebyshev polynomials are expressed as the tensor product function of multiple one-dimensional Chebyshev polynomials, wherein the tensor product function is the product of trigonometric functions of multidimensional angular variables, the values of the multidimensional angular variables are determined according to the zero values of the trigonometric functions, and the values of the multidimensional angular variables are converted into the values of the multidimensional variables as the zero points of the multidimensional Chebyshev polynomials.
[0090] The multidimensional Chebyshev polynomial f(ξ) is expressed as:
[0091]
[0092] In the formula,
[0093] ξ is a k-dimensional variable, ξ1, ζ2, ..., ξ kare the values of ξ in each dimension,
[0094] i1,i2,...,i k are ξ1, ξ2, ..., ξ k The coefficient of k =0,1,2,...,n,
[0095] n is a natural number, which is the highest order of the Chebyshev fitting function selected in step 1.
[0096] s is the order,
[0097] is the tensor product function, expressed as:
[0098]
[0099] Among them, θ1, θ2, …, θ k are respectively related to ξ1, ξ2, ..., ξ k The associated angle variable.
[0100] is a fixed coefficient, expressed as:
[0101]
[0102] in:
[0103]
[0104]
[0105] In the formula,
[0106] j1,j2,...,j k are the coefficients of the variables corresponding to each dimension, j k =1,2,...,m,
[0107] m is the number of Chebyshev zeros, and the value of m is not less than n+1;
[0108] is the k-th dimension variable ξ k The upper limit of the value of
[0109] χ k is the k-th dimension variable ξ k The lower limit of the value of
[0110] When i1~i k When there are m 0s in , the corresponding s=m.
[0111] Use multidimensional Chebyshev polynomials to fit multidimensional system objects. After fitting, use the test data set to calculate the accuracy of the model function, action function and evaluation function. If the accuracy is lower than the set fitting accuracy, the accuracy of the Chebyshev polynomial fitting is improved by increasing the fitting order until it is greater than the set fitting accuracy.
[0112] The specific process of the multidimensional Chebyshev polynomial fitting algorithm is as follows Figure 2 As shown, it can be expressed as:
[0113] 1) Before function fitting, set the parameters of the Chebyshev polynomial, including the Chebyshev polynomial order n and the value range of the variable x = [a, b];
[0114] 2) Calculate the Chebyshev interpolation zero point. First, calculate the angle at the Chebyshev interpolation zero point, and then obtain the Chebyshev interpolation zero point in combination with the value range of the variable;
[0115] 3) Obtain the function value of the fitting object at the interpolation zero point;
[0116] 4) Calculate the fixed coefficients of Chebyshev polynomials according to the formula;
[0117] 5) Use the obtained coefficients and basis functions to construct Chebyshev polynomials to fit the function object.
[0118] Step 3, determine whether the iteration initial values of the action function and the evaluation function have been obtained. If the iteration initial values have been obtained, enter step 4; if the iteration initial values have not been obtained, when the iteration state is the initial state, select the second interpolation point within the value range of the control amount as the iteration initial value.
[0119] It is determined whether it is necessary to obtain the initial value of the iteration according to the iteration state. When the iteration state is the initial state or the iteration state is not converged, the Chebyshev zero point of the action function is selected as the initial value of the iteration. The initial value of the iteration is used to fit the initial fit obtained in step 2, and the Chebyshev polynomial is used to fit and update the controlled object model at the interpolation point until the iteration converges, and the Chebyshev polynomials of the action function and the evaluation function are obtained.
[0120] In a non-restrictive preferred embodiment, the algorithm process is called when it is necessary to obtain the initial value of the iteration, the Chebyshev zero point of the action function is selected as the initial value of the iteration within the value range of the control quantity, and the initial control function and evaluation function are obtained using the initial value of the iteration, providing feasible numerical values for the evaluation value and the control strategy iteration process.
[0121] Specifically, if the iteration initial value is not obtained and the iteration state is a non-initial state, step 3 includes:
[0122] Step 3.1, within the value range of the state quantity and the value range of the control quantity, use the continuous positive definite function of the state quantity and the control quantity and the model function to construct the artificial evaluation function as follows:
[0123]
[0124] In the formula,
[0125] U(x k ,u k ) represents the k-th dimension state quantity x k and the control variable u k The corresponding evaluation function is,
[0126] F(x k ,u k ) represents the k-th dimension state x k , control variable u k The corresponding model function is,
[0127] is the model function F(x k ,u k ) corresponds to a sufficiently large continuous positive definite function, and the positive definite function exists for the Chebyshev zeros of both the action function and the evaluation function, as follows:
[0128]
[0129] In the formula, is the k-th dimension state x k is a continuous positive definite function of .
[0130] Step 3.2, fitting the action function using Chebyshev polynomials at the second interpolation point.
[0131] Step 3.3, update the evaluation function using the control amount corresponding to the action function obtained in step 3.2.
[0132] Step 3.4, select multiple artificial iteration initial values within the range of the control variable; based on different artificial iteration initial values, use the Newton iteration method to obtain the minimum value of the artificial evaluation function and the corresponding control variable.
[0133] Based on the Newton iteration method, the control variable when the artificial evaluation function takes the minimum value is obtained by the following relationship:
[0134]
[0135] In the formula,
[0136] u l and u l+1 are the control variables at the lth and l+1th iterations, respectively.
[0137] g′(x,u l ) and g″(x,u l ) are the first-order derivative and second-order derivative of the artificial evaluation function g(x,u), respectively.
[0138] When the iteration converges, the minimum value of the evaluation function g(x, u l ) state control quantity u k .
[0139] Step 3.5, compare the minimum values of different artificial evaluation functions, and take the control variable corresponding to the minimum value among the minimum values of the artificial evaluation function as the minimum control variable.
[0140] From the minimum value of the evaluation function g(x, u k ) to extract the control quantity corresponding to the minimum value As the minimum control amount.
[0141] Step 3.6, based on the minimum control variable, update the positive definite function. When the updated positive definite function is not greater than the positive definite function before the update, the minimum control variable and its corresponding evaluation function are used as the initial value of the iteration.
[0142] After obtaining the minimum control amount, the positive definite function is updated:
[0143]
[0144] In the formula, are the evaluation functions before and after updating respectively.
[0145] Both represent evaluation functions.
[0146] The present invention realizes the fitting update of the evaluation function and the action function based on the Chebyshev polynomial, and the fitting update process is consistent with the fitting process of the controlled object model based on the Chebyshev polynomial in step 2.
[0147] Step 4, based on the initial value of the iteration, perform inner iteration of the Chebyshev polynomial of the evaluation function at the third interpolation point, obtain the latest evaluation function through the inner iteration, and update the action function with the control amount corresponding to the latest evaluation function under the convergence criterion in the outer iteration to obtain the updated action function.
[0148] The evaluation function and the action function can be understood as a symbiotic relationship. Finding the optimal evaluation function means finding the optimal action function; vice versa. The ultimate goal of the adaptive dynamic programming algorithm is to obtain the optimal action function, which can also be understood as obtaining the optimal evaluation function. The acquisition process is obtained through an iterative method, that is, the inner loop iteration of the evaluation function and the outer loop iteration of the action function. To start the iterative process, it is necessary to input the initial value of the iteration, but at this time the initial value of the iteration is unknown and needs to be determined by the algorithm.
[0149] Therefore, in step 4, based on the initial value of the iteration, the inner layer iteration of the Chebyshev polynomial of the evaluation function is performed at the zero point of the Chebyshev polynomial of the evaluation function; when the number of inner layer iterations reaches the preset number, the latest internal evaluation function is obtained;
[0150] Based on the latest internal evaluation function, the Newton iteration method is used to perform the outer iteration of the control quantity; before the number of outer iterations reaches the preset number, for all the zeros of the Chebyshev polynomials of the internal evaluation function, there are two times when the absolute value of the difference between the internal evaluation functions is less than the set value, then the outer iteration ends;
[0151] When both inner and outer layer iterations are completed, the latest evaluation function and the optimal control function are obtained.
[0152] Among them, the preset number of times is not less than 1 time, and the set value ε is not greater than 0.01.
[0153] Specifically, step 4 includes:
[0154] Step 4.1, using the Chebyshev polynomials of the action function and the evaluation function obtained in step 3 as the initial function of the fitting iteration, perform inner iteration of the evaluation function and outer iteration of the action function at the interpolation point.
[0155] The inner iteration of the evaluation function is performed according to the following relationship:
[0156]
[0157] In the formula,
[0158] i is the number of outer layer iterations,
[0159] j i is the number of inner layer iterations corresponding to the i-th outer layer iteration,
[0160] x k is the k-th dimension state quantity,
[0161] v i (x k ) is the k-th dimension state x at the i-th outer iteration k The corresponding control quantity,
[0162] F(x k , v i (x k )) is the k-th dimension state quantity x k , control quantity v i (x k ) and serves as the k+1th dimensional state quantity x k+1 ,
[0163] is the jth value of the i-th outer iteration i +1 inner iteration k-th dimension state x k The corresponding internal evaluation function is,
[0164] is the jth value of the i-th outer iteration i The k+1th dimension state x at the outer iteration k+1 The corresponding internal evaluation function is,
[0165] U(x k , v i (x k )) is the state quantity x k , control quantity v i (x k ) is the corresponding evaluation function.
[0166] Step 4.2, based on the latest internal evaluation function, use the Newton iteration method to perform outer iteration and fit the control quantity with the following relationship:
[0167]
[0168] In the formula,
[0169] U(x k ,u k ) is the state quantity x k , control variable u k The corresponding evaluation function is,
[0170] F(x k ,u k ) is the k-th dimension state quantity x k , control variable u k The corresponding model function is,
[0171] V i-1 (F(x k ,u k )) is the model function F(x) at the i-1th outer layer iteration k ,u k ) corresponds to the internal evaluation function.
[0172] Step 4.3, perform ai The outer iterations are repeated until the absolute value of the difference between the evaluation functions of two points is less than the preset value ε for all Chebyshev zeros, that is, |V i (x k )-V i-1 (x k )|≤ε, the iteration ends.
[0173] It is considered that the control function obtained at this time is the optimal control function, and the evaluation function obtained is the optimal evaluation function, and the optimal control function and the optimal evaluation function are fitted in the form of Chebyshev polynomials.
[0174] Step 5: Obtain the control quantity according to the state quantity using the updated action function, and output the control quantity to the controlled object model.
[0175] Specifically, given the initial value of the state quantity, after substituting it into the action function, the control quantity of the corresponding state quantity is obtained, and the control quantity is applied to the controlled object. Then, a new state quantity can be obtained in the next sampling period, and the new state quantity is brought into the action function to obtain a new control quantity, and so on, until the state quantity finally tends to a stable value and the algorithm ends.
[0176] The present invention also proposes an adaptive dynamic programming system based on Chebyshev polynomials, comprising: an input module, an interpolation point module, a model function fitting module, an iteration initial value module, an action function and evaluation function fitting module, and an output module;
[0177] An input module is used to obtain the fitting parameters of the Chebyshev polynomials of each function, including: fitting order, value range of state quantity, and value range of control quantity;
[0178] An interpolation point module is used to obtain the zero points of the Chebyshev polynomials of the model function, the action function and the evaluation function as the first interpolation point, the second interpolation point and the third interpolation point within the value range of the state quantity and the control quantity;
[0179] A model function fitting module, used for fitting the model function using Chebyshev polynomials at the first interpolation point;
[0180] The iteration initial value module is used to determine whether the iteration initial values of the action function and the evaluation function have been obtained. If the iteration initial values have been obtained, the evaluation function module is called; if the iteration initial value has not been obtained, when the iteration state is the initial state, the second interpolation point is selected within the value range of the control amount as the iteration initial value;
[0181] The action function and evaluation function fitting module is used to perform inner iteration of the Chebyshev polynomial of the evaluation function at the third interpolation point based on the initial value of the iteration, obtain the latest evaluation function through the inner iteration, and update the action function with the control amount corresponding to the latest evaluation function under the convergence criterion in the outer iteration to obtain the updated action function;
[0182] The output module obtains the control quantity according to the state quantity using the updated action function, and outputs the control quantity to the controlled object model.
[0183] The controlled object is an actual physical entity. The model function only uses polynomials to fit the behavior characteristic function of the controlled object, which is convenient for mathematical description of the problem, so that the action function for control can be obtained by using algorithms. After obtaining the action function, the control signal can be obtained according to the state information, and acted on the controlled object to achieve control of the controlled object.
[0184] The output module finally outputs the control signal. Figure 3 The algorithm architecture diagram shows that after the algorithm iteration is completed, for each state vector x k , the corresponding control vector u can be obtained through the action network k , so that u k Act on the physical system to achieve optimal control of the physical system.
[0185] Further, if Figure 3 The heuristic adaptive dynamic programming (HDP) shown is a specific implementation structure of ADP, which uses Chebyshev polynomials to fit the evaluation function and action function, reduces the size of preset parameters, improves the algorithm convergence speed and storage resource consumption, and makes the obtained optimal evaluation network and action network have a clear polynomial form, which is convenient for the implementation of digital processing chips.
[0186] Dynamic programming is essentially a nonlinear programming method proposed by American mathematician Bellman. The core of this programming algorithm is the Bellman optimality principle. It is relatively easy to solve discrete-time optimal control problems using dynamic programming, but this method has certain limitations in its application due to the "curse of dimensionality" problem when dealing with complex problems. Subsequently, Werbos proposed the theory of adaptive dynamic programming, using the function approximate structure to obtain the solution of optimal control. On this basis, he used Chebyshev polynomials to improve the adaptive dynamic programming algorithm and proposed an adaptive dynamic programming algorithm framework based on Chebyshev polynomials.
[0187] The principle of adaptive dynamic programming is to use the function approximation structure to approximate the performance index function in dynamic programming with Chebyshev, so as to obtain the optimal performance index function and optimal control to meet the optimality principle. The adaptive dynamic programming structure includes three modules: evaluation module, model module and execution module. Each module can be approximated by Chebyshev polynomials. The execution network is used to approximate the optimal control strategy, and the evaluation network is used to approximate the optimal performance index function. After the control / execution acts on the dynamic system (or the controlled object), the reward / penalty generated by the controlled object at different stages affects the judgment function. The Chebyshev polynomials are used to approximate the execution function and the judgment function. The optimal control function can be obtained while obtaining the optimal evaluation function. The parameter update of the judgment function is based on the Bellman optimal principle, which can not only reduce the forward calculation time, but also respond to the dynamic changes of the system online and automatically adjust the fitting parameters in the network structure.
[0188] Reference Figure 4 , is the execution effect of the adaptive dynamic programming method based on Chebyshev function. Assuming that the controlled object is a torsion pendulum system, its discretized system equation can be expressed as:
[0189]
[0190] where [x 1k , x 2k ] T is the system state equation, u k is the system control variable, f d is the friction coefficient, and f is taken during the simulation process d =0.2, the initial state of the simulation is taken as x0=[2, -1], and the adaptive dynamic programming method based on neural network and the method of the present invention are used to iterate respectively to obtain the optimal control rate.
[0191] In the traditional iterative method based on BP neural network, the sampling value of the state quantity is selected as 10000, and it takes 5254.84 seconds and 200MB of memory to iterate the value function and the action function to obtain the optimal action function; while using the method described in the present invention, it only takes 39.617 seconds and 4MB of memory to obtain the optimal action function. After using the method described in the present invention, the calculation time and memory occupied are greatly reduced, and the construction of complex neural network equations is avoided. Finally, the action function in polynomial form can be obtained, which is convenient for application in digital control units. And by Figure 4 It can be seen that under the control rate described in the present invention, the state quantity of the system will eventually converge to [0, 0], thereby achieving control over the object.
[0192] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0193] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0194] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0195] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. An adaptive dynamic programming method based on Chebyshev polynomials for fitting a controlled object model in a digital control unit, wherein: The controlled object model includes a model function, an action function and an evaluation function; and is characterized in that it includes: Step 1, obtaining the fitting parameters of the Chebyshev polynomial of each function, including: fitting order, value range of state quantity, value range of control quantity; Step 2: within the range of the state quantity and the control quantity, obtain the zero points of the Chebyshev polynomials of the model function, the action function and the evaluation function as the first interpolation point, the second interpolation point and the third interpolation point respectively; fit the model function using the Chebyshev polynomial at the first interpolation point; Step 3, determine whether the initial values of the iteration of the action function and the evaluation function have been obtained. If the initial values of the iteration have been obtained, proceed to step 4; if the initial values of the iteration have not been obtained, when the iteration state is the initial state, select the second interpolation point within the value range of the control amount as the initial value of the iteration; Step 4: Based on the initial value of the iteration, an inner iteration of the Chebyshev polynomial of the evaluation function is performed at the third interpolation point, and the latest evaluation function is obtained through the inner iteration. In the outer iteration, the control amount corresponding to the latest evaluation function under the convergence criterion is used to update the action function to obtain the updated action function; Step 5: Obtain the control quantity according to the state quantity using the updated action function, and output the control quantity to the controlled object model.
2. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 1, characterized in that: In step 1, the Chebyshev polynomials include one-dimensional Chebyshev polynomials and multi-dimensional Chebyshev polynomials.
3. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 2, characterized in that: In step 2, for the one-dimensional Chebyshev polynomial, the zero point of the Chebyshev polynomial is determined according to the fitting order; For multidimensional Chebyshev polynomials, the multidimensional Chebyshev polynomials are expressed as tensor product functions of multiple one-dimensional Chebyshev polynomials, wherein the tensor product function is the product of trigonometric functions of multidimensional angular variables, the values of the multidimensional angular variables are determined according to the zero values of the trigonometric functions, and the values of the multidimensional angular variables are converted into values of the multidimensional variables as the zero points of the multidimensional Chebyshev polynomials.
4. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 1, characterized in that: Step 2 also includes: sampling at the first interpolation point to obtain a test data set, using the test data set to calculate the error of the Chebyshev polynomial of the model function, and when the absolute value of the error is greater than a limit value, increasing the fitting order of the model function, wherein the limit value is not greater than 0.
01.
5. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 1, characterized in that: In step 3, if the initial value of the iteration is not obtained and the iteration state is not the initial state, it includes: Step 3.1, within the value range of the state quantity and the value range of the control quantity, construct an artificial evaluation function using a continuous positive definite function and a model function about the state quantity and the control quantity; Step 3.2, fitting the action function using Chebyshev polynomial at the second interpolation point; Step 3.3, using the control amount corresponding to the action function obtained in step 3.2 to update the evaluation function; Step 3.4, selecting multiple artificial iteration initial values within the range of the control variable; based on different artificial iteration initial values, using Newton iteration method to obtain the minimum value of the artificial evaluation function and the corresponding control variable; Step 3.5, comparing the minimum values of different artificial evaluation functions, and taking the control variable corresponding to the minimum value among the minimum values of the artificial evaluation function as the minimum control variable; Step 3.6, based on the minimum control variable, update the positive definite function. When the updated positive definite function is not greater than the positive definite function before the update, the minimum control variable and its corresponding evaluation function are used as the initial value of the iteration.
6. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 5, characterized in that: The manual evaluation function g(x,u) is expressed as: In the formula, U(x k ,u k ) represents the k-th dimension state quantity x k and the control variable u k The corresponding evaluation function is, F(x k ,u k ) represents the k-th dimension state x k , control variable u k The corresponding model function is, is the model function F(x k ,u k ) corresponds to a sufficiently large continuous positive definite function, and the positive definite function exists for the Chebyshev zeros of both the action function and the evaluation function, as follows: In the formula, is the k-th dimension state x k is a continuous positive definite function of .
7. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 6, characterized in that: Based on the Newton iteration method, the control variable when the artificial evaluation function takes the minimum value is obtained by the following relationship: In the formula, u l and u l+1 are the control variables at the lth and l+1th iterations, respectively. g′(x,u l ) and g″(x,u l ) are the first-order derivative and second-order derivative of the artificial evaluation function g(x,u), respectively.
8. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 7, characterized in that: After obtaining the minimum control amount, the positive definite function is updated: In the formula, are positive definite functions before and after updating respectively.
9. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 1, characterized in that: Step 4 includes: performing inner layer iteration of the Chebyshev polynomial of the evaluation function at the zero point of the Chebyshev polynomial of the evaluation function based on the initial value of the iteration; when the number of inner layer iterations reaches a preset number, obtaining the latest internal evaluation function; Based on the latest internal evaluation function, the Newton iteration method is used to perform the outer iteration of the control quantity; before the number of outer iterations reaches the preset number, for all the zeros of the Chebyshev polynomials of the internal evaluation function, there are two times when the absolute value of the difference between the internal evaluation functions is less than the set value, then the outer iteration ends; When both inner and outer layer iterations are completed, the latest evaluation function and the optimal control function are obtained.
10. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 9, characterized in that: The inner iteration of the evaluation function is performed according to the following relationship: In the formula, i is the number of outer layer iterations, j i is the number of inner layer iterations corresponding to the i-th outer layer iteration, x k is the k-th dimension state quantity, v i (x k ) is the k-th dimension state x at the i-th outer iteration k The corresponding control amount, F(x k ,v i (x k )) is the k-th dimension state quantity x k , control quantity v i (x k ) and serves as the k+1th dimensional state quantity x k+1 , is the jth value of the i-th outer iteration i +1 inner iteration k-th dimension state x k The corresponding internal evaluation function is, is the jth value of the i-th outer iteration i The k+1th dimension state x at the outer iteration k+1 The corresponding internal evaluation function is, U(x k ,v i (x k )) is the state quantity x k , control quantity v i (x k ) is the corresponding evaluation function.
11. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 10, characterized in that: Based on the latest internal evaluation function, the Newton iteration method is used for outer iteration to fit the control quantity with the following relationship: In the formula, U(x k ,u k ) is the state quantity x k , control variable u k The corresponding evaluation function is, F(x k ,u k ) is the k-th dimension state quantity x k , control variable u k The corresponding model function is, V i-1 (F(x k ,u k )) is the model function F(x) at the i-1th outer layer iteration k ,u k ) corresponds to the internal evaluation function.
12. The adaptive dynamic programming method based on Chebyshev polynomials according to claim 9, characterized in that: The preset number of times is not less than 1, and the set value ε is not greater than 0.
01.
13. An adaptive dynamic programming system based on Chebyshev polynomials using the method according to any one of claims 1 to 12, characterized in that: The system includes: input module, interpolation point module, model function fitting module, iterative initial value module, action function and evaluation function fitting module, output module, An input module is used to obtain the fitting parameters of the Chebyshev polynomials of each function, including: fitting order, value range of state quantity, and value range of control quantity; An interpolation point module is used to obtain the zero points of the Chebyshev polynomials of the model function, the action function and the evaluation function as the first interpolation point, the second interpolation point and the third interpolation point within the value range of the state quantity and the control quantity; A model function fitting module, used for fitting the model function using Chebyshev polynomials at the first interpolation point; The iteration initial value module is used to determine whether the iteration initial values of the action function and the evaluation function have been obtained. If the iteration initial values have been obtained, the evaluation function module is called; if the iteration initial value has not been obtained, when the iteration state is the initial state, the second interpolation point is selected within the value range of the control amount as the iteration initial value; The action function and evaluation function fitting module is used to perform inner iteration of the Chebyshev polynomial of the evaluation function at the third interpolation point based on the initial value of the iteration, obtain the latest evaluation function through the inner iteration, and update the action function with the control amount corresponding to the latest evaluation function under the convergence criterion in the outer iteration to obtain the updated action function; The output module obtains the control quantity according to the state quantity using the updated action function, and outputs the control quantity to the controlled object model.
14. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-12.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
Citation Information
Patent Citations
Robot motion optimization method and device, computer equipment and storage medium
CN109318230A
Inertial navigation resolving method and system based on matrix form function iteration
CN114996652A
Analog computing implementing arbitrary non-linear functions using Chebyshev-polynomial-interpolation schemes and methods of use
US20200293725A1
