Optimal tracking control method, device and equipment based on adaptive dynamic programming

Abstract

The invention relates to the technical field of optimal tracking control, and provides an optimal tracking control method, device and equipment based on adaptive dynamic programming. Constructing an augmented system according to a tracking error and a reference trajectory of the system (the system is constrained by part of unknown disturbance), and designing a corresponding value function; based on the designed value function, the H infinity tracking problem is converted into a zero sum game, and a corresponding Hamilton-Jacobi-Isaacs (Hamilton-Jacobi-Isaacs, HJI) equation is constructed; and the evaluation-execution-disturbance network approximates the HJI equation, the optimal control strategy and the worst disturbance strategy are solved, and the optimal control strategy is applied to the controller. According to the method, the control input and the disturbance input during neural network training are updated by using a double-event triggering mechanism, and the resource utilization efficiency is improved. According to the optimal control input data, external disturbance factors suffered by the controlled system are considered, so that the robust performance of the obtained optimal control input data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of optimal tracking control, and specifically to an optimal tracking control method, device, and equipment based on adaptive dynamic programming. Background Art

[0002] Adaptive dynamic programming approximately solves a nonlinear system by solving the control input of the controlled system (the control input is the information output from the controller to the controlled system, and the controlled system automatically plans a reference trajectory based on this information). The prior art uses the H-infinity tracking control method to solve the input of the controlled system without considering the situation where the controlled system is partially unknown. Some tracking control methods do not consider the influence of the external disturbance information of the controlled system (the external disturbance information is the interference information from the outside to the controlled system) on the control input, and the external disturbance information is an interference information that actually exists and is inevitable in the working environment of the controlled system. When solving the control input for a system facing disturbances, the prior art has the problem of large resource consumption. Therefore, the control input solved by the prior art tracking control method has low applicability, robustness, and resource utilization efficiency.

[0003] Therefore, the prior art still needs to be improved. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides an optimal tracking control method, device, and equipment based on adaptive dynamic programming, and solves the technical problems of low accuracy and applicability of the control input obtained by the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In the first aspect, the present invention provides an optimal tracking control method based on adaptive dynamic programming, which includes:

[0007] Obtain the system state of the controlled system, apply an execution network to the system state, and the execution network obtains control input data for controlling the controlled system by solving the HJI equation; apply a disturbance network to the system state, and the disturbance network obtains disturbance input data for disturbing the controlled system by solving the HJI equation;

[0008] Apply an evaluation network to the control input data, the disturbance input data, and the system state, and the evaluation network obtains a value function related to the tracking error based on the system state;

[0009] Update the value function by updating the control input data and the disturbance input data until the optimal value function is obtained, and use the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

[0010] In one implementation, the construction method of the HJI equation includes:

[0011] Obtain the drift dynamics, input dynamics, and disturbance dynamics of the controlled system;

[0012] Set the trajectory state and reference trajectory of the controlled system to obtain a corresponding tracking error, where the tracking error is the error between the actual trajectory of the controlled system and the reference trajectory corresponding to the trajectory state;

[0013] Apply the drift dynamics to the tracking error and the trajectory state, and then combine the reference trajectory and the tracking error to obtain a drift force vector;

[0014] Apply the input dynamics to the tracking error and the trajectory state to obtain an input force vector;

[0015] Apply the disturbance dynamics to the tracking error and the trajectory state to obtain a disturbance vector;

[0016] Construct an HJI equation regarding the control input parameters and disturbance input parameters based on the drift force vector, the input force vector, and the disturbance vector.

[0017] In one implementation, applying the drift dynamics to the tracking error and the trajectory state, and then combining the reference trajectory and the tracking error to obtain a drift force vector includes:

[0018] Apply the drift dynamics to the tracking error and the trajectory state to obtain a calculation result;

[0019] Subtract the reference trajectory from the calculation result to obtain a first element;

[0020] Determine the error trajectory corresponding to the tracking error, and use the error trajectory as a second element;

[0021] Construct a drift force vector based on the first element and the second element.

[0022] In one implementation, using the control input data corresponding to the optimal value function as the optimal control input data of the controlled system includes:

[0023] Substitute the optimal value function into the value function expression composed of control input parameters and disturbance input parameters, and solve for the control input parameters to obtain the optimal control input data.

[0024] In one implementation, the construction method of the value function expression includes:

[0025] Determine the control constraint loss composed of the control input parameters;

[0026] Obtain the disturbance attenuation coefficient preset for the controlled system, and construct a value function expression based on the system state, the disturbance attenuation coefficient, the control constraint loss, and the disturbance input parameters.

[0027] In one implementation, the execution network is a neural network. The tuning method during the training of the execution network includes:

[0028] Train the execution network based on the system sample state. During the training process, obtain the control input training data approximated by the evaluation network;

[0029] Obtain the true control input data corresponding to the system sample state; and obtain the input error based on the true control input data and the control input training data;

[0030] Obtain a number of input historical error data of the controlled system, and adjust the parameters of the execution network based on the input error and the input historical error data.

[0031] In one implementation, the evaluation network is a neural network. The tuning method during the training of the evaluation network includes:

[0032] Train the evaluation network based on the system sample state. During the training process, obtain the optimal training value function output by the evaluation network;

[0033] Obtain the true value function corresponding to the system sample state, and perform integration with respect to time based on the optimal training value function and the true value function to obtain the Bellman error;

[0034] Obtain the historical value function errors corresponding to a number of input historical error data of the controlled system, and adjust the parameters of the evaluation network based on the Bellman error and a number of the historical value function errors.

[0035] In a second aspect, an embodiment of the present invention further provides an optimal tracking control device based on adaptive dynamic programming. Wherein, the device includes the following components:

[0036] The HJI equation solving module is used to obtain the system state of the controlled system, apply the execution network to the system state, and the execution network obtains the control input data for controlling the controlled system by solving the HJI equation; apply the disturbance network to the system state, and the disturbance network obtains the disturbance input data for disturbing the controlled system by solving the HJI equation;

[0037] A value function solving module, configured to apply an evaluation network to the control input data, the disturbance input data, and the system state, where the evaluation network obtains a value function related to the tracking error based on the system state;

[0038] An optimal control input data solving module, configured to update the value function by updating the control input data and the disturbance input data until the optimal value function is obtained, and use the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

[0039] In a third aspect, an embodiment of the present invention further provides a terminal device, where the terminal device includes a memory, a processor, and an optimal tracking control program based on adaptive dynamic programming stored in the memory and executable on the processor. When the processor executes the optimal tracking control program based on adaptive dynamic programming, the steps of the above-mentioned optimal tracking control method based on adaptive dynamic programming are implemented.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which an optimal tracking control program based on adaptive dynamic programming is stored. When the optimal tracking control program based on adaptive dynamic programming is executed by a processor, the steps of the above-mentioned optimal tracking control method based on adaptive dynamic programming are implemented.

[0041] Beneficial effects: According to the control input data and the disturbance input data input to the controlled system, the present invention obtains a value function, and updates the value function by updating the control input data and the disturbance input data until the optimal value function is obtained. Then, the control input data corresponding to the optimal value function is used as the optimal control input data, and the optimal control input data is input to the controlled system through a controller, and the controlled system operates based on the optimal control input data. From the above analysis, it can be seen that the present invention uses a dual-event trigger mechanism to update the control input and the disturbance input, improving the resource utilization efficiency. The optimal control input data of the present invention takes into account the external disturbance factors received by the controlled system, thereby improving the robustness of the obtained optimal control input data. Furthermore, the controlled system operating based on the optimal control input data can track the reference trajectory in the presence of disturbances. Description of the Drawings

[0042] Figure 1 is the overall flowchart of the present invention;

[0043] Figure 2 is the optimal tracking control flowchart based on dual-event-triggered adaptive dynamic programming in an embodiment of the present invention;

[0044] Figure 3This is the schematic diagram of optimal tracking control based on dual-event-triggered adaptive dynamic programming in the embodiments of the present invention;

[0045] Figure 4 This is the structural diagram of the optimal tracking control device based on adaptive dynamic programming provided by the present invention;

[0046] Figure 5 This is the internal structure principle block diagram of the terminal device provided by the embodiments of the present invention. Specific embodiments

[0047] The following combines embodiments and the accompanying drawings of the specification to clearly and completely describe the technical solutions in the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0048] It has been found through research that approximate solution of the nonlinear system by adaptive dynamic programming is to solve the control input of the controlled system (the control input is the information output from the controller to the controlled system, and the controlled system automatically plans the reference trajectory based on this information). The prior art uses the H-infinity tracking control method to solve the input of the controlled system, without considering the situation where the controlled system is partially unknown. Some tracking control methods do not consider the influence of the external disturbance information of the controlled system (the external disturbance information is the interference information from the outside to the controlled system) on the control input, and the external disturbance information is an interference information that actually exists and is inevitable in the working environment of the controlled system. When solving the control input for a system facing disturbances, the existing method of adding a disturbance network has the problem of large resource consumption. Therefore, the control input solved by the prior art tracking control method has low applicability, robustness, and resource utilization efficiency.

[0049] To solve the above technical problems, the present invention provides an optimal tracking control method, device, and equipment based on adaptive dynamic programming, which solves the technical problems of low accuracy and applicability of the control input obtained by the prior art.

[0050] The optimal tracking control method based on adaptive dynamic programming in this embodiment can be applied to a terminal device, and the terminal device can be a terminal product with a trajectory planning function, such as an in-vehicle computer, etc. In this embodiment, as Figure 1 shown in, the optimal tracking control method based on adaptive dynamic programming specifically includes the following steps:

[0051] S100, obtain the system state of the controlled system, apply an execution network to the system state, and the execution network obtains control input data for controlling the controlled system by solving the HJI equation; apply a disturbance network to the system state, and the disturbance network obtains disturbance input data for disturbing the controlled system by solving the HJI equation;

[0052] S200, applying an evaluation network to the control input data, the disturbance input data, and the system state, wherein the evaluation network obtains a value function related to the tracking error based on the system state;

[0053] S300, updating the value function by updating the control input data and the disturbance input data until the optimal value function is obtained, and using the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

[0054] The tracking control method of steps S100, S200, and S300 can be applied to the path tracking of a robot. The specific application scenarios are as follows:

[0055] The robot is the controlled system. The reference trajectory of the robot is set, and the corresponding system state x(t) of the robot and the input dynamics g(x) of the controlled system are collected. The system state includes the projection x of the center of gravity of the robot in the inertial coordinate system XOY. r and y r , robot yaw angle δ r , actuator steering angle δ f , center of gravity velocity v, acceleration a, and actuator wheelbase L. The corresponding augmented system is constructed based on the system state, tracking error, input dynamics, drift dynamics f(x), and the set perturbation dynamics k(x). The execution network solves preliminary control input data based on the augmented system state (this control input data is used to control the robot to travel along the reference trajectory). The perturbation network solves preliminary perturbation data based on the augmented system state X. Then, the evaluation network solves the value function based on the preliminary control input data, preliminary perturbation data, and system state. This value function is used to reflect whether the control input data tracks the reference trajectory. If the value function error is within the set range, the value function at this time is the optimal value function, and the control input data at this time is also the optimal control input data. If the reference trajectory cannot be tracked, the control input data and perturbation data are iteratively updated to update the value function until the value function reaches the optimal value function. The control input data corresponding to the optimal value function is the optimal control input data. The robot can track the reference trajectory by driving based on the optimal control input data.

[0056] Example 1: In this example, constructing the HJI equation (the HJI equation is also known as the HJI variance) includes the following specific steps S101 to S109:

[0057] S101 , obtaining the drift dynamics f(x), input dynamics g(x), and disturbance dynamics k(x) of the controlled system.

[0058] x∈R nrepresents the state of the controlled system, and f(x) ∈ R n , g(x) ∈ R n×m , k(x) ∈ R n×q . Both g(x), k(x), and f(x) are prior arts.

[0059] Equation of the controlled system

[0060]

[0061] u = [u1, u2,..., u m represents the control input of the system, and w = [w1, w2,..., w q represents the external disturbance.

[0062] S102. Set the trajectory state x d (t) and the tracking error e d (t). The tracking error e d (t) is the error between the actual trajectory of the controlled system and the reference trajectory corresponding to the trajectory state .

[0063] Reference trajectory state x d (t) ∈ R m , and the trajectory state x d (t) is the desired trajectory of the operating state of the controlled system, and h(·) is the functional relationship between the reference trajectory state and its derivative.

[0064] The tracking error e d (t) is the difference between the actual state of the system and the desired state trajectory, that is:

[0065] e d (t) = x(t) - x d (t)

[0066] The derivative of the tracking error is

[0067]

[0068] S103. Apply the drift dynamics f(x) to the tracking error e d (t) and the trajectory state x d to obtain the calculation result f(e d + x d ).

[0069] S104. Subtract the reference trajectory h(x d + x d ) from the calculation result f(ed ) to obtain the first element \(f(e d + x d ) - h(x d );

[0070] S105. Determine the tracking error \(e d corresponding error trajectory \(h(e d ) and use the error trajectory \(h(e d ) as the second element.

[0071] S106. Construct the drift force vector \(F(X)\) based on the first element and the second element:

[0072]

[0073] S107. Apply the input dynamics \(g(x)\) to the tracking error \(e d and the reference trajectory state \(x d to obtain the input force vector \(G(X)\):

[0074]

[0075] S108. Apply the perturbation dynamics \(k(x)\) to the tracking error \(e d and the trajectory state \(x d to obtain the perturbation vector \(K(X)\):

[0076]

[0077] S109. Construct the corresponding augmented system based on the tracking error \(e d (t) and the reference trajectory state \(x d (t):

[0078]

[0079] Meanwhile, the derivative of the augmented system is:

[0080]

[0081] S110. Construct the HJI equation for the control input parameter \(u\) and the perturbation input parameter \(w\) based on the drift force vector \(F(X)\), the input force vector \(G(X)\) and the perturbation vector \(K(X)\).

[0082] First, construct the Hamiltonian for the control input parameter \(u\) and the perturbation input parameter \(w\) according to the drift force vector \(F(X)\), the input force vector \(G(X)\) and the perturbation vector \(K(X)\):

[0083]

[0084] X T Q A X + C(u) - γ 2 w T w - αV

[0085] Wherein, V X represents the partial derivative of V with respect to X, and V is the value function. Q > 0, γ is the preset disturbance attenuation level for the controlled system and takes a positive constant, and γ satisfies the following inequality:

[0086]

[0087] Wherein, C(u) is the control constraint loss, and the calculation formula of C(u) is as follows:

[0088]

[0089] ε is the saturation boundary of the actuator (the computer with the execution network set in the actuator), is a monotonically increasing odd function with a bounded first derivative, R = diag[r1, r2...., r n > 0, representing a diagonal matrix, R = [r1, r2,..., r n ∈ R 1×n , represents a vector with all elements being 1. When taking this non - quadratic performance function, the control input satisfies the given constraint conditions.

[0090] After obtaining the Hamiltonian, replace the value function V in the Hamiltonian with the optimal value function V * , replace the control input parameter u with the optimal control input parameter u * , replace the disturbance input parameter w with the optimal input parameter w * , then the Hamiltonian can be converted into the following HJI equation (the HJI equation is HJI):

[0091]

[0092] Iteratively solve the HJI equation, and each time solve the values of u and w in the HJI equation. As Figure 3 shown, the evaluation network calculates the value of the value function V according to the values of u and w. If the value of V does not reach the optimal value, then continue to iteratively solve the values of u and w according to the value of V generated in this iteration combined with the HJI equation to update the value of V until V reaches the optimal. At this time, the value of u corresponding to V is the optimal control input data.

[0093] As Figure 2 shown, define two monotonically increasing time series and i, j ∈ N, which respectively represent the control input u k and the disturbance input w k sampling times. At time t i update the control input u k , at time m j update the disturbance input w k . The two asynchronously updated inputs are held by a zero-order hold during the sampling signal interval. The control input and the disturbance input can be expressed as:

[0094]

[0095] Define the event-triggering error as:

[0096]

[0097] Applying the event-triggering mechanism to the control input and the disturbance input, the optimal control and the worst-case disturbance under the event-triggering mechanism can be obtained as:

[0098]

[0099] The non-negative double event-triggering condition is designed as:

[0100]

[0101] where the double event-triggering threshold is:

[0102]

[0103] where Γ1, Γ2 ∈ (0, 1), λ min (Q A ) is the minimum eigenvalue of Q A , and L1, L2 are two positive constants satisfying the following conditions:

[0104]

[0105] Example 2. In this example, substitute the optimal value function into the value function expression composed of the control input parameter and the disturbance input parameter, solve the control input parameter, and obtain the optimal control input data. The optimal control input data is the value of u * :

[0106]

[0107] The disturbance input corresponding to the optimal value function obtained in this example is also optimal:

[0108]

[0109] The construction of the above value function expression includes: determining the control constraint loss C(u) composed of the control input parameters; obtaining the disturbance attenuation coefficient γ preset for the controlled system, and constructing the following value function expression based on the system state X, the disturbance attenuation coefficient, the control constraint loss, and the disturbance input parameters:

[0110]

[0111] In the formula, e is the natural constant.

[0112] In this embodiment, the value function expression can also be:

[0113]

[0114] V * (X(t)) is the optimal value of the value function V(X(t)), and V * (X(t)) and V(X(t)) satisfy the following relational expression:

[0115]

[0116] In the formula,

[0117] Embodiment 3. In this embodiment, the execution network is a neural network. The tuning method during the training of the execution network includes: training the execution network based on the system sample state. During the training process, obtain the control input training data approximated by the evaluation network Obtain the true control input data u′ corresponding to the system sample state X; and based on the true control input data u′ and the control input training data to obtain the input error e a (t); obtain several input historical error data e of the controlled system an , and based on the input error e a (t) and the input historical error data, adjust the parameters of the execution network

[0118]

[0119]

[0120] The error between the control input applied to the system and the input approximated by the evaluation network is:

[0121]

[0122] is The optimized data, where β0, β1, β2, β4, k1, k2 ∈ (0, 1).

[0123]

[0124] In this embodiment, the evaluation network is also a neural network. The optimization method for training the evaluation network includes: training the evaluation network based on the system sample state. During the training process, obtain the optimal training value function V * (X(t)); obtain the true value function corresponding to the system sample state and perform integration with respect to time based on the optimal training value function and the true value function to obtain the Bellman error; obtain the historical value function error e corresponding to several control input historical data of the controlled system cn and adjust the parameters of the evaluation network based on the Bellman error and several historical value function errors

[0125]

[0126] The weight estimation errors of the execution network and the perturbation network are:

[0127]

[0128] The Bellman error is:

[0129]

[0130] is the optimized data.

[0131] Δη = e -αh η c (X(t + h)) - η c (X(t))

[0132] This embodiment also takes the partial derivative of with respect to:

[0133]

[0134] In this example, the perturbation network is also a neural network. The optimization method:

[0135]

[0136] are the parameters of the perturbation network, is the optimized data.

[0137]

[0138]

[0139] The error between the disturbance input applied to the controlled system and the disturbance approximated by the evaluation network is:

[0140]

[0141] Example 4. In this example, an augmented system of the tracking error and the reference trajectory is constructed:

[0142]

[0143] The derivative of the augmented system is

[0144]

[0145] According to the input constraint conditions and the above disturbance attenuation conditions, the value function of the augmented system:

[0146]

[0147] Example 5. In this example, the value function and its derivative are constructed:

[0148]

[0149] represents the weight vector of the neural network, η(·) ∈ R n represents the activation function, d c represents the approximation error of the neural network, and n is the number of neurons.

[0150] The optimal value function can be transformed into:

[0151]

[0152] Approximating the value function therein by a neural network gives an error:

[0153]

[0154] Due to the unknownness of the ideal weights, the value function can be approximated as:

[0155]

[0156] The actual error is:

[0157]

[0158] The approximation error of the actual HJI equation is:

[0159]

[0160] In summary, the present invention uses integral reinforcement learning to find the optimal control strategy and the worst-case perturbation strategy in the presence of input constraints, eliminate the influence of system drift dynamics, and is applicable to systems with partially unknown dynamics. The training method of the present invention reduces resource consumption during the training process, adopts a dual-event triggering mechanism, separates the triggering conditions of the control input and the perturbation input, and avoids mutual interference. The present invention can improve the utilization rate of data and enhance the training efficiency.

[0161] This embodiment also provides an optimal tracking control device based on adaptive dynamic programming, as Figure 4 shown. The device includes the following components:

[0162] The HJI equation solving module 01 is used to obtain the system state of the controlled system, apply an execution network to the system state, and the execution network obtains control input data for controlling the controlled system by solving the HJI equation; apply a perturbation network to the system state, and the perturbation network obtains perturbation input data for interfering with the controlled system by solving the HJI equation;

[0163] The value function solving module 02 is used to apply an evaluation network to the control input data, the perturbation input data, and the system state, and the evaluation network obtains a value function related to the tracking error based on the system state;

[0164] The optimal control input data solving module 03 is used to update the value function by updating the control input data and the perturbation input data until the optimal value function is obtained, and use the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

[0165] Based on the above embodiment, the present invention also provides a terminal device, and its principle block diagram can be as Figure 5 shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements an optimal tracking control method based on adaptive dynamic programming. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0166] Those skilled in the art can understand, Figure 5The principle block diagram shown only shows the block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0167] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and an optimal tracking control program based on adaptive dynamic programming stored in the memory and executable on the processor. When the processor executes the optimal tracking control program based on adaptive dynamic programming, the following operation instructions are implemented:

[0168] Obtain the system state of the controlled system, apply an execution network to the system state, and the execution network obtains control input data for controlling the controlled system by solving the HJI equation; apply a disturbance network to the system state, and the disturbance network obtains disturbance input data for disturbing the controlled system by solving the HJI equation;

[0169] Apply an evaluation network to the control input data, the disturbance input data, and the system state, and the evaluation network obtains a value function related to the tracking error based on the system state;

[0170] Update the value function by updating the control input data and the disturbance input data until the optimal value function is obtained, and use the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

[0171] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyncHlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimal tracking control method based on adaptive dynamic programming, characterized in that Including: Obtain the system state of the controlled system, and apply an execution network to the system state, where the execution network obtains control input data for controlling the controlled system by solving the HJI equation; Apply a perturbation network to the system state, where the perturbation network obtains perturbation input data for disturbing the controlled system by solving the HJI equation; Apply an evaluation network to the control input data, the perturbation input data, and the system state, where the evaluation network obtains a value function related to the tracking error based on the system state; Update the value function by updating the control input data and the perturbation input data until the optimal value function is obtained, and use the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

2. The optimal tracking control method based on adaptive dynamic programming according to claim 1, characterized in that, The construction method of the HJI equation includes: Obtain the drift dynamics, input dynamics, and perturbation dynamics of the controlled system; Set the trajectory state and reference trajectory of the controlled system to obtain a corresponding tracking error, where the tracking error is the error between the actual trajectory of the controlled system and the reference trajectory corresponding to the trajectory state; Apply the drift dynamics to the tracking error and the trajectory state, and combine with the reference trajectory and the tracking error to obtain a drift force vector; Apply the input dynamics to the tracking error and the trajectory state to obtain an input force vector; Apply the perturbation dynamics to the tracking error and the trajectory state to obtain a perturbation vector; Construct an HJI equation about the control input parameter and the perturbation input parameter based on the drift force vector, the input force vector, and the perturbation vector.

3. The optimal tracking control method based on adaptive dynamic programming according to claim 2, characterized in that, Applying the drift dynamics to the tracking error and the trajectory state, and combining with the reference trajectory and the tracking error to obtain a drift force vector includes: Apply the drift dynamics to the tracking error and the trajectory state to obtain a calculation result; Subtract the reference trajectory from the calculation result to obtain a first element; Determine the error trajectory corresponding to the tracking error, and use the error trajectory as a second element; Construct a drift force vector based on the first element and the second element.

4. The optimal tracking control method based on adaptive dynamic programming according to claim 1, characterized in that Using the control input data corresponding to the optimal value function as the optimal control input data of the controlled system includes: Substitute the optimal value function into a value function expression composed of a control input parameter and a perturbation input parameter, and solve the control input parameter to obtain the optimal control input data.

5. The optimal tracking control method based on adaptive dynamic programming according to claim 4, characterized in that, The construction method of the value function expression includes: Determine a control constraint loss composed of the control input parameter; Obtain a preset perturbation attenuation coefficient for the controlled system, and construct a value function expression based on the system state, the perturbation attenuation coefficient, the control constraint loss, and the perturbation input parameter.

6. The optimal tracking control method based on adaptive dynamic programming according to claim 1, characterized in that The execution network is a neural network. The tuning method during training of the execution network includes: Train the execution network based on system sample states. During the training process, obtain the control input training data approximated by the evaluation network; Obtain the true control input data corresponding to the system sample state; and obtain the input error based on the true control input data and the control input training data. Obtain a number of input historical error data of the controlled system, and adjust the parameters of the execution network based on the input error and the input historical error data.

7. The optimal tracking control method based on adaptive dynamic programming according to claim 1, characterized in that, The evaluation network is a neural network. The tuning method for training the evaluation network includes: Based on the system sample state, train the evaluation network. During the training process, obtain the optimal training value function output by the evaluation network. Obtain the true value function corresponding to the system sample state, and perform integration with respect to time based on the optimal training value function and the true value function to obtain the Bellman error. Obtain the historical value function error corresponding to a number of input historical error data of the controlled system, and adjust the parameters of the evaluation network based on the Bellman error and a number of the historical value function errors.

8. An optimal tracking control device based on adaptive dynamic programming, characterized in that, The device includes the following components: The HJI equation solving module is used to obtain the system state of the controlled system, apply the execution network to the system state, and the execution network obtains the control input data for controlling the controlled system by solving the HJI equation. Apply the perturbation network to the system state, and the perturbation network obtains the perturbation input data for disturbing the controlled system by solving the HJI equation. The value function solving module is used to apply the evaluation network to the control input data, the perturbation input data, and the system state, and the evaluation network obtains the value function related to the tracking error based on the system state. The optimal control input data solving module is used to update the control input data and the perturbation input data to update the value function until the optimal value function is obtained, and use the control input data corresponding to the optimal value function as the optimal control input data of the controlled system.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an optimal tracking control program based on adaptive dynamic programming stored in the memory and executable on the processor. When the processor executes the optimal tracking control program based on adaptive dynamic programming, the steps of the optimal tracking control method based on adaptive dynamic programming according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, An optimal tracking control program based on adaptive dynamic programming is stored on the computer-readable storage medium. When the optimal tracking control program based on adaptive dynamic programming is executed by the processor, the steps of the optimal tracking control method based on adaptive dynamic programming according to any one of claims 1-7 are implemented.

Images