Online multi-step heuristic dynamic planning method, medium and equipment

Abstract

The invention discloses an online multi-step heuristic dynamic planning method, a medium and equipment, belongs to the field of nonlinear system control, and develops an online multi-step heuristic dynamic planning method for a nonlinear affine system. The learning speed is improved by combining an actor approval network and an experience playback technology, and more efficient strategy evaluation can be realized on the premise of not depending on acceptable control conditions. In order to enhance the diversity of data in the learning process, an execution evaluation neural network structure integrated with an ER technology is provided, so that an agent can effectively explore on fewer samples, and the dependence on training data is reduced. Finally, the superiority of the proposed algorithm is verified through a simulation result, and an experiment shows that under the same condition, the ER-MSHDP algorithm has a faster learning process compared with a traditional HDP algorithm, and the learning time is exponentially reduced along with the increase of the step length.

Description

Technical Field

[0001] The present invention relates to the field of optimal control, and particularly to an online multi-step heuristic dynamic programming method, medium and device. Background Art

[0002] As a core area in control theory, optimal control aims to improve system performance by designing optimal control strategies, and shows broad prospects especially in applications such as autonomous driving, unmanned aerial vehicle flight control and industrial automation. The optimal control problem usually involves various constraints, such as time, resource limitations and safety requirements, and its solution process is both complex and challenging. The Bellman optimality principle provides a theoretical basis for solving the optimal control problem, but the existence of high dimensions, nonlinear systems and uncertainties makes the solution process more difficult. In addition, the solution of continuous-time optimal control problems often relies on the Hamilton-Jacobi-Bellman (HJB) equation. However, in many practical problems, the analytical solution of the HJB equation is difficult to obtain, which has prompted researchers to turn to numerical optimization and approximation methods.

[0003] In response to these challenges, many studies have attempted to provide solutions. Reinforcement learning (RL) and algorithms based on approximate dynamic programming (ADP) are the main technical approaches to solve the optimal control problem. Reinforcement learning learns the optimal control strategy step by step through interaction with the environment, and shows unique advantages especially in environments with incomplete information and dynamic changes. The policy iteration (PI) and value iteration (VI) algorithms under the ADP framework, which rely on the feedback adjustment of system states and control strategies, have become two core methods in the optimal control theory. In addition, researchers have also proposed various variant algorithms, such as generalized value iteration (GVI) and multi-step policy evaluation (MPE), which can improve the convergence speed and robustness of control strategies and play an important role in the optimal control of complex systems.

[0004] Despite the significant progress made by these methods, there are still many unsolved problems in practical applications. First, existing reinforcement learning algorithms often require a large amount of training data and computing resources, which is a major bottleneck for application scenarios with high real-time requirements. Second, although the use of models such as neural networks can effectively improve the accuracy of control strategies, problems such as the generalization ability of the model, overfitting and long training time still restrict its practical application. More complicatedly, how to design control algorithms with strong robustness and high adaptability under various uncertainty factors is still an open research area. In addition, many current control strategies often rely on offline training, which makes them lack sufficient adaptability when facing new and unknown environments. Therefore, how to develop more efficient and flexible online optimal control methods remains a major challenge in optimal control research. Summary of the Invention

[0005] An online multi-step heuristic dynamic programming method, device and storage medium proposed by the present invention can at least solve one of the technical problems in the background art.

[0006] To achieve the above object, the present invention adopts the following technical solutions: An online multi-step heuristic dynamic programming method, comprising the following steps: S1. Construct a cost function according to a non-linear affine system; S2. Process the cost function by a multi-step value iteration method to obtain an extended cost function; S3. Construct a critic-execution neural network to approximate the cost function and design a neural network iteration formula; S4. Define a buffer according to the experience replay technique; S5. Initialize the buffer and start online learning to solve the control law , and complete the dynamic programming.

[0007] Further, in step S1 of the present invention, the method for constructing a cost function according to a non-linear affine system includes: The non-linear affine system includes: (1) Wherein, , , The subscript represents the state of the system at the th step, The subscript represents the control input of the system at the th step represents the control input output by the Actor network and acting on the system; represents the corresponding control strategy calculated from the state ; The cost function is defined as a performance index for measuring the quality of the controller: (2) Wherein, , and are both positive semi-definite matrices; Definition 1: If can stabilize the system (1) within , wherein, represents the stability domain of the system state , and ensures the finiteness of the cost function for all , where and are within If it is continuous above, then is an admissible control set; Optimal cost function satisfies the discrete-time HJB equation: (3) According to (3), the optimal control input has the following expression: (4) Assume that is continuously differentiable to obtain the optimal control input; (5) The case of the linear system is as follows: (6) Consider the quadratic cost function , where .

[0008] Furthermore, the method for obtaining the extended cost function in step S2 of the present invention includes: According to 's definition, the extended cost function is: (7) where is an integer greater than 1, representing the learning step size. In multi-step value iteration, the utility function of several future time steps is considered, and the cost function contains more system information; at the beginning of the iteration, is initialized to a non-negative value; similar to VI, multi-step value iteration is used for policy improvement; (8) Policy evaluation (9) When , MSVI is equivalent to VI, and when , MSVII is equivalent to PI.

[0009] Furthermore, the method for constructing the judgment-execution neural network to approximate the cost function and designing the neural network iteration formula of the present invention includes: The cost function is expressed as (10) where is the optimal weight vector, is the neural network approximation error, which is related to the number of neurons , because ; Approximate cost function using neural network approximation It is expressed as: (11) where is the estimated value of the optimal weight vector ; The estimation error is defined as: (12) where is the future estimated state calculated according to the model information and ; The loss function of the evaluation network is defined as: (13) Then, the estimated value of the optimal weight vector is updated by the gradient descent method : (14) where the learning rate , and the estimation error is defined as: (15) According to the update rule (14) of the evaluation neural network weights, it is expressed as follows: (16) The neural network structure is a classic actor-critic network; by constructing the actor network, it is approximated as (17) where, , + represents the number of neurons in the actor neural network, represents the approximation error, the estimated value of (18) According to (8), (11), the control strategy approximated by the actor network is (19) Then, the estimation error of the control input (20) The network training aims to minimize the loss function of the actor network , which is defined as follows (21) The method for adjusting the weights of the execution network is the same as that in Equation (14); (22) where the learning rate satisfies , and the estimation error is defined as: (23).

[0010] Furthermore, in step S5 of the present invention, solving the control law , and the process of completing the dynamic programming method includes: Initializing the buffer data includes the experience replay buffer size L, Q, and , the initial weight , the step size , the maximum time step , and the number of single-step iterations h; During the online learning process, when , , then a sample is randomly selected from the experience pool according to a uniform distribution, and the weights are alternately updated according to Equations (14) and (22), is incremented; afterwards, the current data sample will be added to the database; if there is probing noise, the control law is calculated through Equation (18), and the system state is updated through Equation (1). During this process, if the size of the experience pool exceeds the preset maximum value , then the earliest generated data in the experience pool is deleted; if the condition is satisfied, then adding probing noise is stopped. Finally, is incremented, and the execution continues until the maximum time step is reached.

[0011] On the other hand, the present invention also discloses a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0012] On yet another aspect, the present invention also discloses a computer device including a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the above method.

[0013] Compared with the prior art, the present invention has the following technical effects: For non-linear affine systems, an online multi-step heuristic dynamic programming method is developed. By combining the actor-critic network and experience replay technology, the learning speed is improved, and more efficient policy evaluation can be achieved without relying on acceptable control conditions. To enhance the diversity of data during the learning process, an actor-critic neural network structure incorporating ER technology is proposed, enabling the agent to effectively explore with fewer samples, thus reducing the dependence on training data. Finally, the superiority of the proposed algorithm is verified through simulation results. Experiments show that under the same conditions, the ER-MSHDP algorithm has a faster learning process compared with the traditional HDP algorithm, and as the step size increases, the learning time decreases exponentially. Description of the Drawings

[0014] Figure 1 It is a flowchart of an online multi-step heuristic dynamic programming method of the present invention; Figure 2 It is a graph of the change of weights during the training process of Example 1 of the present invention; Figure 3 Under the proposed algorithm of Example 1 of the present invention of the system state and control input; Figure 4 Under different step sizes of Example 1 of the present invention of the norm value; Figure 5 It is the number of learning steps in different cases of Example 1 of the present invention; Figure 6 It is the change of weights during the training process of Example 1 of the present invention; Figure 7 It is the system state and control input under the proposed algorithm of Example 2 of the present invention; Figure 8 It is the approximate optimal value function of Example 2 of the present invention; Figure 9 It is the parameter modulus value under different step sizes of Example 2 of the present invention; Figure 10 It is the number of learning steps required for convergence under different step sizes of Example 2 of the present invention. Detailed Embodiment

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0016] As Figure 1 shown, this embodiment discloses an online multi-step heuristic dynamic programming method, including the following steps: S1. Construct a cost function according to the non - linear affine system; S2. Process the cost function through a multi - step value iteration method to obtain an extended cost function; S3. Construct a critic - actor neural network to approximate the cost function and design a neural network iteration formula; S4. Define a buffer according to the experience replay technique; S5. Initialize the buffer and start online learning to solve the control law ; The following is a detailed description of each step: S1. Construct a cost function according to the non - linear affine system.

[0017] Consider the non - linear affine system as: (1) where , , The subscript represents the state of the system at the th step, The subscript represents the control input of the system at the th step represents the control input output by the Actor network and acting on the system. represents the corresponding control policy calculated from the state . For the non - linear affine system, the goal of optimal control is to find a control policy that can both reach the equilibrium point and minimize the performance index . Therefore, we define the following cost function as the performance index to measure the quality of the controller: (2) where , and are both positive semi - definite matrices.

[0018] Definition 1: If can stabilize the system (1) within , where represents the domain of attraction of the system state and ensures the finiteness of the cost function for all , where and are continuous on , then is an admissible control set.

[0019] The optimal cost function satisfies the discrete - time HJB equation: (3) According to (3), the optimal control input has the expression (4) Assume is continuously differentiable, and the optimal control input is obtained.

[0020] (5) The case of the linear system is as follows: (6) Consider the quadratic cost function , where . The problem of solving the optimal process can be equivalently represented as the problem of solving the Riccati equation, which has been widely studied. For equation (6), can be obtained by numerical solution or iterative solution according to the system model information. However, due to the non - linear characteristics of equation (3), solving this equation is usually challenging and sometimes even infeasible. The iterative algorithm is one of the key tools for solving non - linear partial differential (HJB) equations. Therefore, combining the iterative algorithm with ADP and RL methods can provide a more flexible and powerful method for solving such problems.

[0021] S2. Process the cost function through the multi - step value iteration method to obtain the extended cost function; VI and PI are two methods for obtaining the optimal control. Both methods involve alternately evaluating the control strategy and improving the control strategy. When the difference between adjacent cost functions is less than a certain small range, that is , where is a constant greater than 0 ( varies according to the system complexity, required accuracy, etc. The higher the required accuracy the smaller it will be selected). We then consider that we have obtained and . However, PI must start the iteration from an acceptable control strategy, and directly implementing this requirement is challenging, especially for non - linear systems. However, since PI emphasizes policy evaluation, usually only a few iterations are required to obtain the optimal cost function. On the contrary, VI improves the strategy starting from the initialized cost function, relaxing the strict requirement for admissible control. However, it cannot accurately evaluate the current strategy, so the iteration efficiency is often relatively low. The multi - step value iteration (MSVI), as a bridge between PI and VI, can effectively make up for the deficiencies of both. According to 's definition, the extended cost function is: (7) where is an integer greater than 1, representing the learning step size. In multi-step value iteration, the utility function for several future time steps is considered. That is to say, the cost function contains more system information, which is beneficial to improving the learning speed. At the beginning of the iteration, is usually initialized to a non-negative value. Similar to VI, multi-step value iteration is used for policy improvement.

[0022] (8) Policy evaluation (9) It should be noted that when , MSVI is equivalent to VI, and when , MSVII is equivalent to PI. As Figure 1 shown, MSVI involves policy improvement and policy evaluation: predicting the utility function for future steps.

[0023] S3. Construct a judgment-execution neural network to approximate the cost function and design a neural network iteration formula.

[0024] A neural network usually consists of an input layer, a hidden layer, and an output layer. The input layer and the output layer usually correspond to the independent variable and the dependent variable of the function to be approximated respectively. The hidden layer mainly performs feature extraction and calculation. The activation function of the hidden layer is a key part because it determines the ability of the network to map non-linear relationships. These activation functions usually introduce non-linearity, enabling the network to adapt to complex non-linear data. The present invention adopts a polynomial activation function. Therefore, the cost function can be expressed as (10) where is the optimal weight vector. is the neural network approximation error, which is related to the number of neurons . Since . Generally speaking, is difficult to obtain directly. Therefore, the approximate cost function approximated by the neural network can be expressed as: (11) where is the estimated value of the optimal weight vector . Then we define the estimation error as (12) where is the future estimated state calculated according to the model information and .

[0025] The loss function of the evaluation network is defined as (13) Then, the optimal weight vector is updated by gradient descent method of the estimate : (14) where the learning rate . The estimation error is defined as: (15) According to the update rule (14) of the weights of the evaluation neural network, it can be expressed as follows: (16) The neural network structure adopted by the present invention is a classic Actor-Critic network. By constructing an execution network, it can be approximated as (17) where, . represents the number of neurons in the actor neural network, represents the approximation error. The estimate of can be expressed as: (18) According to (8), (11), the control policy approximated by the execution network is (19) Then, we can obtain the estimation error of the control input (20) The network training aims to minimize the loss function of the execution network , which is defined as follows (21) The method of adjusting the weights of the execution network is the same as that in formula (14).

[0026] (22) where the learning rate satisfies . The estimation error is defined as: (23) In the design of neural networks, the selection of activation functions is a crucial step, which directly affects the representational ability, convergence speed, and overall performance of the model. Different activation functions have unique characteristics and are thus suitable for different tasks and network architectures. Commonly used activation functions include the sigmoid function and the tanh function. Both of these functions are continuously differentiable and are therefore often used in adaptive dynamic programming. For an n-dimensional linear system with a given performance criterion, if the corresponding Riccati equation has a solution, , then its form is a quadratic polynomial. Therefore, to verify the effectiveness of ER-MSHDP, the present invention adopts a polynomial activation function.

[0027] S4. Define a buffer according to the experience replay technique; Experience replay is a technique widely used in RL that can improve the efficiency of the learning process simultaneously. Its basic concept is to store the agent's experiences in a buffer, which can be reused later to reduce the correlation between consecutive observations and improve the sampling efficiency. During the online training process, by utilizing the stored trajectories instead of relying solely on the current state, the convergence speed can be increased. This method enables the training of neural networks to utilize experiences other than the latest policy. Therefore, it helps prevent the algorithm from getting stuck in local minima or diverging. Based on previous studies, we incorporated the experience replay technique to further improve the learning efficiency. We define the ER buffer as (24) where L represents the buffer capacity, and its specific value is determined through experimental debugging. The present invention does not adopt the prioritized experience replay technique but uniformly samples the experience tuples. During each iteration of the network weights, the weights are updated using the states sampled from the buffer.

[0028] The size of the replay buffer should be large enough to store various different experiences, covering as many state-action pairs as possible, thereby enhancing the generalization ability of the training process. If the buffer is too small, it may lead to insufficient sample diversity, causing the model to be prone to overfitting. Although increasing the size of the buffer can provide a more diverse sample set, it also brings additional storage requirements and computational overhead. An overly large buffer may consume a large amount of memory, especially in high-dimensional environments. The cost of storing a large amount of historical experience should match the hardware resources of the system. In addition, an overly large buffer will increase the time required for sample retrieval, thus affecting the training efficiency. Moreover, the number of experiences selected at each learning step is also a key factor affecting the model performance.

[0029] S5. Initialize the buffer and start online learning to solve the control law , and complete the dynamic programming; The initialized buffer data includes the experience replay buffer size L, Q, and , initial weight , step size , maximum time step , number of single-step iterations h.

[0030] During the online learning process, when , , then a sample is randomly selected from the experience pool according to a uniform distribution, and the weights are alternately updated according to equations (14) and (22) , increment . After that, the current data sample will be added to the database. If there is probing noise, the control law is calculated by equation (18) , and the system state is updated by equation (1) . During this process, if the size of the experience pool exceeds the preset maximum value , then the earliest generated data in the experience pool is deleted. If the condition is satisfied, then the addition of probing noise is stopped. Finally, increment , and continue to execute until the maximum time step is reached.

[0031] Next, to further prove and demonstrate stability, a reasonable assumption and a theorem are given.

[0032] Assumption 1: and as well as and are bounded, that is , , and .

[0033] Theorem 1: Vectors of the same dimension and . satisfy the following inequality: (25) Lemma 1: Assuming Assumption 1 holds, and obtain approximate values through equations (11) and (18) respectively. Let and be updated according to equations (14) and (22). In addition, it can be proven that , as shown in equations (15) and (23), is UUB.

[0034] Proof The following is the construction process of the Lyapunov function: (26) Among them . Subsequently, the standard will be obtained. It can be expressed as (27) Then, according to Equation (15), it can be obtained.

[0035] (28) According to Equation (10) and Equation (11), it can be further deduced as (29) According to Theorem 1, satisfies (30) Next, it can be expressed as (31) Similarly, according to Equation (17) and Equation (18), it can be deduced as (32) We set . Satisfies the following inequality: (33) Therefore, if Equation (34) holds, .

[0036] (34) At the same time, assuming 1 is satisfied. We can obtain (35) If the inequality (35) holds, then is obtained. Therefore, and are UUB. This completes the proof.

[0037] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: various media such as ROM, RAM, magnetic disk, or optical disc that can store program codes.

[0038] Next, a simulation example is given.

[0039] The present invention will verify the effectiveness of an online multi-step heuristic dynamic programming method through a linear power system: (36) where and represent the time constants of the governor, turbine, and generator models, respectively. In addition, represents the feedback regulation constant, while and represent the gain constants of the generator and turbine models, respectively. Table 1 lists the parameters in (36).

[0040] Table 1 Power system parameters

[0041] Regarding the frequency deviation output increment , the generator output and the governor position value as state variables, and the positive deviation speed change increment input vector. Then, using the sampling period to discretize the power system (36), the following formula is obtained: (37) For the system (37), and are respectively set to and . All network weights are initially set to 0. The solution of the DARE can be obtained using MATLAB2020a . For the linear system (37), given the performance index parameters and R, P can be obtained through the "idare( )" function in the Matlab library: (38) The relevant optimal policy , is . We set the activation function to and . Then, the optimal weights can be obtained as: (39) and (40) To implement the ER-MSHDP online algorithm, we selected the relevant parameters and listed them in Table 2.

[0042] Table 2 Relevant parameters of the ASVI integration algorithm

[0043] The PE condition ensures that the agent explores a sufficiently diverse set of state-action pairs during interaction, preventing the system from converging to a local optimum or an unvisited state. Additionally, in ADP, many theoretical convergence results rely on the satisfaction of the PE condition. For example, when using gradient descent or other learning algorithms, the persistence of excitation guarantees the accuracy of the model parameters or value function estimates, thus ensuring convergence to the optimal solution. The present invention designs the PE condition , and terminates the exploration noise after the first 500 steps of the total sampling period. To fully train the agent, we add exploration noise to the system, i.e., persistent excitation. It should be noted that the magnitude of the exploration noise needs to be adjusted according to the relevant system or experience. Being too large may cause the system to diverge, while being too small may lead to insufficient training. Under persistent excitation, and 's change trajectories are respectively as shown in Figure 2 and Figure 3 , and the initial state is defined as . The trained weights converge to and respectively, which are approximately equal to the theoretical values.

[0044] In addition, we also design experiments with four different step sizes. As shown in Figure 4 , the convergence of weights under different step sizes. In fact, by adjusting some parameters and the magnitude of the exploration noise, the weights can converge to the optimal weights under different step sizes. To only verify the influence of the step size on the iteration efficiency, other parameter settings are the same. The experimental results show that under certain conditions, as the step size increases, the learning efficiency of the neural network improves, and the error between the estimated weights after training and the optimal weights decreases. The experimental results show that under sufficient PE conditions, the neural network weights with different step sizes can converge to near-optimal values.

[0045] In addition, we also design an experimental comparison of experienced replay and non-experienced replay with different step sizes. As shown in Figure 5 , using the experience replay technique can effectively improve the data utilization rate and accelerate the learning speed.

[0046] Consider the following nonlinear system: (41) where and . The parameters in the utility function are respectively selected as and To implement the online ER-MSHDP algorithm, we set the activation functions to and . In this case, the step size, the length of the experience replay buffer, and the sample size are selected as 10, 8, and 5 respectively. It is worth mentioning that the setting of relevant parameters is not unique and can be adjusted according to the system model and practical experience. In the proposed online algorithm, and the neural network weights, are updated simultaneously. As shown in Figure 6 , Figure 7 , the weights, as well as the state under the exploration noise and control input, . As shown in Figure 8 , the approximate optimal cost function . As shown in Figure 9 , the convergence of the critic network weights affected by different step sizes. In addition, under the specified acceptable policy, as shown in Figure 10 , the relationship between the step size and the required number of iterations. It should be noted that when the step size is small, as the step size increases, the required number of iterations decreases exponentially.

[0047] On the other hand, the present invention also discloses a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to execute the steps of the above method.

[0048] On yet another aspect, the present invention also discloses a computer device including a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, it causes the processor to execute the steps of the above method.

[0049] In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions, which when run on a computer causes the computer to execute any one of the online multi-step heuristic dynamic programming methods in the above embodiments.

[0050] It can be understood that the systems, devices, and storage media provided by the embodiments of the present invention correspond to the methods provided by the embodiments of the present invention, and the explanations, examples, and beneficial effects of relevant content can refer to the corresponding parts in the above methods.

[0051] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0052] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0053] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0054] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online multi-step heuristic dynamic programming method, characterized in that, Including the following steps: S1. Construct a cost function according to the non - linear affine system; S2. Process the cost function by the multi - step value iteration method to obtain an extended cost function; S3. Construct a critic - actor neural network to approximate the cost function and design a neural network iteration formula; S4. Define a buffer according to the experience replay technique; S5. Initialize the buffer and start online learning to solve the control law μ k , and complete dynamic programming.

2. The online multi-step heuristic dynamic programming method according to claim 1, wherein The method for constructing a cost function according to the non - linear affine system in step S1 includes: The non - linear affine system includes: x k+1 = f(x k ) + g(x k )μ k (1) where \(x\in R\) n and \(\mu\in R\) m , \(x\) k with subscript representing the state of the system at the \(k\)-th step, and \(\mu\) k with subscript representing the control input of the system at the \(k\)-th step. \(\mu\) represents the output of the Actor network and acts on the control input of the system; \(\mu(x\) k ) represents the corresponding control policy calculated from the state \(x\) k . The cost function J(x k ) is defined as a performance metric for measuring the quality of the controller: where \(U(x,\mu)=x\) T Qx+\mu T R\mu, Q\in\mathbb{R} n×n and \(R\in\mathbb{R}\) m×m are both positive semi - definite matrices; Definition 1: If can stabilize system (1) within Ω x , where Ω x represents the stability domain of the system state x, and ensures the finiteness of the cost function J(x k ) for all x ∈ Ω x , where μ(0) = 0 and μ(x) is continuous on Ω x , then Γ is the admissible control set; Optimal cost function J * (x k ) satisfies the discrete-time HJB equation: J * (x k ) = min μ∈Γ {U(x k , μ k ) + J * (x k+1 )}(3) According to (3), the optimal control input μ * (x k ) is expressed as: Assume J * (x k ) is continuously differentiable to obtain the optimal control input; The situation of the linear system is as follows: x k+1 = Ax k + Bu k (6) Consider the quadratic cost function J(x k ) = x T Px, where P ∈ R n×n .

3. The online multi-step heuristic dynamic programming method according to claim 1, wherein The method for obtaining the extended cost function in step S2 includes: According to the definition of J(x k ), the extended cost function is as follows: where N is an integer greater than 1, representing the learning step size. In multi-step value iteration, the utility function considering future several time steps, and the cost function contains more system information; at the beginning of the iteration, J (0) (x k ) is initialized to a non-negative value; similar to VI, multi-step value iteration is used for policy improvement; μ (i) (x k ) = argmin{U(x k , μ k ) + J (i) (x k+1 )} (8) Policy evaluation When N = 1, MSVI is equivalent to VI, and when N = ∞, MSVII is equivalent to PI.

4. The online multi-step heuristic dynamic programming method according to claim 1, characterized in that The method for constructing a critic - actor neural network to approximate the cost function and design a neural network iteration formula includes: The cost function J(x k ) is expressed as J(x k )=(w c ) T σ(x k )+λ c (10) where is the optimal weight vector, and λ c is the neural network approximation error, which is related to the number of neurons H c , since H c →∞, λ c →0; Approximate cost function using neural network approximation Expressed as: wherein is the estimated value of the optimal weight vector w c ; Define the estimation error as follows: where x′ is the future estimated state calculated according to the model information and μ(x k ). The loss function Ξ of the evaluation network c (x k ) is defined as: Then, update the estimated value of the optimal weight vector w by the gradient descent method c of the where the learning rate α c ∈(0, 1), and the estimation error is defined as: According to the update rule (14) for judging the neural network weights, which is expressed as follows: The neural network structure is a classic actor-critic network; by constructing an actor network, μ k is approximated as μ k = (w a ) T φ(x k ) + λ a (17) Among them, +H a represents the number of neurons in the actor neural network, and λ a represents the approximation error, and the estimated value of μ k is expressed as: According to (8) and (11), execute the control strategy μ for network approximation k For Then, obtain the estimation error of the control input Network training aims to minimize the loss function Ξ of the execution network a , which is defined as follows The method for adjusting the weights of the actor network is the same as formula (14); where the learning rate α a satisfies 0 < α a < 1, and the estimation error is defined as:

5. The online multi-step heuristic dynamic programming method according to claim 1, characterized in that: The method for solving the control law μk and completing dynamic programming in step S5 includes: The initialized buffer data includes the experience replay buffer sizes L, Q, and R, and the initial weights The step size N, the maximum time step T max , and the number of single-step iterations h; During the online learning process, when k < T max , and i < h, a sample is randomly selected from the experience pool H according to the uniform distribution, and the weights are alternately updated according to Equations (14) and (22) Increment i; afterwards, the current data sample will be added to the database; if there is probing noise, the control law u is calculated through Equation (18) k , and the system state x is updated through Equation (1) k+1 . During this process, if the size of the experience pool H exceeds the preset maximum value L, the earliest generated data in the experience pool H is deleted; if the condition is satisfied, the addition of probing noise is stopped. Finally, increment k and continue to execute until the maximum time step T is reached max .

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is caused to execute the method according to any one of claims 1 to 5.

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the method according to any one of claims 1 to 5.

Images