An online multi-step heuristic dynamic programming method, medium and device

Abstract

The application discloses an online multi-step heuristic dynamic programming method, medium and equipment, belongs to the field of nonlinear system control, and comprises the following steps: for a nonlinear affine system, an online multi-step heuristic dynamic programming method is developed. By combining the actor critic network and the experience replay technology, the learning speed is improved, and more efficient policy evaluation can be realized without relying on acceptable control conditions. In order to enhance the diversity of data in the learning process, an execution evaluation neural network structure integrating the ER technology is proposed, so that the agent can effectively explore on fewer samples, thereby reducing the dependence on training data. Finally, the superiority of the proposed algorithm is verified through simulation results, and experiments show that, under the same conditions, the ER-MSHDP algorithm has a faster learning process than the traditional HDP algorithm, and the learning time decreases exponentially with the increase of the step length.

Description

Technical Field

[0001] This invention relates to the field of optimal control, and in particular to an online multi-step heuristic dynamic programming method, medium, and device. Background Technology

[0002] Optimal control, a core area of ​​control theory, aims to improve system performance by designing optimal control strategies, showing great promise in applications such as autonomous driving, drone flight control, and industrial automation. Optimal control problems typically involve multiple constraints, such as time, resource limitations, and safety requirements, making their solution process both complex and challenging. The Bellman optimality principle provides a theoretical foundation for solving optimal control problems, but the presence of high-dimensional, nonlinear systems and uncertainties further complicates the solution process. Furthermore, solving continuous-time optimal control problems often relies on the Hamilton-Jacobi-Bellman (HJB) equations; however, analytical solutions to the HJB equations are difficult to obtain in many practical problems, prompting researchers to turn to numerical optimization and approximation methods.

[0003] To address these challenges, various studies have attempted to provide solutions. Reinforcement learning (RL) and algorithms based on approximate dynamic programming (ADP) are the main technical approaches for solving optimal control problems. RL learns optimal control policies progressively through interaction with the environment, exhibiting unique advantages, especially in environments with incomplete information and dynamic changes. Policy iteration (PI) and value iteration (VI) algorithms within the ADP framework, relying on feedback adjustments to the system state and control policy, have become two core methods in optimal control theory. Furthermore, researchers have proposed several variant algorithms, such as generalized value iteration (GVI) and multi-step policy evaluation (MPE). These methods can improve the convergence speed and robustness of control policies, playing a crucial role in the optimal control of complex systems.

[0004] Despite significant progress, these methods still face numerous unresolved issues in practical applications. First, existing reinforcement learning algorithms often require substantial training data and computational resources, posing a major bottleneck for applications with high real-time requirements. Second, while neural networks and other models can effectively improve the accuracy of control strategies, issues such as model generalization ability, overfitting, and excessively long training times still limit their practical application. More complexly, designing robust and adaptable control algorithms under various uncertainties remains an open research area. Furthermore, many current control strategies rely on offline training, resulting in insufficient adaptability to new and unknown environments. Therefore, developing more efficient and flexible online optimization control methods remains a significant challenge in optimal control research. Summary of the Invention

[0005] The present invention proposes an online multi-step heuristic dynamic programming method, device, and storage medium, which can at least solve one of the technical problems in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An online multi-step heuristic dynamic programming method includes the following steps:

[0008] S1. Construct the cost function based on the nonlinear affine system;

[0009] S2. Process the cost function using a multi-step value iteration method to obtain the extended cost function;

[0010] S3. Construct an approximate cost function for the evaluation-execution neural network and design the neural network iterative formula;

[0011] S4. Define a buffer based on experience-based replay techniques;

[0012] S5. Initialize the buffer and begin online learning to solve the control law. , and complete the dynamic programming.

[0013] Furthermore, the method for constructing the cost function based on the nonlinear affine system in step S1 of the present invention includes:

[0014] Nonlinear affine systems include:

[0015] (1)

[0016] in, , , The subscript indicates the system number. The state of the step, The subscript indicates the system number. Step control input This represents the output of the Actor network and its control input to the system. Indicates the state The corresponding control strategy obtained from the calculation;

[0017] Cost function Defined as a performance metric for measuring controller quality:

[0018] (2)

[0019] in, , and They are all positive semidefinite matrices;

[0020] Definition 1: If able to Internally stable system (1), wherein, Indicates system state The stability region of the cost function, and ensure the cost function For all The finiteness of which and exist If the above is continuous, then It is an allowable control set;

[0021] Optimal cost function Satisfying the discrete-time HJB equation:

[0022] (3)

[0023] According to (3), the optimal control input The expression is:

[0024] (4)

[0025] Assumption It is continuously differentiable, thus yielding the optimal control input;

[0026] (5)

[0027] The case for linear systems is as follows:

[0028] (6)

[0029] Consider quadratic cost function ,in .

[0030] Furthermore, the method for obtaining the extended cost function in step S2 of the present invention includes:

[0031] according to The extended cost function is defined as follows:

[0032] (7)

[0033] in It is an integer greater than 1, representing the learning step size. In multi-step value iteration, the utility function for the next few time steps is considered, and the cost function contains more system information; at the beginning of the iteration, Initialize to a non-negative value; similar to VI, multi-step value iteration is used for policy improvement;

[0034] (8)

[0035] Strategy Evaluation

[0036] (9)

[0037] when At that time, MSVI is equivalent to VI, when At that time, MSVII is equivalent to PI.

[0038] Furthermore, the method for constructing an approximate cost function for a judgment-execution neural network and designing an iterative formula for the neural network in this invention includes:

[0039] Cost function Represented as

[0040] (10)

[0041] in It is the optimal weight vector. It is the approximation error of the neural network, which is related to the number of neurons. ,because ;

[0042] Approximate cost function using neural network approximation Represented as:

[0043] (11)

[0044] in It is the optimal weight vector The estimated value;

[0045] Estimation error Defined as:

[0046] (12)

[0047] in It is based on model information and The calculated estimated future state;

[0048] Evaluation network loss function Defined as:

[0049] (13)

[0050] Then, the optimal weight vector is updated using gradient descent. The estimated value :

[0051] (14)

[0052] Learning rate estimation error Defined as:

[0053] (15)

[0054] According to the update rule (14) for evaluating the weights of the neural network, It is expressed as follows:

[0055] (16)

[0056] Neural network architecture is a classic execution-evaluation network; by constructing an execution network, Approximately

[0057] (17)

[0058] in, ,+ This represents the number of neurons in the actor's neural network. Indicates the approximation error. The estimated value is expressed as:

[0059] (18)

[0060] Based on (8) and (11), the control strategy for network approximation is implemented. for

[0061] (19)

[0062] Then, the estimation error of the control input is obtained.

[0063] (20)

[0064] Network training aims to minimize the loss function used to execute the network. Its definition is as follows:

[0065] (twenty one)

[0066] The method for adjusting the execution network weights is the same as in equation (14);

[0067] (twenty two)

[0068] Learning rate satisfy estimation error Defined as:

[0069] (twenty three).

[0070] Furthermore, in step S5 of the present invention, the control law is solved. The methods for completing dynamic programming include:

[0071] Initialize the buffer data, including the empirical replay buffer size L, Q, and Initial weights Step length Maximum time step The number of iterations per step, h;

[0072] During online learning, when , Then, from the experience pool A sample is randomly selected from the middle according to a uniform distribution, and the weights are updated alternately according to equations (14) and (22). Increasing Then, the current data sample will be added to the database; if there is detection noise, the control law will be calculated using equation (18). The system state is updated using equation (1). During this process, if the experience pool The size exceeds the preset maximum value. Then delete the experience pool. The earliest generated data; if the conditions are met. Then stop adding detection noise, and finally, increment... And continue execution until the maximum time step is reached. .

[0073] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0074] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0075] Compared with existing technologies, this invention offers the following technical advantages: For nonlinear affine systems, an online multi-step heuristic dynamic programming method is developed. By combining actor critique networks and experience replay techniques, the learning speed is improved, and more efficient policy evaluation can be achieved without relying on acceptable control conditions. To enhance the diversity of data during the learning process, an executive evaluation neural network structure incorporating ER technology is proposed, enabling the agent to explore effectively on fewer samples, thereby reducing dependence on training data. Finally, simulation results verify the superiority of the proposed algorithm. Experiments show that, under the same conditions, the ER-MSHDP algorithm has a faster learning process than the traditional HDP algorithm, and the learning time decreases exponentially with increasing step size. Attached Figure Description

[0076] Figure 1 This is a flowchart of an online multi-step heuristic dynamic programming method according to the present invention;

[0077] Figure 2 This is a graph showing the changes in weights during the training process of Example 1 of the present invention;

[0078] Figure 3 Example 1 of the proposed algorithm of this invention System status and control inputs;

[0079] Figure 4 Example 1 of the present invention: Different sizing The standard value;

[0080] Figure 5 This refers to the number of learning steps in different cases of Example 1 of the present invention;

[0081] Figure 6 This is an example of the weight changes during the training process of Example 1 of the present invention;

[0082] Figure 7 The system state and control inputs under the proposed algorithm in Example 2 of this invention;

[0083] Figure 8 This is the approximate optimal value function in Example 2 of the present invention;

[0084] Figure 9 The parameter magnitudes under different phase lengths in Example 2 of this invention;

[0085] Figure 10 This represents the number of learning steps required for convergence under different time lengths in Example 2 of this invention. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0087] like Figure 1 As shown, this embodiment discloses an online multi-step heuristic dynamic programming method, including the following steps:

[0088] S1. Construct the cost function based on the nonlinear affine system;

[0089] S2. Process the cost function using a multi-step value iteration method to obtain the extended cost function;

[0090] S3. Construct an approximate cost function for the evaluation-execution neural network and design the neural network iterative formula;

[0091] S4. Define a buffer based on experience-based replay techniques;

[0092] S5. Initialize the buffer and begin online learning to solve the control law. ;

[0093] The following provides a detailed explanation of each step:

[0094] S1. Construct the cost function based on the nonlinear affine system.

[0095] Consider a nonlinear affine system as follows:

[0096] (1)

[0097] in, , , The subscript indicates the system number. The state of the step, The subscript indicates the system number. Step control input This represents the output of the Actor network and the control input that acts on the system. Indicates the state The corresponding control strategy is calculated. For nonlinear affine systems, the objective of optimal control is to find a control strategy that both reaches the equilibrium point and minimizes the performance index. Therefore, we will use the following cost function Defined as a performance metric for measuring controller quality:

[0098] (2)

[0099] in, , and They are all positive semidefinite matrices.

[0100] Definition 1: If able to Internally stable system (1), wherein, Indicates system state The stability region of the cost function, and ensure the cost function For all The finiteness of which and exist If the above is continuous, then It is an allowable control set.

[0101] Optimal cost function Satisfying the discrete-time HJB equation:

[0102] (3)

[0103] According to (3), the optimal control input The expression is

[0104] (4)

[0105] Assumption It is continuously differentiable, thus yielding the optimal control input.

[0106] (5)

[0107] The case for linear systems is as follows:

[0108] (6)

[0109] Consider quadratic cost function ,in The problem of solving the optimal process can be equivalently represented as the problem of solving the Riccati equation, which has been extensively studied. For equation (6), The equation (3) can be obtained through numerical or iterative solutions based on the system model information. However, due to the nonlinear characteristics of equation (3), solving this equation is often challenging, and sometimes even infeasible. Iterative algorithms are one of the key tools for solving nonlinear partial differential (HJB) equations. Therefore, combining iterative algorithms with ADP and RL methods can provide a more flexible and powerful approach for solving such problems.

[0110] S2. Process the cost function using a multi-step value iteration method to obtain the extended cost function;

[0111] VI and PI are two methods for achieving optimal control. Both methods involve alternately evaluating and improving the control strategy. When the difference between adjacent cost functions is less than a certain small range, i.e. ,in A constant greater than 0 ( It varies depending on the system complexity and required accuracy. Higher accuracy requirements... The smaller the value selected, the better. We then consider that we have obtained it. and However, PI (Progressive Injection) requires iterative iteration starting from an acceptable control policy, a requirement that is challenging to meet directly, especially for nonlinear systems. However, because PI emphasizes policy evaluation, the optimal cost function is usually obtained in just a few iterations. Conversely, VI (Variable Injection) improves the policy from an initial cost function, removing the strict requirement for admissible control. However, it cannot accurately evaluate the current policy, thus its iterative efficiency is often relatively low. Multistep Value Iteration (MSVI) serves as a bridge between PI and VI, effectively compensating for the shortcomings of both. The extended cost function is defined as follows:

[0112] (7)

[0113] in This is an integer greater than 1, representing the learning step size. In multi-step value iteration, the utility function for several future time steps is considered; that is, the cost function contains more system information, which is beneficial for improving learning speed. At the beginning of the iteration, It is typically initialized to a non-negative value. Similar to VI, multi-step value iteration is used for policy improvement.

[0114] (8)

[0115] Strategy Evaluation

[0116] (9)

[0117] It is worth noting that when At that time, MSVI is equivalent to VI, when At that time, MSVII is equivalent to PI. For example... Figure 1 As shown, MSVI involves strategy improvement and strategy evaluation: predicting the future. The utility function of each step.

[0118] S3. Construct an approximate cost function for the evaluation-execution neural network and design the neural network iterative formula.

[0119] Neural networks typically consist of an input layer, hidden layers, and an output layer. The input and output layers usually correspond to the independent and dependent variables of the function to be approximated, respectively. The hidden layers primarily perform feature extraction and computation. The activation function of the hidden layer is a crucial component because it determines the network's ability to map nonlinear relationships. These activation functions often introduce nonlinearity, enabling the network to adapt to complex nonlinear data. This invention employs a multinomial activation function. Therefore, the cost function... It can be represented as

[0120] (10)

[0121] in It is the optimal weight vector. It is the approximation error of the neural network, which is related to the number of neurons. .because Generally speaking, It is difficult to obtain directly. Therefore, an approximate cost function using a neural network is employed. It can be represented as:

[0122] (11)

[0123] in It is the optimal weight vector The estimated value. Then we will estimate the error. Defined as

[0124] (12)

[0125] in It is based on model information and The calculated estimated future state.

[0126] Evaluation network loss function Defined as

[0127] (13)

[0128] Then, the optimal weight vector is updated using gradient descent. The estimated value :

[0129] (14)

[0130] Learning rate Estimation error Defined as:

[0131] (15)

[0132] According to the update rule (14) for evaluating the weights of the neural network, It can be represented as follows:

[0133] (16)

[0134] The neural network structure used in this invention is the classic actor-critic network. By constructing the actor network, It can be approximated as

[0135] (17)

[0136] in, . This represents the number of neurons in the actor's neural network. This indicates the approximation error. The estimated value can be expressed as:

[0137] (18)

[0138] Based on (8) and (11), the control strategy for network approximation is implemented. for

[0139] (19)

[0140] Then, we can obtain the estimation error of the control input.

[0141] (20)

[0142] Network training aims to minimize the loss function used to execute the network. Its definition is as follows:

[0143] (twenty one)

[0144] The method for adjusting the execution network weights is the same as that in equation (14).

[0145] (twenty two)

[0146] Learning rate satisfy Estimation error Defined as:

[0147] (twenty three)

[0148] In neural network design, the choice of activation function is a crucial step, directly impacting the model's representational power, convergence speed, and overall performance. Different activation functions possess unique characteristics, making them suitable for various tasks and network architectures. Commonly used activation functions include the sigmoid and tanh functions, both of which are continuously differentiable and therefore frequently used in adaptive dynamic programming. For an n-dimensional linear system with a given performance criterion, if the corresponding Riccati equation has a solution, Therefore, its form is a quadratic polynomial. Thus, to verify the effectiveness of ER-MSHDP, this invention employs a polynomial activation function.

[0149] S4. Define a buffer based on experience-based replay techniques;

[0150] Experience replay is a widely used technique in RL that can improve the efficiency of the learning process. Its basic concept is to store the agent's experience in a buffer, which can be reused later to reduce the correlation between consecutive observations and improve sampling efficiency. During online training, by utilizing the stored trajectory instead of relying entirely on the current state, the convergence speed can be improved. This method allows the training of neural networks to utilize experience beyond the latest policy. Therefore, it helps prevent the algorithm from getting trapped in local minima or diverging. Building on previous research, we incorporate the experience replay technique to further improve learning efficiency. We define the ER buffer as...

[0151] (twenty four)

[0152] Where L represents the buffer capacity, the specific value of which is determined through experimental debugging. This invention does not employ a priority experience replay technique, but instead performs uniform sampling of experience tuples. During each iteration of the network weights, the weights are updated using the state sampled from the buffer.

[0153] The replay buffer should be large enough to store diverse experiences, covering as many state-action pairs as possible to enhance the generalization ability of the training process. If the buffer is too small, it may lead to insufficient sample diversity, making the model prone to overfitting. While increasing the buffer size provides a more diverse sample set, it also introduces additional storage requirements and computational overhead. An excessively large buffer can consume significant amounts of memory, especially in high-dimensional environments. The cost of storing a large amount of historical experience should be commensurate with the system's hardware resources. Furthermore, an excessively large buffer increases the time required for sample retrieval, thus impacting training efficiency. Additionally, the number of experiences selected for each learning step is also a key factor affecting model performance.

[0154] S5. Initialize the buffer and begin online learning to solve the control law. Complete dynamic programming;

[0155] Initialize the buffer data, including the empirical replay buffer size L, Q, and Initial weights Step length Maximum time step The number of iterations per step, h.

[0156] During online learning, when , Then, from the experience pool A sample is randomly selected from the middle according to a uniform distribution, and the weights are updated alternately according to equations (14) and (22). Increasing Then, the current data sample will be added to the database. If probe noise is present, the control law is calculated using equation (18). The system state is updated using equation (1). During this process, if the experience pool The size exceeds the preset maximum value. Then delete the experience pool. The earliest generated data. If the conditions are met. If the noise level drops, then stop adding detection noise. Finally, increment... And continue execution until the maximum time step is reached. .

[0157] The stability is then further proven and demonstrated, with a reasonable assumption and a theorem provided.

[0158] Assumption 1: and as well as and It is bounded, that is... , , and .

[0159] Theorem 1: Vectors of the same dimension and The following inequalities must be satisfied:

[0160] (25)

[0161] Lemma 1: Assumption 1 holds. and Approximate values ​​are obtained using equations (11) and (18) respectively. Let... and Update according to equations (14) and (22). Furthermore, it can be proven that… As shown in equations (15) and (23), it is UUB.

[0162] The following is proof of the construction process of the Lyapunov function:

[0163] (26)

[0164] in The result will then be... The standard. It can be represented as

[0165] (27)

[0166] Then, according to equation (15), we can obtain the result.

[0167] (28)

[0168] According to equations (10) and (11), This can be further deduced as

[0169] (29)

[0170] According to Theorem 1, satisfy

[0171] (30)

[0172] Next, It can be represented as

[0173] (31)

[0174] Similarly, according to equation (17) and equation (18), It can be deduced as

[0175] (32)

[0176] We set . The following inequalities must be satisfied:

[0177] (33)

[0178] Therefore, if equation (34) holds, .

[0179] (34)

[0180] Meanwhile, assumption 1 is satisfied. We can obtain

[0181] (35)

[0182] If inequality (35) holds, then we get .therefore, and It's UUB. That completes the proof.

[0183] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0184] The following is a simulation example.

[0185] This invention will verify the effectiveness of an online multi-step heuristic dynamic programming method using a linear power system:

[0186] (36)

[0187] in and These represent the time constants of the governor, turbine, and generator models, respectively. Furthermore, This represents the feedback adjustment constant, while and These represent the gain constants of the generator and turbine models, respectively. Table 1 lists the parameters in (36).

[0188] Table 1 Power System Parameters

[0189]

[0190] Output frequency deviation increment Generator output and governor position value Treating it as a state variable, the incremental change in positive deviation velocity is input into the vector. Then, the sampling period is used. Discretizing the power system (36) yields the following formula:

[0191] (37)

[0192] For system (37), and Set as follows and All network weights are initially set to 0. The solution for DARE can be obtained using MATLAB 2020a. For a linear system (37), given the performance index parameters... R and P can be obtained through the "idare" library in Matlab. The function yields:

[0193] (38)

[0194] Related optimal strategy , for We set the activation function to and Then, the optimal weights can be obtained as follows:

[0195] (39)

[0196] and (40)

[0197] To implement the ER-MSHDP online algorithm, we selected relevant parameters and listed them in Table 2.

[0198] Table 2. Relevant parameters of the ASVI ensemble algorithm

[0199]

[0200] The PE condition ensures that the agent explores sufficiently diverse state-action pairs during interactions, preventing the system from converging to local optima or unvisited states. Furthermore, in ADP, many theoretical convergence results depend on the satisfaction of the PE condition. For example, when using gradient descent or other learning algorithms, the persistence of incentives guarantees the accuracy of model parameter or value function estimates, thus ensuring convergence to the optimal solution. This invention designs the PE condition. The detection noise is terminated after the first 500 steps of the total sampling period. To ensure the agent is adequately trained, we introduce detection noise, i.e., continuous excitation, into the system. It is worth noting that the magnitude of the detection noise needs to be adjusted based on the relevant system or experience. Too much noise may cause the system to diverge, while too little may lead to undertraining. Under continuous excitation, and The trajectories of change are as follows: Figure 2 and Figure 3 As shown, the initial state is defined as The trained weights converge to... and It is approximately equal to the theoretical value.

[0201] In addition, we designed four experiments with different durations. For example... Figure 4 The figure shows the convergence of weights under different step lengths. In fact, by adjusting some parameters and the magnitude of the probe noise, the weights can converge to the optimal weights under different step lengths. To verify only the effect of step length on iteration efficiency, other parameters were set consistently. Experimental results show that, under certain conditions, as the step length increases, the learning efficiency of the neural network improves, and the error between the estimated weights after training and the optimal weights decreases. Experimental results also show that, under sufficient PE conditions, neural network weights with different step lengths can converge to near-optimal values.

[0202] In addition, we designed experimental comparisons of experienced and unexperienced responses at different time lengths. For example... Figure 5 As shown, using experience replay technology can effectively improve data utilization and accelerate learning.

[0203] Consider the following nonlinear system:

[0204] (41)

[0205] in and Utility function The parameters in the text are selected as follows: and To implement the online ER-MSHDP algorithm, we set the activation function to... and In this case, the step size, empirical replay buffer length, and sample size are chosen to be 10, 8, and 5, respectively. It's worth noting that the settings for these parameters are not unique and can be adjusted based on the system model and practical experience. In the proposed online algorithm, and neural network weights, They are updated simultaneously. For example... Figure 6 , Figure 7 As shown, the weights and the state under probe noise and control input are... .like Figure 8 As shown, the approximate optimal cost function .like Figure 9 As shown, the convergence of the critical network weights is affected by different phase lengths. Furthermore, under a specified acceptable policy, such as... Figure 10 The diagram illustrates the relationship between step size and the number of iterations required. It is noteworthy that when the step size is small, the number of iterations required decreases exponentially as the step size increases.

[0206] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0207] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0208] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the online multi-step heuristic dynamic programming methods described in the above embodiments.

[0209] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0210] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0211] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0212] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0213] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online multi-step heuristic dynamic programming method, characterized in that, Includes the following steps: S1. Construct the cost function based on the nonlinear affine system; S2. Process the cost function using a multi-step value iteration method to obtain the extended cost function; S3. Construct an approximate cost function for the evaluation-execution neural network and design the neural network iterative formula; S4. Define a buffer based on experience-based replay techniques; S5. Initialize the buffer and begin online learning to solve the control law. Complete dynamic programming; The method for obtaining the extended cost function in step S2 includes: according to The extended cost function is defined as follows: (7) in It is an integer greater than 1, representing the learning step size. In multi-step value iteration, the utility function for the next few time steps is considered, and the cost function contains more system information; at the beginning of the iteration, Initialize to a non-negative value; similar to VI, multi-step value iteration is used for policy improvement; (8) Strategy Evaluation (9) when At that time, MSVI is equivalent to VI, when At that time, MSVII is equivalent to PI; Solving the control law in step S5 The methods for completing dynamic programming include: Initialize the buffer data, including the empirical replay buffer size L, Q, and Initial weights Step length Maximum time step The number of iterations per step, h; During online learning, when , Then, from the experience pool A sample is randomly selected from the middle according to a uniform distribution, and the weights are updated alternately according to equations (14) and (22). Increasing Then, the current data sample will be added to the database; if there is detection noise, the control law will be calculated using equation (18). The system state is updated using equation (1). During this process, if the experience pool The size exceeds the preset maximum value. Then delete the experience pool. The earliest generated data; if the conditions are met. Then stop adding detection noise, and finally, increment... And continue execution until the maximum time step is reached. .

2. The online multi-step heuristic dynamic programming method according to claim 1, characterized in that, Step S1, the method for constructing the cost function based on the nonlinear affine system, includes: Nonlinear affine systems include: (1) in, , , The subscript indicates the system number. The state of the step, The subscript indicates the system number. Step control input This represents the output of the Actor network and its control input to the system. Indicates the state The corresponding control strategy obtained through calculation; Cost function Defined as a performance metric for measuring controller quality: (2) in, , and They are all positive semidefinite matrices; like able to Internally stable system (1), wherein, Indicates system state The stability region of the cost function, and ensure the cost function For all The finiteness of which and exist If the above is continuous, then It is an allowable control set; Optimal cost function Satisfying the discrete-time HJB equation: (3) According to (3), the optimal control input The expression is: (4) Assumption It is continuously differentiable, thus yielding the optimal control input; (5) The case for linear systems is as follows: (6) Consider quadratic cost function ,in .

3. The online multi-step heuristic dynamic programming method according to claim 1, characterized in that, The methods for constructing an approximate cost function for a judgment-execution neural network and designing iterative formulas for the neural network include: Cost function Represented as (10) in It is the optimal weight vector. It is the approximation error of the neural network, which is related to the number of neurons. ,because ; Approximate cost function using neural network approximation Represented as: (11) in It is the optimal weight vector The estimated value; Estimation error Defined as: (12) in It is based on model information and The calculated estimated future state; Evaluation network loss function Defined as: (13) Then, the optimal weight vector is updated using gradient descent. The estimated value : (14) Learning rate estimation error Defined as: (15) According to the update rule (14) for evaluating the weights of the neural network, It is expressed as follows: (16) Neural network architecture is a classic execution-evaluation network; by constructing an execution network, Approximately (17) in, ,+ This represents the number of neurons in the actor's neural network. Indicates the approximation error. The estimated value is expressed as: (18) Based on (8) and (11), the control strategy for network approximation is implemented. for (19) Then, the estimation error of the control input is obtained. (20) Network training aims to minimize the loss function used to execute the network. Its definition is as follows: (21) The method for adjusting the execution network weights is the same as in equation (14); (22) Learning rate satisfy estimation error Defined as: (23)。 4. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the processor performs the method as described in any one of claims 1 to 3.

5. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 3.

Images