An approximate dynamic programming control method based on support vector regression

Abstract

The present invention provides an approximate dynamic programming control method based on support vector regression, which relates to the technical field of approximate dynamic programming. The present invention proposes a method for optimal control problems of continuous nonlinear systems. The method integrates the continuous-time IRL Bellman equation and support vector regression (SVR) technology. The use of IRL allows the algorithm to estimate the value function through integral terms when some system dynamics are unknown, providing a powerful tool for online learning. Furthermore, the present invention applies SVR to ADP for the first time, transforming the complex function approximation problem into a convex optimization problem, ensuring the existence of the optimal solution and the stability of the algorithm. By adopting HESVR, the present invention's algorithm can more accurately approximate the cost function and control strategy. The least squares method is used to adjust the HESVR weights, improving the efficiency and accuracy of the algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of approximate dynamic programming, in particular to an approximate dynamic programming control method based on support vector regression. Background Art

[0002] Optimal control, a key branch of control theory, aims to identify control strategies that maximize or minimize system performance under specific constraints. Since the mid-20th century, continuous advances in space technology, automation, and various engineering applications have significantly fueled the development of optimal control theory, playing a vital role in numerous fields. From optimizing spacecraft trajectories to fine-tuning industrial processes, optimal control strategies are widely used.

[0003] In the energy sector, especially when it comes to large-scale or high-dimensional energy systems such as smart grids, wind farms, or solar power generation systems, traditional control strategies often struggle to cope with their complexity and dynamic changes. For example, smart grids need to respond in real time to changes in supply and demand, power quality control, and the volatility of renewable energy. These systems are often highly nonlinear and uncertain, placing strict requirements on the response time of control algorithms. With the increasing complexity of modern systems, especially the emergence of nonlinear systems, it has become difficult to use traditional linear optimal control methods when dealing with problems. Due to the inherently complex dynamic characteristics of nonlinear systems, the design of control strategies has become extremely difficult. The Hamilton-Jacobi-Bellman (HJB) equation is a nonlinear partial differential equation whose solution can provide the optimal feedback control law for the system. However, since the analytical solution of the HJB equation is often difficult to obtain in high-dimensional spaces or complex dynamic systems, this poses a major obstacle to solving the optimal control problem.

[0004] To address this challenge, researchers have developed a series of approximate analytical methods: Fuzzy logic control utilizes fuzzy set theory to handle system uncertainty and ambiguity, approximating optimal control strategies through fuzzy rules. Neural network control leverages its powerful nonlinear mapping capabilities to learn control strategies from training data. ADP, a more direct numerical method, solves the optimal control strategy by iteratively approximating the solution to the HJB equation.

[0005] The ADP method, based on the principles of reinforcement learning and dynamic programming, progressively optimizes control strategies through policy iteration or value iteration algorithms. Policy iteration algorithms begin with a good initial policy and, through successive policy evaluation and refinement steps, gradually approach the optimal policy. Value iteration algorithms, on the other hand, start with an initial value function and, through iterative updates, gradually improve the estimate of the value function until convergence to the optimal value function. These methods have achieved some success in discrete-time and certain types of continuous-time systems, but they still face challenges such as slow convergence and high computational costs when dealing with high-dimensional or highly dynamic systems. Furthermore, existing approximate analytical methods face other challenges in practical applications. For example, while fuzzy logic control can handle uncertainty, its design often relies on empirical experience, making it difficult to adapt to rapidly changing or unforeseen system behavior. While neural network control offers good generalization capabilities, its training requires a large amount of data, and the choice of network architecture significantly impacts performance. While the ADP method is theoretically attractive, practical applications still require further research on issues such as selecting an appropriate iterative algorithm, designing an approximator, and ensuring convergence and stability.

[0006] Optimal control is a core topic in control theory, and the exploration of its solution strategies has always been a focus of engineering and scientific research. Within this field, methods such as dynamic programming, fuzzy logic control, neural network control, and ADP have been proposed and applied to various control scenarios. However, in the energy context, especially systems involving renewable energy, model uncertainty is often a problem. The unpredictability of wind speed and solar radiation can lead to fluctuations in energy output, and traditional control methods, often based on deterministic models, are unable to adapt to this uncertainty.

[0007] Dynamic programming is a classic method for solving optimal control strategies. It solves complex problems by breaking them down into a series of simpler problems. As system size increases, such as with the expansion of power grids or the integration of multiple energy sources, the dimensionality of the state space increases dramatically. This causes traditional dynamic programming-based methods to face the "curse of dimensionality," leading to exponential growth in computing resources and storage space requirements. Dynamic programming algorithms typically require comprehensive knowledge of the system, which is often difficult to meet in practical applications.

[0008] Fuzzy logic control, as an effective means of handling uncertainty, uses fuzzy set theory to describe system states and control rules, thereby addressing uncertainty and ambiguity. While fuzzy logic control performs well in certain nonlinear or difficult-to-model systems, its generalization capabilities are limited, making it difficult to adapt to system behaviors that exceed the designer's expectations. Furthermore, the design of fuzzy logic controllers often relies on expert experience, which limits their application in a wider range of fields.

[0009] Neural network control leverages the powerful nonlinear mapping capabilities of neural networks to approximate complex functional relationships, thereby achieving effective system control. However, neural network control suffers from insufficient robustness and model transparency, limiting its application in many safety and reliability-critical applications. Furthermore, neural network training typically requires extensive data, and the choice of network structure significantly impacts control performance, complicating design and application.

[0010] ADP, a numerical method that combines reinforcement learning and dynamic programming, solves optimal control problems by iteratively approximating the HJB equation. The policy iteration (PI) algorithm in the ADP method begins with a good initial policy and approaches the optimal policy through successive policy evaluation and improvement steps. However, the PI algorithm can have difficulty converging to the global optimal solution, especially in non-convex situations or when multiple local optima exist. Furthermore, the PI algorithm can converge slowly and is sensitive to the choice of initial policy.

[0011] As an alternative to the ADP method, value iteration algorithms can approach the optimal value function by iteratively updating the value function starting from arbitrary initial values. However, value iteration algorithms can face stability and convergence issues, especially in continuous-time or high-dimensional systems. Furthermore, value iteration algorithms can require a large number of iterations in practical applications, which undoubtedly increases computational costs.

[0012] It is worth noting that existing methods also have limitations in terms of computing resources and real-time performance. As the scale of the system increases, the required computing resources may make the method infeasible, especially in real-time systems that require fast response. Real-time systems have strict requirements on the response time of the control algorithm, and existing methods may have difficulty meeting these requirements due to their high computational complexity. Existing methods also face the challenges of model uncertainty and external disturbances in practical applications. In actual engineering applications, system models often have uncertainties and are easily disturbed by the external environment. Existing methods may perform poorly in dealing with these uncertainties and disturbances, and further improvements are needed to improve their robustness. Summary of the Invention

[0013] In view of the shortcomings of the prior art, the present invention aims to propose an approximate dynamic programming control method based on support vector regression, comprising:

[0014] Step 1: Construct the state space of a nonlinear dynamic system with a time-invariant affine structure. The state space of the nonlinear dynamic system is expressed as:

[0015] ;

[0016] in, t Indicates the moment, represents the state space of the nonlinear dynamic system, Represents a nonlinear dynamic system in t The state vector at time t, represents the nonlinear drift term, represents the control gain matrix, express t The control input vector at time t;

[0017] Step 2: Based on the state space of the nonlinear dynamic system, construct the formula of the value function and the control strategy based on the continuous-time iterative algorithm;

[0018] The formula of the value function is expressed as:

[0019] ;

[0020] in, Indicates in i At the +1 iteration, the nonlinear dynamic system has a state vector The state value at , that is, the terminal state value, T represents the transpose of the matrix, Represents a nonlinear dynamic system in The state vector at time t, Indicates that at the i-th iteration, the nonlinear dynamic system is in state The optimal control strategy at , R is the symmetric positive definite control weight matrix, Q is the state positive definite function;

[0021] The control strategy is expressed as follows:

[0022] ;

[0023] in, Indicates the i Iterations, the nonlinear dynamic system in the state vector The optimal control strategy at Represented in state space The set of admissible control strategies on , For the preset time interval, T represents transpose, Represents a nonlinear dynamic system in The state vector at time t, express The control input vector at time , Indicates in i At the iteration, the nonlinear dynamic system has a state vector The state value of the location;

[0024] Step 3: According to the formula of the value function and the formula of the control strategy, determine the final optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system.

[0025] Optionally, step 3 specifically includes iterating based on the formula of the value function and the formula of the control strategy to obtain the final optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system, specifically including the following steps:

[0026] Step A1: Set initial state value , set the initial iteration number i=0, use the initial iteration number as the current iteration number, and use the initial state value as the state value of the current iteration number ;

[0027] Step A2: Set the current iteration state value Substitute into the control strategy formula and calculate the optimal control strategy for the current number of iterations ;

[0028] Step A3: The optimal control strategy for the current number of iterations Substitute the formula of the value function to calculate the state value of the next iteration of the current iteration number ;

[0029] Step A4: Calculate the first difference , specifically implemented through the following formula:

[0030] ;

[0031] Step A5: Determine the first difference Is it less than the preset threshold ε? When it is less than the preset threshold ε, As the final state value, the optimal control strategy for the current number of iterations is As the final optimal control strategy, in the first difference If the number of iterations is less than or equal to the preset number, the current number of iterations is judged to be less than or equal to the preset number. If the current number of iterations is less than or equal to the preset number, As the final state value, the optimal control strategy for the current number of iterations is As the final optimal control strategy, when the current number of iterations is greater than the preset number, the current number of iterations is increased by one as the new current number of iterations, and the process returns to step 3.2.

[0032] Optionally, step 3 specifically includes improving the formula of the value function and the formula of the control strategy, and then iterating based on the improved formula to obtain the final optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system, specifically including the following steps:

[0033] Step B1: Based on the hard ε-bond support vector regression machine HESVR, the formula of the value function and the formula of the control strategy are improved to obtain the HESVR-based value function formula and the HESVR-based control strategy formula;

[0034] Step B2: Based on the least squares method, the value function formula based on HESVR is processed to obtain the weight formula of HESVR;

[0035] Step B3: According to the HESVR weight formula, the HESVR-based value function formula and the HESVR-based control strategy formula, the optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system are calculated through an iterative algorithm.

[0036] Optionally, the value function formula based on HESVR in step B1 is expressed as:

[0037] ;

[0038] in, is the state value based on HESVR, For the i At the iteration, the nonlinear dynamic system has a state vector Approximate state value, In state The nonlinear basis function vector at .

[0039] Optionally, the control strategy formula based on HESVR in step B1 is expressed as:

[0040] ;

[0041] in, represents the optimal control strategy based on HESVR, is the inverse matrix of the control weight, is the system dynamic gradient, For the i The gradient of the value function of the iteration, Indicates that the status x ( t ), the transpose of the gradient matrix of the nonlinear basis function vector at For the i The weight vector obtained by HESVR training in the iteration.

[0042] Optionally, the weight formula of HESVR in step B2 is expressed as:

[0043] ;

[0044] in,

[0045] ；

[0046] ;

[0047] in, In state The nonlinear basis function vector at ， In state The nonlinear basis function vector at , M is the number of sampling points in the least squares method, In state The nonlinear basis function vector at , From the state x ( t ) to execute the control strategy exist The total trajectory cost in time, From the state Start and execute control strategy exist The total trajectory cost in time.

[0048] Optionally, step B3 specifically includes:

[0049] Step B3.1: Set initial HESVR weights , set the initial state value , set the initial iteration number i=0, use the initial iteration number as the current iteration number, and use the initial HESVR weight as the current HESVR weight , take the initial state value as the state value of the current iteration ;

[0050] Step B3.2: Determine whether the current number of iterations is greater than the preset number , when the current number of iterations is greater than the preset number In the case of , execute step 3.3.9, and if the current number of iterations is not greater than the preset number If yes, proceed to step 3.3.3;

[0051] Step B3.3: Set HESVR weights Substitute into the HESVR-based control strategy formula to calculate the optimal control strategy based on HESVR for the current number of iterations ;

[0052] Step B3.4: Obtain the state vector using the Runge-Kutta algorithm ;

[0053] Step B3.5: Transform the state vector Substitute it into the HESVR weight formula to calculate the updated HESVR weight;

[0054] Step B3.6: Substitute the updated HESVR weight into the HESVR-based value function formula to calculate the HESVR-based state value for the next iteration of the current iteration. ;

[0055] Step B3.7: Calculate the second difference , specifically implemented through the following formula:

[0056] ;

[0057] Step B3.8: Determine the second difference Is it greater than the preset threshold ε? If the difference is greater than the preset threshold ε, execute step 3.3.9 and If the value is not greater than the preset threshold ε, the current number of iterations is incremented by one and the new number of iterations is returned to step 3.3.2.

[0058] Step B3.9: As the final state value, the optimal control strategy based on HESVR for the current number of iterations is As the final optimal control strategy.

[0059] The beneficial effects of adopting the above technical solution are:

[0060] The present invention proposes a method for the optimal control problem of continuous nonlinear systems. The method combines the continuous-time IRL Bellman equation and support vector regression (SVR) technology to form a novel numerical iterative method. The use of IRL allows the algorithm to estimate the value function through integral terms when the dynamics of some systems are unknown, providing a powerful tool for online learning. In addition, the present invention applies SVR to ADP for the first time, transforming the complex function approximation problem into a convex optimization problem, ensuring the existence of the optimal solution and the stability of the algorithm. By adopting HESVR, the algorithm can more accurately approximate the cost function and control strategy, and at the same time use the least squares method to adjust the weights of HESVR, thereby improving the efficiency and accuracy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic flow chart of an approximate dynamic programming control method based on support vector regression in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0063] To address the challenges of existing technologies, the present invention aims to develop an innovative control strategy to effectively solve the optimal control problem of continuous-time nonlinear systems. Traditional control methods often struggle to cope with the system complexity and dynamic characteristics, particularly in large-scale smart grids or complex energy networks integrating multiple energy sources. To address this issue, the present invention employs a hard-ε-bond support vector regression (HESVR) machine to approximate the cost function and control strategy of continuous-time systems.

[0064] As an advanced regression technology, HESVR has shown significant advantages in dealing with cost-benefit analysis and control decisions of energy systems due to its superior generalization ability and robustness. By introducing HESVR, the present invention can accurately capture the cost function and control strategy of the system, and maintain a high-precision approximation effect even in the face of highly nonlinear system behavior. Furthermore, the present invention proposes a new numerical iterative method by combining SVR technology with the ADP algorithm. As an approximate method for dynamic programming, the iterative characteristics of the ADP algorithm combined with the approximation ability of HESVR form a powerful tool for solving the optimal control problem of continuous-time nonlinear systems. This method not only improves the approximation accuracy of the optimal control strategy, but also gradually optimizes the control strategy through an iterative process until it reaches the optimal state.

[0065] The numerical iteration method proposed in the present invention pays special attention to convergence and stability. Convergence ensures that the algorithm can find a satisfactory control strategy within a limited number of iterations, and stability means that the obtained control strategy can remain effective even when the energy system changes dynamically or there are external interferences. This dual focus on convergence and stability makes the present invention more reliable and practical in practical applications, and can be widely used in different types of energy systems, including but not limited to power systems, energy production, distribution and storage systems. Whether in terms of improving energy efficiency, optimizing energy distribution, or promoting the integration of renewable energy, the present invention can provide an effective optimal control strategy to promote the intelligent and sustainable development of energy systems.

[0066] Specifically, the present invention provides an approximate dynamic programming control method based on support vector regression, combined with Figure 1 , which may include the following steps:

[0067] Step 1: Construct a state space of a nonlinear dynamic system with a time-invariant affine structure, wherein the nonlinear dynamic system may be a mechanical system. The state space of the nonlinear dynamic system is expressed as:

[0068] ;

[0069] in, t Indicates the moment, represents the state space of the nonlinear dynamic system, , represents the system state vector (such as the position / speed of the mechanical system, the voltage / frequency of the power grid, etc.), Represents a nonlinear dynamic system in t The state vector at time t, represents the nonlinear drift term, where , represents the control gain matrix, where , express t The control input vector at time , , where m is the dimension of the control output, representing the control input vector (such as motor torque, reactive compensation device output, etc.);

[0070] in, In the compact set The Lipschitz continuity condition is satisfied, where n is the dimension of the state variable, and They represent the nonlinear drift term and the control gain matrix, respectively, describing the natural dynamics of the system and characterizing the control ability of the actuator, and there is an equilibrium point (Right now ). Assume that the system satisfies the controllability condition, that is, there is at least one continuous control law , can be Achieve asymptotic stability of the closed-loop system.

[0071] Step 2: Based on the state space of the nonlinear dynamic system, construct the formula of the value function and the control strategy based on the continuous-time iterative algorithm;

[0072] The formula of the value function is expressed as:

[0073] ;

[0074] in, Indicates in i At the +1 iteration, the nonlinear dynamic system has a state vector The state value at , that is, the terminal state value, T represents the transpose of the matrix, Represents a nonlinear dynamic system in The state vector at time t, Indicates that at the i-th iteration, the nonlinear dynamic system is in state The optimal control strategy at , R is the symmetric positive definite control weight matrix, R is used to represent the control weight, Q is the state positive definite function, Q is used to represent the state weight;

[0075] The control strategy is expressed as follows:

[0076] ;

[0077] in, Indicates the i Iterations, the nonlinear dynamic system in the state vector The optimal control strategy at Represented in state space The set of admissible control strategies on , For the preset time interval, T represents transpose, Represents a nonlinear dynamic system in The state vector at time t, express The control input vector at time , Indicates in i At the iteration, the nonlinear dynamic system has a state vector The state value of the location;

[0078] The formulas for the value function and the control strategy are derived and constructed through the following steps:

[0079] The infinite time domain performance index of the system is defined as:

[0080] ;

[0081] in, is the initial state of the system at the initial time t=0, is the state positive definite function, which represents the state weight, is a symmetric positive definite control weight matrix, and represents the control weights. The core of this optimal control problem is to find a control strategy that satisfies the following conditions: 1. Ensure the asymptotic stability of the closed-loop system; 2. Make the performance index reach the global minimum.

[0082] For admissible control strategies , its mathematical characteristics should satisfy: 1. The control law is Continuously differentiable; 2. Satisfy at the equilibrium point 3. Value Function limited.

[0083] Based on the value function, given an admissible strategy , for any initial time t and finite time interval T, the value function (i.e., the performance index of the system) can be constructed as a time domain rolling optimization form:

[0084] ;

[0085] in, represents the state tracking cost, represents the cost of controlling energy consumption, represents the terminal cost, Indicates zero initial condition, the integral term is any moment in the rolling time domain (dynamic time variable), T represents the vector / matrix transpose (e.g. is the transpose of the state vector).

[0086] This formula essentially constructs the Bellman equation in the framework of continuous-time inverse reinforcement learning. Its core feature is the combination of finite-time domain cost and terminal cost. According to the Bellman optimality principle, the optimal value function must satisfy the recursive minimization condition: ;

[0087] in, represents the state tracking cost, represents the cost of controlling energy consumption, represents the optimal terminal cost.

[0088] The corresponding derived explicit optimal control strategy can be expressed in a dual optimization form:

[0089] ;

[0090] ;

[0091] In order to obtain the optimal solution of the above three formulas, the continuous time value iteration algorithm will iterate in the following two steps. Given the initial value function , for the number of iterations Algorithm execution:

[0092] (Strategy Improvement) For the convenience of distinction, the present invention expresses the control law as follows:

[0093] ;

[0094] in, Indicates that at the i-th iteration, the system is in state The optimal control strategy at Consistent definitions.

[0095] represents the finite time domain cost, Represents the terminal cost of the i-th iteration represents the inverse control weight matrix, represents the system dynamic gradient, represents the value function gradient.

[0096] (Value Update) The value function estimation method is:

[0097] ;

[0098] in, Represents the rolling optimization cost Represents the previous iteration value function, and then we can deduce:

[0099] ;

[0100] in, represents the actual trajectory cost, Represents a terminal state value.

[0101] Change the integral range of the above formula from [t, t+ ] to [t, ], then there is no terminal state value, and the final formula for the value function is:

[0102] .

[0103] Step 3: According to the formula of the value function and the formula of the control strategy, determine the final optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system.

[0104] In the first implementation, step 3 specifically includes iterating based on the formula of the value function and the formula of the control strategy to obtain the final optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system, which specifically includes the following steps:

[0105] Step A1: Set initial state value , set the initial iteration number i=0, use the initial iteration number as the current iteration number, and use the initial state value as the state value of the current iteration number ;

[0106] Step A2: Set the current iteration state value Substitute into the control strategy formula and calculate the optimal control strategy for the current number of iterations ;

[0107] Step A3: The optimal control strategy for the current number of iterations Substitute the formula of the value function to calculate the state value of the next iteration of the current iteration number ;

[0108] Step A4: Calculate the first difference , specifically implemented through the following formula:

[0109] ;

[0110] Step A5: Determine the first difference Is it less than the preset threshold ε? When it is less than the preset threshold ε, As the final state value, the optimal control strategy for the current number of iterations is As the final optimal control strategy, in the first difference If the number of iterations is less than or equal to the preset number, the current number of iterations is judged to be less than or equal to the preset number. If the current number of iterations is less than or equal to the preset number, As the final state value, the optimal control strategy for the current number of iterations is As the final optimal control strategy, when the current number of iterations is greater than the preset number, the current number of iterations is increased by one as the new current number of iterations, and the process returns to step 3.2.

[0111] Among them, the initial value function selection conditions are: ,in Represents the set of Lyapunov functions. Convergence guarantee condition: When the system satisfies , .

[0112] In the first implementation, the idea of ​​the IRL method is used. In each iteration, the newly updated function Can be expressed as a time interval The cost integral term on the function obtained in the previous iteration step sum.

[0113] In the second implementation, step 3 specifically involves improving the formulas for the value function and the control strategy, and then iterating based on the improved formulas to obtain the final optimal control strategy and final state value of the nonlinear dynamic system. The second implementation is an innovative numerical optimization method specifically designed to adjust the weights of HESVR. When faced with complex nonlinear systems or when online learning is required, the present invention uses an algorithm that combines SVR to achieve efficient approximation. This algorithm is the core technology for implementing key steps in continuous-time ADP, aiming to accurately approximate the value function and control strategy of complex systems. The second implementation can be seen as an extension of the first, and the introduction of SVR addresses the traditional method's reliance on model accuracy. Both are based on the value iteration framework, but the second implementation achieves broader applicability through machine learning.

[0114] Specifically, the second implementation method includes the following steps:

[0115] Step B1: Improve the value function formula and the control strategy formula based on the hard ε-bond support vector regression machine (HESVR) to obtain the HESVR-based value function formula and the HESVR-based control strategy formula;

[0116] Among them, the value function formula based on HESVR is expressed as:

[0117] ;

[0118] in, is the state value based on HESVR, For the i At the iteration, the nonlinear dynamic system has a state vector Approximate state value, In state The nonlinear basis function vector at .

[0119] Among them, the control strategy formula based on HESVR is expressed as:

[0120] ;

[0121] in, represents the optimal control strategy based on HESVR, is the inverse matrix of the control weight, is the system dynamic gradient, is the gradient of the value function at the i-th iteration, Indicates that the status x ( t ), the transpose of the gradient matrix of the nonlinear basis function vector at is the weight vector obtained through HESVR training in the i-th iteration.

[0122] Step B2: Based on the least squares method, the value function formula based on HESVR is processed to obtain the weight formula of HESVR;

[0123] Among them, the weight formula of HESVR is expressed as:

[0124] ;

[0125] in,

[0126] ;

[0127] ;

[0128] in, In state The nonlinear basis function vector at , In state The nonlinear basis function vector at , M is the number of sampling points in the least squares method, In state The nonlinear basis function vector at , From the state x ( t ) to execute the control strategy exist The total trajectory cost in time, From the state Start and execute control strategy exist The total trajectory cost in time.

[0129] Specifically, the value function formula of HESVR is processed to obtain the weight formula of HESVR, including the following steps:

[0130] Since the value iteration algorithm based on HESVR To start, set the initial value to , control strategy It will be approximated by the control strategy formula based on HESVR, It will be evaluated by the value function formula based on HESVR, which means that only one HESVR is used in the approximation process. In each iteration, the HESVR weight It is adjusted in the value function formula based on HESVR, and the training sample is known to be .

[0131] Notice The rotation is actually looking for the approximate optimal solution of HESVR with an error of ε. The least squares format is used in the HESVR tuning process, which requires considering the system dynamics at the same time. and According to the value function formula based on HESVR, we have:

[0132] ;

[0133] in, To start from state x(t), execute the control strategy exist The total trajectory cost in time.

[0134] In the i In the iterations, keep the strategy Unchanged, sample M points from the state space, the sampling time interval t is the same, where the parameter M should satisfy , making can be identified, that is, in the least squares sense, the weight update of HESVR is , where:

[0135] ;

[0136] ;

[0137] Define the system dynamics parameters f, g and the cost function matrices Q, R; integration time interval ; The number of samples per iteration M; The initial state vector ; Nonlinear mapping function ; Iteration termination criteria ; Maximum number of iterations ; Approximate optimal cost function sequence ; Approximate optimal control strategy sequence .

[0138] Step B3: According to the HESVR weight formula, the HESVR-based value function formula and the HESVR-based control strategy formula, the optimal control strategy of the nonlinear dynamic system and the final state value of the nonlinear dynamic system are calculated through an iterative algorithm.

[0139] Step B3.1: Set initial HESVR weights , set the initial state value , set the initial iteration number i=0, use the initial iteration number as the current iteration number, and use the initial HESVR weight as the current HESVR weight , take the initial state value as the state value of the current iteration ;

[0140] Step B3.2: Determine whether the current number of iterations is greater than the preset number , when the current number of iterations is greater than the preset number In the case of , execute step 3.3.9, and if the current number of iterations is not greater than the preset number If yes, proceed to step 3.3.3;

[0141] Step B3.3: Set HESVR weights Substitute into the HESVR-based control strategy formula to calculate the optimal control strategy based on HESVR for the current number of iterations ;

[0142] Step B3.4: Obtain the state vector using the Runge-Kutta algorithm ;

[0143] Step B3.5: Transform the state vector Substitute it into the HESVR weight formula to calculate the updated HESVR weight;

[0144] Step B3.6: Substitute the updated HESVR weight into the HESVR-based value function formula to calculate the HESVR-based state value for the next iteration of the current iteration. ;

[0145] Step B3.7: Calculate the second difference , specifically implemented through the following formula:

[0146] ;

[0147] Step B3.8: Determine the second difference Is it greater than the preset threshold ε? If the difference is greater than the preset threshold ε, execute step 3.3.9 and If the value is not greater than the preset threshold ε, the current number of iterations is incremented by one and the new number of iterations is returned to step 3.3.2.

[0148] Step B3.9: As the final state value, the optimal control strategy based on HESVR for the current number of iterations is As the final optimal control strategy.

[0149] The solution proposed in this invention includes two implementation methods. The first implementation method is responsible for the value iteration process based on the continuous-time system, while the second implementation method optimizes the HESVR weights through the least squares method. The second implementation method is an engineering extension of the first implementation method. The two work together to combine the theoretical framework with machine learning tools, solving the computational bottleneck in the actual system and providing an efficient and stable numerical solution for solving the optimal control problem of continuous-time nonlinear systems.

[0150] The present invention significantly improves the computational efficiency of the solution process by transforming the optimal control problem into a convex optimization problem, effectively reducing the dependence on computing resources when dealing with large-scale or complex systems. This transformation not only optimizes the calculation process, but also ensures the existence and accessibility of the global optimal solution through the application of convex optimization theory, thereby improving the quality and reliability of the solution. Compared with traditional optimal control methods, the present invention achieves significant simplification in the design process of the control strategy. By introducing advanced mathematical programming technology, the present invention reduces the dependence on complex mathematical modeling and control theory, making the design of the control strategy more intuitive and convenient, and suitable for a variety of applications in energy systems, such as load forecasting of smart grids, optimization of energy distribution, control strategies for renewable energy integration, and management of energy storage systems. Guided by simulation results, the present invention provides a clear path for the conversion between theoretical algorithms and actual energy control systems, greatly reducing the difficulty from theoretical research to engineering application and accelerating the speed of technology transformation.

[0151] Another advantage of the present invention is its wide applicability and flexibility. It is not limited to a specific system type or structure, but can be flexibly applied to different systems, including linear systems, nonlinear systems, discrete-time systems, and continuous-time systems. This versatility makes the present invention have potential application value in multiple fields and industries, such as aerospace, robotics, intelligent manufacturing, economic system optimization, etc. This adaptability not only improves the practicality of the algorithm, but also provides energy system operators with more options and flexibility when facing different control challenges, thereby better responding to fluctuations in the energy market and policy changes, and achieving efficient, stable and optimized operation of the energy system.

[0152] The above description is merely an illustration of the preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. An approximate dynamic programming control method based on support vector regression, characterized in that: include: Step 1: Construct a state space of a power grid with a time-invariant affine structure. The state space of the power grid is expressed as: Where t represents the time, represents the state space of the power grid, x(t) represents the voltage or frequency of the power grid at time t, f(x(t)) represents the nonlinear drift term, g(x(t)) represents the control gain matrix, and u(t) represents the motor torque or reactive compensation device output at time t; Step 2: Based on the state space of the power grid, construct the formula of the value function and the formula of the control strategy based on the continuous-time iterative algorithm; The formula of the value function is expressed as: Among them, V i+1 (x(t)) represents the state value of the grid at the grid voltage or grid frequency x(t) at the i+1th iteration, that is, the terminal state value, T represents the transpose of the matrix, x(τ) represents the grid voltage or grid frequency at time τ, v i (x(τ)) represents the optimal control strategy of the power grid at state x(τ) at the i-th iteration, R is the symmetric positive definite control weight matrix, and Q is the state positive definite function; The control strategy is expressed as follows: Among them, v i (x(t)) represents the optimal control strategy of the power grid at the grid voltage or grid frequency x(t) in the i-th iteration, U(Ω) represents the set of permissible control strategies in the state space Ω, T1 is the preset time interval, T represents the transpose, x(τ) represents the grid voltage or grid frequency at time τ, u(τ) represents the motor torque or reactive compensation device output at time τ, V i (x(t+T1)) represents the state value of the grid at the grid voltage or grid frequency x(t+T1) at the i-th iteration; Step 3: According to the formula of the value function and the formula of the control strategy, determine the final optimal control strategy of the power grid and the final state value of the power grid.

2. The approximate dynamic programming control method based on support vector regression according to claim 1, characterized in that: Step 3 specifically includes iterating based on the formula of the value function and the formula of the control strategy to obtain the final optimal control strategy of the power grid and the final state value of the power grid, which specifically includes the following steps: Step A1: Set the initial state value V0(x(t)) = 0, set the initial iteration number i = 0, use the initial iteration number as the current iteration number, and use the initial state value as the state value V of the current iteration number. i (x(t)); Step A2: Set the current iteration state value V i Substitute (x(t)) into the control strategy formula to calculate the optimal control strategy v for the current number of iterations i (x(t)); Step A3: The optimal control strategy v of the current iteration number i Substitute (x(t)) into the formula of the value function to calculate the state value V of the next iteration of the current iteration number i+1 (x(t)); Step A4: Calculate the first difference δ, which is specifically achieved by the following formula: δ=||V i+1 (x(t))-V i (x(t))||; Step A5: Determine whether the first difference δ is less than a preset threshold ε. If the first difference δ is less than the preset threshold ε, set V i+1 (x) is taken as the final state value, and the optimal control strategy v of the current iteration number is taken as i (x(t)) is used as the final optimal control strategy. When the first difference δ is not less than the preset threshold ε, it is determined whether the current number of iterations is less than or equal to the preset number. If the current number of iterations is less than or equal to the preset number, V i+1 (x) is taken as the final state value, and the optimal control strategy v of the current iteration number is taken as i (x(t)) is used as the final optimal control strategy. When the current number of iterations is greater than the preset number, the current number of iterations is increased by one as the new current number of iterations, and the process returns to step 3.

2.

3. The approximate dynamic programming control method based on support vector regression according to claim 1, characterized in that: Step 3 specifically includes improving the formula of the value function and the formula of the control strategy, and then iterating based on the improved formula to obtain the final optimal control strategy of the power grid and the final state value of the power grid, which specifically includes the following steps: Step B1: Based on the hard ε-bond support vector regression machine HESVR, the formula of the value function and the formula of the control strategy are improved to obtain the HESVR-based value function formula and the HESVR-based control strategy formula; Step B2: Based on the least squares method, the value function formula based on HESVR is processed to obtain the weight formula of HESVR; Step B3: According to the HESVR weight formula, the HESVR-based value function formula and the HESVR-based control strategy formula, the optimal control strategy of the power grid and the final state value of the power grid are calculated through an iterative algorithm.

4. The approximate dynamic programming control method based on support vector regression according to claim 3 is characterized in that: The value function formula based on HESVR in step B1 is expressed as: in, is the state value based on HESVR, is the approximate state value of the power grid at the voltage of the power grid or the frequency of the power grid x(t+T1) at the i-th iteration, and Φ(x(t+T1)) is the nonlinear basis function vector at the state x(t+T1).

5. The approximate dynamic programming control method based on support vector regression according to claim 3 is characterized in that: The control strategy formula based on HESVR in step B1 is expressed as: in, represents the optimal control strategy based on HESVR, R -1 is the inverse matrix of the control weight, g T (x(t)) is the system dynamic gradient, is the gradient of the value function at the i-th iteration, Represents the transpose of the gradient matrix of the nonlinear basis function vector at state x(t) is the weight vector obtained through HESVR training in the i-th iteration.

6. The approximate dynamic programming control method based on support vector regression according to claim 3, characterized in that: The weight formula of HESVR in step B2 is expressed as: in, A=[φ(x(t))-Φ(x(t+T1)),···,Φ(x(t+(M-1)T1))-Φ(x(t+MT1))]; B=[Γ(x(t),v i (x(t))),···,Γ(x(t+(M-1)T1),v i (x(t+(M-1)T1)))] T ; Among them, Φ(x(t)) is the nonlinear basis function vector at state x(t), Φ(x(t+(M-1)T1)) is the nonlinear basis function vector at state x(t+(M-1)T1), M is the number of sampling points in the least squares method, Φ(x(t+MT1)) is the nonlinear basis function vector at state x(t+MT1), Γ(x(t),v i (x(t))) is the execution of control strategy v starting from state x(t) i The total trajectory cost of (x(t)) in time T1, Γ(x(t+(M-1)T1),v i (x(t+(M-1)T1))) starts from the state x(t+(M-1)T1) and executes the control strategy v i The total trajectory cost (x(t+(M-1)T1)) in time T1.

7. The approximate dynamic programming control method based on support vector regression according to claim 3 is characterized in that: Step B3 specifically includes: Step B3.1: Set initial HESVR weights Set the initial state value V0(x(t)) = 0, set the initial iteration number i = 0, use the initial iteration number as the current iteration number, and use the initial HESVR weight as the current HESVR weight The initial state value is used as the state value V of the current iteration i (x(t)); Step B3.2: Determine whether the current number of iterations is greater than the preset number N max , when the current number of iterations is greater than the preset number N max In the case of , execute step 3.3.9, and if the current number of iterations is not greater than the preset number N max If yes, proceed to step 3.3.3; Step B3.3: Set HESVR weights Substitute into the HESVR-based control strategy formula to calculate the optimal control strategy based on HESVR for the current number of iterations Step B3.4: Obtain the voltage or frequency of the grid {x(t k +T1),…,x(t k +MT1)}; Step B3.5: Set the grid voltage or grid frequency {x(t k +T1),…,x(t k +MT1)} is brought into the weight formula of HESVR to calculate the updated weight of HESVR; Step B3.6: Substitute the updated HESVR weight into the HESVR-based value function formula to calculate the HESVR-based state value for the next iteration of the current iteration. Step B3.7: Calculate the second difference ρ, which is specifically achieved using the following formula: Step B3.8: Determine whether the second difference ρ is greater than a preset threshold ε. If so, proceed to Step 3.3.

9. If not, increment the current iteration count by one, and return to Step 3.3.

2. Step B3.9: As the final state value, the optimal control strategy based on HESVR for the current number of iterations is As the final optimal control strategy.

Images