Command output control method and device for unknown continuous-time nonlinear system

Abstract

The application discloses a kind of unknown continuous time nonlinear system-oriented instruction output control method and device, it is related to power grid control technical field, for DFIG wind farm grid-connected system under the condition of unknown model, parameter time-varying provide a kind of instruction output control scheme with control performance, adaptive capacity and engineering practicability.It described method includes: acquisition operation data, constructs system state variable, and constructs additional control instruction as control input;Define instantaneous performance index function, construct performance evaluation function;Approximate representation of performance evaluation function is formed using Critic network;Online acquisition online sampling data, construct Bellman error function, using gradient descent method to update Critic network weight;The control input that minimizes the estimated value of approximate representation is calculated as the optimization control input of current time;In the case where it is determined that convergence condition is satisfied, the optimization control input is output as additional control instruction.

Description

Technical Field

[0001] This application relates to the field of power grid control technology, and in particular to a command output control method and device for unknown continuous-time nonlinear systems. Background Technology

[0002] With the continuous expansion of new energy power generation, doubly-fed induction generator (DFIG) wind power systems have become an important component of grid-connected wind farms. In actual operation, DFIG wind power systems play a crucial role in wind power grid connection, voltage support, and fault recovery control by using rotor-side and grid-side converters to flexibly regulate active and reactive power. However, DFIG wind farm grid-connected systems are essentially typical continuous-time nonlinear systems. Under the influence of multiple factors such as grid faults, operating condition switching, wind speed fluctuations, and external disturbances, they exhibit strong nonlinearity, strong coupling, and significant uncertainty. Especially after a short-circuit fault, the grid connection voltage and the active power of the wind farm will fluctuate drastically. At this time, without effective additional reactive power control measures, the system's voltage recovery speed, dynamic regulation performance, and grid connection stability may be severely affected. Therefore, how to design a control command output method that can adapt to changes in operating conditions and ensure rapid and stable voltage recovery under unknown or partially unknown system model conditions has become an important technical problem that urgently needs to be solved in the field of wind power grid connection control.

[0003] In related technologies, various solutions have been proposed for the control problem of the aforementioned continuous-time nonlinear systems. One mainstream approach is the optimal control method based on an accurate model. This involves establishing an accurate mathematical model of the system, constructing state equations, and solving the Hamilton-Jacobi-Bellman (HJB) equations to obtain the optimal control law. Another improved approach is to use adaptive dynamic programming (ADP) or reinforcement learning methods based on value functions. This involves approximating the value function using online sampled data and then obtaining an approximate optimal control strategy through policy iteration. Furthermore, in engineering practice, for the voltage recovery problem of DFIG wind farms, an additional reactive power control method is typically used. This involves collecting the voltage deviation signal at the grid connection point, outputting reactive power compensation through a fixed-parameter controller, and superimposing this compensation onto the steady-state reactive power reference value to form an actual reactive power command, thereby providing voltage support after a fault.

[0004] In the process of implementing the relevant technologies, it was recognized that the technologies at least have the following technical problems: For optimal control methods based on accurate models, the exact dynamic model of the DFIG wind farm grid-connected system is often difficult to obtain due to factors such as random wind speed fluctuations, electrical parameter perturbations, unmodeled dynamics, and converter control coupling. Furthermore, the analytical solution of the HJB equations for nonlinear continuous-time systems is challenging, resulting in high computational complexity and making it difficult to meet real-time engineering requirements. Simultaneously, when system parameters or operating conditions change, the performance of control laws designed based on fixed models significantly deteriorates, even leading to oscillations or slow voltage recovery. While adaptive dynamic programming methods based on value functions reduce the need for complete model information, in unknown continuous-time nonlinear systems, the learning process of value functions still relies on some dynamic information such as the system input gain matrix. When this information is unknown or difficult to accurately identify, control law design still faces a fundamental model dependency problem. Moreover, its Actor-Critic dual-network structure involves large online computations, resulting in excessively high engineering implementation complexity. The core drawback of traditional reactive power control methods lies in their use of fixed-parameter or offline-tuned control structures. When system operating conditions deviate, fault severity changes, or model parameters are perturbed, fixed-parameter controllers struggle to adaptively adjust, resulting in poor voltage recovery speed, dynamic regulation capability, and system stability across various scenarios. Therefore, related technologies struggle to balance control performance, adaptive capability, and engineering practicality when dealing with continuous-time nonlinear systems with unknown models, time-varying parameters, and diverse operating conditions. A control method capable of achieving effective command output under unknown system model conditions is urgently needed. Summary of the Invention

[0005] In view of this, this application provides a command output control method and device for unknown continuous-time nonlinear systems, the main purpose of which is to solve the current problem of difficulty in balancing control performance, adaptive capability and engineering practicality.

[0006] According to a first aspect of this application, a command output control method for an unknown continuous-time nonlinear system is provided, the method comprising: Collect the operating data of the target continuous-time nonlinear system, construct system state variables based on the operating data, and construct additional control commands as control inputs; Define an instantaneous performance index function based on the system state variables and the control input, and construct a performance evaluation function based on the instantaneous performance index function; The estimated value of the performance evaluation function is represented by a Critic network as a combination of Critic network weights and basis function vectors, thus forming an approximate representation of the performance evaluation function. During system operation, the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment are collected online as online sampling data. The Bellman error function is constructed using the online sampling data, and the weights of the Critic network are updated using the gradient descent method in combination with the Bellman error function. Combining the updated Critic network weights and the preset optimality conditions, the control input that minimizes the estimated value of the approximate representation is calculated as the optimal control input at the current moment; Determine whether the optimized control input meets the convergence condition, and if the convergence condition is met, output the optimized control input as an additional control command.

[0007] According to a second aspect of this application, a command output control device for an unknown continuous-time nonlinear system is provided, the device comprising: The acquisition module is used to acquire the operating data of the target continuous-time nonlinear system, construct system state variables based on the operating data, and construct additional control commands as control inputs. The function construction module is used to define an instantaneous performance index function based on the system state variables and the control input, and to construct a performance evaluation function based on the instantaneous performance index function; An approximation representation module is used to represent the estimated value of the performance evaluation function as a combination of Critic network weights and basis function vectors using a Critic network, so as to form an approximate representation of the performance evaluation function; The function construction module is also used to collect the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment as online sampling data during system operation, construct the Bellman error function using the online sampling data, and update the weights of the Critic network using the gradient descent method in combination with the Bellman error function. The solution module is used to combine the updated Critic network weights and the preset optimality conditions to calculate the control input that minimizes the estimated value of the approximate representation as the optimal control input at the current time. The output module is used to determine whether the optimized control input meets the convergence condition, and if it is determined that the convergence condition is met, to output the optimized control input as an additional control command.

[0008] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0009] According to a fourth aspect of this application, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0010] By employing the above technical solutions, this application provides a command output control method and apparatus for unknown continuous-time nonlinear systems. This application transforms the traditional continuous-time optimal control problem, which relies on an accurate system model for solving the HJB equations, into a data-driven performance evaluation function approximation and optimization problem by constructing a performance evaluation function that simultaneously includes state variables and control inputs. This avoids the technical obstacle of accurately obtaining the system dynamics model due to factors such as wind speed fluctuations, parameter perturbations, and unmodeled dynamics. The controller can complete the online construction of the control law without needing to know the specific expressions of the system's internal dynamics and input gain. Simultaneously, a Critic network is used to approximate the performance evaluation function. By constructing Bellman error using online sampling data to drive network weight updates, this approach not only avoids the problems of high online computation and excessive engineering complexity caused by the Actor-Critic dual-network structure in related technologies, but also enables the controller to continuously correct network weights based on real-time system operating data. This allows for adaptive adjustment of the control strategy when system operating conditions deviate or model parameters are perturbed, effectively improving the insufficient voltage recovery speed and dynamic adjustment capability of fixed-parameter controllers in different scenarios. It provides a command output control scheme for DFIG wind farm grid-connected systems under conditions of unknown models and time-varying parameters, taking into account control performance, adaptive capability, and engineering practicality.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This paper presents a schematic flowchart of a command output control method for an unknown continuous-time nonlinear system according to an embodiment of this application. Figure 2 This paper presents a schematic flowchart of another instruction output control method for unknown continuous-time nonlinear systems provided in an embodiment of this application. Figure 3 This paper shows a schematic diagram of the structure of a command output control device for an unknown continuous-time nonlinear system according to an embodiment of this application. Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0013] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0014] This application provides a command output control method for unknown continuous-time nonlinear systems, such as... Figure 1 As shown, the method includes: S10: Collect the operating data of the target continuous-time nonlinear system, construct the system state variables based on the operating data, and construct additional control commands as control inputs.

[0015] In this embodiment, during the operation of the target continuous-time nonlinear system, it is necessary to collect the system's operational data in real time. The operational data includes at least measurable variables characterizing the system's current operating state and pre-given steady-state reference commands. Specifically, when constructing system state variables, the deviation is obtained based on the difference between the measurable variables and the corresponding target reference values. This deviation constitutes the system state variables, used to describe the system's dynamic response characteristics after disturbances. When constructing additional control commands, they are constructed as additional compensation quantities superimposed on the steady-state reference commands. These additional control commands serve as control inputs, subsequently forming, together with the steady-state reference commands, the actual control commands sent to the system's actuators.

[0016] Thus, in the above process, by constructing the system state as a deviation quantity and the control input as an additional compensation quantity, the subsequent control optimization process directly targets disturbance suppression and command compensation, naturally aligning with the control objective.

[0017] For example, taking a doubly fed induction generator wind farm grid-connected system as an example, the actual value of the grid connection point voltage, the actual value of the wind farm output active power, and the steady-state reactive power reference value are collected in real time. The difference between the grid connection point voltage and the target voltage is used as the voltage deviation, and the difference between the active power and the target power is used as the power deviation. The two are used to form the system state variables, and the additional reactive power reference command is used as the control input.

[0018] S20: Define the instantaneous performance index function based on the system state variables and control input, and construct the performance evaluation function based on the instantaneous performance index function.

[0019] In this embodiment, an instantaneous performance index function needs to be defined based on the system state variables and control inputs. This instantaneous performance index function is a quadratic form of the system state variables and control inputs. The quadratic term of the system state variables measures the degree to which the system state deviates from the desired state, and the quadratic term of the control input measures the magnitude of the control cost. Based on this, a continuous-time performance index function is further defined, which is the integral of the instantaneous performance index function from the current moment to infinity, serving as a comprehensive evaluation of the system's long-term cumulative performance. To avoid directly solving the Hamilton-Jacobi-Bellman equations related to this continuous-time performance index function, this embodiment also constructs a performance evaluation function that simultaneously includes system state variables and control inputs. This performance evaluation function combines the value function with the Hamiltonian function, allowing the control input to directly enter the performance evaluation process. This enables subsequent control command optimization to be directly based on this performance evaluation function.

[0020] For example, taking a doubly fed induction generator wind farm grid-connected system as an example, the weighted sum of the squares of voltage deviation and power deviation and the weighted sum of the squares of the additional reactive power control input are used as instantaneous performance indicators. Then, the sum of these weighted sums over an infinite time domain is used as long-term performance indicators. Based on this, a Q function containing state and control inputs is constructed as a performance evaluation function.

[0021] S30: The estimated value of the performance evaluation function is represented by a combination of the Critic network weights and the basis function vectors, thus forming an approximate representation of the performance evaluation function.

[0022] In this embodiment, a Critic network is used to approximate the performance evaluation function, i.e., a parameterized function approximator is constructed to estimate the true performance evaluation function. Specifically, the estimated value of the performance evaluation function needs to be represented as a combination of the Critic network weight vector and the basis function vector. The basis function vector is composed of system state variables and control inputs, and each basis function in the basis function vector is a nonlinear function of the system state variables and / or control inputs. Each weight coefficient in the Critic network weight vector corresponds one-to-one with the corresponding basis function. By adjusting the Critic network weights, this linear combination can approximate the true performance evaluation function.

[0023] Therefore, it can be seen that the technical means used in the above process are different from the scheme using the Actor-Critic dual network structure. Here, only a single Critic network is needed to approximate the performance evaluation function. The network structure is simple and the online computation is small.

[0024] For example, taking a doubly fed induction generator wind farm grid-connected system as an example, a basis function vector consisting of a voltage deviation quadratic term, a voltage deviation and power deviation cross term, a power deviation quadratic term, a voltage deviation and additional reactive power control input coupling term, a power deviation and additional reactive power control input coupling term, and an additional reactive power control input quadratic term is selected and combined with the Critic network weight vector to form the estimated value of the Q function.

[0025] S40: During system operation, the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment are collected online as online sampling data. The Bellman error function is constructed using the online sampling data, and the gradient descent method is used to update the weights of the Critic network in combination with the Bellman error function.

[0026] In this embodiment, during system operation, it is necessary to collect the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment to form online sampling data. This online sampling data directly originates from the actual operating trajectory of the system and does not depend on the system's mathematical model. Then, based on the continuous-time Bellman equation, a Bellman error function is constructed using the difference between the estimated value and the target value of the performance evaluation function. The Bellman error function measures the accuracy of the current Critic network weights' estimation of the performance evaluation function. Based on this, the square of the Bellman error function is used as the loss function to be minimized, and the Critic network weights are updated using a gradient descent method. That is, each update adjusts the weight values ​​along the negative gradient direction of the loss function with respect to the Critic network weights, causing the Critic network weights to gradually converge to the optimal value that minimizes the Bellman error. This update process relies only on the online sampling data and instantaneous performance index data, and does not require knowledge of the system's internal dynamics or the specific expression of the input gain.

[0027] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, the voltage and power deviations at the current moment, the additional reactive power control commands output at the current moment, and the voltage and power deviations at the next moment are collected. Based on this, the Bellman error is calculated, and then the gradient of the Bellman error is used to drive the online correction of the Critic network weights.

[0028] S50: Combining the updated Critic network weights and the preset optimality conditions, calculate the control input that minimizes the estimated value of the approximate representation as the optimized control input at the current moment.

[0029] In this embodiment, after the Critic network weights are updated, it is necessary to solve for the control input that minimizes the performance evaluation function estimate based on the performance evaluation function estimate determined by the updated Critic network weights, using the control input as the optimization variable. Specifically, since the performance evaluation function estimate contains a quadratic term of the control input, which is a convex function with respect to the control input, this embodiment uses the zero first-order partial derivative of the performance evaluation function estimate with respect to the control input as the optimality condition. Based on this, the control input value that minimizes the performance evaluation function estimate is obtained, and this control input value is used as the optimized control input-output at the current moment. Thus, the calculation of the control input depends only on the updated Critic network weights and the system state variables at the current moment, without needing to explicitly use the system's input gain matrix or solve the Hamilton-Jacobi-Bellman equation.

[0030] For example, taking a doubly fed induction generator wind farm grid-connected system as an example, based on the updated Critic weights and the current voltage and power deviations, the additional reactive power control command that minimizes the Q function estimate is directly calculated by taking the partial derivative of the additional reactive power control command with the Q function and setting it to zero.

[0031] S60: Determine whether the optimized control input meets the convergence condition, and if it is determined that the convergence condition is met, output the optimized control input as an additional control command.

[0032] In this implementation, after obtaining the optimized control input at the current moment, it is necessary to determine whether the optimized control input meets the preset convergence condition. Specifically, it is necessary to compare the overall change between the estimated values ​​of the performance evaluation function corresponding to two adjacent iterations. If the overall change is less than a pre-given convergence threshold, it is determined that the current iteration result has met the convergence condition, indicating that the current optimized control input has tended to a stable optimal value. At this time, the optimized control input is output as an additional control command. If the overall change is greater than or equal to the convergence threshold, it is determined that the current iteration result has not yet met the convergence condition, and the iteration process of data acquisition, Bellman error construction, Critic weight update, and control input calculation is returned to continue until the convergence condition is met. In this way, by setting a convergence judgment mechanism, it can be ensured that the final output additional control command is a stable result after sufficient iterative optimization.

[0033] For example, taking a doubly fed induction generator wind farm grid-connected system as an example, the norm of the difference between the estimated values ​​of the Q function in two adjacent iterations is calculated. If the norm is less than a preset threshold, the additional reactive power control command calculated at the moment is used as the final output, superimposed on the steady-state reactive power reference value and sent to the reactive power control channel of the rotor-side converter.

[0034] Optionally, in this embodiment, the process of collecting operational data of the target continuous-time nonlinear system, constructing system state variables based on the operational data, and constructing additional control commands as control inputs includes: real-time acquisition of operational data during the operation of the target continuous-time nonlinear system, wherein the target continuous-time nonlinear system is a doubly-fed induction generator wind farm grid-connected system, and the operational data includes at least the actual value of the grid connection point voltage. Actual value of active power output from wind farm Steady-state reactive power reference value The additional reactive power reference command output by the controller at the previous moment; using the following formula, based on the actual value of the grid connection point voltage... With target voltage Calculate the voltage deviation at the grid connection point : ; Using the following formula, based on the actual value of the active power output of the wind farm... With the target active power Calculate active power deviation : ; Based on the voltage deviation at the grid connection point and active power deviation The system state variables constituted by the following expression System state variables Used to describe the dynamic response process of a target continuous-time nonlinear system after a fault disturbance: ; Define the additional reactive power reference command according to the following expression. As control input : .

[0035] In this embodiment, in the doubly-fed induction generator wind farm grid-connected system, it is necessary to collect operational data in real time during system operation. The operational data should include at least the actual value of the grid connection point voltage. Actual value of active power output from wind farm Steady-state reactive power reference value And the additional reactive power reference command output by the controller at the previous moment, wherein the actual value of the grid connection point voltage. Reflecting the current voltage level at the connection point between the wind farm and the grid, the actual value of the active power output of the wind farm. Reflecting the current active power supplied by the wind farm to the grid, steady-state reactive power reference value. It is the reactive power command reference value of the wind farm during steady-state operation, and the additional reactive power reference command output by the controller at the previous moment is the additional compensation amount calculated in the previous control cycle.

[0036] After collecting the above data, in one aspect of this application embodiment, the actual value of the grid connection point voltage is calculated using the following formula 1. With target voltage Calculate the voltage deviation at the grid connection point : Formula 1: ; Among them, the target voltage in Formula 1 This represents the expected operating value of the grid connection point voltage.

[0037] On the other hand, the actual value of the active power output of the wind farm is used according to the following formula 2. With the target active power Calculate active power deviation : Formula 2: ; Among them, the target active power in Formula 2 This represents the expected output value of the active power of the wind farm.

[0038] Subsequently, the embodiments of this application use the grid connection point voltage deviation and active power deviation The system state variables constitute the expressions shown in Formula 3 below. : Formula 3: ; Among them, the superscript in Formula 3 Represents vector transpose, system state variable This is used to describe the dynamic response process of a doubly-fed induction generator wind farm grid-connected system after a fault disturbance. Voltage deviation reflects the degree to which the grid connection point voltage deviates from the target value after the fault, and power deviation reflects the degree to which the active power output deviates from the target value. Together, they characterize the deviation of the system's operating state after the disturbance.

[0039] Furthermore, the embodiments of this application also define an additional reactive power reference command according to the expression shown in Formula 4 below. As control input : Formula 4: ; Among them, the control input in formula 4 This is the additional compensation amount to be optimized, used to be superimposed on the steady-state reactive power reference value in subsequent steps. The above constitutes the actual reactive power command sent to the rotor-side converter.

[0040] In this way, by constructing the system state as voltage and power deviations, the system state directly reflects the deviation of the control target after the fault disturbance, which is beneficial for the subsequent control optimization process to focus on the elimination of deviations. Moreover, by constructing the control input as an additional compensation quantity, the controller can be connected in a compensation manner on the basis of the original underlying vector control structure, avoiding large-scale modification of the existing control system architecture.

[0041] For example, when a short-circuit fault occurs in the grid-connected system of a doubly-fed induction generator wind farm, the actual value of the grid connection point voltage... It will drop to the target voltage Below, the voltage deviation at this time The value is negative and has a large absolute value, while the actual value of the active power output of the wind farm is... It will also deviate from the target active power due to voltage drops. Active power deviation Correspondingly, system state variables are generated. This plotted the dynamic deviation after the fault disturbance in real time, providing a basis for subsequent control optimization based on state feedback, and adding reactive power reference commands. Then, as the control variable to be learned, its optimization objective is to reduce the deviations in the state vector by additional compensation of reactive power after the fault is cleared, thereby supporting the grid connection point voltage to recover to near the target value.

[0042] In this embodiment of the application, optionally, an instantaneous performance index function is defined based on the system state variables and control input, and a performance evaluation function is constructed based on the instantaneous performance index function, including: establishing an unknown continuous-time nonlinear affine system description of the target continuous-time nonlinear system as shown in the following formula. : ; in, For system state variables, To control the input, For unknown nonlinear dynamic terms within the system, For unknown control input gain terms; based on system state variables and control input Define the instantaneous performance index function as shown in the following formula. : ; in, for transpose, This is the state weight matrix used to apply weighted penalties to the components of the system's state variables. for transpose, This is the control weight matrix used to weight and penalize the amplitude of the control input; based on the instantaneous performance index function... Define the continuous-time performance index function as shown in the following formula. : ; in, Indicates from the current moment Integral operations up to infinity From the current moment The initial future time variable, For at any time The system state variables, For at any time Control input; Combined with continuous-time performance index function Define value function And determine the partial derivatives of the value function. Description of unknown continuous-time nonlinear affine systems Instantaneous performance index function and partial derivatives Construct the Hamiltonian function as shown in the following formula: ; Associative value function Construct the performance evaluation function shown in the following formula using the Hamilton function. : .

[0043] In this embodiment, before constructing the performance evaluation function, it is necessary to first establish the unknown continuous-time nonlinear affine system description of the target continuous-time nonlinear system as shown in Equation 5 below. : Formula 5: ; Among them, in formula 5 For the system state variables constructed above, For the control inputs defined above; This refers to the unknown nonlinear dynamic term within the system, used to characterize the input-independent part of the system's own state evolution law; The unknown control input gain term characterizes how the control input affects the rate of change of the system state. Due to factors such as parameter perturbations, wind speed fluctuations, external disturbances, and unmodeled dynamics in real-world systems... and The specific mathematical expression is difficult to obtain accurately. Therefore, in the embodiments of this application, the affine system description will serve as the formal basis for subsequent theoretical derivation, and will not need to be identified or solved during the algorithm execution process.

[0044] After establishing this formal description, it is necessary to further define the instantaneous performance index function as shown in Equation 6 below, based on the system state variables and control inputs. : Formula 6: ; Among them, in formula 6 System state variables transpose, This is the state weight matrix used to apply weighted penalties to the components of the system's state variables. This is the control weight matrix used to weight and penalize the amplitude of the control input; in this instantaneous performance index, This term is used to quantify the degree to which the system state deviates from the target state. This item is used to limit the amplitude of the control input to avoid excessive control input.

[0045] Furthermore, in this embodiment of the application, the continuous-time performance index function shown in Formula 7 below is also defined based on the instantaneous performance index function. : Formula 7: ; Among them, in formula 7 From the current moment The initial future time variable, For the future moment The system state variables, For at any time The control input, this continuous-time performance index function is used to evaluate the long-term cumulative performance of the system state and control input over the entire future time domain from the current moment.

[0046] Then, the value function is defined in conjunction with the continuous-time performance index function. Value function Indicates starting from the current state Starting with the minimum cumulative performance index achievable in the future, and further determining the partial derivatives of the value function with respect to the state variables. The partial derivative The rate of change of the time-valued function used to describe minute changes in state is then used. Subsequently, based on the description of the unknown continuous-time nonlinear affine system, the instantaneous performance index function, and partial derivatives... Construct the Hamiltonian function as shown in Equation 8 below: Formula 8: ; The Hamilton function shown in Equation 8 couples instantaneous performance indicators with the system dynamics through-valued partial derivatives, providing an intermediate bridge for the construction of performance evaluation functions.

[0047] Finally, the value function With Hamilton function Add them together to construct the performance evaluation function shown in Formula 9 below. : Formula 9: ; The performance evaluation function shown in Equation 9 includes both system state variables and control inputs, allowing the control inputs to directly enter the performance evaluation process. As a result, in subsequent steps, the control law can be directly obtained from the minimization condition of the performance evaluation function with respect to the control inputs, eliminating the need to explicitly use unknown internal system dynamic terms and input gain terms.

[0048] In this way, through the above process, the problem of solving the HJB equation in traditional continuous-time optimal control is transformed into the problem of learning and optimizing the performance evaluation function, which avoids the difficulty of directly facing the solution of nonlinear partial differential equations.

[0049] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, when establishing the affine system description, it is not necessary to obtain the specific model parameters of the wind turbine's mechanical dynamics, electrical dynamics, and converter control links. Instead, the state vector composed of voltage deviation and power deviation can be used directly. and additional reactive power control input Based on this, the instantaneous performance index is defined as the weighted sum of the squares of voltage deviation and power deviation and the square of the additional reactive power control input. Then, a Q function containing state and control is constructed as the objective function for subsequent Critic network approximation, so that the entire control scheme can be optimized by relying solely on online running data in the case of a completely unknown system model.

[0050] Optionally, in this embodiment of the application, a Critic network is used to represent the estimated value of the performance evaluation function as a combination of Critic network weights and basis function vectors, thereby forming an approximate representation of the performance evaluation function, including: Constructing from system state variables and control input The basis function vectors formed , where the basis function vector It includes at least the quadratic terms of the system state variables, the cross-product terms between the system state variables, the coupling product terms between the system state variables and the control input, and the quadratic terms of the control input; Combining basis function vectors The performance evaluation function is approximated using a Critic network. The estimated value of the performance evaluation function is constructed as a vector of Critic network weights and basis functions. A linear combination of these forms yields an approximate representation as shown in the following formula: ; in, This is an estimate of the performance evaluation function. For Critic network weights, This is the transpose of the Critic network weights.

[0051] In this embodiment, after the theoretical construction of the performance evaluation function is completed, a parameterized function approximator is needed to approximate the performance evaluation function in order to facilitate online software implementation in the actual system. Specifically, it is first necessary to construct a function approximation based on the system state variables. and control input The basis function vector formed together basis function vectors The selection principle is to ensure that the basis function vectors can fully characterize the contribution components in the performance evaluation function. In the embodiments of this application, the basis function vectors are... It should include at least quadratic terms of the system state variables, representing the cost of the state deviating from the target value; cross-product terms between system state variables, representing the impact of coupling effects between different state variables on performance evaluation; coupling product terms between system state variables and control input, representing the contribution of the interaction between state and control to performance evaluation; and quadratic terms of control input, representing the contribution of control costs to performance evaluation. For example, basis function vectors. It can be shown in Formula 10 below: Formula 10: ; Formula 10 includes quadratic terms of the system state variables. Cross-product terms between system state variables The coupling product term between system state variables and control inputs as well as The quadratic term of the control input .

[0052] After constructing the basis function vectors, this embodiment uses a Critic network to approximate the performance evaluation function, that is, to estimate the performance evaluation function. Constructed as Critic network weights With basis function vectors The linear combination form is shown in Formula 11 below: Formula 11: ; Among them, in formula 11 The estimated value of the performance evaluation function represents the approximation result of the true performance evaluation function under given state and control input; is the weight vector of the Critic network, where each weight coefficient corresponds one-to-one with the corresponding basis function in the basis function vector, and is used to adjust the contribution of each basis function to the estimated value; Critic network weight vector The transpose operation.

[0053] Thus, in the above process, by using a Critic network for approximation, the originally unknown performance evaluation function is transformed into a parameterized expression of the Critic network weights. The subsequent learning process then becomes an adjustment process of the Critic network weights, allowing this linear combination form to gradually approximate the true performance evaluation function value through online correction of the weight coefficients. Unlike related technologies that require maintaining a dual-network structure of both the Actor and Critic networks, this embodiment only requires a single Critic network to approximate the performance evaluation function. This results in fewer network parameters and less online computation, which helps reduce the computational burden on the control system and improve real-time response capabilities.

[0054] Optionally, in this embodiment, during system operation, the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment are collected online as online sampling data. A Bellman error function is constructed using the online sampling data, and the Critic network weights are updated using a gradient descent method based on the Bellman error function. This includes: during system operation, collecting the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment as online sampling data; determining the current control strategy; and, based on the continuous-time Bellman equation, using the estimated value of the performance evaluation function... With target value The difference between them is used to construct the Bellman error function in the form of the performance evaluation function shown in the following formula. : ; Among them, the target value It is constructed based on online sampling data and instantaneous performance index function; combined with Bellman error function. A weight update expression is constructed. During each update, the Bellman error and gradient vector are calculated in real time based on the online sampling data at the current moment, according to the weight update expression, to complete an online update of the Critic network weights.

[0055] In this embodiment, during system operation, it is necessary to collect the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment as online sampling data. The system state variables and control input at the current moment characterize the system's operating state and control action at that moment, while the system state variables at the next moment reflect the evolution of the system state under the current control action after a sampling interval. After data acquisition is completed, the current control strategy is fixed, and the estimated value of the performance evaluation function is used according to the continuous-time Bellman equation. With target value The difference between them is used to construct the Bellman error function shown in Equation 12 below: Formula 12: ; Among them, in formula 12 This represents the current estimate of the performance evaluation function by the Critic network. The target value is constructed based on online sampling data and the instantaneous performance index function. This target value represents the expected value that the performance evaluation function should satisfy under the current sampling data. Bellman error function. The physical meaning lies in measuring the accuracy of the current Critic network weights in estimating the performance evaluation function. When the value approaches zero, it means that the estimate of the Critic network satisfies the optimality condition described by the Bellman equation, that is, the current estimate is close to the true performance evaluation function.

[0056] After constructing the Bellman error, it is necessary to further construct the update expression for the Critic network weights by combining the Bellman error function. This update expression aims to minimize the square of the Bellman error. During each update, the Bellman error and gradient vector are calculated in real time based on the online sampling data at the current moment according to this weight update expression, thus completing an online update of the Critic network weights.

[0057] Thus, throughout the update process, the correction of the Critic network weights depends on the calculation results of online sampling data and instantaneous performance indicators, without needing to obtain the system's internal dynamic model and input gain matrix.

[0058] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, after the fault is cleared, the software control system collects the voltage deviation at the current moment in real time. Power deviation and additional reactive power control commands and the voltage deviation at the next moment. and power deviation Based on these data, the Bellman error is calculated. If the estimated Q value is higher than the target value, the Bellman error is positive. Gradient descent updates will adjust the Critic weights in the direction of reducing the error, so that the next estimate is closer to the true Q function. This process is repeated iteratively, driving the Critic weights to gradually converge to the optimal value that minimizes the Bellman error.

[0059] Optionally, in this application embodiment, the Bellman error function is incorporated. Constructing the weight update expression includes: combining the Bellman error function. Construct the squared error function as shown in the following formula. : ; Using the squared error function Weights of the Critic network The negative gradient direction is used as the weight update direction. The Critic network weights are updated using the gradient descent method according to the following formula. Perform iterative updates: ; in, For the first Critic network weights during step update For the first The updated Critic network weights The preset Critic learning rate, Squared error function The gradient vector of the Critic network weights; the squared error function Substituting the expression into the update formula, we obtain the weight update expression shown in the following formula: .

[0060] In this embodiment, the Bellman error function is constructed. Next, in order to update the weights of the Critic network, the squared error function shown in Equation 13 is first constructed using half of the square of the Bellman error: Formula 13: ; Squared error function It is a scalar loss function used to comprehensively measure the overall deviation of the current Critic network from the performance evaluation function estimate.

[0061] After constructing the squared error function, continue using this squared error function Weights of the Critic network The negative gradient direction is used as the weight update direction, and the gradient descent method is used to iteratively update the Critic network weights. The update formula is shown in Formula 14 below: Formula 14: ; Formula 14 For the first Critic network weights during step update For the first The updated Critic network weights; The preset Critic learning rate is used to control the step size of each weight update to avoid overshoot. Squared error function The gradient vector of the weights in the Critic network points in the direction where the squared error function rises the fastest in the weight space. Therefore, updating along its negative direction can reduce the value of the error function.

[0062] Furthermore, the squared error function Substituting the expression into the update formula, we can obtain the following result according to the chain rule: Therefore, the weight update expression shown in Equation 15 can be obtained as follows: Formula 15: .

[0063] Thus, the weight update process described above only uses online sampled data to calculate the Bellman error and its gradient in real time. It does not require any information about the system's internal dynamics or input gain; the weights can be corrected based solely on the current Critic weights, state variables, and control input. The Bellman error... The direction and magnitude of the weight update are driven by the Bellman error. When the Bellman error is large, the weight adjustment magnitude increases accordingly to accelerate convergence. When the Bellman error approaches zero, the weight adjustment magnitude approaches zero, indicating that the estimation has become accurate.

[0064] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, assuming that the Critic weight is at a certain sampling time... From the current state and control input Calculate the Bellman error , Calculated based on the basis function vector, assuming a preset learning rate. If the value is 0.1, then the change in this update is New Critic weights After one correction based on this change, and through multiple iterative updates at sampling points, the Critic weights gradually converge to the optimal value that minimizes the Bellman error. This continuously improves the estimation accuracy of the performance evaluation function by the Critic network, thus providing a reliable basis for subsequent control law calculations based on this estimate.

[0065] In this embodiment of the application, optionally, the control input that minimizes the approximate estimated value is calculated as the optimization control input at the current time, combining the updated Critic network weights and preset optimality conditions. This includes: decomposing the performance evaluation function to obtain the following expression: ; Using the control input as the optimization variable, and with the first partial derivative of the approximate estimate with respect to the control input being zero, we obtain the optimality condition shown in the following expression, which determines the necessary condition for the approximate estimate to reach a minimum: ; Using the following formula and based on the optimality condition, solve for the estimated value that approximates the representation at the current time. Relative to control input The control input value that reaches the minimum value : ; Control input values As the optimal control input at the current moment.

[0066] In this embodiment, after the Critic network weights are updated, the policy improvement phase begins, where the control policy is updated based on the performance evaluation function estimate determined by the updated Critic network weights. Because the performance evaluation function... It also includes state items that are only related to the system state. State-control coupling terms that are related to both system state and control input. and control input items that are only related to control input. ,in The contribution of the characterization state itself to the target value to performance evaluation. The contribution of the interaction between characterization state and control to performance evaluation. To characterize the contribution of control costs to performance evaluation, the performance evaluation function can be decomposed according to the above three terms, resulting in the expression shown in Formula 16 below: Formula 16: .

[0067] After completing the decomposition of the performance evaluation function, the embodiments of this application use control input. To optimize the variables, since the performance evaluation function is a quadratic form with respect to the control input, and its graph is an upward-opening parabola, its minimum point lies at the location where the first-order partial derivative is zero. Based on this, the performance evaluation function is estimated using the expression in Formula 17 below. Regarding control input The first-order partial derivative being zero is used as an optimality condition: Formula 17: ; The condition shown in Formula 17 is used to determine the necessary mathematical conditions for minimizing the performance evaluation function estimate. This embodiment of the application solves for the performance evaluation function estimate at the current time according to this optimality condition. Relative to control input The control input value that reaches the minimum value is shown in Formula 18 below: Formula 18: ; The control input values ​​shown in Formula 18 This is the optimal control input at the current moment, due to the weight matrix corresponding to the quadratic term of the control input in the performance evaluation function. Since it is a positive definite matrix, the local minimum point is also the global minimum point, ensuring the uniqueness and optimality of the solution.

[0068] Thus, the calculation of the entire control law in the above process can be completed using only the currently updated Critic network weights and the system state variables at the current moment, without needing to obtain the system's input gain matrix. It also does not require solving any system dynamic equations or Hamilton-Jacobi-Bellman equations.

[0069] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, when the system state variable Control input After updating the Critic network weights to their current values, calculate the estimated value of the performance evaluation function, and use this estimated value as the optimization variable for the quadratic expression with respect to... Taking the partial derivative and setting it to zero, we can directly solve for the optimal control input that minimizes the estimated value of the performance evaluation function. This calculation only involves the weight coefficients in the Critic network and the voltage deviation collected at the current time. and power deviation It does not rely on the mechanical dynamic model, electrical parameters, and converter control model of the doubly-fed induction generator, thus achieving true model-free control command optimization.

[0070] In this embodiment of the application, optionally, determining whether the optimization control input satisfies the convergence condition, and outputting the optimization control input as an additional control command if the convergence condition is satisfied, includes: in the current iteration, obtaining the estimated values ​​of the approximate representations corresponding to two adjacent iterations. and ,in, This represents the iteration round number. Indicates the first The approximate representation of the weights of the Critic network after rounds of iteration is estimated. Indicates the first The performance evaluation function estimate is determined by the updated Critic network weights after each iteration; the overall change in the approximate estimate between two adjacent iterations is calculated using the following formula, and the overall change is compared with a preset convergence threshold. Comparison: ; in, This represents the difference between the approximate estimates obtained from two consecutive iterations. This represents the overall change obtained after performing norm operations on the difference. The method uses a pre-defined positive scalar to determine whether the iterative process tends to be stable. If the overall change is less than the convergence threshold, the current iteration result is determined to meet the convergence condition, and the optimized control input obtained in the current iteration is output as an additional control instruction. Correspondingly, the method also includes: if the overall change is greater than or equal to the convergence threshold, the current iteration result is determined not to meet the convergence condition, and the steps of updating the Critic network weights and calculating the optimized control input are re-executed until the convergence condition is met.

[0071] In this embodiment, after updating the Critic network weights and calculating the optimized control input in the current iteration, a convergence determination phase is needed to ensure the reliability of the final output control commands. This phase checks whether the current iteration result has reached a stable state. In the current iteration... In this process, it is necessary to obtain the estimated values ​​of the performance evaluation function for two consecutive iterations. and ,in This represents the iteration round number. Indicates the first The estimated value of the performance evaluation function determined by the weights of the Critic network after rounds of iteration. Indicates the first The performance evaluation function is estimated based on the updated Critic network weights after each iteration. Then, convergence is determined using Equation 19 below: Formula 19: ; in, This represents the difference between the estimated values ​​of the performance evaluation function corresponding to two consecutive iterations. This represents the overall change obtained after taking the norm of the difference. This norm operation can take the Euclidean norm or the infinity norm, and is used to compress the vector-like difference into a single scalar to measure the degree of overall change. It is a pre-defined positive scalar used to determine whether the iterative process tends to be stable. Its value can be set comprehensively according to the control accuracy requirements and convergence speed requirements of the actual system.

[0072] If the overall change is less than the convergence threshold If the current iteration result satisfies the convergence condition, it indicates that the estimated value of the performance evaluation function no longer changes significantly between two adjacent iterations, and the weights of the Critic network tend to stabilize. At this point, the optimized control input obtained in the current iteration is used as the additional control command output. If the overall change is greater than or equal to the convergence threshold... If the current iteration result does not meet the convergence condition, it indicates that the Critic network weights have not yet converged sufficiently and the estimated value of the performance evaluation function is still changing significantly. At this time, the update step of the Critic network weights and the calculation step of the optimized control input are re-executed to enter the next iteration until the overall change meets the condition of being less than the convergence threshold. This ensures that the final output control command is a stable result after sufficient iteration optimization, and avoids the output of suboptimal control quantity due to insufficient iteration, which may have an adverse effect on the system operation.

[0073] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, assuming a convergence threshold... for ,calculate and The norm of the difference, if the norm is less than a preset convergence threshold. This indicates that the estimated value of the Q function has changed sufficiently in this iteration, satisfying the convergence condition. Therefore, the additional reactive power control command calculated in this round can be applied. As an effective output, it is superimposed on the steady-state reactive power reference value and sent to the rotor-side converter control channel; if the norm is still greater than the threshold, it continues to iterate until the condition is met, thereby ensuring that the additional reactive power command finally applied to the converter control channel is a stable control quantity after sufficient optimization and convergence.

[0074] Optionally, in this embodiment of the application, after determining whether the optimized control input meets the convergence condition, and outputting the optimized control input as an additional control command when the convergence condition is met, the method further includes: using the optimized control input output when the convergence condition is met as an additional control command. ; Obtain the pre-set steady-state reference command The additional control commands will be generated according to the following formula. Superimposed on steady-state reference command The above constitutes the actual control command. : ; The actual control commands are output to the actuator control channel of the target continuous-time nonlinear system, and the actuator adjusts the operating state of the target continuous-time nonlinear system according to the actual control commands.

[0075] In this embodiment, after determining that the current iteration result meets the convergence condition, it is necessary to input the optimization control when the convergence condition is met. As an additional control command ,in This is an additional compensation quantity to be superimposed on the original steady-state command of the system, used to provide additional control action while maintaining the original control architecture of the system unchanged. Subsequently, a pre-set steady-state reference command needs to be obtained. The steady-state reference command is the baseline value of the command used by the system when it is operating normally under fault-free or disturbance-free conditions. It is given in advance by the system operation scheduling layer or steady-state control layer, and the additional control command is applied according to the following formula 20. Superimposed on steady-state reference command The above constitutes the actual control command. : Formula 20: ; The result calculated in Formula 20 The comprehensive control commands ultimately sent to the system actuators include both the baseline components required for the steady-state operation of the system and the additional compensation components obtained through online learning and optimization. Together, they determine the actual adjustment targets of the actuators.

[0076] After the superposition is completed, the embodiments of this application will actually control the instructions. The output is sent to the actuator control channel of the target continuous-time nonlinear system, and the actuator adjusts the operating state of the target continuous-time nonlinear system according to the actual control command.

[0077] In this way, the entire superposition process described above does not change the original underlying control structure and the generation method of steady-state reference commands. It only superimposes the learned additional control commands as compensation signals into the original command channels. Therefore, it can be easily embedded into the existing control system architecture without large-scale modification, reducing the difficulty and cost of engineering deployment.

[0078] For example, taking a doubly-fed induction generator wind farm grid-connected system as an example, the actuator of this system is the rotor-side converter, and the steady-state reference command... This refers to the reactive power reference value of the wind farm during steady-state operation, provided by the upper-level dispatching or wind farm monitoring system. After meeting the convergence conditions, the learned optimal control input will be... As an additional reactive power reference command According to Formula 20, it is compared with the steady-state reactive power reference value. The superposition constitutes the actual reactive power reference command. Then the The reactive power is fed into the reactive power control channel of the doubly-fed induction generator rotor-side converter. The vector control loop inside the rotor-side converter adjusts the reactive power output of the doubly-fed induction generator according to the actual reactive power reference command, thereby providing voltage support to the grid connection point after a fault disturbance. The entire process retains the original vector control structure of the rotor-side converter, only introducing an additional compensation quantity optimized through online learning into its reactive power reference channel, making the control scheme have good engineering compatibility and deployability.

[0079] In summary, the logical process of the technical solution in the embodiments of this application is summarized as follows: Figure 2 As shown, the initialization process begins by setting the Critic network weights, learning rate, state weight matrix, control weight matrix, and convergence threshold, and providing a steady-state reactive power reference value. Next, operational data of the doubly-fed induction generator grid-connected system is collected, including grid connection point voltage and wind farm output active power. Based on this, state variables and control inputs are constructed to establish a description of the unknown continuous-time nonlinear system. Then, a performance index function and a continuous-time state-action Q-function (i.e., performance evaluation function) are constructed, and the Critic network is used to approximate the Q-function. Based on this, a Bellman error function is constructed using online sampling data, the Critic network weights are updated using gradient descent, and the control law is reconstructed based on the Q-function minimization condition, i.e., selecting the control input that minimizes the current Q-function as the control quantity for the next time step. Finally, a convergence check is performed, comparing whether the estimated Q-function values ​​of two adjacent iterations satisfy the convergence condition shown in Equation 19. If satisfied, an additional reactive power reference command is output; otherwise, iteration continues until convergence.

[0080] The method provided in this application transforms the HJB equation solving problem, which relies on an accurate system model in traditional continuous-time optimal control, into a data-driven performance evaluation function approximation and optimization problem by constructing a performance evaluation function that simultaneously includes state variables and control inputs. This avoids the technical obstacle of accurately obtaining the system dynamics model due to factors such as wind speed fluctuations, parameter perturbations, and unmodeled dynamics. The controller can complete the online construction of the control law without needing to know the specific expressions of the system's internal dynamics and input gains. Furthermore, by using a Critic network to approximate the performance evaluation function and constructing a Bellman error using online sampling data to drive network weight updates, the method not only avoids the problems of high online computational load and excessive engineering complexity caused by the Actor-Critic dual-network structure in related technologies, but also enables the controller to continuously correct the network weights based on real-time system operating data. This allows for adaptive adjustment of the control strategy when the system operating conditions deviate or model parameters are perturbed, effectively improving the insufficient voltage recovery speed and dynamic adjustment capability of fixed-parameter controllers in different scenarios. This provides a command output control scheme for DFIG wind farm grid-connected systems under conditions of unknown models and time-varying parameters, balancing control performance, adaptive capability, and engineering practicality.

[0081] Furthermore, as Figure 1 In a specific implementation of the method, this application provides a command output control device for unknown continuous-time nonlinear systems, such as... Figure 3 As shown, the device includes: a data acquisition module 301, a function construction module 302, an approximate representation module 303, a solution module 304, and an output module 305.

[0082] The acquisition module 301 is used to acquire the operating data of the target continuous-time nonlinear system, construct system state variables based on the operating data, and construct additional control commands as control inputs. The function construction module 302 is used to define an instantaneous performance index function based on the system state variables and the control input, and to construct a performance evaluation function based on the instantaneous performance index function; The approximate representation module 303 is used to represent the estimated value of the performance evaluation function as a combination of Critic network weights and basis function vectors using a Critic network, so as to form an approximate representation of the performance evaluation function; The function construction module 302 is also used to collect the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment as online sampling data during system operation, construct the Bellman error function using the online sampling data, and update the weights of the Critic network using the gradient descent method in combination with the Bellman error function. The solution module 304 is used to combine the updated Critic network weights and the preset optimality conditions to calculate the control input that minimizes the estimated value of the approximate representation as the optimal control input at the current time. The output module 305 is used to determine whether the optimized control input meets the convergence condition, and if it is determined that the convergence condition is met, to output the optimized control input as an additional control command.

[0083] In a specific application scenario, the acquisition module 301 is used to acquire the operating data in real time during the operation of the target continuous-time nonlinear system. The target continuous-time nonlinear system is a doubly-fed induction generator wind farm grid-connected system, and the operating data includes at least the actual value of the grid connection point voltage. Actual value of active power output from wind farm Steady-state reactive power reference value The additional reactive power reference command output by the controller at the previous moment; using the following formula, based on the actual value of the grid connection point voltage... With target voltage Calculate the voltage deviation at the grid connection point : ; Using the following formula, based on the actual value of the active power output of the wind farm... With the target active power Calculate active power deviation : ; Based on the voltage deviation at the grid connection point and the active power deviation The system state variables constituted by the following expression The system state variables Used to describe the dynamic response process of the target continuous-time nonlinear system after a fault disturbance: ; Define the additional reactive power reference command according to the following expression. As the control input : .

[0084] In specific application scenarios, the function construction module 302 is used to establish an unknown continuous-time nonlinear affine system description of the target continuous-time nonlinear system as shown in the following formula. : ; in, For the system state variables, For the control input, For unknown nonlinear dynamic terms within the system, For unknown control input gain terms; based on the system state variables and the control input Define the instantaneous performance index function as shown in the following formula. : ; in, for transpose, This is the state weight matrix used to apply weighted penalties to the components of the system state variables. for transpose, This is a control weight matrix used to weight and penalize the amplitude of the control input; based on the instantaneous performance index function. Define the continuous-time performance index function as shown in the following formula. : ; in, From the current moment The initial future time variable, For at any time The system state variables, For at any time Control input; combined with the continuous-time performance index function Define value function And determine the partial derivatives of the value function. Description based on the unknown continuous-time nonlinear affine system The instantaneous performance index function and the partial derivatives Construct the Hamiltonian function as shown in the following formula: ; Combined with the value function Using the Hamilton function, construct the performance evaluation function shown in the following formula. : .

[0085] In specific application scenarios, the approximate representation module 303 is used to construct a representation based on the system state variables. and the control input The basis function vectors constituted , where the basis function vector It includes at least the quadratic term of the system state variables, the cross product term between the system state variables, the coupling product term between the system state variables and the control input, and the quadratic term of the control input; combined with the basis function vector. The performance evaluation function is approximated using the Critic network, and the estimated value of the performance evaluation function is constructed as the weights of the Critic network and the basis function vector. A linear combination of these forms yields the approximate representation shown in the following formula: ; in, This is an estimate of the performance evaluation function. The weights of the Critic network are... This is the transpose of the Critic network weights.

[0086] In specific application scenarios, the function construction module 302 is also used to collect the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment as the online sampling data during system operation; determine the current control strategy, and, based on the continuous-time Bellman equation, use the estimated value of the performance evaluation function. With target value The difference between them is used to construct the Bellman error function in the form of the performance evaluation function shown in the following formula. : ; Wherein, the target value It is constructed based on the online sampling data and the instantaneous performance index function; combined with the Bellman error function. A weight update expression is constructed. During each update, the Bellman error and gradient vector are calculated in real time based on the online sampling data at the current moment, according to the weight update expression, to complete an online update of the Critic network weights.

[0087] In specific application scenarios, the function construction module 302 is used to combine the Bellman error function. Construct the squared error function shown in the following formula. : ; With the squared error function For the weights of the Critic network The negative gradient direction is used as the weight update direction, and the Critic network weights are updated using the gradient descent method according to the following formula. Perform iterative updates: ; in, For the first Critic network weights during step update For the first The updated Critic network weights The preset Critic learning rate, The squared error function The gradient vector of the Critic network weights; the squared error function Substituting the expression into the update formula, we obtain the weight update expression shown in the following formula: .

[0088] In a specific application scenario, the solution module 304 is used to decompose the performance evaluation function to obtain the following expression: ; Using the control input as the optimization variable, and with the first partial derivative of the approximate estimate with respect to the control input being zero, the optimality condition shown in the following expression is obtained, thus determining the necessary condition for the approximate estimate to reach a minimum: ; According to the following formula and the aforementioned optimality condition, the estimated value that makes the approximate representation at the current time is solved. Relative to the control input The control input value that reaches the minimum value : ; Take the control input value As the optimal control input at the current moment.

[0089] In a specific application scenario, the output module 305 is used to obtain the estimated values ​​of the approximate representation corresponding to two adjacent iterations in the current iteration round. and ,in, This represents the iteration round number. Indicates the first The approximate representation of the weights of the Critic network determined after rounds of iteration. Indicates the first The performance evaluation function estimate is determined by the updated Critic network weights after each iteration; the overall change in the approximate estimate between two adjacent iterations is calculated using the following formula, and the overall change is compared with a preset convergence threshold. Comparison: ; in, This represents the difference between the approximate estimates corresponding to two consecutive iterations obtained through computation. This represents the overall change obtained after performing norm operations on the difference. A pre-defined positive scalar is used to determine whether the iterative process tends to be stable; if the overall change is less than the convergence threshold, the current iteration result is determined to meet the convergence condition, and the optimized control input obtained in the current iteration is output as the additional control command. Accordingly, the output module 305 is further configured to determine that the current iteration result does not meet the convergence condition if the overall change is greater than or equal to the convergence threshold, and return to re-execute the calculation steps of updating and optimizing the control input of the Critic network weights until the convergence condition is met.

[0090] In specific application scenarios, the output module 305 is further used to take the optimized control input output when the convergence condition is met as the additional control command. ; Obtain the pre-set steady-state reference command The additional control commands are processed according to the following formula. Superimposed on the steady-state reference command The above constitutes the actual control command. : ; The actual control command is output to the actuator control channel of the target continuous-time nonlinear system, and the actuator adjusts the operating state of the target continuous-time nonlinear system according to the actual control command.

[0091] The apparatus provided in this application transforms the HJB equation solving problem, which relies on an accurate system model in traditional continuous-time optimal control, into a data-driven performance evaluation function approximation and optimization problem by constructing a performance evaluation function that simultaneously includes state variables and control inputs. This avoids the technical obstacle of accurately obtaining the system dynamics model due to factors such as wind speed fluctuations, parameter perturbations, and unmodeled dynamics. The controller can complete the online construction of the control law without needing to know the specific expressions of the system's internal dynamics and input gains. Furthermore, by using a Critic network to approximate the performance evaluation function and constructing a Bellman error using online sampling data to drive network weight updates, it not only avoids the problems of high online computation and excessive engineering complexity caused by the Actor-Critic dual-network structure in related technologies, but also enables the controller to continuously correct the network weights based on real-time system operating data. This allows for adaptive adjustment of the control strategy when the system operating conditions deviate or model parameters are perturbed, effectively improving the insufficient voltage recovery speed and dynamic adjustment capability of fixed-parameter controllers in different scenarios. This provides a command output control scheme for DFIG wind farm grid-connected systems under conditions of unknown models and time-varying parameters, balancing control performance, adaptive capability, and engineering practicality.

[0092] It should be noted that other corresponding descriptions of the functional units involved in the instruction output control device for unknown continuous-time nonlinear systems provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in [the document] will not be repeated here.

[0093] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data that have been used with the user's permission or have been fully permitted by all parties.

[0094] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been known and agreed to by the relevant parties or has been fully permitted by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0095] The above embodiments and the technical features in the embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

[0097] In an exemplary embodiment, see Figure 4 An electronic device is also provided, comprising a bus 401, a processor 402, a memory 403, and a communication interface 404. It may also include an input / output interface 405 and a display device 406. The various functional units can communicate with each other via the bus. The memory 403 stores a computer program, which, from bottom to top, includes a kernel 4034 that directly manages the hardware, middleware 4033 that provides general services for applications, an application programming interface 4032 that defines calling specifications, and an application 4031 that implements specific business logic. The processor 402 is used to read from the memory 403 via the bus 401 and execute the program stored in the memory, performing the instruction output control method for unknown continuous-time nonlinear systems described in the above embodiments.

[0098] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the instruction output control method for an unknown continuous-time nonlinear system.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0100] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0101] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0102] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0103] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A command output control method for unknown continuous-time nonlinear systems, characterized in that, include: Collect the operating data of the target continuous-time nonlinear system, construct system state variables based on the operating data, and construct additional control commands as control inputs; Define an instantaneous performance index function based on the system state variables and the control input, and construct a performance evaluation function based on the instantaneous performance index function; The estimated value of the performance evaluation function is represented by a Critic network as a combination of Critic network weights and basis function vectors, thus forming an approximate representation of the performance evaluation function. During system operation, the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment are collected online as online sampling data. The Bellman error function is constructed using the online sampling data, and the weights of the Critic network are updated using the gradient descent method in combination with the Bellman error function. Combining the updated Critic network weights and the preset optimality conditions, the control input that minimizes the estimated value of the approximate representation is calculated as the optimal control input at the current moment; Determine whether the optimized control input meets the convergence condition, and if the convergence condition is met, output the optimized control input as an additional control command.

2. The method according to claim 1, characterized in that, The acquisition of operational data from the target continuous-time nonlinear system, the construction of system state variables based on the operational data, and the construction of additional control commands as control inputs include: During the operation of the target continuous-time nonlinear system, the operating data is collected in real time. The target continuous-time nonlinear system is a doubly-fed induction generator wind farm grid-connected system, and the operating data includes at least the actual value of the grid connection point voltage. Actual value of active power output from wind farm Steady-state reactive power reference value , and the additional reactive power reference command output by the controller at the previous moment; Using the following formula, based on the actual value of the grid connection point voltage... With target voltage Calculate the voltage deviation at the grid connection point , ； Using the following formula, based on the actual value of the active power output of the wind farm... With the target active power Calculate active power deviation , ； Based on the voltage deviation at the grid connection point and the active power deviation The system state variables constituted by the following expression The system state variables This is used to describe the dynamic response process of the target continuous-time nonlinear system after a fault disturbance. ； Define the additional reactive power reference command according to the following expression. As the control input , 。 3. The method according to claim 1, characterized in that, The step of defining an instantaneous performance index function based on the system state variables and the control input, and constructing a performance evaluation function based on the instantaneous performance index function, includes: Establish an unknown continuous-time nonlinear affine system description for the target continuous-time nonlinear system as shown in the following formula. , ， in, For the system state variables, For the control input, For unknown nonlinear dynamic terms within the system, The control input gain term is unknown; According to the system state variables and the control input Define the instantaneous performance index function as shown in the following formula. , ， in, for transpose, This is the state weight matrix used to apply weighted penalties to the components of the system state variables. for transpose, This is a control weight matrix used to apply a weighted penalty to the amplitude of the control input; According to the instantaneous performance index function Define the continuous-time performance index function as shown in the following formula. , , in, From the current moment The initial future time variable, For at any time The system state variables, For at any time Control input; Combined with the continuous-time performance index function Define value function And determine the partial derivatives of the value function. ; Based on the description of the unknown continuous-time nonlinear affine system The instantaneous performance index function and the partial derivatives Construct the Hamiltonian function as shown in the following formula. ； Combined with the value function Using the Hamilton function, construct the performance evaluation function shown in the following formula. , 。 4. The method according to claim 1, characterized in that, The step of using a Critic network to represent the estimated value of the performance evaluation function as a combination of Critic network weights and basis function vectors to form an approximate representation of the performance evaluation function includes: Constructing the system state variables and the control input The basis function vectors formed , where the basis function vector It includes at least the quadratic term of the system state variable, the cross product term between the system state variables, the coupling product term between the system state variable and the control input, and the quadratic term of the control input; Combining the basis function vector The performance evaluation function is approximated using the Critic network, and the estimated value of the performance evaluation function is constructed as the weights of the Critic network and the basis function vector. A linear combination of these forms yields the approximate representation shown in the following formula. ， in, This is an estimate of the performance evaluation function. The weights of the Critic network are... This is the transpose of the Critic network weights.

5. The method according to claim 1, characterized in that, The process of collecting system state variables, control inputs, and system state variables at the current and next time moments online during system operation as online sampling data, constructing a Bellman error function using this online sampling data, and updating the Critic network weights using a gradient descent method based on the Bellman error function includes: During system operation, the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment are collected as the online sampling data; Determine the current control strategy, and based on the continuous-time Bellman equation, use the estimated value of the performance evaluation function. With target value The difference between them is used to construct the Bellman error function in the form of the performance evaluation function shown in the following formula. , ， Wherein, the target value It is constructed based on the online sampling data and the instantaneous performance index function; Combined with the Bellman error function A weight update expression is constructed. During each update, the Bellman error and gradient vector are calculated in real time based on the online sampling data at the current moment, according to the weight update expression, to complete an online update of the Critic network weights.

6. The method according to claim 5, characterized in that, The combination of the Bellman error function Construct the weight update expression, including: Combined with the Bellman error function Construct the squared error function as shown in the following formula. , ； With the squared error function For the weights of the Critic network The negative gradient direction is used as the weight update direction, and the Critic network weights are updated using the gradient descent method according to the following formula. Perform iterative updates. ， in, For the first Critic network weights during step update For the first The updated Critic network weights The preset Critic learning rate, The squared error function The gradient vector of the Critic network weights; The squared error function Substituting the expression into the update formula, we obtain the weight update expression shown in the following formula. 。 7. The method according to claim 1, characterized in that, The step of combining the updated Critic network weights and preset optimality conditions to calculate the control input that minimizes the approximate estimate as the optimal control input for the current time step includes: The performance evaluation function is decomposed to obtain the following expression: ； Using the control input as the optimization variable, and with the first partial derivative of the approximate estimate with respect to the control input being zero, the optimality condition shown in the following expression is obtained, thus determining the necessary condition for the approximate estimate to reach a minimum value. ； According to the following formula and the aforementioned optimality condition, the estimated value that makes the approximate representation at the current time is solved. Relative to the control input The control input value that reaches the minimum value , ； Take the control input value As the optimal control input at the current moment.

8. The method according to claim 1, characterized in that, The step of determining whether the optimized control input meets the convergence condition, and outputting the optimized control input as an additional control command when the convergence condition is met, includes: In the current iteration, obtain the estimated values ​​of the approximate representation corresponding to two adjacent iterations. and ,in, This represents the iteration round number. Indicates the first The approximate representation of the weights of the Critic network determined after rounds of iteration. Indicates the first The estimated value of the performance evaluation function determined by the updated Critic network weights after each iteration; The overall change in the approximate estimate between two consecutive iterations is calculated using the following formula, and the overall change is compared with a preset convergence threshold. Comparison, ， in, This represents the difference between the approximate estimates corresponding to two consecutive iterations obtained through computation. This represents the overall change obtained after performing norm operations on the difference. A pre-defined positive scalar value used to determine whether the iterative process tends to be stable; If the overall change is less than the convergence threshold, the current iteration result is determined to meet the convergence condition, and the optimized control input obtained in the current iteration is output as the additional control command. Accordingly, the method further includes: If the overall change is greater than or equal to the convergence threshold, the current iteration result is determined not to meet the convergence condition, and the process of updating the Critic network weights and calculating the optimization control input is repeated until the convergence condition is met.

9. The method according to claim 1, characterized in that, After determining whether the optimized control input meets the convergence condition, and outputting the optimized control input as an additional control command if the convergence condition is met, the method further includes: The optimized control input output when the convergence condition is met is used as the additional control command. ; Obtain the pre-set steady-state reference command The additional control commands are processed according to the following formula. Superimposed on the steady-state reference command The above constitutes the actual control command. , ； The actual control command is output to the actuator control channel of the target continuous-time nonlinear system, and the actuator adjusts the operating state of the target continuous-time nonlinear system according to the actual control command.

10. A command output control device for unknown continuous-time nonlinear systems, characterized in that, include: The acquisition module is used to acquire the operating data of the target continuous-time nonlinear system, construct system state variables based on the operating data, and construct additional control commands as control inputs. The function construction module is used to define an instantaneous performance index function based on the system state variables and the control input, and to construct a performance evaluation function based on the instantaneous performance index function; An approximation representation module is used to represent the estimated value of the performance evaluation function as a combination of Critic network weights and basis function vectors using a Critic network, so as to form an approximate representation of the performance evaluation function; The function construction module is also used to collect the system state variables at the current moment, the control input at the current moment, and the system state variables at the next moment as online sampling data during system operation, construct the Bellman error function using the online sampling data, and update the weights of the Critic network using the gradient descent method in combination with the Bellman error function. The solution module is used to combine the updated Critic network weights and the preset optimality conditions to calculate the control input that minimizes the estimated value of the approximate representation as the optimal control input at the current time. The output module is used to determine whether the optimized control input meets the convergence condition, and if it is determined that the convergence condition is met, to output the optimized control input as an additional control command.

Images