Converter power and DC voltage regulation and control method and system

Abstract

The invention discloses a method and a system for regulating and controlling the power and the direct-current voltage of a converter. The method comprises the following steps: acquiring the power and the direct-current voltage of the converter in a power grid; an error comprehensive index PSVRE is constructed; training the intelligent agent by adopting reinforcement learning based on the error comprehensive index PSVRE; according to the converter power and the DC voltage, obtaining a DC voltage reference value variable quantity of the converter based on the trained intelligent agent; and regulating and controlling the power and the direct-current voltage of the converter based on the direct-current voltage reference value variable quantity. According to the method, a real-time control decision can be made under the condition of not depending on a comprehensive system model through an error comprehensive index PSVRE and a PPO algorithm in reinforcement learning, so that the dynamic characteristics of large fluctuation amplitude and high speed of offshore wind power are effectively dealt with, safe and stable operation of a multi-terminal flexible direct current system is ensured, in addition, the method does not depend on a communication system, and the operation is simple and convenient. The system can still work under the condition of communication deficiency, and the stability of the multi-terminal flexible direct current system is not affected.

Description

Technical Field

[0001] The present invention belongs to the technical field of high-voltage flexible direct current grid operation control, and in particular relates to a method and system for regulating converter power and direct current voltage. Background Art

[0002] It is crucial for high-voltage flexible DC power grids to consider both DC voltage regulation and active power distribution after encountering sudden faults. At present, the control mode of DC systems is mainly active power-voltage droop control. However, due to the limitation of the number of converters based on active power-voltage droop control, that is, the number of control degrees of freedom, it is difficult to balance the regulation of all node voltages and the distribution of active power of each converter under traditional analytical algorithms. At present, the average DC voltage of all nodes is widely used as the control object to alleviate the problem of insufficient control freedom. However, the average DC voltage cannot fully reflect the characteristics of the DC voltage of the entire network, and the distribution of active power will also be affected by line resistance and DC droop coefficient. How to simultaneously achieve the regulation of DC voltage of all nodes and the reasonable distribution of converter active power is a problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides a method and system for regulating converter power and DC voltage, which can achieve regulation of DC voltages at all nodes and reasonable allocation of converter active power.

[0004] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] In a first aspect, a method for regulating converter power and DC voltage is provided, comprising:

[0006] Obtain converter power and DC voltage in the power grid;

[0007] Construct the comprehensive error index PSVRE;

[0008] Reinforcement learning is used to train the agent based on the comprehensive error index PSVRE;

[0009] According to the converter power and DC voltage, the DC voltage reference value change of the converter is obtained based on the trained intelligent agent;

[0010] The power and DC voltage of the converter are regulated based on the variation of the DC voltage reference value.

[0011] In combination with the first aspect, further, the error comprehensive index PSVRE is expressed as:

[0012]

[0013] Where ΔP is the active power variation matrix of the droop control converter, H is the available margin matrix of the droop control converter, H = [H1, H2, ..., H m ] T , ΔP=[ΔP1,ΔP2,…,ΔP m ] T , ΔP m is the power variation of the mth droop control converter, H m is the available power margin of the mth droop control converter, P mis is the power difference, m is the number of droop control converters, H j is the available power margin of the jth droop control converter, V * and V sch are the DC voltage vector of the droop control converter after the fault occurs and the DC voltage vector of the set droop control converter, P rated is the rated power of the droop control converter, V rated Rated voltage of the converter for droop control.

[0014] Combined with the first aspect, further training of the agent includes:

[0015] The PPO algorithm in reinforcement learning is used to train the agent. During the agent training process, the error comprehensive index PSVRE is minimized to the greatest extent. The objective function is as follows:

[0016]

[0017] Among them, F(π) represents the objective function to be minimized under strategy π, that is, the value of the error comprehensive index PSVRE; π * represents the optimal strategy that satisfies all constraints; C i (π) represents the violation of the i-th constraint under strategy π; M represents the number of constraints; θ is the strategy parameter;

[0018] The optimization of the objective function is transformed into an unconstrained optimization problem by the following formula:

[0019]

[0020] Among them, L(π,λ) is the behavior value function, λ is the Lagrange multiplier, and λ i is the Lagrange multiplier included in the i-th constraint, λ * is the optimal value of the Lagrange multiplier, (π * ,λ * ) is the joint optimal solution that satisfies the objective function F(π) and all constraints.

[0021] Combined with the first aspect, further, the behavior value function in the PPO algorithm is expressed as follows:

[0022] L(a,θ μ ,λ,μ)=f(a)+λ T G(a)+μ T H(a) (4)

[0023] Among them, L(a,θ μ ,λ,μ) is the representation of the behavior value function L(π,λ) in the PPO algorithm, a is the behavior of the agent, μ is the non-negative coefficient of the inequality constraint, θ μ is the strategy parameter related to the non-negative coefficient μ of the inequality, f(a) is the representation of the objective function F(π) in the PPO algorithm, G(a) is the equality constraint, and H(a) is the inequality constraint.

[0024] Combined with the first aspect, further, in the process of training the intelligent agent, the policy parameters are obtained by calculating the policy network gradient. The policy network gradient calculation formula is as follows:

[0025]

[0026] in, Expressing expectation, r t (θ) is the current strategy and the old strategy in a given state s t and behavior a t The probability ratio under the condition, clip is the clipping function, ∈ is the hyperparameter controlling the clipping range, is the advantage function;

[0027] The policy parameters are updated as follows:

[0028]

[0029] in, is the policy network gradient, θ n is the updated policy parameter, α a is the learning rate.

[0030] In combination with the first aspect, further, the policy network gradient can be decomposed into the objective function gradient related to the behavior and the behavior gradient with respect to the policy parameters The expression is as follows:

[0031]

[0032] in, is the behavior-related gradient, is the gradient associated with the policy parameters, h k(a) is the kth inequality constraint, μ k is the coefficient of the kth inequality constraint, N λ is the number of inequality constraints, is the functional form of the strategy parameter θ after the nth iteration, Respectively represent in a given state s t The droop control converter DC power, droop control converter DC voltage and voltage reference value of the lth round of training.

[0033] Combined with the first aspect, further, in the process of training the agent, the constraint policy parameter gradient is also considered Adjust the strategy parameters through the Lagrange multiplier gradient Adjust the Lagrange multipliers to take into account the gradient of the policy parameters under the constraints and the Lagrange multiplier gradient The expression is as follows:

[0034]

[0035] Where N is the number of samples, s i 、a i They represent the state and behavior corresponding to the i-th constraint, M is the number of constraints, and C i (s i ,a i ) is the constraint cost.

[0036] In a second aspect, the present invention further provides a control system for converter power and DC voltage, comprising:

[0037] A data acquisition module, used to obtain converter power and DC voltage in the power grid;

[0038] The indicator construction module is used to construct the error comprehensive indicator PSVRE;

[0039] The training module is used to train the agent using reinforcement learning based on the comprehensive error index PSVRE;

[0040] A control module is used to obtain a DC voltage reference value adjustment amount of the converter based on the trained intelligent agent according to the converter power and DC voltage;

[0041] The power and DC voltage of the converter are regulated based on the DC voltage reference value adjustment amount.

[0042] Beneficial technical effects: The present invention adopts the PPO algorithm in reinforcement learning, and trains the intelligent agent by constructing the error comprehensive index PSVRE to find the multi-objective optimal control parameters of the voltage source converter, so as to achieve the purpose of optimizing the DC voltage of the converter and the reasonable distribution of power. Through the error comprehensive index PSVRE and the PPO algorithm in reinforcement learning, real-time control decisions can be made without relying on the comprehensive system model, thereby effectively coping with the dynamic characteristics of large fluctuation amplitude and fast speed of offshore wind power, and ensuring the safe and stable operation of the multi-terminal flexible DC system. In addition, the present invention does not rely on the communication system, can still work in the absence of communication, and does not affect the stability of the multi-terminal flexible DC system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a single line diagram of a multi-terminal flexible DC test system with five nodes;

[0044] Figure 2 It is a control block diagram of a flexible control method for power-DC voltage of a high-voltage flexible DC system converter based on reinforcement learning proposed by the present invention;

[0045] Figure 3a To verify the active power diagram of the voltage source converter in Example 1;

[0046] Figure 3b To verify the DC voltage schematic diagram in Example 1;

[0047] Figure 3c To verify the DC average voltage schematic diagram in Example 1;

[0048] Figure 3d This is a schematic diagram of the comprehensive error index PSVRE in verification example 1;

[0049] Figure 4a To verify the active power diagram of the voltage source converter in Example 2;

[0050] Figure 4b To verify the DC voltage schematic diagram in Example 2;

[0051] Figure 4c To verify the DC average voltage schematic diagram in Example 2;

[0052] Figure 4d Schematic diagram of the comprehensive error index PSVRE in verification example 2. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0054] Example 1

[0055] The present invention proposes a method for regulating converter power and DC voltage based on reinforcement learning, which takes the regulation of DC voltage of all nodes (converters) and the distribution of active power of converters as control targets, and is achieved by changing the DC voltage reference value of the converter under droop control. The control method is based on the proximal policy optimization (PPO) algorithm of reinforcement learning, embedded in the basic framework of decentralized control, and does not rely on the communication system.

[0056] The distributed control framework includes a secondary control layer, a primary control layer, and a physical layer.

[0057] The control method of the present invention is mainly embodied in the secondary control layer. The adjustment amount of the variable voltage reference value, that is, the change amount of the DC voltage reference value of the converter, is calculated through the proximal policy optimization (PPO) algorithm based on reinforcement learning. Then, the droop controller located in the primary control layer outputs a signal according to the change amount of the DC voltage reference value, and then outputs the control signal to the physical layer after PWM modulation to control the converter located in the layer, thereby completing the adjustment of the DC voltage of the converter and then reasonably allocating the active power of the converter.

[0058] The process of the converter power and DC voltage control method provided by the present invention is mainly as follows:

[0059] Obtain converter power and DC voltage in the power grid;

[0060] The error comprehensive index PSVRE is constructed to train the intelligent agent in the PPO algorithm. The expression of the error comprehensive index PSVRE is as follows:

[0061]

[0062] Where ΔP is the active power variation matrix of the droop control converter, H is the available margin matrix of the droop control converter, H = [H1, H2, ..., H m ] T , ΔP=[ΔP1,ΔP2,…,ΔP m ] T , ΔP m is the power variation of the mth droop control converter, H m is the available power margin of the mth droop control converter, P mis is the power difference, m is the number of droop control converters, H j is the available power margin of the jth droop control converter, V * and V sch are the DC voltage vector of the droop control converter after the fault occurs and the DC voltage vector of the set droop control converter, P ratedis the rated power of the droop control converter, V rated Rated voltage of the converter for droop control.

[0063] The training of the agent includes taking the converter power and DC voltage as the state of the agent in the PPO algorithm (s t ) is input to the agent, and the agent outputs behavior a according to the obtained state t (DC voltage reference value change ΔV ref ), the change in the DC voltage reference value is used as an output signal and then modulated by PWM to output the control signal to the power grid to control the converter involved in the droop control. The corresponding converter power and DC voltage will be fed back to the intelligent agent, and the intelligent agent will continue to adjust the output according to the feedback, and this cycle will be repeated until the goal is achieved and the training is completed.

[0064] The training specifically includes: minimizing the error comprehensive index PSVRE to the greatest extent during the agent training process. The objective function is as follows:

[0065]

[0066] Among them, F(π) represents the objective function to be minimized under strategy π, that is, the value of the error comprehensive index PSVRE; π * represents the optimal strategy that satisfies all constraints; C i (π) represents the violation of the i-th constraint under strategy π; M represents the number of constraints; θ is the strategy parameter;

[0067] The optimization of the objective function is transformed into an unconstrained optimization problem by the following formula:

[0068]

[0069] Among them, L(π,λ) is the behavior value function, λ is the Lagrange multiplier, and λ i is the Lagrange multiplier included in the i-th constraint, λ * is the optimal value of the Lagrange multiplier, (π * ,λ * ) is the joint optimal solution that satisfies the objective function F(π) and all constraints.

[0070] Among them, the behavior value function is expressed in the PPO algorithm as follows:

[0071] L(a,θ μ ,λ,μ)=f(a)+λ T G(a)+μ T H(a) (4)

[0072] Among them, L(a,θ μ,λ,μ) is the representation of the behavior value function in the PPO algorithm, a is the behavior of the agent, μ is the non-negative coefficient of the inequality constraint, θ μ is the strategy parameter related to the non-negative coefficient μ of the inequality, f(a) is the representation of the objective function in the PPO algorithm, G(a) is the equality constraint, and H(a) is the inequality constraint.

[0073] In the process of training the agent, the policy parameters are obtained by calculating the policy network gradient. The policy network gradient calculation formula is as follows:

[0074]

[0075] in, Expressing expectation, r t (θ) is the current strategy and the old strategy in a given state s t and behavior a y The probability ratio under the condition, clip is the clipping function, ∈ is the hyperparameter controlling the clipping range, is the advantage function;

[0076] The policy parameters are updated as follows:

[0077]

[0078] in, is the policy network gradient, θ n is the updated policy parameter, α a is the learning rate.

[0079] In the present invention, after the agent training is completed, the agent is trained according to the strategy parameter θ n The corresponding behavior can be output, that is, the DC voltage reference value change ΔV ref , thereby achieving the ultimate goal of regulating the DC voltage and reasonably distributing the active power of the converter.

[0080] The policy network gradient can be decomposed into the objective function gradient associated with the behavior and the behavior gradient with respect to the policy parameters The expression is as follows:

[0081]

[0082] in, is the behavior-related gradient, is the gradient associated with the policy parameters, h k (a) is the kth inequality constraint, μ k is the coefficient of the kth inequality constraint, N λ is the number of inequality constraints, is the functional form of the strategy parameter θ after the nth iteration, Respectively represent in a given state s t The droop control converter DC power, droop control converter DC voltage and voltage reference value of the lth round of training.

[0083] In the process of training the agent, the policy parameter gradient of the constraint is also considered Adjust the strategy parameters through the Lagrange multiplier gradient Adjust the Lagrange multipliers to take into account the gradient of the policy parameters under the constraints and the Lagrange multiplier gradient The expression is as follows:

[0084]

[0085] Where N is the number of samples, s i 、a i They represent the state and behavior corresponding to the i-th constraint, M is the number of constraints, and C i (s i ,a i ) is the constraint cost.

[0086] Example 2

[0087] The present invention also provides a control system for converter power and DC voltage, comprising:

[0088] A data acquisition module, used to obtain converter power and DC voltage in the power grid;

[0089] The indicator construction module is used to construct the error comprehensive indicator PSVRE;

[0090] The training module is used to train the agent using reinforcement learning based on the comprehensive error index PSVRE;

[0091] A control module is used to obtain a DC voltage reference value adjustment amount of the converter based on the trained intelligent agent according to the converter power and DC voltage;

[0092] The power and DC voltage of the converter are regulated based on the DC voltage reference value adjustment amount.

[0093] Below we verify the effect of the present invention through two specific cases.

[0094] like Figure 1As shown, a five-node multi-terminal flexible DC transmission system connected to an offshore wind farm (OWF, Offshore Wind Farm) is used to verify the voltage source converter (VSC) power and DC voltage regulation method based on reinforcement learning proposed in the present invention. In this system, the voltage source converter (Voltage Source Converter) VSCs-3 is connected to the offshore wind farm OWF-3, the voltage source converter VSCs-5 is connected to the offshore wind farm OWF-5, and the voltage source converters VSCs-1, VSCs-2 and VSCs-4 are connected to the three AC grids AC-1, AC-2 and AC-4 represented by the ideal voltage source model through impedance, forming a multi-terminal flexible DC grid architecture. The voltage source converter connected to the offshore wind farm operates in the active power control mode, while the voltage source converter connected to the AC grid adopts an adaptive voltage droop control strategy.

[0095] Verification Example 1

[0096] In the case of large power fluctuations in offshore wind power, the control method adopted by the present invention can distribute active power in proportion and adjust the DC voltage of each node of the system to a set reference value. The power generation of the offshore wind farm OWF-5 connected to the voltage source converter VSC-5 increases from -500MW to -850MW at time time=0.5s, and then the control method of the present invention is started at time=1s to achieve optimal control of system power distribution and voltage regulation. Figure 3a-3d It can be seen from the figure that the error of the control method adopted by the present invention is very small. Figure 3a It can be seen that after the power P5 of the voltage source converter VSC-5 increases, the power distribution of the voltage source converters VSC-1, VSC-2 and VSC-4 under fixed droop control is not ideal, because VSC-1 still has a relatively large active power margin compared with the other two droop controlled voltage source converters, while the active power P4 of the voltage source converter VSC-4 exceeds the limit value P after the fault. r (460WM). After the control method of the present invention is started at time time=1s, the active power of the voltage source converter is distributed according to the proportion of its power margin. Figure 3b It can be seen that the DC voltage curve increases as the power of the voltage source converter VSC-5 changes. In particular, the voltage of VSC-5 is close to the upper voltage limit (640kV). Figure 3b as well as Figure 3c As shown in FIG. 1 , after the control method of the present invention is used to adjust the voltage to the set value, the increase of the voltage curve is significantly alleviated and approaches the steady-state set value. Figure 3dAs shown in FIG. 1 , after the control method of the present invention is enabled, the established comprehensive index PSVRE is significantly optimized, indicating that the control method provided by the present invention can effectively realize the proportional distribution of converter power and the optimal regulation of DC voltage when the control freedom is insufficient (note that in the present invention, the converter power and DC voltage before the fault are not taken into account in the calculation of the established index PSVRE, so Figure 3d Only the second half of the optimization control is captured).

[0097] Verification Example 2

[0098] In the event of a fault where the converter connected to the AC grid stops operating, the entire system will lose a degree of control freedom. The control method provided by the present invention can still distribute active power in proportion in the absence of control freedom, and adjust the DC voltage of each node of the system to a value close to the set value. The voltage source converter VSC-2 completely stops operating at time = 0.5s, and then the control method provided by the present invention is started at t = 1s to achieve optimal control of system power distribution and voltage regulation. Figure 4a-4d It can be seen from the above that the control method provided by the present invention has a small error. Figure 4a It can be seen that after the voltage source converter VSC-2 is completely shut down, the power distribution of the remaining two voltage source converters VSC-1 and VSC-4 based on adaptive droop control is not ideal under fixed droop control, because the voltage source converter VSC-1 still has a relatively large active power margin compared with VSC-4, while the power of VSC-4 exceeds the limit value P after the fault. r (460WM). After the control method provided by the present invention is started at time time=1s, the active power of the two voltage source converters is distributed according to the ratio of their power margins. Figure 4b and Figure 4c It can be seen that the DC voltage curve is significantly optimized with the introduction of the control method provided by this invention, close to the steady-state set value, and the risk of each voltage (V1, V2, V3, V4, V5) exceeding the limit is significantly reduced. Figure 4d As shown in the figure, after the control method proposed in the present invention is used, the established comprehensive index PSVRE is also significantly optimized, with a reduction of up to 20 percentage points. This shows that the control method proposed in the present invention can effectively achieve the proportional distribution of converter power and the optimal regulation of DC voltage when the control freedom is insufficient (note that in the present invention, the converter power and DC voltage before the fault are not taken into account in the calculation of the established index PSVRE, so Figure 4d Only the second half of the optimization control is captured).

[0099] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for regulating converter power and DC voltage, characterized in that: include: Obtain converter power and DC voltage in the power grid; Construct the comprehensive error index PSVRE; Reinforcement learning is used to train the agent based on the comprehensive error index PSVRE; According to the converter power and DC voltage, the DC voltage reference value change of the converter is obtained based on the trained intelligent agent; The power and DC voltage of the converter are regulated based on the variation of the DC voltage reference value.

2. A method for controlling converter power and DC voltage according to claim 1, characterized in that: The error comprehensive index PSVRE expression is: Where ΔP is the active power variation matrix of the droop control converter, H is the available margin matrix of the droop control converter, H = [H1, H2, ..., H m ] T , ΔP=[ΔP1,ΔP2,…,ΔP m ] T , ΔP m is the power variation of the mth droop control converter, H m is the available power margin of the mth droop control converter, P mis is the power difference, m is the number of droop control converters, H j is the available power margin of the jth droop control converter, V * and V sch are the DC voltage vector of the droop control converter after the fault occurs and the DC voltage vector of the set droop control converter, P rated is the rated power of the droop control converter, V rated Rated voltage of the converter for droop control.

3. The method for controlling converter power and DC voltage according to claim 1, characterized in that: The training of the agent includes: The PPO algorithm in reinforcement learning is used to train the agent. During the agent training process, the error comprehensive index PSVRE is minimized to the greatest extent. The objective function is as follows: Among them, F(π) represents the objective function to be minimized under strategy π, that is, the value of the error comprehensive index PSVRE; π * represents the optimal strategy that satisfies all constraints; C i (π) represents the violation of the i-th constraint under strategy π; M represents the number of constraints; θ is the strategy parameter; The optimization of the objective function is transformed into an unconstrained optimization problem by the following formula: Among them, L(π,λ) is the behavior value function, λ is the Lagrange multiplier, and λ i is the Lagrange multiplier included in the i-th constraint, λ * is the optimal value of the Lagrange multiplier, (π * ,λ * ) is the joint optimal solution that satisfies the objective function F(π) and all constraints.

4. A method for controlling converter power and DC voltage according to claim 3, characterized in that: The behavior value function in the PPO algorithm is expressed as follows: L(a,θ μ ,λ,μ)=f(a)+λ T G(a)+μ T H(a) (4) Among them, L(a,θ μ ,λ,μ) is the representation of the behavior value function in the PPO algorithm, a is the behavior of the agent, μ is the non-negative coefficient of the inequality constraint, θ μ is the strategy parameter related to the non-negative coefficient μ of the inequality, f(a) is the representation of the objective function in the PPO algorithm, G(a) is the equality constraint, and H(a) is the inequality constraint.

5. A method for controlling converter power and DC voltage according to claim 4, characterized in that: In the process of training the agent, the policy parameters are obtained by calculating the policy network gradient. The policy network gradient calculation formula is as follows: in, Expressing expectation, r t (θ) is the current strategy and the old strategy in a given state s t and behavior a t The probability ratio under the condition, clip is the clipping function, ∈ is the hyperparameter controlling the clipping range, is the advantage function; The policy parameters are updated as follows: in, is the policy network gradient, θ n is the updated policy parameter, α a is the learning rate.

6. A method for controlling converter power and DC voltage according to claim 5, characterized in that: The policy network gradient can be decomposed into the objective function gradient associated with the behavior and the behavior gradient with respect to the policy parameters The expression is as follows: in, is the behavior-related gradient, is the gradient associated with the policy parameters, h k (a) is the kth inequality constraint, μ k is the coefficient of the kth inequality constraint, N λ is the number of inequality constraints, is the functional form of the strategy parameter θ after the nth iteration, Respectively represent in a given state s t The droop control converter DC power, droop control converter DC voltage and voltage reference value of the lth round of training.

7. A method for controlling converter power and DC voltage according to claim 3, characterized in that: In the process of training the agent, the gradient of the policy parameters constrained by Adjust the strategy parameters through the Lagrange multiplier gradient Adjust the Lagrange multipliers to take into account the gradient of the policy parameters under the constraints and the Lagrange multiplier gradient The expression is as follows: Where N is the number of samples, s i 、a i They represent the state and behavior corresponding to the i-th constraint, M is the number of constraints, and C i (s i ,a i ) is the constraint cost.

8. A control system for converter power and DC voltage, characterized in that: include: A data acquisition module, used to obtain converter power and DC voltage in the power grid; The indicator construction module is used to construct the error comprehensive indicator PSVRE; The training module is used to train the agent using reinforcement learning based on the comprehensive error index PSVRE; A control module is used to obtain a DC voltage reference value adjustment amount of the converter based on the trained intelligent agent according to the converter power and DC voltage; The power and DC voltage of the converter are regulated based on the DC voltage reference value adjustment amount.

Images