A method and system for regulating converter power and dc voltage

Abstract

The application discloses a kind of converter power and DC voltage regulation method and system, comprising: obtaining converter power and DC voltage in power grid;Error comprehensive index PSVRE is constructed;Based on error comprehensive index PSVRE, agent is trained using reinforcement learning;According to the converter power and DC voltage, the change amount of DC voltage reference value of converter is obtained based on the trained agent;The power and DC voltage of converter are regulated based on DC voltage reference value change amount.The application can make real-time control decisions without relying on comprehensive system model through error comprehensive index PSVRE and PPO algorithm in reinforcement learning, thereby effectively dealing with the dynamic characteristics of large amplitude and fast speed of offshore wind power fluctuation, ensuring the safe and stable operation of multi-terminal flexible DC system.In addition, the application does not rely on communication system, and can still work in the absence of communication, and does not affect the stability of multi-terminal flexible DC system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of high-voltage flexible DC power grid operation control, and particularly relates to a converter power and DC voltage regulation method and system. BACKGROUND

[0002] It is crucial to simultaneously consider the regulation of DC voltage and the distribution of active power after the high-voltage flexible DC power grid encounters a sudden fault, and the current control mode of the DC system is mainly active power-voltage droop control. However, due to the limitation of the number of converters based on active power-voltage droop control, i.e., the number of control degrees of freedom, it is difficult to balance the regulation of all node voltages and the distribution of active power of each converter under the traditional analytical algorithm. At present, the average DC voltage of all nodes is widely used as a control object to alleviate the problem of insufficient control degrees of freedom, but the average DC voltage cannot fully reflect the characteristics of the DC voltage of the entire network, and the distribution of active power will also be affected by the line resistance and DC droop coefficient to produce deviation. How to simultaneously realize the regulation of all node DC voltages and the reasonable distribution of converter active power is a problem that needs to be solved at present. SUMMARY

[0003] In order to solve the problems existing in the prior art, the application provides a converter power and DC voltage regulation method and system, which can realize the regulation of all node DC voltages and the reasonable distribution of converter active power.

[0004] The technical problem to be solved by the application is solved by the following technical scheme:

[0005] In a first aspect, a converter power and DC voltage regulation method is provided, comprising:

[0006] Obtaining converter power and DC voltage in a power grid;

[0007] Constructing an error comprehensive index PSVRE;

[0008] Training an agent based on the error comprehensive index PSVRE using reinforcement learning;

[0009] According to the converter power and DC voltage, the DC voltage reference value change of the converter is obtained based on the trained agent;

[0010] Regulating the power and DC voltage of the converter based on the DC voltage reference value change.

[0011] In combination with the first aspect, further, the error comprehensive index PSVRE expression is:

[0012]

[0013] Wherein, ΔP is the active power change matrix of the droop control converter, H is the available margin matrix of the droop control converter, H = [H1, H2, …, Hm], ΔP = [ΔP1, ΔP2, …, ΔPm], P = [P1, P2, …, Pm], m is the number of the droop control converters, Hj is the available power margin of the jth droop control converter, V is the DC voltage vector of the droop control converter, V is the set DC voltage vector of the droop control converter, P is the rated power of the droop control converter, V is the rated voltage of the droop control converter. m ] T , ΔP = [ΔP1, ΔP2, …, ΔP m ] T , ΔP m is the power change amount of the mth droop control converter, H m is the available power margin of the mth droop control converter, P mis is the power difference, m is the number of the droop control converters, H j is the available power margin of the jth droop control converter, V * and V sch are the DC voltage vector of the droop control converter after the fault occurs and the set DC voltage vector of the droop control converter respectively, P rated is the rated power of the droop control converter, V rated is the rated voltage of the droop control converter.

[0014] In combination with the first aspect, further, the training of the intelligent agent includes:

[0015] The PPO algorithm in the reinforcement learning is adopted to train the intelligent agent, and the error comprehensive index PSVRE is reduced to the maximum extent in the intelligent agent training process. The objective function is as follows:

[0016]

[0017] Wherein, F(π) represents the objective function to be minimized under the policy π, that is, the value of the error comprehensive index PSVRE; π * represents the optimal policy that meets all the constraints; C i (π) represents the violation of the ith constraint under the policy π; M represents the number of constraints; θ is the policy parameter.

[0018] The optimization of the objective function is converted into an unconstrained optimization problem by the following formula:

[0019]

[0020] Wherein, L(π, λ) is the behavior value function, λ is the Lagrange multiplier, λ i is the Lagrange multiplier contained in the ith constraint, λ * is the optimal value of the Lagrange multiplier, (π * , λ * ) is the joint optimal solution that meets the optimal objective function F(π) and meets all the constraints.

[0021] With the first aspect, further, the behavior value function in the PPO algorithm is shown as follows:

[0022] L(a, θ μ , λ, μ) = f(a) + λ μ G(a) + μ t H(a) (4)

[0023] Wherein, L(a, θ t , λ, μ) is the form of behavior value function L(π, λ) in the PPO algorithm, a is the behavior of the agent, μ is the non-negative coefficient of inequality constraint, θ t is the policy parameter related to the inequality non-negative coefficient μ, f(a) is the form of objective function F(π) in the PPO algorithm, G(a) is the equality constraint, and H(a) is the inequality constraint.

[0024] With the first aspect, further, the policy parameter is obtained by calculating the policy network gradient in the process of training the agent, and the policy network gradient calculation formula is shown as follows:

[0025]

[0026] Wherein, represents expectation, r n (θ) is the probability ratio of the current policy and the old policy under the given state s a and behavior a k , clip is a clipping function, ∈ is a hyperparameter for controlling the clipping range, is an advantage function;

[0027] The policy parameter is updated as shown below:

[0028]

[0029] Wherein, is the policy network gradient, θ k is the updated policy parameter, and α λ is the learning rate.

[0030] With the first aspect, further, the policy network gradient can be decomposed into the behavior-related objective function gradient and the policy parameter-related behavior gradient The expression is shown as follows:

[0031]

[0032] Wherein, is the behavior-related gradient, is the policy parameter-related gradient, and h t(a) is the kth inequality constraint, μ k is the coefficient of the kth inequality constraint, N λ is the number of inequality constraints, is the function form of the policy parameter θ after the nth iteration, respectively represent the DC power of the droop control converter, the DC voltage of the droop control converter and the voltage reference value in the given state s t after the lth round of training.

[0033] In combination with the first aspect, further, in the process of training the agent, the policy parameter gradient considering the constraint is adjusted, the Lagrange multiplier gradient is adjusted, and the policy parameter gradient considering the constraint and the Lagrange multiplier gradient are expressed as follows:

[0034]

[0035] wherein N is the number of samples, s i , a i respectively represent the state and behavior corresponding to the ith constraint, M is the number of constraints, C i (s i , a i ) is the constraint cost.

[0036] In the second aspect, the application further provides a converter power and DC voltage regulation system, comprising:

[0037] a data acquisition module for acquiring the converter power and DC voltage in the power grid;

[0038] an index construction module for constructing an error comprehensive index PSVRE;

[0039] a training module for training an agent based on the error comprehensive index PSVRE using reinforcement learning;

[0040] a regulation module for obtaining a DC voltage reference value adjustment amount of the converter based on the trained agent according to the converter power and DC voltage;

[0041] regulating the power and DC voltage of the converter based on the DC voltage reference value adjustment amount.

[0042] Beneficial technical effects: the PPO algorithm in the reinforcement learning is adopted, the multi-objective optimal control parameters of the voltage source converter are found by training the agent through the construction of the error comprehensive index PSVRE, so that the purpose of optimizing and regulating the DC voltage of the converter and reasonably distributing the power is achieved, the real-time control decisions can be made without relying on the comprehensive system model through the error comprehensive index PSVRE and the PPO algorithm in the reinforcement learning, so that the dynamic characteristics of the large amplitude and fast speed of the offshore wind power fluctuation are effectively coped with, the safe and stable operation of the multi-terminal flexible DC system is ensured, in addition, the application does not rely on the communication system, and still can work in the case of communication loss, and does not affect the stability of the multi-terminal flexible DC system. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a single-line diagram of a multi-terminal flexible DC test system containing five nodes;

[0044] Figure 2 is a control block diagram of the flexible power-DC voltage regulation method of the high-voltage flexible DC system converter based on reinforcement learning proposed by the application;

[0045] Figure 3a is an active power schematic diagram of the voltage source converter in verification example 1;

[0046] Figure 3b is a DC voltage schematic diagram in verification example 1;

[0047] Figure 3c is a DC average voltage schematic diagram in verification example 1;

[0048] Figure 3d is an error comprehensive index PSVRE schematic diagram in verification example 1;

[0049] Figure 4a is an active power schematic diagram of the voltage source converter in verification example 2;

[0050] Figure 4b is a DC voltage schematic diagram in verification example 2;

[0051] Figure 4c is a DC average voltage schematic diagram in verification example 2;

[0052] Figure 4d is an error comprehensive index PSVRE schematic diagram in verification example 2. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application more clear, the application is further described in detail below in combination with the drawings and examples.

[0054] Example 1

[0055] The application provides a converter power and DC voltage regulation method based on reinforcement learning, which takes the regulation of DC voltages of all nodes (converters) and the distribution of active power of the converters as control targets and is realized by changing the DC voltage reference values of the converters under droop control.

[0056] The distributed control framework comprises a secondary control layer, a primary control layer and a physical layer.

[0057] The regulation method mainly reflects the secondary control layer, and the regulation amount of the variable voltage reference value is calculated by the reinforcement learning-based proximal policy optimization (PPO) algorithm, that is, the DC voltage reference value change amount of the converter, and then the droop controller in the primary control layer outputs a signal according to the DC voltage reference value change amount, and the control signal is output to the physical layer after PWM modulation to control the converter in the layer, so as to regulate the DC voltage of the converter and reasonably distribute the active power of the converter.

[0058] The process of the converter power and DC voltage regulation method provided by the application mainly comprises the following steps:

[0059] The converter power and DC voltage in the power grid are obtained.

[0060] An error comprehensive index PSVRE is constructed for training the agent in the PPO algorithm, and the expression of the error comprehensive index PSVRE is as follows:

[0061]

[0062] Wherein, ΔP is the active power change matrix of the droop control converter, H is the available margin matrix of the droop control converter, H=[H1, H2, …, Hm], ΔP=[ΔP1, ΔP2, …, ΔPm], V=[V1, V2, …, Vm], V=[V1, V2, …, Vm], P=[P1, P2, …, Pm], m is the number of the droop control converters, Hj is the available power margin of the jth droop control converter, Vj is the DC voltage vector of the droop control converter after the fault occurs, and Vj is the set DC voltage vector of the droop control converter. m ] T , ΔP=[ΔP1, ΔP2, …, ΔP m ] T , ΔP m is the power change amount of the mth droop control converter, H m is the available power margin of the mth droop control converter, P mis is the power difference, m is the number of the droop control converters, H j is the available power margin of the jth droop control converter, V * and V sch are the DC voltage vectors of the droop control converters after the fault occurs and the set DC voltage vectors of the droop control converters, respectively, and P ratedTo control the rated power of the droop converter, V rated To control the rated voltage of the droop converter.

[0063] The training of the agent includes inputting the converter power and DC voltage and DC voltage as the state (s t ) of the agent in the PPO algorithm, and the agent outputs the behavior a t (DC voltage reference value change ΔV ref ), which is output to the grid as an output signal after PWM modulation to control the converter participating in droop control, and the corresponding converter power and DC voltage are fed back to the agent, and the agent continues to adjust the output according to the feedback. Repeat this cycle until the target is reached to complete the training.

[0064] The training specifically includes: minimizing the error comprehensive index PSVRE to the greatest extent during the agent training process, and the objective function is as follows:

[0065]

[0066] Wherein, F(π) represents the objective function to be minimized under the policy π, that is, the value of the error comprehensive index PSVRE; π * represents the optimal policy that meets all constraints; C i (π) represents the violation of the i-th constraint under the policy π; M represents the number of constraints; θ is the policy parameter;

[0067] The optimization of the objective function is converted into an unconstrained optimization problem by the following formula:

[0068]

[0069] Wherein, L(π,λ) is the behavior value function, λ is the Lagrange multiplier, λ i is the Lagrange multiplier contained in the i-th constraint condition, λ * is the optimal value of the Lagrange multiplier, (π * ,λ * ) is the joint optimal solution that meets the optimal objective function F(π) and meets all constraints.

[0070] Wherein, the expression form of the behavior value function in the PPO algorithm is as follows:

[0071] L(a,θ μ ,λ,μ)=f(a)+λ T G(a)+μ T H(a) (4)

[0072] Wherein, L(a,θ μ, λ, μ) is the form of the behavior value function in the PPO algorithm, a is the behavior of the agent, μ is the non-negative coefficient of the inequality constraint, θ μ is the policy parameter related to the inequality non-negative coefficient μ, f(a) is the form of the objective function in the PPO algorithm, G(a) is the equality constraint, and H(a) is the inequality constraint.

[0073] The policy parameter is obtained by calculating the policy network gradient in the process of training the agent, and the policy network gradient calculation formula is as follows:

[0074]

[0075] Among them, represents expectation, r t (θ) is the probability ratio of the current policy and the old policy under the given state s t and the behavior a y , clip is a clipping function, ∈ is a hyperparameter for controlling the clipping range, is an advantage function;

[0076] The policy parameter is updated as follows:

[0077]

[0078] Among them, is the policy network gradient, θ n is the updated policy parameter, α a is the learning rate.

[0079] In the present application, when the agent training is completed, the agent can output the corresponding behavior according to the policy parameter θ n , that is, the direct current voltage reference value change ΔV ref , so as to achieve the final purpose of adjusting the direct current voltage and reasonably distributing the active power of the converter.

[0080] The policy network gradient can be decomposed into the behavior-related objective function gradient and the behavior gradient related to the policy parameter The expression is as follows:

[0081]

[0082] Among them, is the behavior-related gradient, is the policy parameter-related gradient, h k (a) is the kth inequality constraint, μ k is the coefficient of the kth inequality constraint, N λ is the number of inequality constraints, The function form of the strategy parameter theta after the nth iteration, respectively represent the state s t The DC power of the droop control converter, the DC voltage of the droop control converter and the voltage reference value of the lth round of training.

[0083] In the process of training the agent, the constrained strategy parameter gradient Adjust the strategy parameter, and adjust the Lagrange multiplier gradient Adjust the Lagrange multiplier, and consider the constrained strategy parameter gradient And the Lagrange multiplier gradient The expression is as follows:

[0084]

[0085] Wherein, N is the number of samples, s i , a i Respectively represent the state and behavior corresponding to the ith constraint, M is the number of constraints, C i (s i , a i ) is the constraint cost.

[0086] Embodiment 2

[0087] The application also provides a converter power and DC voltage regulation system, comprising:

[0088] A data acquisition module is configured to acquire the converter power and DC voltage in the power grid.

[0089] An index construction module is configured to construct an error comprehensive index PSVRE.

[0090] A training module is configured to train an agent based on the error comprehensive index PSVRE using reinforcement learning.

[0091] A regulation module is configured to obtain a DC voltage reference value adjustment amount of the converter based on the trained agent according to the converter power and DC voltage.

[0092] The power and DC voltage of the converter are regulated based on the DC voltage reference value adjustment amount.

[0093] In the following, we will verify the effect of the application through two specific cases.

[0094] For example Figure 1As shown, the five-node multi-terminal flexible HVDC system connected to offshore wind farms (OWF) is used to verify the proposed method for regulating the power and DC voltage of voltage source converters (VSCs) based on reinforcement learning. In this system, voltage source converters (VSCs-3) are connected to offshore wind farm OWF-3, voltage source converters (VSCs-5) are connected to offshore wind farm OWF-5, and voltage source converters (VSCs-1, VSCs-2, and VSCs-4) are connected to three AC grids (AC-1, AC-2, and AC-4) represented by ideal voltage source models through impedances, forming a multi-terminal flexible HVDC grid architecture. The voltage source converters connected to offshore wind farms operate in active power control mode, while the voltage source converters connected to AC grids use adaptive voltage droop control strategy.

[0095] Verification Example 1

[0096] Under the condition of large-scale power fluctuation of offshore wind power, the regulation method adopted by the present application can proportionally allocate active power and regulate the DC voltage of each node of the system to the set reference value. The power generation of offshore wind farm OWF-5 connected to voltage source converter VSC-5 increases from -500 MW to -850 MW at time time = 0.5 s, and then the regulation method of the present application is started at time time = 1 s to realize the optimal control of power distribution and voltage regulation of the system. From Figures 3a-3d It can be seen that the regulation method adopted by the present application has very small error. From Figure 3a It can be seen that after the power P5 of voltage source converter VSC-5 increases, the power distribution of voltage source converters VSC-1, VSC-2, and VSC-4 under fixed droop control is not ideal, because VSC-1 still has a relatively large active power margin compared with the other two droop-controlled voltage source converters, while the active power P4 of voltage source converter VSC-4 exceeds the limit value P r (460 WM) after the fault. After starting the regulation method of the present application at time time = 1 s, the active power of the voltage source converters is distributed in proportion to its power margin. On the other hand, from Figure 3b It can be seen that the DC voltage curve increases with the change of the power of voltage source converter VSC-5. In particular, the voltage of VSC-5 approaches the upper limit of voltage (640 kV). As Figure 3b and Figure 3c shown, after the regulation method of the present application is enabled to regulate the voltage to the set value, the increase of the voltage curve is obviously alleviated and tends to be a steady-state set value. As Figure 3dAs shown, after the control method of the present application is enabled, the established comprehensive index PSVRE is obviously optimized, indicating that the control method provided by the present application can effectively realize proportional distribution of converter power and optimal adjustment of DC voltage when the control freedom is insufficient (note that in the present application, the converter power and DC voltage before the fault are not considered in the calculation of the established index PSVRE, so Figure 3d only the latter half of the optimal control is intercepted).

[0097] Verification Example 2

[0098] In the fault condition of shutdown of the converter connected to the AC power grid, the entire system loses one control freedom, and the control method provided by the present application can still proportionally distribute active power and adjust the DC voltage of each node of the system to be close to the set value in the case of lack of control freedom. The voltage source converter VSC-2 is completely shut down at time time = 0.5s, and then the control method provided by the present application is started at t = 1s to realize optimal control of power distribution and voltage adjustment of the system. From Figures 4a-4d it can be seen that the control method provided by the present application is very close to the optimal value. From Figure 4a it can be seen that after the voltage source converter VSC-2 is completely shut down, the power distribution of the remaining two voltage source converters VSC-1 and VSC-4 based on adaptive droop control under fixed droop control is not ideal, because the voltage source converter VSC-1 still has a relatively large active power margin compared with VSC-4, while VSC-4 exceeds the limit value P r (460WM) after the fault. After the control method provided by the present application is started at time time = 1s, the active power of the two voltage source converters is distributed in proportion to its power margin. On the other hand, from Figure 4b and Figure 4c it can be seen that the DC voltage curve is obviously optimized with the introduction of the control method provided by the present application, close to the steady-state set value, and the risk of each voltage (V1, V2, V3, V4, V5) exceeding the limit is significantly reduced. As Figure 4d shown, after the control method proposed by the present application is enabled, the established comprehensive index PSVRE is also obviously optimized, with a decrease of up to 20 percentage points. The control method proposed by the present application can effectively realize proportional distribution of converter power and optimal adjustment of DC voltage when the control freedom is insufficient (note that in the present application, the converter power and DC voltage before the fault are not considered in the calculation of the established index PSVRE, so Figure 4d only the latter half of the optimal control is intercepted).

[0099] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0100] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0101] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0102] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the technical solutions of the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A method of regulating inverter power and DC voltage, characterized by The method comprises the following steps: obtaining the converter power and the DC voltage in the power grid; constructing an error comprehensive index PSVRE; training an agent based on the error comprehensive index PSVRE by using reinforcement learning; obtaining the DC voltage reference value change of the converter based on the trained agent according to the converter power and the DC voltage; controlling the power and the DC voltage of the converter based on the DC voltage reference value change; the error comprehensive index PSVRE is expressed as: Wherein, ΔP is the active power change matrix of the droop control converter, H is the available margin matrix of the droop control converter, H=[H1, H2, …, Hm]T, ΔP=[ΔP1, ΔP2, …, ΔPm]T, P=[P1, P2, …, Pm]T, m is the number of the droop control converters, Hj is the available power margin of the jth droop control converter, V is the DC voltage vector of the droop control converter after the fault occurs, V is the set DC voltage vector of the droop control converter, P is the power difference, P is the rated power of the droop control converter, V is the rated voltage of the droop control converter. m ] T , ΔP=[ΔP1, ΔP2, …, ΔP m ] T , ΔP m is the power change amount of the mth droop control converter, H m is the available power margin of the mth droop control converter, P mis is the power difference, m is the number of the droop control converters, H j is the available power margin of the jth droop control converter, V * and V sch are the DC voltage vector of the droop control converter after the fault occurs and the set DC voltage vector of the droop control converter respectively, P rated is the rated power of the droop control converter, V rated is the rated voltage of the droop control converter. The training of the agent comprises the following steps: training the agent by using the PPO algorithm in the reinforcement learning, and reducing the error comprehensive index PSVRE to the maximum extent in the agent training process, and the objective function is as follows: where F(π) represents the objective function to be minimized under policy π, i.e., the value of the error synthesis indicator PSVRE; π * represents the optimal policy that satisfies all constraints; C i represents the violation of the i-th constraint under policy π; M represents the number of constraints; θ is the policy parameter; the optimization of the objective function is converted into an unconstrained optimization problem by using the following formula: where L(π, λ) is the behavior value function, λ is the Lagrange multiplier, λ i is the Lagrange multiplier contained in the ith constraint, λ * is the optimal value of the Lagrange multiplier, (π * , λ * ) is the joint optimal solution that satisfies the optimality of the objective function F(π) and satisfies all constraints.

2. The method of claim 1, wherein the DC voltage is regulated by the converter power. the expression form of the behavior value function in the PPO algorithm is as follows: L(a, θ μ , λ, μ) = f(a) + λ T G(a) + μ T H(a) (4) wherein L(a, θ μ , λ, μ) is a form of the behavior value function in the PPO algorithm, a is the behavior of the agent, μ is a non-negative coefficient of the inequality constraint, θ μ is a policy parameter related to the inequality non-negative coefficient μ, f(a) is a form of the objective function in the PPO algorithm, G(a) is an equality constraint, and H(a) is an inequality constraint.

3. A method of regulating converter power and DC voltage according to claim 2, characterized in that, the policy network gradient is calculated to obtain the policy parameter in the process of training the agent, and the policy network gradient calculation formula is as follows: where, represents the expectation, r t (θ) is the probability ratio of the current policy and the old policy in a given state s t and action a t , clip is a clipping function, and ∈ is a hyperparameter that controls the clipping range, is the advantage function; the policy parameter is updated as follows: where, is the policy network gradient, θ n is the updated policy parameter, a a is the learning rate.

4. A method of regulating converter power and DC voltage according to claim 3, characterized in that, The policy network gradient can be decomposed into a target function gradient related to the behavior and a behavior gradient related to the policy parameters The expression is as follows: wherein, is the gradient related to the behavior, is the gradient related to the policy parameter, h k (a) is the kth inequality constraint, μ k is the coefficient of the kth inequality constraint, N λ is the number of inequality constraints, is the function form of the policy parameter θ after the nth iteration, respectively represent the droop-controlled converter DC power, the droop-controlled converter DC voltage and the voltage reference value in the lth round of training under the given state s t ​ 5. The method of claim 1, wherein the DC voltage is regulated by the converter power. considering the constrained policy parameter gradient adjusting the policy parameter by the Lagrange multiplier gradient adjusting the Lagrange multiplier considering the constrained policy parameter gradient and the Lagrange multiplier gradient The expression is as follows: Where N is the number of samples, s i , a i represents the state and behavior corresponding to the ith constraint, M is the number of constraints, C i (s i , a i ) is the constraint cost.

6. The control system of a control method of a converter power and a DC voltage according to claim 1, characterized by, The method comprises the following steps: a data acquisition module is arranged to obtain the converter power and the DC voltage in the power grid; an index construction module is arranged to construct an error comprehensive index PSVRE; a training module is arranged to train an agent based on the error comprehensive index PSVRE by using reinforcement learning; a control module is arranged to obtain the DC voltage reference value adjustment amount of the converter based on the trained agent according to the converter power and the DC voltage; the power and the DC voltage of the converter are controlled based on the DC voltage reference value adjustment amount.

Images