Direct current power distribution network power quality improvement method and system

Abstract

The application provides a direct current power distribution network power quality improvement method and system, comprising the following steps: S1: initializing parameters, setting initial parameters; based on the initial parameters, selecting an action in a discrete action space that maximizes the evaluation network Q value according to the current system state; executing the action to obtain a reward value and a next time state; storing the action, the current time state, the next time state and the reward value in the form of a four-tuple to an experience pool; randomly sampling a batch of experiences from the experience pool, calculating a corresponding time difference target according to the experiences; back propagating according to the time difference target, calculating a loss, and then updating the evaluation network parameters until the power grid power quality improvement is completed. The application is based on a model-free Double DQN algorithm, and an intelligent agent does not need to perform any explicit mathematical modeling on the power distribution network environment in the learning process, directly learns the mapping relationship between the state and the reward through the interaction with the environment, and obtains an optimal strategy.

Description

Technical Field

[0001] This invention belongs to the field of power system control technology, specifically relating to a method and system for improving power quality in DC distribution networks. More specifically, it relates to a DC distribution network power quality improvement technology based on the Double DQN algorithm. Background Technology

[0002] With the call for carbon reduction, the demand for increasing the proportion of distributed renewable energy generation such as wind and solar power in the power grid is growing daily. However, distributed generation is highly volatile and uncertain, and its large-scale application and integration pose significant challenges to traditional power distribution systems. DC power distribution networks, capable of efficiently absorbing diverse and abundant distributed renewable energy sources and solving the grid connection problem, have become a research hotspot both domestically and internationally. Among these, DC power quality management is one of the key issues in the design, planning, and operation control of DC power distribution systems.

[0003] In DC distribution networks, there are a large number of constant power loads, or CPLs, which are closed-loop controlled power electronic converters with extremely strong negative impedance and nonlinear characteristics. These constant power loads are usually strictly regulated by high-bandwidth local controllers, which can easily lead to rapid changes in load power over a wide range, resulting in DC bus voltage fluctuations and a deterioration in system power quality.

[0004] Currently, voltage control methods for DC distribution networks can be divided into linear and nonlinear control methods; however, these methods typically require a known mathematical model of the controlled object. As a nonlinear and high-dimensional dynamic system, the power system is extremely difficult to model accurately. Therefore, intelligent control strategies that do not rely on precise modeling have attracted attention in recent years. In the field of DC-DC converters, many intelligent controllers have emerged that combine traditional control methods with artificial intelligence methods such as fuzzy neural networks, particle swarm optimization, and recurrent neural networks. These intelligent controllers reference the experience of previous supervised learning models, optimizing dynamic performance while ensuring system stability margin. However, the learning and adaptability of these methods are limited, necessitating the search for a more efficient intelligent control strategy.

[0005] In summary, the problem of power quality degradation in DC distribution networks caused by constant power loads urgently needs to be addressed.

[0006] Reference 1: Dai Yuxuan, Cui Chenggang. Control strategy for Boost converter based on deep reinforcement learning [J]. Journal of System Simulation, 2023, 35(05): 1109-1119, discloses an intelligent control strategy based on model-free deep reinforcement learning to control the converter voltage. However, the action space of the agent is different under different operating conditions, which requires retraining, resulting in low efficiency and strong limitations.

[0007] Reference 2: Gheisarnejad M, Farsizadeh H, Khooban M HA Novel Nonlinear DeepReinforcement Learning Controller for DC–DC Power Buck Converters[J].IEEE Transactions on Industrial Electronics, 2021, 68(8):6849-6858, discloses a DDPG controller to compensate for a proportional-integral sliding mode controller, which can adjust the output voltage of a single DC-DC buck converter equipped with a constant power load. This method enhances the controller's ability to handle nonlinear loads, but if the system encounters operating conditions not covered during training during online operation, it cannot guarantee system stability. In addition, this method still relies on system modeling. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for improving the power quality of DC distribution networks.

[0009] A method for improving power quality in a DC distribution network according to the present invention includes:

[0010] Step S1: Initialize parameters and set initial parameters;

[0011] Step S2: Based on the initial parameters and the current system state, select the action in the discrete action space that maximizes the Q-value of the evaluation network;

[0012] Step S3: Execute the action to obtain the reward value and the state at the next moment;

[0013] Step S4: Store the action, the current state, the next state, and the reward value in the experience pool as a quadruple;

[0014] Step S5: Randomly sample a batch of experiences from the experience pool, calculate the corresponding time-series differential target based on the experiences; backpropagate according to the time-series differential target to calculate the loss, then update the evaluation network parameters, and repeat step S2 until the power quality of the power grid is improved.

[0015] Preferably, in step S1, the parameters include: evaluation network parameters, target network parameters, experience pool, and learning rate; the initial parameters include: training epochs, initial system state, and sampling time.

[0016] Step S2 includes:

[0017] Step S2.1: Input the current system state into the evaluation network;

[0018] Step S2.2: Estimate the Q-value corresponding to each discrete action in the current system state by evaluating the network;

[0019] Step S2.3: Select the action that maximizes the Q value.

[0020] Preferably, in step S3, the agent performs an action to update the state to the next moment, generating voltage error values ​​and current error values; the voltage error values ​​and current error values ​​are substituted into the reward function to obtain the reward value at the current moment;

[0021] The reward function is expressed mathematically as follows:

[0022] r(t)=r1(t)+r2(t) (6)

[0023] Where r(t) represents the reward function, t represents time; r1(t) represents the reward function for the first sub-objective; and r2(t) represents the reward function for the second sub-objective.

[0024] The reward function for the first sub-objective is expressed mathematically as follows:

[0025]

[0026] Among them, e v (t) represents the voltage error signal;

[0027] The reward function for the second sub-objective is expressed mathematically as follows:

[0028]

[0029] Among them, e i (t) represents the difference in the converter output current.

[0030] Preferably, in step S5, the mathematical expression for the time-series difference objective is:

[0031]

[0032] in, Let r(t) represent the time-series difference objective; r(t) represent the reward value; γ represent the discount factor; and Q represent the action value function. This represents the agent action that maximizes the Q-value; 'a' represents the learning rate; and 'w' represents the network parameters being evaluated. tar Indicates the target network parameters;

[0033] The loss is obtained through a loss function; the mathematical expression of the loss function is:

[0034]

[0035] Where L(w) represents the loss function; Represents the time-series difference objective at time t; s t Indicates the system state at time t; a t This represents the action performed by the agent at time t;

[0036] In step S5, based on the loss, the network parameters are updated and evaluated by minimizing the loss function, the mathematical expression of which is:

[0037]

[0038] Where, the symbol ← indicates assignment; α represents the learning rate; |B| is the number of experience points sampled from the experience pool; This indicates finding the gradient of Q with respect to w; s t Indicates the system state at time t; a t B represents the action taken by the agent at time t; B represents a batch of experience obtained by random sampling.

[0039] In step S5, the target network parameters and the updated evaluation network parameters are weighted and averaged, and the result is assigned to the target network; the target network is a neural network.

[0040] The target network parameter w tar The mathematical expression is:

[0041] w tar ←(1-α smooth )w tar +α smooth w tar (11)

[0042] Where, α smooth This represents the target smoothing factor.

[0043] A power quality improvement system for a DC distribution network according to the present invention includes:

[0044] Module M1: Initializes parameters, sets initial parameters;

[0045] Module M2: Based on the initial parameters and the current system state, select the action in the discrete action space that maximizes the Q-value of the evaluation network;

[0046] Module M3: Executes the action to obtain the reward value and the state at the next moment;

[0047] Module M4: Stores the action, current state, next state, and reward value in the experience pool as a quadruple;

[0048] Module M5: Randomly samples a batch of experiences from the experience pool, calculates the corresponding time-series differential target based on the experiences, backpropagates based on the time-series differential target, calculates the loss, updates and evaluates the network parameters, and re-triggers module M2 to work until the power quality of the power grid is improved.

[0049] Preferably, in module M1, the parameters include: evaluation network parameters, target network parameters, experience pool, and learning rate; the initial parameters include: training epochs, initial system state, and sampling time.

[0050] The module M2 includes:

[0051] Module M2.1: Inputs the current system state into the evaluation network;

[0052] Module M2.2: Estimates the Q-value for each discrete action in the current system state by evaluating the network;

[0053] Module M2.3: Select the action that maximizes the Q value.

[0054] Preferably, in module M3, the agent performs an action to update the state to the next moment, generating voltage error values ​​and current error values; the voltage error values ​​and current error values ​​are substituted into the reward function to obtain the reward value at the current moment;

[0055] The reward function is expressed mathematically as follows:

[0056] r(t)=r1(t)+r2(t) (6)

[0057] Where r(t) represents the reward function, t represents time; r1(t) represents the reward function for the first sub-objective; and r2(t) represents the reward function for the second sub-objective.

[0058] The reward function for the first sub-objective is expressed mathematically as follows:

[0059]

[0060] Among them, e v (t) represents the voltage error signal;

[0061] The reward function for the second sub-objective is expressed mathematically as follows:

[0062]

[0063] Among them, e i (t) represents the difference in the converter output current.

[0064] Preferably, in module M5, the timing difference objective is mathematically expressed as:

[0065]

[0066] in, Let r(t) represent the time-series difference objective; r(t) represent the reward value; γ represent the discount factor; and Q represent the action value function. This represents the agent action that maximizes the Q-value; 'a' represents the learning rate; and 'w' represents the network parameters being evaluated. tar Indicates the target network parameters;

[0067] The loss is obtained through a loss function; the mathematical expression of the loss function is:

[0068]

[0069] Where L(w) represents the loss function; Represents the time-series difference objective at time t; s t Indicates the system state at time t; a t This represents the action performed by the agent at time t;

[0070] In module M5, based on the loss, the network parameters are updated and evaluated by minimizing the loss function, the mathematical expression of which is:

[0071]

[0072] Where, the symbol ← indicates assignment; α represents the learning rate; |B| is the number of experience points sampled from the experience pool; This indicates finding the gradient of Q with respect to w; s t Indicates the system state at time t; a t B represents the action taken by the agent at time t; B represents a batch of experience obtained by random sampling.

[0073] In module M5, the target network parameters and the updated evaluation network parameters are weighted and averaged, and the result is assigned to the target network; the target network is a neural network.

[0074] The target network parameter w tar The mathematical expression is:

[0075] w tar ←(1-α smooth )w tar +α smooth w tar (11)

[0076] Where, α smooth This represents the target smoothing factor.

[0077] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the steps of the DC distribution network power quality improvement method are implemented.

[0078] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the DC power distribution network power quality improvement method.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. This invention is based on the model-free Double DQN algorithm. During the learning process, the agent does not need to perform any explicit mathematical modeling of the power distribution network environment. It directly learns the mapping relationship between the state and reward through interaction with the environment to obtain the optimal strategy, which is simple and efficient.

[0081] 2. The operating frequency of the DRL agent provided by this invention matches the switching frequency of the high-frequency MOSFET, which can quickly restore the bus voltage and greatly shorten the voltage recovery time compared with the traditional control strategy.

[0082] 3. This invention can autonomously learn control strategies without any prior knowledge and can cope with a variety of complex operating conditions, reduce bus voltage deviation, and has good robustness. In other words, under conditions of few known parameters, inaccurate models, and external disturbances, compared with the prior art, this invention can ensure the stability of large signals in DC distribution networks and respond quickly over a wider operating range, which is conducive to building an intelligent operation system for power systems.

[0083] 4. This invention takes a parallel Buck converter equipped with a constant power load as the control target, and realizes precise regulation of the bus voltage and power sharing between converters. This control scheme is also applicable to other converter topologies. Attached Figure Description

[0084] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0085] Figure 1 A simplified schematic diagram of a DC distribution network system provided by the present invention;

[0086] Figure 2 A schematic diagram of the Double DQN algorithm framework provided by this invention;

[0087] Figure 3 This is a schematic diagram of constant power load disturbance provided by the present invention;

[0088] Figure 4 A schematic diagram of the output power of the Buck converter provided by this invention;

[0089] Figure 5 A schematic diagram of the simulated bus voltage waveform under constant power load variation provided by the present invention;

[0090] Figure 6 This is a schematic diagram of the bus voltage simulation waveform under nominal voltage variation provided by the present invention;

[0091] Figure 7 A schematic diagram of the bus voltage simulation waveform when the parameters are uncertain, as provided by this invention. Detailed Implementation

[0092] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0093] This invention focuses on parallel Buck converters equipped with constant power loads, proposing a DC distribution network power quality improvement technology based on deep reinforcement learning. By collecting data and training an intelligent agent online, a control strategy is obtained, accelerating the response speed to external disturbances, rapidly restoring the bus voltage, and improving system power quality. More notably, it reduces the DC bus voltage deviation caused by constant power loads and achieves a reasonable power configuration for the source-side converter.

[0094] In other words, the solution provided by this invention is a DC distribution network bus voltage control based on a parallel Buck circuit. The voltage control problem is modeled as a Markov decision process, and by designing reasonable reward functions, observation spaces, and action spaces, the reinforcement learning agent is guided to learn effective control strategies. This generates PWM signals to control the switching transistors of each converter, achieving precise control of the DC bus voltage and improving the system's power quality.

[0095] According to the present invention, a method for improving power quality in a DC distribution network includes the following software system operation steps:

[0096] Step 1: Initialize parameters, set the training rounds to M, initialize system state, and sampling time;

[0097] The parameters include: the evaluation network parameters, i.e., w; the target network parameters, i.e., w. tar Experience pool, i.e., D, learning rate;

[0098] Step 2: Based on the current state, select the action that maximizes the Q-value of the evaluation network from the discrete action space; the Q-value is the action value function.

[0099] Step 3: Execute the action, and then obtain the reward and the state at the next moment based on the reward function;

[0100] Step 4: Store the action, the current state, the next state, and the reward value in the experience pool as a quadruple.

[0101] Step 5: Randomly sample a batch of experiences from the experience pool, denoted as B, calculate the corresponding temporal difference objective, perform backpropagation based on the temporal difference objective, calculate the loss, and update the evaluation network parameters, i.e., w; D is the experience pool, and B is a batch of experiences sampled from the experience pool.

[0102] Step 6: Repeat steps 2-5 to complete several updates to the evaluation network, then update the target network parameters, i.e., w. tar To improve the power quality of the power grid.

[0103] Specifically, in step 2, actions are selected through a discrete action space as follows: the current state is input into an evaluation network, which estimates the Q-value corresponding to each discrete action performed in that state, and the agent selects the action that maximizes the Q-value. The evaluation network is a neural network.

[0104] Specifically, in step 3, the process is as follows: after the agent performs the action at the current moment, the system updates to the state at the next moment, and substitutes the voltage error value and the current error value into the reward function to obtain the reward value at the current moment.

[0105] The reward function is expressed mathematically as follows:

[0106] r(t)=r1(t)+r2(t)(1)

[0107] Where r(t) represents the reward function, t represents time; r1(t) represents the reward function for the first sub-objective; and r2(t) represents the reward function for the second sub-objective.

[0108] The reward function for the first sub-objective is expressed mathematically as follows:

[0109]

[0110] Among them, e v (t) represents the voltage error signal;

[0111] The reward function for the second sub-objective is expressed mathematically as follows:

[0112]

[0113] Among them, e i (t) represents the difference in converter output current;

[0114] In step 5, the mathematical expression for the time-series difference objective is:

[0115]

[0116] in, The time-series differential objective is represented by r(t); the reward value is represented by γ; the discount factor is represented by Q; and the action value function is represented by Q. This represents the agent action that maximizes the Q-value; 'a' represents the learning rate; and 'w' represents the network parameters being evaluated. tar Indicates the target network parameters;

[0117] The loss is obtained through a loss function, the mathematical expression of which is:

[0118]

[0119] Where L(w) represents the loss function; Represents the time-series difference objective at time t; s t Indicates the system state at time t; a t This represents the action performed by the agent at time t;

[0120] After obtaining the loss, the neural network parameters are updated by minimizing the loss function using equation (9), which is to evaluate the network parameters. The mathematical expression is:

[0121]

[0122] Where, the symbol ← indicates assignment; α represents the learning rate; |B| is the number of experience points sampled from the experience pool; This indicates calculating the gradient of Q with respect to w; s t Indicates the system state at time t; a t B represents the action taken by the agent at time t; B represents a batch of experience obtained by random sampling.

[0123] In step 6, the target network parameters, w, are obtained in the following way. tar The weighted average of the old target network parameters and the new evaluation network parameters is assigned to the target network.

[0124] The target network parameter w tar The mathematical expression is:

[0125] w tar ←(1-α smooth )wtar +α smooth w tar (11)

[0126] Where, α smooth Indicates the target smoothing factor;

[0127] The specific implementation methods are as follows:

[0128] A. DC distribution network modeling

[0129] Considering only the output characteristics of distributed generation, it is equivalent to a stable DC voltage source; the closed-loop controlled power electronic converter is equivalent to a constant power load, i.e., CPLs; the load that needs to operate with constant impedance in its working mode is equivalent to a constant impedance load, i.e., CILs; the power converter connects the distributed generation to the DC bus, thereby supplying energy to the load side. Therefore, the DC distribution network is simplified as follows: Figure 1 The cascaded system shown consists of a distributed power source, a Buck converter, a constant power load, and a constant impedance load. For ease of analysis, the effect of parasitic resistance is ignored, and it is considered an ideal circuit. The system operates in continuous inductor current mode.

[0130] B. Establishing a Markov Decision Process

[0131] Based on the MATLAB / Simulink platform and reinforcement learning toolbox, a Markov decision process model is used to model this problem, transforming the voltage control problem into a sequential decision problem, which facilitates subsequent agent training.

[0132] A Markov decision process typically consists of a state space, an action space, a reward function, a state transition function, and a discount rate, denoted as (S, A, P, R, γ). Here, S is the state space, referring to the set of all possible states; A is the action space, referring to the set of all possible actions of the agent; P is the set of state transition functions p; and R is the set of reward functions r.

[0133] like Figure 2 As shown, under the current policy π, in state s t When performing action a t It transitions to the next state s with probability p. t+1 And the environment returns a reward r to the agent. t This process exhibits the Markov property, meaning that given the present state, the conditional probability distribution of future states depends only on the current state and not on all past states. The mathematical expression is:

[0134] p(s t+1 |s t ,…,s0)=p(s t+1 |st (2)

[0135] Where p represents the state transition probability function, which is determined by the environment and is used to describe the state transition.

[0136] (1) State-space design

[0137] The purpose of control is to reduce the bus voltage V bus Stable at the nominal value V ref Nearby, power sharing between converters is simultaneously achieved. Therefore, the voltage error signal e v (t) and the difference e between the output currents of the two converters i (t) serves as the fundamental observation signal. Considering that inductors and capacitors, as energy storage elements, cannot be instantaneously charged and discharged, and simultaneously observing the delayed signal and the differential signal to comprehensively characterize the environmental state, the state space S is designed as follows, with the mathematical expression as follows:

[0138]

[0139] Among them, v bus_del This represents the bus voltage delayed by one sampling period; e v_del (t) represents the voltage error signal delayed by one sampling period; e i_del (t) represents the current error signal delayed by one sampling period;

[0140] (2) Action Space Design

[0141] The duty cycles d1 and d2 of the two Buck converter transistors are used as the output actions of the agent. The upper limit of the granularity of the action space discretization is the original continuous space, and the lower limit is the minimum control accuracy to ensure functional completeness. After comprehensive consideration, this chapter stipulates that the duty cycle range of the two transistors is A1 = A2 = {0.42, 0.43, ..., 0.55}. The action space A is designed as follows, and the mathematical expression is:

[0142] A = A1 × A2 (4)

[0143] Where A represents the operating space; A1 represents the duty cycle range of the first switch; and A2 represents the duty cycle range of the second switch.

[0144] (3) Reward function design

[0145] The reward function is designed based on the error between the current state and the control objective, which is e in this chapter. v (t) and e i (t). The reward function r(t) with sub-objectives is designed as follows, and its mathematical expression is:

[0146]

[0147] r(t)=r1(t)+r2(t) (7)

[0148] C. Solving the algorithm based on deep Q-learning

[0149] The DQN algorithm is a classic value-based algorithm that uses a deep Q-network (DQN) to fit the action value function.

[0150] DQN is denoted as Q(s,a;w), where w are the neural network weight parameters, i.e., the evaluation network parameters. As a value learning algorithm, DQN takes the state of the environment as its input and outputs the Q-value of each action in the discrete action space A. As an off-policy algorithm, DQN training can be divided into two independent parts: collecting training data and updating the parameters w.

[0151] 1. Collecting Training Data: The policy function π that controls the agent's interaction with the environment is called the behavioral policy. To ensure that the agent simultaneously explores unknown actions and acquired action information, balancing exploration and utilization, an ε-greedy policy is often used as the behavioral policy. Based on the behavioral policy, the agent interacts with the environment n times in one round, denoted as: s1, a1, r1, s2, a2, r2, ..., s n ,a n ,r n Divide the n interactions in the trajectory into n groups of the form (s t ,a t ,r t ,s t+1 The quadruplets of ) are stored in the experience pool, denoted as D. DQN adopts an experience replay mechanism, that is, each set of data in the experience pool can be repeatedly extracted. During each round of training, a batch of experience is extracted from the experience pool, denoted as B. This can significantly improve the sample utilization rate, eliminate the correlation between data, and make the data distribution more stable, which is conducive to the training of neural networks.

[0152] 2. Update parameter w.

[0153] The DQN is trained using the temporal difference algorithm. At time t, the neural network provides a prediction. Temporal difference objective It is the prediction made by the neural network at time t+1, based on the reward r actually observed at time t. t .

[0154]

[0155] in, Represents the discrete action space;

[0156] Compare More reliable, therefore DQN encourages near

[0157] Therefore, the loss function is defined as shown in equation (8):

[0158]

[0159] The neural network parameters are updated by minimizing the loss function using equation (9).

[0160]

[0161] Where α is the learning rate.

[0162] Too small a learning rate will slow down training, while too large a learning rate will make training difficult to converge. However, the original DQN algorithm mentioned above usually suffers from the problem of value overestimation.

[0163] Therefore, a target network is added to the original DQN. Initially, the target network has the same structure and parameters as the original neural network. The original neural network is called the evaluation network. During the learning process, the target network is used to predict the Q-value, which is taken as the learning objective, as shown in the following equation.

[0164]

[0165] Meanwhile, during the weight update process, only the weights w of the evaluation network are updated, not the weights w of the target network. tar After updating the network a certain number of times, the weights of the evaluation network are then assigned to the target network. When updating the target network, Polyak averaging is used, and a target smoothing factor α is introduced. smooth According to equation (11), the weighted average value of the old target network parameters and the new evaluation network parameters is assigned to the target network.

[0166] w tar ←(1-α smooth )w tar +α smooth w tar (12)

[0167] The introduction of the target network also reduces model fluctuations, making the learning process more stable. However, the use of the target network cannot alleviate the overestimation caused by the max operator, and further improvements to DQN are needed. Expanding equation (10) yields:

[0168]

[0169] As can be seen, the next action and the selection of the target value are both completed by the target network. To alleviate overestimation, the following improvements are made:

[0170]

[0171] In other words, the next action is determined by the evaluation network, while the target value is calculated by the target network. This algorithm is called Double DQN. The Double DQN algorithm framework is as follows: Figure 2 As shown.

[0172] D. Training process based on Double DQN algorithm

[0173] Based on the above analysis, this scheme transforms the DC distribution network control model into a sequential decision model, with each 1e -4 s makes a decision and solves it using the Double DQN algorithm.

[0174] The distribution network control process based on the Double DQN algorithm is as follows:

[0175]

[0176] Implementation Example 1:

[0177] In such Figure 1 The simulation system shown verifies the proposed DC distribution network power quality improvement technology under three operating conditions. The system parameters are shown in Table 1, and the training parameters for the Double DQN algorithm are shown in Table 2.

[0178] Table 1 Parameters of DC Distribution Network Model

[0179]

[0180]

[0181] Table 2 Training parameters of the Double DQN algorithm

[0182] hyperparameters symbol numerical values Learning rate α 1e-4 Discount factor γ 0.9 Experience pool capacity D 2e5 Small batch experience capacity B 256 Target smoothing factor <![CDATA[α smooth ]]> 1e-3 Optimizer Adam Explore parameters ε 0.1

[0183] Operating Condition 1: Constant Power Load Disturbance Test.

[0184] Constant power loads such as Figure 3 The step disturbance shown occurs. Figure 4 As can be seen, the output powers P1 and P2 of the two Buck converters are basically the same, indicating that the proposed DRL control strategy achieves current sharing among the parallel modules. Figure 5It is known that when the load power changes, the bus voltage under DRL control can stabilize near the nominal voltage value. The bus overshoot under DRL control and dual-loop PI control is almost the same, but under DRL control, the bus voltage recovery time is extremely short, and the dynamic response performance is excellent. At the initial moment, the settling time using DRL control is 3.6ms, a reduction of 64.62% compared to dual-loop PI control. Analyzing a partial graph under load changes, the bus voltage recovery time using DRL control is 1.493ms, a reduction of 84.51% compared to dual-loop PI control.

[0185] Operating Condition 2: Test for changes in nominal voltage value.

[0186] With a constant power load of 300W, the nominal voltage jumps from 100V to 105V in 0.1 seconds and drops from 110V to 90V in 0.23 seconds. The dynamic response process of the system bus voltage is as follows: Figure 6 As can be seen, when the nominal voltage rises, it only takes 3ms to track the specified nominal voltage value. When the nominal voltage drops, it can also track the required nominal voltage value after 56ms, with almost no voltage overshoot.

[0187] Operating Condition 3: System Parameter Uncertainty Test.

[0188] With the nominal voltage constant, a constant power load disturbance experiment was conducted based on different inductance and voltage deviation values ​​to verify the robustness of the proposed DRL control strategy under uncertain distribution network model parameters. The constant power load jumped from an initial value of 500W to 800W in 0.2 seconds and dropped from 800W to 500W in 0.3 seconds. Figure 7 The dynamic response of the system bus voltage is shown when L1, L2, C1, and C2 differ from their nominal values ​​by +20%, 0%, and -20%, respectively. It can be seen that under DRL control, the system can quickly track the nominal bus voltage value within 2ms, demonstrating good robustness.

[0189] In summary, this invention improves the system's dynamic response speed and tracking accuracy, effectively enhancing the power quality of DC distribution networks. Furthermore, it eliminates the need for precise power system modeling and exhibits strong adaptability.

[0190] The present invention also provides a DC distribution network power quality improvement system, which can be implemented by executing the process steps of the DC distribution network power quality improvement method. That is, those skilled in the art can understand the DC distribution network power quality improvement method as a preferred embodiment of the DC distribution network power quality improvement system.

[0191] A power quality improvement system for a DC distribution network according to the present invention includes:

[0192] Module M1: Initializes parameters, sets initial parameters;

[0193] Module M2: Based on the initial parameters and the current system state, select the action in the discrete action space that maximizes the Q value of the evaluation network;

[0194] Module M3: Executes the action to obtain the reward value and the state at the next moment;

[0195] Module M4: Stores the action, current state, next state, and reward value in the experience pool as a quadruple;

[0196] Module M5: Randomly samples a batch of experiences from the experience pool, calculates the corresponding time-series differential target based on the experiences, backpropagates based on the time-series differential target, calculates the loss, updates and evaluates the network parameters, and re-triggers module M2 to work until the power quality of the power grid is improved.

[0197] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0198] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for improving power quality in a DC distribution network, characterized in that, include: Step S1: Initialize parameters and set initial parameters; Step S2: Based on the initial parameters and the current system state, select the action in the discrete action space that maximizes the Q value of the evaluation network; Step S3: Perform the action to obtain the reward value and the state at the next moment; Step S4: Store the action, the current state, the next state, and the reward value in the experience pool as a quadruple; Step S5: Randomly sample a batch of experiences from the experience pool, and calculate the corresponding time-series difference target based on the experiences; The loss is calculated by backpropagation based on the time-series differential objective, and then the network parameters are updated and evaluated. Step S2 is then re-executed until the power quality of the power grid is improved.

2. The method for improving power quality in a DC distribution network according to claim 1, characterized in that, In step S1, the parameters include: evaluation network parameters, target network parameters, experience pool, and learning rate; the initial parameters include: training epochs, initial system state, and sampling time. Step S2 includes: Step S2.1: Input the current system state into the evaluation network; Step S2.2: Estimate the Q-value corresponding to each discrete action in the current system state by evaluating the network; Step S2.3: Select the action that maximizes the Q value.

3. The method for improving power quality in a DC distribution network according to claim 2, characterized in that, In step S3, the agent performs an action to update the state to the next time step, generating voltage error values ​​and current error values; the voltage error values ​​and current error values ​​are substituted into the reward function to obtain the reward value at the current time step. The reward function is expressed mathematically as follows: r(t)=r1(t)+r2(t) (6) Where r(t) represents the reward function, t represents time; r1(t) represents the reward function for the first sub-objective; and r2(t) represents the reward function for the second sub-objective. The reward function for the first sub-objective is expressed mathematically as follows: Among them, e v (t) represents the voltage error signal; The reward function for the second sub-objective is expressed mathematically as follows: Among them, e i (t) represents the difference in the converter output current.

4. The method for improving power quality in a DC distribution network according to claim 3, characterized in that, In step S5, the mathematical expression for the time-series difference objective is: in, The time-series differential objective is represented by r(t); the reward value is represented by γ; the discount factor is represented by Q; and the action value function is represented by Q. This represents the agent action that maximizes the Q-value; 'a' represents the learning rate; and 'w' represents the network parameters being evaluated. tar Indicates the target network parameters; The loss is obtained through a loss function; the mathematical expression of the loss function is: Where L(w) represents the loss function; Represents the time-series difference objective at time t; s t Indicates the system state at time t; a t This represents the action performed by the agent at time t; In step S5, based on the loss, the network parameters are updated and evaluated by minimizing the loss function, the mathematical expression of which is: Where, the symbol ← indicates assignment; α represents the learning rate; |B| is the number of experience points sampled from the experience pool; This indicates finding the gradient of Q with respect to w; s t Indicates the system state at time t; a t B represents the action taken by the agent at time t; B represents a batch of experience obtained by random sampling. In step S5, the target network parameters and the updated evaluation network parameters are weighted and averaged, and the result is assigned to the target network; the target network is a neural network. The target network parameter w tar The mathematical expression is: w tar ←(1-a smooth )w tar +a smooth w tar (11) Where, α smooth This represents the target smoothing factor.

5. A DC distribution network power quality improvement system, characterized in that, include: Module M1: Initializes parameters, sets initial parameters; Module M2: Based on the initial parameters and the current system state, select the action in the discrete action space that maximizes the Q value of the evaluation network; Module M3: Executes the action to obtain the reward value and the state at the next moment; Module M4: Stores the action, current state, next state, and reward value in the experience pool as a quadruple; Module M5: Randomly samples a batch of experiences from the experience pool, calculates the corresponding time-series differential target based on the experiences, backpropagates based on the time-series differential target, calculates the loss, updates and evaluates the network parameters, and re-triggers module M2 to work until the power quality of the power grid is improved.

6. The DC distribution network power quality improvement system according to claim 5, characterized in that, In module M1, the parameters include: evaluation network parameters, target network parameters, experience pool, and learning rate; the initial parameters include: training epochs, initial system state, and sampling time. The module M2 includes: Module M2.1: Inputs the current system state into the evaluation network; Module M2.2: Estimates the Q-value for each discrete action in the current system state by evaluating the network; Module M2.3: Select the action that maximizes the Q value.

7. The DC distribution network power quality improvement system according to claim 6, characterized in that, In module M3, the agent performs actions to update the state to the next moment, generating voltage error values ​​and current error values; the voltage error values ​​and current error values ​​are substituted into the reward function to obtain the reward value at the current moment; The reward function is expressed mathematically as follows: r(t)=r1(t)+r2(t) (6) Where r(t) represents the reward function, t represents time; t1(t) represents the reward function for the first sub-objective; and r2(t) represents the reward function for the second sub-objective. The reward function for the first sub-objective is expressed mathematically as follows: Among them, e v (t) represents the voltage error signal; The reward function for the second sub-objective is expressed mathematically as follows: Among them, e i (t) represents the difference in the converter output current.

8. The DC distribution network power quality improvement system according to claim 7, characterized in that, In module M5, the mathematical expression for the time-series differential objective is: in, The time-series differential objective is represented by r(t); the reward value is represented by γ; the discount factor is represented by Q; and the action value function is represented by Q. This represents the agent action that maximizes the Q-value; 'a' represents the learning rate; and 'w' represents the network parameters being evaluated. tar Indicates the target network parameters; The loss is obtained through a loss function; the mathematical expression of the loss function is: Where L(w) represents the loss function; Represents the time-series difference objective at time t; s t Indicates the system state at time t; a t This represents the action performed by the agent at time t; In module M5, based on the loss, the network parameters are updated and evaluated by minimizing the loss function, the mathematical expression of which is: Where, the symbol ← indicates assignment; α represents the learning rate; |B| is the number of experience points sampled from the experience pool; This indicates finding the gradient of Q with respect to w; s t Indicates the system state at time t; a t B represents the action taken by the agent at time t; B represents a batch of experience obtained by random sampling. In module M5, the target network parameters and the updated evaluation network parameters are weighted and averaged, and the result is assigned to the target network; the target network is a neural network. The target network parameter w tar The mathematical expression is: w tar ←(1-a smooth )w tar +a smooth w tar (11) Where, α smooth This represents the target smoothing factor.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the DC distribution network power quality improvement method according to any one of claims 1 to 4.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the DC distribution network power quality improvement method according to any one of claims 1 to 4.

Images