MMC intelligent setting and cooperative control method and system based on deep reinforcement learning

Abstract

The invention discloses an MMC intelligent setting and cooperative control method and system based on deep reinforcement learning, and relates to the technical field of power transmission system regulation and control, and the method comprises the steps: obtaining the current operation state data of a converter in a power transmission system, the current operation state data comprising a system frequency, a frequency reference value, a frequency deviation and a frequency change rate; establishing a third-order active closed-loop small signal transfer function based on an active transient feedback compensation algorithm, and decoupling a damping coefficient and a primary frequency modulation coefficient in VSG control through the third-order active closed-loop small signal transfer function; inputting the current operation state data into the trained DDPG intelligent agent for processing, and obtaining an optimization result of the virtual inertia; the optimized virtual inertia is applied to a rotational inertia control link of VSG control, and self-adaptive adjustment of the virtual inertia is carried out through the trained DDPG intelligent agent; the problem that control parameters of a traditional virtual synchronous generator VSG are fixed and are difficult to adapt to system operation state changes is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power transmission system regulation, more particularly, it relates to a method and system for MMC intelligent setting and collaborative control based on deep reinforcement learning. BACKGROUND

[0002] At present, the global energy structure is in a structural transformation stage, and the proportion of renewable energy in the energy system continues to increase. The large-scale access of intermittent and volatile power sources such as wind and solar energy has put forward new key requirements for the stability of the power system and the efficient consumption of electric energy. Under the unique energy pattern in China, energy resources and load demand are inversely distributed, and the complex power grid structure with high proportion of new energy grid connection and multi-direct current feeding makes the flexible direct current transmission technology play an increasingly important role in improving the flexibility and reliability of the power grid.

[0003] In the flexible direct current transmission system based on the modular multi-level converter (MMC), the traditional vector control strategy has inherent limitations: it cannot provide the necessary inertia support for the system, and it is difficult to effectively participate in grid frequency regulation. This problem becomes more and more prominent with the increasing penetration of new energy, bringing serious challenges to the safe and stable operation of the power grid. Under this background, the virtual synchronous generator (VSG) technology simulates the key characteristics of the synchronous generator (SG) such as active-frequency regulation, reactive-voltage regulation and rotor motion equation, so that the grid-connected converter can be equivalent to a traditional synchronous generator in terms of operation mechanism, thereby providing the much-needed inertia and damping support for the system and effectively enhancing the dynamic stability of the power grid.

[0004] Virtual inertia and virtual damping coefficient are two core parameters in VSG control, which jointly determine the dynamic characteristics and steady-state performance of the system. By reasonably coordinating these two parameters, the ability of the system to respond to power surges and frequency disturbances can be improved, and a more robust and flexible new energy-friendly power grid can be built. Specifically, virtual inertia simulates the rotor inertia effect, enhances the ability of the system to resist frequency fluctuations, and enhances frequency stability. The damping coefficient adjusts the damping characteristics of the system, speeds up the frequency recovery speed, and suppresses power oscillation and transient instability. The coordination of these two constitutes the theoretical basis for VSG technology to enhance the stability of the power grid.

[0005] However, the current VSG control parameter design strategy still faces severe challenges; most existing researches are based on fixed parameter design, which is difficult to adapt to the actual operating environment with random fluctuations of new energy output and complex and variable load characteristics; especially when the grid frequency fluctuates sharply, imperfect adaptive mechanism is prone to cause "parameter spike" phenomenon, leading to system regulation lag, response overshoot and other adverse dynamic behaviors, and even may cause subsynchronous oscillation and other new stability problems. Therefore, how to construct a virtual inertia and damping coefficient coordinated control strategy that can optimize dynamic response and steady-state performance simultaneously under the premise of fully considering system operation uncertainty has become a research hotspot in the field of current VSG. SUMMARY

[0006] The purpose of the present application is to provide a method and system for MMC intelligent setting and collaborative control based on deep reinforcement learning, by introducing an active transient feedback compensation algorithm and a virtual inertia intelligent adaptive mechanism based on deep deterministic policy gradient, a clear reward-punishment mechanism is constructed to realize the adaptive setting of VSG control parameters; and according to the real-time operating state of system frequency fluctuation, load change, etc., the control strategy can be dynamically adjusted to effectively improve the transient response capability and steady-state operating performance of the system, so as to solve the problem of fixed VSG control parameters in traditional virtual synchronous generator, which is difficult to adapt to the change of system operating state.

[0007] The above technical purpose of the present application is realized by the following technical scheme:

[0008] In a first aspect, the present application provides a method for MMC intelligent setting and collaborative control based on deep reinforcement learning, comprising the following specific steps:

[0009] Obtain the current operating state data of the converter in the power transmission system, the current operating state data including system frequency, frequency reference value, frequency deviation and frequency change rate;

[0010] Establish a three-order active closed-loop small-signal transfer function based on the active transient feedback compensation algorithm, and decouple the damping coefficient and primary frequency modulation coefficient in the VSG control through the three-order active closed-loop small-signal transfer function;

[0011] Input the current operating state data into the trained DDPG agent for processing to obtain the optimization result of virtual inertia;

[0012] Apply the optimized virtual inertia to the rotational inertia control link of VSG control, and perform adaptive adjustment of virtual inertia through the trained DDPG agent.

[0013] On the basis of the above technical scheme, the present application can also be improved as follows.

[0014] Further, the third-order active closed-loop small-signal transfer function described above comprises a rotor motion equation and an active power steady-state deviation equation of the improved AFC-VSG, and is specifically as follows: ; wherein, T F is a lag time constant of the AVSG, K F is a transient power compensation coefficient, P m and P e are the MMC DC side power and output power respectively, ΔP es is a steady-state deviation of the VSG control; G P (s) and G ω (s) are closed-loop transfer functions of the VSG control output power under active disturbance and grid frequency disturbance; ΔP ref is a given change of active power; Δω g is a given change of grid frequency, J is a virtual inertia; ω0 is a rated angular frequency; ω is a system angular frequency; K P is a primary frequency modulation coefficient.

[0015] Further, the DDPG agent described above comprises an Actor network and a Critic network, wherein:

[0016] the Actor network is configured to generate a deterministic action according to a current state, and the action comprises a virtual inertia value;

[0017] the Critic network is configured to evaluate a value of a state-action pair and guide policy optimization of the Actor network.

[0018] Further, the DDPG agent described above is trained in the following manner:

[0019] a Simulink simulation environment is constructed, and a VSG control system model based on an active transient feedback compensation algorithm is established, system parameters and disturbance scenarios are set, and the disturbance scenarios at least include load mutation and grid frequency fluctuation;

[0020] an experience replay buffer is initialized and configured to store training samples generated by the agent in the process of interacting with the environment;

[0021] the agent interacts with the simulation environment: at each time step, the agent to be trained selects an action according to a current state, the environment executes the action and returns a reward and a next state, and a four-tuple composed of the state, the action, the reward and the next state is stored in the experience replay buffer;

[0022] a batch of samples are randomly extracted from the experience replay buffer to form training samples for training the agent to be trained;

[0023] The training sample is input into the agent to be trained for processing, a TD target value is calculated, a loss function based on mean square error is used to update the Critic network parameters, and the network parameters of the Actor network are updated through a policy gradient method;

[0024] The Actor network and the Critic network in the agent to be trained are updated through a soft update strategy until the cumulative reward converges or the maximum number of training rounds is reached, and the trained network when the cumulative reward converges or the maximum number of training rounds is reached is determined as the DDPG agent.

[0025] Further, the above reward is calculated in the following manner: ; In the formula, φ1 is a first interval frequency deviation threshold value; φ2 is a second interval frequency deviation threshold value; φ3 is a third interval frequency deviation threshold value; φ lim is an extreme frequency deviation threshold value; R max is a maximum reward reference value; k1 is an exponential decay coefficient; k2 is a heavy penalty increment coefficient; R is a heavy penalty base coefficient; α is a slope of a linear segment in a medium deviation interval; β is an intercept of the linear segment in the medium deviation interval; z1 is a slope of a linear segment in a large deviation interval; and z2 is an intercept of the linear segment in the large deviation interval.

[0026] Further, the above TD target value is specifically: ; In the formula, denotes a discount factor, is a TD target value, denotes a reward corresponding to the i th sample, denotes target evaluation network parameters, denotes a next state corresponding to the i th sample, is a target policy network parameter, is a state-action value of a target Q network, is an output action of the target policy network;

[0027] The loss function based on mean square error is specifically: ; In the formula, is a value of the loss function, denotes a number of training samples, is a TD target value, denotes a state corresponding to the i th sample, denotes an action corresponding to the i th sample, An online evaluation network parameter is represented.

[0028] Further, the soft update strategy is specifically as follows: ; In the formula, is a soft update coefficient, is an online evaluation network parameter in the Critic network, represents a network parameter in the Actor network.

[0029] In a second aspect, the application provides a system for MMC intelligent setting and collaborative control based on deep reinforcement learning, applied to the method for MMC intelligent setting and collaborative control based on deep reinforcement learning in any one of the first aspect, comprising:

[0030] A data acquisition module is configured to acquire current operating state data of a converter in a power transmission system, and the current operating state data comprises system frequency, frequency reference value, frequency deviation and frequency change rate.

[0031] A coefficient decoupling module is configured to establish a third-order active closed-loop small-signal transfer function based on an active transient feedback compensation algorithm, and decouple damping coefficients and primary frequency modulation coefficients in VSG control through the third-order active closed-loop small-signal transfer function.

[0032] A virtual inertia optimization module is configured to input the current operating state data into a trained DDPG agent for processing, and obtain an optimization result of virtual inertia.

[0033] An adaptive adjustment module is configured to apply the optimized virtual inertia to a rotational inertia control link of VSG control, and perform adaptive adjustment of the virtual inertia through the trained DDPG agent.

[0034] In a third aspect, the application provides an electronic device, comprising at least one processor, at least one memory and a data bus;

[0035] The processor and the memory complete mutual communication through the data bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the method for MMC intelligent setting and collaborative control based on deep reinforcement learning in any one of the first aspect.

[0036] In a fourth aspect, the application provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions cause a computer to execute the method for MMC intelligent setting and collaborative control based on deep reinforcement learning in any one of the first aspect.

[0037] Compared with the prior art, the application has at least the following beneficial effects:

[0038] 1. Strong intelligent adaptive capability: By introducing the DDPG deep reinforcement learning algorithm, the proposed method can learn autonomously and dynamically output the optimal virtual inertia value according to the real-time operating status of the system, thereby realizing intelligent adaptive control and effectively avoiding the limitations of traditional control that relies on human experience to design complex control rules.

[0039] 2. Taking into account multiple state variables such as frequency deviation and frequency change rate, a composite reward mechanism of "deviation penalty + proximity reward" was designed. This mechanism balances the transient response performance and steady-state control accuracy of the system, improves the convergence speed and learning efficiency of reinforcement learning, enhances training efficiency, and makes the trained strategy more in line with actual control requirements.

[0040] 3. Excellent generalization ability: The DDPG algorithm is trained with a large amount of simulation data and has the ability to adapt to different operating conditions and various disturbance scenarios. It does not require redesigning the controller structure or parameters for specific operating conditions, thus enhancing the applicability of the system in complex power grid environments.

[0041] 4. A simulation model of MMC inverter based on AFC-VSG grid-connected control was built using Matlab / Simulink. The grid parameters were set, and the system was simulated under different disturbance conditions such as grid frequency disturbance, local load change and active power disturbance. This method avoids the repeated manual adjustment of virtual inertia and damping parameters in the traditional method, which greatly reduces the difficulty and time cost of controller design and debugging. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a schematic diagram of the network structure of the DDPG agent in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the overall structure of the AFC-VSG virtual inertia intelligent adaptive control system based on DDPG in an embodiment of the present invention;

[0045] Figure 3 This is a comparison diagram of system active power and frequency response under power grid frequency disturbance conditions in an embodiment of the present invention, wherein: Figure 3 (c) is the virtual inertia adaptive adjustment curve of the system under the condition of power grid frequency disturbance in the embodiment of the present invention;

[0046] Figure 4 This is a comparison diagram of the system active power and frequency response under active power disturbance conditions in an embodiment of the present invention, wherein: Figure 4(c) is the virtual inertia adaptive adjustment curve of the system under active power disturbance conditions in the embodiment of the present invention;

[0047] Figure 5 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0050] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0051] In the description of the embodiments of the present invention, "multiple" means at least two.

[0052] Example 1: Addressing the core issues of fixed control parameters and insufficient dynamic response adaptability in existing VSG control technologies when dealing with complex power grid conditions, this example proposes an intelligent adaptive control architecture integrating active power transient feedback compensation and deep reinforcement learning. This architecture introduces an AFC control loop to decouple the damping coefficient from the primary frequency regulation coefficient, reducing the system coupling effect. Simultaneously, it constructs an online virtual inertia optimization strategy based on DDPG, autonomously adjusting the virtual inertia parameters according to the real-time system state. Furthermore, it designs a clearly oriented segmented reward-penalty mechanism to guide the agent to achieve a balance between frequency deviation suppression and system stability, thereby comprehensively improving the frequency support capability and dynamic regulation quality of the VSG under various disturbance conditions. The deep reinforcement learning-based MMC intelligent tuning and cooperative control method provided in this example is as follows: Figure 5 As shown, the specific steps include the following:

[0053] S1: Obtain the current operating status data of the converter in the power transmission system. The current operating status data includes system frequency, frequency reference value, frequency deviation, and frequency change rate.

[0054] S2, a third-order active power closed-loop small-signal transfer function is established based on the active power transient feedback compensation algorithm, and the damping coefficient and the primary frequency regulation coefficient in VSG control are decoupled through the third-order active power closed-loop small-signal transfer function.

[0055] Among them, the AFC algorithm (active transient feedback compensation algorithm) introduces an active transient compensation term and sets the damping coefficient to 0. This solves the technical contradictions in the traditional VSG control strategy, such as the mutual coupling between the damping coefficient and the primary frequency regulation coefficient, the increase in active steady-state deviation due to a large damping coefficient, and the inability of a small damping coefficient to effectively suppress the dynamic oscillation of the system. It forms a coordinated control with the virtual inertia adjustment of the DDPG agent, which significantly improves the frequency response speed and anti-disturbance capability of the system.

[0056] Optionally, the above-mentioned third-order active power closed-loop small-signal transfer function includes the improved AFC-VSG rotor motion equation and active power steady-state deviation equation, specifically: ; In the formula, T F K is the lag time constant of AVSG. F P is the transient power compensation coefficient. m P e These represent the DC-side power and output power of the MMC, ΔP es For steady-state deviation of VSG control; G P (s) and G ω (s) is the closed-loop transfer function of the VSG control output power under active power disturbance and grid frequency disturbance; ΔP ref The change in active power is given by Δω. g The change in the grid frequency is given by J, where J is the virtual inertia; ω0 is the rated angular frequency; ω is the system angular frequency; and K is the variable. P This is the primary frequency modulation coefficient.

[0057] S3 inputs the current running status data into the trained DDPG agent for processing to obtain the optimization results of virtual inertia.

[0058] Optional, Figure 1 The DDPG agent network structure consists of an Actor network and a Critic network, where:

[0059] Actor networks are used to generate deterministic actions based on the current state, including virtual inertia values.

[0060] Critic networks are used to evaluate the value of state-action pairs and guide policy optimization in Actor networks.

[0061] In the above, the Actor network includes:

[0062] State space: ;

[0063] The first hidden layer has 256 neurons, configured with layer normalization and ReLU activation function. Layer normalization helps stabilize the training process and accelerate convergence.

[0064] The second hidden layer has 128 neurons, configured with layer normalization, ReLU activation function, and a Dropout layer (dropout rate 0.1). Dropout is used to prevent overfitting and improve generalization ability.

[0065] The third hidden layer consists of 64 neurons, configured with the ReLU activation function.

[0066] Output layer: Fully connected layer (1 neuron) + tanh + scaling layer.

[0067] The scaling layer maps the output of tanh [-1, 1] to the virtual inertia range [1, 295].

[0068] The Critic network employs a dual-path structure:

[0069] State path: Input state vector (4-dimensional) → Fully connected layer (256 neurons) → ReLU;

[0070] Action path: Input action vector (1D) → Fully connected layer (256 neurons);

[0071] Fusion path: splicing state and action features → fully connected layer (256 neurons) → layer normalization → ReLU → Dropout (0.1) → fully connected layer (128 neurons) → layer normalization → ReLU → fully connected layer (1 neuron, output Q-value); the output layer of the Actor network uses a scaling mechanism, the formula is: ; Where x is the output of the network's front layer, the tanh function maps x to [-1, 1], and through scaling factor 147 and bias 148, the output range of virtual inertia J is finally made to be [1, 295].

[0072] Optionally, the above DDPG agent is trained in the following way:

[0073] S31, initialize the parameters of the Actor network, Critic network, target Actor network, and target Critic network.

[0074] S32, initialize the experience playback buffer to be empty.

[0075] S33, for each training round:

[0076] a. Reset the simulation environment and randomly set the initial disturbance of the power grid frequency.

[0077] b. Obtain the initial state s0.

[0078] c. For each time step t = 0, 1, ..., 1099:

[0079] Based on the current state s t The Actor network outputs action a. t = μ(s t |θ μ ).

[0080] Add OU noise: a t' = a t + N t , where N t This is OU noise.

[0081] Perform action a t' Environment returns reward r t and the next state s t+1 .

[0082] Store the experience in the experience replay buffer.

[0083] If the number of samples in the buffer is greater than or equal to n, and n=128, then perform a parameter update once:

[0084] n samples are randomly drawn from the experience pool [s] i a i r i s i+1 ], i=1,…,n-1, calculate the TD objective: ; In the formula: γ is the discount factor, These are the network parameters for the target policy. To evaluate the network parameters for the target.

[0085] The Critic network is updated, and its loss function is minimized as follows: ; In the formula: To evaluate network parameters online.

[0086] Update the Actor network and its network parameters. The update process is as follows: ; In the formula: is the learning rate of the online policy network. This is the gradient of the loss function with respect to the network parameters.

[0087] Soft update target network: ; In the formula: τ is the soft update coefficient.

[0088] d. Record the cumulative rewards for this round.

[0089] S34. After training is complete, save the trained Actor network parameters.

[0090] As the cumulative reward gradually increases during training (while the absolute value of the penalty term decreases), it indicates that the agent has learned an effective control strategy. After approximately 200 rounds, the cumulative reward essentially converges, and the network loss tends to stabilize.

[0091] Online control implementation:

[0092] After training, the trained Actor network is deployed to the real-time controller. The online control process is as follows:

[0093] S35. Construct the state vector s = [f, f] ref ,Δf,df / dt].

[0094] S36. Input the state vector into the Actor network and propagate it forward to obtain the virtual inertia J.

[0095] S37. Apply the virtual inertia J to the rotational inertia element of the AFC-VSG control algorithm.

[0096] S38. Output control signal to inverter to complete one control cycle.

[0097] Optionally, the above rewards are calculated in the following way: ; In the formula, φ1 is the frequency deviation threshold for the first interval; φ2 is the frequency deviation threshold for the second interval; φ3 is the frequency deviation threshold for the third interval; φ lim R is the threshold for extreme frequency deviation. max Here, k1 is the maximum reward baseline value; k2 is the exponential decay coefficient; R is the heavy penalty increment coefficient; α is the slope of the linear segment in the medium deviation interval; β is the intercept of the linear segment in the medium deviation interval; z1 is the slope of the linear segment in the larger deviation interval; z2 is the intercept of the linear segment in the larger deviation interval; specifically, φ1 = 0.05Hz, φ2 = 0.2Hz, φ3 = 0.5Hz, φ lim = 2Hz; R max= 80, k1 = 670, k2 = 100, R = 40; α ≈ -133, β ≈ 21.67, z1 ≈ -116.67, z2 ≈ 18.33.

[0098] The reward function employs a segmented design strategy to adaptively adjust the rewards and penalties for the agent's behavior. Specifically, within the small frequency deviation range, an exponentially decaying reward function is used to provide a large positive incentive; as the deviation expands to the medium range, the reward value decreases linearly and transitions to a slight penalty; when the deviation further increases, the penalty intensity increases linearly with the deviation; if the deviation enters a large range, the penalty effect is strengthened through a quadratic function; and in the case of extreme deviations, a fixed upper limit for the penalty is set to prevent numerical overflow and ensure training stability. The boundaries of each segmented interval are smoothed to ensure that the function is continuously differentiable. This design aims to effectively guide the agent to dynamically adjust the virtual inertia, prioritizing control strategies that can quickly suppress frequency deviations, thereby improving the system's transient response performance.

[0099] Optionally, to fully explore the action space in the early stages of training, the DDPG algorithm introduces exploration noise. This invention employs an Ornstein-Uhlenbeck (OU) noise process, which has mean-regression characteristics and can generate time-dependent noise, making it more suitable for continuous control tasks. In the early stages of training, the addition of noise helps the agent explore more of the policy space. As training progresses, the noise gradually decays, causing the agent to tend towards a stable optimal policy.

[0100] S4 applies the optimized virtual inertia to the rotational inertia control loop of VSG control, and performs adaptive adjustment of the virtual inertia through the trained DDPG agent.

[0101] The following examples will provide further explanation and analysis:

[0102] like Figure 2 As shown in Table 1, the simulation model of the AFC-VSG system was built in the Matlab / Simulink environment. The specific simulation parameters are shown in Table 2.

[0103] Table 1 Parameter Settings

[0104] Table 2 Hyperparameter Settings for DDPG

[0105] To verify the effectiveness of the method of this invention, a simulation comparison experiment was first conducted under the condition of power grid frequency disturbance: given a rated active power of 5MW, at t=1.5s, the power grid frequency dropped from 50Hz to 49.9Hz; at t=2.5s, the power grid frequency recovered from 49.9Hz to 50Hz. Two control methods were compared:

[0106] Method 1: Traditional VSG control (fixed virtual inertia J=265 kg·m², damping coefficient D=2536 N·m·s / rad, no AFC);

[0107] Method 2: Adapt the virtual inertia J of AFC-VSG using the DDPG algorithm, i.e., DA-VSG.

[0108] like Figure 3 As shown, when the system is subjected to power grid frequency disturbances:

[0109] Method 1, under the influence of the damping coefficient D, Figure 3 (d) The active power showed a steady-state deviation of 1MW, accompanied by an overshoot of 0.02MW; Figure 3 (e) The frequency response also showed a certain degree of overshoot. The adjustment time required for the system frequency to recover to steady state after experiencing a disturbance was about 0.6s.

[0110] In contrast, Method 2 (the present invention) Figure 3 (a) It not only effectively overcomes the steady-state deviation caused by the damping coefficient, reducing the steady-state deviation by 0.5MW, but also ensures that there is no overshoot or oscillation in the active power and frequency output, and shortens the steady-state adjustment time to 0.3s. The control performance is significantly better than that of traditional methods.

[0111] Figure 4 This sets the operating conditions when the system experiences active power disturbances. The specific settings for the operating conditions are shown in Table 3 below.

[0112] Table 3 Simulation settings for sudden increase in active power

[0113] Depend on Figure 4 It can be seen that when the active power of the system suddenly increases from 5MW to 8MW at 1.5s, Method 1, due to its use of fixed virtual inertia and damping coefficient, is unable to adapt to the dynamic changes of the system, leading to... Figure 4 (d) The output active power showed an overshoot of 0.03MW. Figure 4 (e) Frequency also exhibits overshoot; while Method 2 effectively maintains system stability and adapts to complex operating conditions by adjusting virtual inertia in real time, and neither the output active power nor the frequency exhibits overshoot or oscillation.

[0114] Further from suchFigure 3 (c) and Figure 4 (c) shows the adaptive adjustment process of virtual inertia in DA-VSG. It can be seen that when the system is disturbed, the agent based on DDPG in the proposed method can quickly adjust the value of virtual inertia. By increasing the virtual inertia J, the inertial support of the system is enhanced, and the rapid decline after the disturbance is effectively suppressed. As the system tends to stabilize, the virtual inertia gradually converges to a stable value, thereby ensuring the smooth operation of the system.

[0115] Simulation results show that the proposed DDPG-based intelligent virtual inertia adjustment method for AFC-VSG can dynamically adjust virtual inertia parameters, effectively improving the frequency response speed and operational stability of the VSG system, and significantly suppressing frequency fluctuations during transient processes. Compared with traditional fixed-parameter control strategies, the proposed method exhibits superior overall performance in terms of settling time, overshoot suppression, and steady-state accuracy.

[0116] Example 3: This application provides a system for intelligent tuning and cooperative control of MMC based on deep reinforcement learning, applied to the method for intelligent tuning and cooperative control of MMC based on deep reinforcement learning in Example 1, including:

[0117] The data acquisition module is used to acquire the current operating status data of the converter in the power transmission system. The current operating status data includes system frequency, frequency reference value, frequency deviation and frequency change rate.

[0118] The coefficient decoupling module is used to establish a third-order active power closed-loop small-signal transfer function based on the active power transient feedback compensation algorithm, and to decouple the damping coefficient and the primary frequency regulation coefficient in VSG control through the third-order active power closed-loop small-signal transfer function.

[0119] The virtual inertia optimization module is used to input the current running status data into the trained DDPG agent for processing and to obtain the optimization results of the virtual inertia.

[0120] The adaptive adjustment module is used to apply the optimized virtual inertia to the rotational inertia control loop of VSG control, and to perform adaptive adjustment of the virtual inertia through a trained DDPG agent.

[0121] Example 3: This application provides an electronic device, including: at least one processor, at least one memory, and a data bus;

[0122] In this system, the processor and memory communicate with each other via a data bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method of MMC intelligent tuning and cooperative control based on deep reinforcement learning as described in Example 1.

[0123] Example 4: This application provides a non-transitory computer-readable storage medium that stores computer instructions, which cause the computer to execute the method of MMC intelligent tuning and cooperative control based on deep reinforcement learning in Example 1.

[0124] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0128] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0129] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent setting and collaborative control of MMC based on deep reinforcement learning, characterized in that, The method comprises the following specific steps: acquiring current operating state data of a converter in a power transmission system, the current operating state data comprising system frequency, frequency reference value, frequency deviation and frequency change rate; establishing a third-order active closed-loop small-signal transfer function based on an active transient feedback compensation algorithm, and decoupling a damping coefficient and a primary frequency modulation coefficient in VSG control through the third-order active closed-loop small-signal transfer function; inputting the current operating state data into a trained DDPG agent for processing to obtain an optimization result of virtual inertia; applying the optimized virtual inertia to a rotational inertia control link in VSG control, and performing adaptive adjustment of the virtual inertia through the trained DDPG agent.

2. The method of intelligent setting and collaborative control of MMC based on deep reinforcement learning according to claim 1, characterized in that, The third-order active closed-loop small-signal transfer function comprises an improved rotor motion equation and an active power steady-state deviation equation of the AFC-VSG, and specifically comprises: ； In the formula, T F is the time constant of the AVSG hysteresis, K F is the transient power compensation coefficient, P m , P e are the MMC DC side power and output power, respectively, ΔP es is the steady-state deviation of the VSG control; G P (s) and G ω (s) are the closed-loop transfer functions of the VSG control of the grid-connected output power under active disturbance and grid frequency disturbance; ΔP ref is the change amount given by the active power; Δω g is the change amount given by the grid frequency, J is the virtual inertia, ω0 is the rated angular frequency, ω is the system angular frequency, K P is the primary frequency modulation coefficient.

3. The method of intelligent setting and collaborative control of MMC based on deep reinforcement learning according to claim 1, characterized in that, The DDPG agent comprises an Actor network and a Critic network, wherein: The Actor network is configured to generate a deterministic action according to a current state, and the action comprises a virtual inertia value; The Critic network is configured to evaluate the value of a state-action pair, and guide policy optimization of the Actor network.

4. The method of intelligent setting and collaborative control of MMC based on deep reinforcement learning according to claim 3, characterized in that, The DDPG agent is trained in the following manner: a Simulink simulation environment is constructed, a VSG control system model based on the active transient feedback compensation algorithm is established, system parameters and disturbance scenarios are set, and the disturbance scenarios at least comprise load mutation and power grid frequency fluctuation; an experience replay buffer is initialized to store training samples generated by the agent in the process of interacting with the environment; the agent interacts with the simulation environment: at each time step, the agent to be trained selects an action according to a current state, the environment executes the action and returns a reward and a next state, and a four-tuple of the state, the action, the reward and the next state is stored in the experience replay buffer; a batch of samples are randomly extracted from the experience replay buffer to form training samples for training the agent to be trained; the training samples are input into the agent to be trained for processing, a TD target value is calculated, a loss function based on mean square error is used to update parameters of the Critic network, and a policy gradient method is used to update network parameters of the Actor network; the Actor network and the Critic network in the agent to be trained are updated through a soft update strategy until the cumulative reward converges or a maximum number of training rounds is reached, and the agent to be trained whose cumulative reward converges or reaches the maximum number of training rounds is determined as the DDPG agent.

5. The method of intelligent setting and collaborative control of MMC based on deep reinforcement learning according to claim 4, characterized in that, The reward is calculated in the following manner: ； In the formula, φ1 is a first interval frequency deviation threshold value; φ2 is a second interval frequency deviation threshold value; φ3 is a third interval frequency deviation threshold value; φ4 is an extreme frequency deviation threshold value; R is a maximum reward reference value; k1 is an exponential attenuation coefficient; k2 is a heavy penalty increasing coefficient; R is a heavy penalty base coefficient; α is a linear segment slope in a medium deviation interval; β is a linear segment intercept in the medium deviation interval; z1 is a linear segment slope in a large deviation interval; and z2 is a linear segment intercept in the large deviation interval. lim In the formula, φ1 is a first interval frequency deviation threshold value; φ2 is a second interval frequency deviation threshold value; φ3 is a third interval frequency deviation threshold value; φ max In the formula, φ1 is a first interval frequency deviation threshold value; φ2 is a second interval frequency deviation threshold value; φ3 is a third interval frequency deviation threshold value; φ In the formula, φ1 is a first interval frequency deviation threshold value; φ2 is a second interval frequency deviation threshold value; φ3 is a third interval frequency deviation threshold value; φ 6. The method of intelligent setting and collaborative control of MMC based on deep reinforcement learning according to claim 4, characterized in that, The TD target value is specifically: ； In the formula, denotes a discount factor, is a TD target value, denotes the reward corresponding to the th sample, denotes the target evaluation network parameter, denotes the next state corresponding to the th sample, is a target policy network parameter, is a state-action value of the target Q network, is an output action of the target policy network; The loss function based on mean square error is specifically: ； In the formula, The value of the loss function. Indicates the number of training samples. For the TD target value, Indicates the first The state corresponding to each sample Indicates the first The action corresponding to each sample This indicates that the network parameters are evaluated online.

7. The method of intelligent setting and collaborative control of MMC based on deep reinforcement learning according to claim 4, characterized in that, The soft update strategy is specifically: ； wherein is a soft update coefficient, is an online critic network parameter in the Critic network, represents a network parameter in the Actor network.

8. A system for intelligent setting and collaborative control of MMC based on deep reinforcement learning, characterized in that, comprises: a data acquisition module configured to acquire current operating state data of a converter in a power transmission system, the current operating state data comprising system frequency, frequency reference value, frequency deviation and frequency change rate; The coefficient decoupling module is configured to establish a third-order active closed-loop small signal transfer function based on an active transient feedback compensation algorithm, and to decouple a damping coefficient and a primary frequency modulation coefficient in VSG control through the third-order active closed-loop small signal transfer function. The virtual inertia optimization module is configured to input the current operating state data into the trained DDPG agent for processing to obtain an optimization result of virtual inertia. The adaptive adjustment module is configured to apply the optimized virtual inertia to a rotational inertia control link of VSG control, and to perform adaptive adjustment of the virtual inertia through the trained DDPG agent.

9. An electronic device, comprising: It comprises: at least one processor, at least one memory and a data bus; wherein the processor and the memory complete mutual communication through the data bus; The memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the method for MMC intelligent setting and collaborative control based on deep reinforcement learning according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method for MMC intelligent setting and collaborative control based on deep reinforcement learning according to any one of claims 1-7.

Images