Virtual synchronous generator parameter determination method and device

Abstract

The invention discloses a virtual synchronous generator parameter determination method and equipment, and relates to the field of power equipment. The rotational inertia and the damping coefficient can be dynamically determined, and the parameter accuracy is improved. The virtual synchronous generator parameter determination method comprises the steps that the frequency difference, the power difference and the frequency change rate of a power grid are collected in real time, a state vector at each moment is determined and input into a reinforcement learning agent, the reinforcement learning agent comprises an action network and an evaluation network, the action network outputs corresponding action parameters, and the evaluation network outputs a Q value; calculating a reward value according to the state vector and the action parameter, and extracting an experience tuple when the reward value meets a preset condition; obtaining a target Q value of the experience tuple, and calculating a time difference error between the Q value and the target Q value to update the evaluation network; calculating the strategy gradient of the action parameters to update the action network; and determining action parameters corresponding to the current state vector through the optimized reinforcement learning agent, wherein the action parameters are rotational inertia and a damping coefficient.

Description

Technical Field

[0001] The present application relates to the field of power equipment, and in particular to a method and device for determining parameters of a virtual synchronous generator. Background Art

[0002] As renewable energy penetration increases, the inertia of power systems decreases, leading to increasingly prominent grid frequency stability issues. Virtual synchronous generator (VSG) technology, which simulates the inertia and damping characteristics of synchronous machines, has become an important means of improving the stability of renewable energy grid-connected systems.

[0003] Existing VSG parameter adjustment methods primarily rely on fixed parameter methods, which determine the moment of inertia J and damping coefficient D through offline simulation. For example, patent CN202410953321.7, "Adaptive Control Technology for the Moment of Inertia and Damping Coefficient of a Photovoltaic Storage VSG System," proposes dynamically adjusting J and D using a PID controller. The technical solution involves selecting an oscillation cycle from a simulated power-frequency curve under fixed parameter control mode and dividing the oscillation cycle into four intervals: ① t1-t2, ② t2-t3, ③ t3-t4, and ④ t4-t5. The parameter variations in each interval are analyzed separately. Based on the analysis results, a control parameter adjustment strategy is applied to adjust the moment of inertia and damping coefficient. However, grid disturbances are random and diverse (such as sudden load changes and intermittent renewable energy output). Preset fixed intervals cannot cover all dynamic scenarios, potentially leading to delayed or ineffective parameter adjustment. Furthermore, adjustment rules based on the arctan function and fixed thresholds can only handle disturbances within a specific range and cannot adapt to complex nonlinear coupled systems. Therefore, the VSG control with fixed parameters J and D is difficult to cope with complex and changeable grid conditions (such as power mutation and load fluctuation), and is prone to problems such as frequency overshoot and long adjustment time. Summary of the Invention

[0004] This application provides a method and device for determining the parameters of a virtual synchronous generator, which dynamically adjusts the moment of inertia J and damping coefficient D of the VSG through a deep reinforcement learning algorithm to make the parameters more accurate.

[0005] In a first aspect, the present application provides a method for determining parameters of a virtual synchronous generator, comprising: Collect the frequency difference, power difference and frequency change rate of the power grid in real time, and determine the state vector at each moment based on the frequency difference, power difference and frequency change rate at each moment; Inputting the state vector at each moment into a reinforcement learning agent, the reinforcement learning agent comprising an action network and an evaluation network, the action network outputting corresponding action parameters based on the state vector, and the evaluation network outputting a Q value based on the state vector and the action parameters; Calculate the reward value at each moment based on the state vector and action parameters at each moment. When the reward value meets the preset conditions, extract the state vector, action parameters and reward value at the target moment, as well as the state vector at the next moment as the experience tuple. The target moment is the moment when the reward value meets the preset conditions. Obtaining a target Q value of the experience tuple through a target network corresponding to the evaluation network, calculating a time differential error between the Q value of the experience tuple and the target Q value, and updating the evaluation network based on the time differential error; wherein the evaluation network and the corresponding target network are initially identical, and the target network is also updated when the evaluation network is updated; Calculating the policy gradient of the action parameters output by the action network for the experience tuple, and updating the action network using the policy gradient to obtain an optimized reinforcement learning agent; The action parameters corresponding to the current state vector are determined by the optimized reinforcement learning agent, and the action parameters are the moment of inertia and the damping coefficient.

[0006] According to the virtual synchronous generator parameter determination method of this embodiment, reinforcement learning is integrated with virtual synchronous generator control technology. Through the reinforcement learning agent, the rotational inertia and damping coefficient under different power grid states can be determined in real time, making the parameters more accurate and truly reflecting the characteristics of the generator. This is conducive to dynamically adjusting the rotational inertia and damping coefficient of the virtual synchronous generator and improving the accuracy of the virtual synchronous machine's response.

[0007] In the above scheme, the reward value at each moment is calculated based on the state vector and action parameters at each moment, including: Calculate the reward value corresponding to the state vector and action parameters through the reward function; The reward function is:

[0008] Among them, k1, k2, k3 are the penalty coefficients of frequency difference, power difference, and frequency change rate respectively, k4 is the penalty coefficient of motion parameter change, J is the moment of inertia, D is the damping coefficient, and t is time.

[0009] In the above solution, determining the state vector at each moment based on the frequency difference, power difference, and frequency change rate at each moment includes: The frequency difference, power difference and frequency change rate at each moment are normalized, and the normalized values ​​are converted into vectors to obtain the corresponding state vector.

[0010] The above scheme also includes: The parameters of the target networks corresponding to the evaluation network and the action network are updated according to a preset soft update coefficient; the action network and the corresponding target network are initially the same.

[0011] In the above solution, the time difference error is:

[0012] in To evaluate the network, To evaluate the parameters of the network; is the target Q value, is the i-th state vector, is the parameter of the i-th action;

[0013] in, is the i-th reward value, γ is the discount factor, is the target network corresponding to the evaluation network, To evaluate the parameters of the target network corresponding to the network, is the target network corresponding to the action network, are the target network parameters corresponding to the action network.

[0014] The above scheme also includes: Determine the parameter gradient by minimizing the time difference error, and update the parameters of the evaluation network using the gradient descent method based on the parameter gradient; The parameters of the action network are updated using the gradient ascent method based on the policy gradient.

[0015] The above solution further includes: adjusting the parameters of the virtual synchronous generator through the rotational inertia and damping coefficient.

[0016] In a second aspect, the present application provides a device for determining parameters of a virtual synchronous generator, comprising: The state acquisition module is used to collect the frequency difference, power difference and frequency change rate of the power grid in real time, and determine the state vector at each moment based on the frequency difference, power difference and frequency change rate at each moment; A parameter output module, configured to input the state vector at each moment into a reinforcement learning agent, wherein the reinforcement learning agent includes an action network and an evaluation network, wherein the action network outputs corresponding action parameters based on the state vector, and the evaluation network determines the Q value of the state vector and the action parameter; A reward calculation module is used to extract the state vector, action parameters and reward value at a target moment, and the state vector at the next moment as an experience tuple when the reward value meets the preset conditions; the target moment is the moment when the reward value meets the preset conditions; An evaluation network online optimization module is configured to obtain a target Q value of an experience tuple using a target network corresponding to the evaluation network, calculate a time differential error between the Q value of the experience tuple and the target Q value, and update the evaluation network based on the time differential error; wherein the evaluation network and the corresponding target network are initially identical, and the target network is also updated when the evaluation network is updated; An action network online optimization module is used to calculate the policy gradient of the action parameters output by the action network for the experience tuple, and to update the action network using the policy gradient to obtain an optimized reinforcement learning agent; The parameter determination module is used to determine the action parameters corresponding to the current state vector through the optimized reinforcement learning agent, where the action parameters are the moment of inertia and the damping coefficient.

[0017] In a third aspect, the present application provides an electronic device comprising a memory and one or more processors. The memory stores one or more computer programs, each of which includes instructions that, when executed by the processor, cause the electronic device to perform the method for determining parameters of a virtual synchronous generator as described in the first aspect.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored therein. When the instructions are executed on an electronic device, the electronic device executes the virtual synchronous generator parameter determination method as in the first aspect.

[0019] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the virtual synchronous generator parameter determination method as described in the first aspect.

[0020] It can be understood that the beneficial effects that can be achieved by the virtual synchronous generator parameter determination device, electronic device, computer-readable storage medium, and computer program product provided above can refer to the beneficial effects in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of a method for determining parameters of a virtual synchronous generator provided in an embodiment of the present application; Figure 2 A control principle structure diagram of a virtual synchronous generator in a method for determining parameters of a virtual synchronous generator provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a device for determining parameters of a virtual synchronous generator provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] To facilitate a clear description of the technical solutions of the embodiments of this application, terms such as "first" and "second" are used in the embodiments of this application to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not limit their order of precedence. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and do not necessarily imply differences. It should be noted that in the embodiments of this application, terms such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more" refers to two or more.

[0023] It should be noted that the “at…” in the embodiments of the present application can be the instant when a certain situation occurs, or it can be a period of time after the occurrence of a certain situation. The embodiments of the present application do not make specific limitations on this.

[0024] Below, some technical terms involved in this application are explained: Virtual synchronous generator (VSG): A virtual synchronous generator (VSG) simulates the external characteristics of a synchronous generator (such as inertia and damping) through a power electronic converter to achieve frequency and voltage support functions for grid-connected equipment.

[0025] Deep Deterministic Policy Gradient (DDPG): A reinforcement learning algorithm that combines deep neural networks with deterministic policies and is suitable for optimization problems in continuous action spaces.

[0026] Moment of inertia (J): A parameter that characterizes the inertia of the rotor of a VSG-simulated synchronous generator and affects the rate at which the system frequency changes.

[0027] Damping coefficient (D): A parameter that characterizes the damping effect of the VSG simulating synchronous generator and affects the decay rate of the system oscillation.

[0028] The implementation of this embodiment will be described in detail below with reference to the accompanying drawings.

[0029] This embodiment provides a method for determining parameters of a virtual synchronous generator. By way of example, the method can be applied to various electronic devices such as computers (PCs), tablet computers, virtual reality / augmented reality devices, wearable devices, industrial computers, and vehicle computers; it can also be applied to servers, cloud computing, server clusters, etc., and this embodiment does not impose any special restrictions on this.

[0030] Figure 1 A flow chart of a method for determining parameters of a virtual synchronous generator provided in an embodiment of the present application is shown.

[0031] like Figure 1 As shown, the method for determining parameters of a virtual synchronous generator may include the following steps: Step 101: collecting the frequency difference, power difference and frequency change rate of the power grid in real time, and determining the state vector at each moment based on the frequency difference, power difference and frequency change rate at each moment.

[0032] Specifically, the frequency difference, power difference and frequency change rate at each moment are normalized, and the normalized values ​​are converted into vectors to obtain the corresponding state vector. , power difference and frequency change rate , normalized to the state vector:

[0033] The normalization process can be expressed as: (1) After normalization, the numerical range of each parameter is [-1, 1], which can avoid the impact of dimensional differences on algorithm convergence.

[0034] Step 102: Input the state vector at each moment into a reinforcement learning agent, wherein the reinforcement learning agent includes an action network and an evaluation network. The action network outputs corresponding action parameters based on the state vector, and the evaluation network determines the Q value of the state vector and the action parameters.

[0035] The reinforcement learning agent refers to a reinforcement learning model that uses DDPG decision-making. This reinforcement learning agent consists of an action (actor) network and a critic (critic) network. These action and critic networks are online networks, and the agent also includes a corresponding target network. During initial model construction, the online network is identical to the target network: the action network is identical to the corresponding target action network, and the critic network is identical to the corresponding target critic network. The online network is used to process online collected power grid data, namely the state vector, and perform online optimization based on the processing results.

[0036] For example, the state vector s t Input the Actor network and get the network output continuous action , the action generation formula is: (2) in is the Actor strategy function, is the Actor network parameter, To explore noise (such as OU noise), constrain the adjustment range of action parameters. .

[0037] The critic network determines the Q value based on the state vector and action parameters, which is used to evaluate the benefits of the action output by the actor network.

[0038] The Actor network is equivalent to an "intelligent decision maker" and is based on the current state of the power grid ( 、 、 ) Generates adjustment strategies for J and D in real time, and their output actions affect the electromechanical characteristics of the VSG.

[0039] The Critic network is equivalent to a "strategy evaluator", which evaluates the long-term benefits of the current action through the Q-value function and guides the Actor to optimize the strategy.

[0040] Step 103: Calculate the reward value at each moment based on the state vector and action parameters at each moment. When the reward value meets the preset conditions, extract the state vector, action parameters, and reward value at the target moment, as well as the state vector at the next moment as the experience tuple. The target moment is the moment when the reward value meets the preset conditions.

[0041] Calculate the reward value corresponding to the state vector and action parameters through the reward function ; For example, the reward function is: (3) in, k 1, k 2, k 3 are the penalty coefficients for frequency difference, power difference, and frequency change rate, respectively. k 4 is the penalty coefficient for action parameter changes, J is the moment of inertia, D is the damping coefficient, t For time.

[0042] When a certain moment t When the reward value meets the preset conditions, for example, it is greater than the preset threshold, then t The target time is the time at which t The state vector st， The action parameters a t And the state vector at the next moment s t+1 As a set of experience values, that is, experience tuple. Exemplarily, the experience tuple may also include the moment t The reward value, that is, the experience tuple is expressed as: ( s t , a t , r t , s t+1 ). The experience tuple can be saved into the experience replay pool to optimize the reinforcement learning agent.

[0043] Step 104: Obtain a target Q value of the experience tuple through the target network corresponding to the evaluation network, calculate the time differential error between the Q value of the experience tuple and the target Q value, and update the evaluation network based on the time differential error; wherein the evaluation network and the corresponding target network are initially the same, and the target network is also updated when the evaluation network is updated.

[0044] Step 105: Calculate the policy gradient of the action parameters output by the action network for the experience tuple, and update the action network using the policy gradient to obtain an optimized reinforcement learning agent.

[0045] The process of obtaining the grid frequency difference, power difference, and frequency change rate is carried out in real time. This application determines the corresponding action parameters, namely the moment of inertia and damping coefficient, through the real-time state vector, adjusts the virtual synchronous generator, and then updates the reinforcement learning agent through the accumulated data (ie, experience tuples). The action at the next time is determined by the updated reinforcement learning agent.

[0046] Specifically, the target Q value of the experience tuple is calculated using the target network corresponding to the evaluation network, and the temporal differential error between the Q value of the experience tuple and the target Q value is calculated. The target network is initially identical to the evaluation network and the action network. The evaluation network is updated based on the temporal differential error, and the target network is also updated synchronously. The target network is updated by updating the parameters of the target network according to a preset soft update coefficient. For the action network, the policy gradient of the action network is calculated, and the action network is updated using this policy gradient.

[0047] The goal of the critic network update is to minimize the error (TD error) between the predicted Q value and the target Q value. The time difference error is used as the loss function, and the expression is: (4) in To evaluate the network, To evaluate the parameters of the network; is the target Q value, is the i-th state vector, is the parameter of the i-th action; (5) in, is the i-th reward value, γ is the discount factor, is the target network corresponding to the evaluation network, To evaluate the parameters of the target network corresponding to the network, is the target network corresponding to the action network, are the target network parameters corresponding to the action network.

[0048] In this embodiment, the parameter gradient is determined by minimizing the time difference error, and the parameters of the evaluation network are updated by gradient descent. Minimizing the time difference error means substituting the calculated target Q value into formula (4) to solve the parameter gradient of the evaluation network that minimizes the time difference error. The specific update process is: when the amount of experience pool data is sufficient, randomly sample batch data Update the network, N is the batch size (number of samples). Sample from the experience tuple to obtain the sampling batch, calculate the target Q value through the sampling batch, and minimize L( θ Q ) Determine the parameter gradient and use gradient descent to update the evaluation network parameters θ Q′ .

[0049] The goal of actor network update is to maximize the Q value evaluated by the critic. Specifically, the parameters are updated along the policy gradient direction. The policy gradient is: (6) Where, is the policy gradient, is the gradient of the Critic output to the action, is the gradient of the Actor's output with respect to its parameters.

[0050] When the actor network is updated, the gradient of the critic network output needs to be determined, so the update process is: fix the critic parameters , calculate the policy gradient according to the critic parameters, and use the gradient ascent method based on the policy gradient to update the parameters of the actor network.

[0051] Every time the actor network and critic network are updated, the corresponding target network is also updated synchronously. The target network is updated as follows: (7) in, is the soft update coefficient, , to ensure training stability. is the target network parameter corresponding to the updated online Critic, is the parameter of the target network corresponding to the updated online Actor network, To update the target network parameters corresponding to the critic network before, The target network parameters corresponding to the Actor network before updating.

[0052] The parameters of the online network and the target network are initially the same. When the target network is updated, a part (proportion τ ) parameters are retained, and the remaining part (i.e. 1- τ ) parameters are randomly updated to form the updated parameters of the target network. That is, the updated network parameters of the target network corresponding to the evaluation network are Depend on and Constitute, accordingly, the target network corresponding to the action network updates the network parameters Depend on and constitute.

[0053] Step 106: Determine the action parameters corresponding to the current state vector through the optimized reinforcement learning agent, where the action parameters are the moment of inertia and the damping coefficient.

[0054] The state vector of the power grid is collected, and the corresponding action parameters are output through the action network. These action parameters are used as the moment of inertia and damping coefficient of the virtual synchronous generator, thereby achieving frequency control of the virtual synchronous generator. This process can be carried out in real time. Therefore, after adjusting the parameters of the virtual synchronous generator, the accuracy of the parameters at the previous moment can be verified by the state vector at the next moment, thus forming a closed loop. The experience tuple corresponding to the accurate parameters is used to update and optimize the reinforcement learning model. The updated and optimized model can then be used to determine the next action parameters, achieving real-time and dynamic updating of the action parameters, so that the action parameters can best reflect the actual characteristics and improve the accuracy of the action parameters.

[0055] This embodiment further includes adjusting parameters of the virtual synchronous generator using the moment of inertia and damping coefficient. The moment of inertia and damping coefficient of the virtual synchronous generator are adjusted to the values ​​output by the optimized reinforcement learning agent, thereby more accurately responding to power grid fluctuations and regulating the stability of the power grid system.

[0056] VSG control simulates the external characteristics of a synchronous generator, enabling the converter to regulate voltage and frequency. The system provides damping and inertia support, and the damping inertia can be adjusted. VSG control is primarily divided into active frequency control, reactive voltage control, and virtual impedance control.

[0057] Active frequency control mainly takes into account the rotor motion equation and primary frequency modulation characteristics of the synchronous generator.

[0058] The rotor motion equation of a synchronous generator is: (8) in, J is the moment of inertia; D is the damping coefficient; ω is the synchronous angular velocity, ω 0 is the angular velocity reference value; T m Mechanical torque; T e is the electromagnetic torque; T d is the damping torque; P e is the electromagnetic power; P m is the mechanical power.

[0059] The frequency regulation characteristics of synchronous generator are: (9) in, P ref is the active power reference value; k p is the droop coefficient.

[0060] Substituting formula (9) into formula (8) yields: (10) By “digitally creating zero” in Equation (10) and performing Laplace transform, the active frequency control of the virtual synchronous generator can be obtained: (11) After determining the moment of inertia and damping coefficient, the active frequency control of the virtual synchronous generator can be determined according to the rotor motion equation and primary frequency modulation characteristics. The active frequency control structure of the virtual synchronous generator is shown in the figure below. Figure 2 shown.

[0061] For example, when the power grid experiences power fluctuations (such as a sudden increase in load), the frequency deviation and rate of change Increase, the Critic network feeds back the penalty signal through the reward function, driving the Actor to adjust J andD : Increase the moment of inertia J , slow down the frequency change speed, suppress overshoot; increase the damping coefficient D , accelerating oscillation decay and shortening adjustment time. Under steady-state conditions, the parameter change penalty term (k4) in the reward function suppresses unnecessary parameter adjustments and maintains system stability.

[0062] This implementation method obtains power grid data in real time, DDPG agent outputs action parameters, parameter execution, J and D Real-time optimization and reward feedback are used to optimize the DDPG agent, forming a closed-loop control chain of "perception-decision-execution-feedback". This can update the policy network in real time and respond more promptly and effectively to complex grid conditions, such as sudden load changes and fluctuations in renewable energy processing. Compared with the method of adjusting parameters using static preset rules, this implementation method uses a deep reinforcement learning algorithm for real-time optimization, breaking through the limitations of static adjustment and providing efficient and adaptive support for the frequency stability of the power coefficient. In particular, for complex scenarios such as high-proportion renewable energy grid-connected systems and isolated microgrids, it can timely capture the dynamic coupling characteristics of the grid and improve the speed and accuracy of response to grid changes.

[0063] Furthermore, this embodiment also provides a device for determining parameters of a virtual synchronous generator, which can be used to execute the above-mentioned method for determining parameters of a virtual synchronous generator.

[0064] like Figure 3As shown, the virtual synchronous generator parameter determination device 300 specifically includes: a state acquisition module 301, which is used to collect the frequency difference, power difference and frequency change rate of the power grid in real time, and determine the state vector at each moment according to the frequency difference, power difference and frequency change rate at each moment; a parameter output module 302, which is used to input the state vector at each moment into the reinforcement learning agent, and the reinforcement learning agent includes an action network and an evaluation network, and the action network outputs corresponding action parameters based on the state vector, and the evaluation network determines the Q value of the state vector and the action parameter; a reward calculation module 303, which is used to extract the state vector, action parameters and reward value at the target moment, and the state vector at the next moment as an experience tuple when the reward value meets the preset conditions, and the target moment is the time when the reward value meets the preset conditions. the moment of the component; the evaluation network online optimization module 304 is used to obtain the target Q value of the experience tuple through the target network corresponding to the evaluation network, calculate the time difference error between the Q value of the experience tuple and the target Q value, and update the evaluation network based on the time difference error; wherein, the evaluation network and the corresponding target network are initially the same, and the target network is also updated when the evaluation network is updated; the action network online optimization module 305 is used to calculate the policy gradient of the action parameters output by the action network for the experience tuple, update the action network through the policy gradient, and obtain the optimized reinforcement learning agent; the parameter determination module 306 is used to determine the action parameters corresponding to the current state vector through the optimized reinforcement learning agent, and the action parameters are the moment of inertia and the damping coefficient.

[0065] The specific details of each module or unit in the above-mentioned virtual synchronous generator parameter determination device have been described in detail in the corresponding virtual synchronous generator parameter determination method, and therefore will not be repeated here.

[0066] The embodiment of the present application also provides an electronic device, Figure 3 A schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. Figure 3 The electronic device 600 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0067] like Figure 3 As shown, electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for system operation are also stored in RAM 603. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0068] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 608 including devices such as a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read from the removable media can be installed in the storage section 608 as needed.

[0069] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from removable media 611. When executed by the central processing unit (CPU) 601, the computer program performs the aforementioned functions defined in the embodiments of the present application.

[0070] For example, when the computer program is executed by the central processing unit (CPU) 601, it can be executed as follows: collecting the frequency difference, power difference and frequency change rate of the power grid in real time, and determining the state vector at each moment based on the frequency difference, power difference and frequency change rate at each moment; inputting the state vector at each moment into the reinforcement learning agent, the reinforcement learning agent includes an action network and an evaluation network, the action network outputs corresponding action parameters based on the state vector, and the evaluation network outputs a Q value based on the state vector and the action parameters; calculating the reward value at each moment based on the state vector and action parameters at each moment, and when the reward value meets the preset conditions, extracting the state vector, action parameters and reward value at the target moment, as well as the state vector at the next moment The vector is used as an experience tuple, and the target moment is the moment when the reward value meets the preset conditions; the target Q value of the experience tuple is obtained through the target network corresponding to the evaluation network, the time difference error between the Q value of the experience tuple and the target Q value is calculated, and the evaluation network is updated based on the time difference error; wherein, the evaluation network and the corresponding target network are initially the same, and the target network is also updated when the evaluation network is updated; the policy gradient of the action parameter output by the action network for the experience tuple is calculated, and the action network is updated by the policy gradient to obtain an optimized reinforcement learning agent; the action parameters corresponding to the current state vector are determined by the optimized reinforcement learning agent, and the action parameters are the moment of inertia and the damping coefficient.

[0071] It should be noted that the computer-readable medium described in this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0073] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, and the units described may also be provided in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0074] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, each of which includes instructions that, when executed by an electronic device, cause the electronic device to implement the methods described in the above embodiments.

[0075] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0076] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for determining parameters of a virtual synchronous generator, characterized in that: include: Collect the frequency difference, power difference and frequency change rate of the power grid in real time, and determine the state vector at each moment based on the frequency difference, power difference and frequency change rate at each moment; Inputting the state vector at each moment into a reinforcement learning agent, the reinforcement learning agent comprising an action network and an evaluation network, the action network outputting corresponding action parameters based on the state vector, and the evaluation network outputting a Q value based on the state vector and the action parameters; Calculate the reward value at each moment based on the state vector and action parameters at each moment. When the reward value meets the preset conditions, extract the state vector, action parameters and reward value at the target moment, as well as the state vector at the next moment as the experience tuple. The target moment is the moment when the reward value meets the preset conditions. The reward value at each moment is calculated based on the state vector and action parameters at each moment, including: calculating the reward value corresponding to the state vector and action parameters through a reward function; the reward function is: in, k 1, k 2, k 3 are the penalty coefficients for frequency difference, power difference, and frequency change rate, respectively. k 4 is the penalty coefficient for action parameter changes, J is the moment of inertia, D is the damping coefficient, t For time; Obtaining a target Q value of the experience tuple through a target network corresponding to the evaluation network, calculating a time differential error between the Q value of the experience tuple and the target Q value, and updating the evaluation network based on the time differential error; wherein the evaluation network and the corresponding target network are initially identical, and the target network is also updated when the evaluation network is updated; Calculating the policy gradient of the action parameters output by the action network for the experience tuple, and updating the action network using the policy gradient to obtain an optimized reinforcement learning agent; The action parameters corresponding to the current state vector are determined by the optimized reinforcement learning agent, and the action parameters are the moment of inertia and the damping coefficient.

2. The method for determining parameters of a virtual synchronous generator according to claim 1, characterized in that: Determining the state vector at each moment according to the frequency difference, power difference, and frequency change rate at each moment includes: The frequency difference, power difference and frequency change rate at each moment are normalized, and the normalized values ​​are converted into vectors to obtain the corresponding state vector.

3. The method for determining parameters of a virtual synchronous generator according to claim 1, wherein: Also includes: Updating the parameters of the target network corresponding to the evaluation network and the action network respectively according to a preset soft update coefficient; The action network and the corresponding target network are initially identical.

4. The method for determining parameters of a virtual synchronous generator according to claim 3, characterized in that: The time difference error is: in To evaluate the network, To evaluate the parameters of the network; is the target Q value, is the i-th state vector, is the parameter of the i-th action; in, is the i-th reward value, γ is the discount factor, is the target network corresponding to the evaluation network, To evaluate the parameters of the target network corresponding to the network, is the target network corresponding to the action network, are the target network parameters corresponding to the action network.

5. The method for determining parameters of a virtual synchronous generator according to claim 1, wherein: Also includes: Determine the parameter gradient by minimizing the time difference error, and update the parameters of the evaluation network using the gradient descent method based on the parameter gradient; The parameters of the action network are updated using the gradient ascent method based on the policy gradient.

6. The method for determining parameters of a virtual synchronous generator according to claim 1, characterized in that: Also includes: The parameters of the virtual synchronous generator are adjusted by the rotational inertia and the damping coefficient.

7. A device for determining parameters of a virtual synchronous generator, characterized in that: include: The state acquisition module is used to collect the frequency difference, power difference and frequency change rate of the power grid in real time, and determine the state vector at each moment based on the frequency difference, power difference and frequency change rate at each moment; A parameter output module, configured to input the state vector at each moment into a reinforcement learning agent, wherein the reinforcement learning agent includes an action network and an evaluation network, wherein the action network outputs corresponding action parameters based on the state vector, and the evaluation network determines the Q value of the state vector and the action parameter; A reward calculation module is used to calculate the reward value at each moment based on the state vector and action parameters at each moment. When the reward value meets the preset conditions, the state vector, action parameters and reward value at the target moment, as well as the state vector at the next moment, are extracted as an experience tuple. The target moment is the moment when the reward value meets the preset conditions. The reward value at each moment is calculated based on the state vector and action parameters at each moment, including: calculating the reward value corresponding to the state vector and action parameters through a reward function; the reward function is: in, k 1, k 2, k 3 are the penalty coefficients for frequency difference, power difference, and frequency change rate, respectively. k 4 is the penalty coefficient for action parameter changes, J is the moment of inertia, D is the damping coefficient, t For time; An evaluation network online optimization module is configured to obtain a target Q value of an experience tuple using a target network corresponding to the evaluation network, calculate a time differential error between the Q value of the experience tuple and the target Q value, and update the evaluation network based on the time differential error; wherein the evaluation network and the corresponding target network are initially identical, and the target network is also updated when the evaluation network is updated; An action network online optimization module is used to calculate the policy gradient of the action parameters output by the action network for the experience tuple, and to update the action network using the policy gradient to obtain an optimized reinforcement learning agent; The parameter determination module is used to determine the action parameters corresponding to the current state vector through the optimized reinforcement learning agent, where the action parameters are the moment of inertia and the damping coefficient.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the electronic device, the electronic device executes the virtual synchronous generator parameter determination method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to execute the method for determining parameters of a virtual synchronous generator according to any one of claims 1 to 6.

Images