Federal learning-based active transformer area power equipment regulation and control method and power system

Abstract

The invention provides an active transformer area power equipment regulation and control method based on federated learning and a power system, and relates to the field of power engineering, the field of artificial intelligence and the field of federated learning. The method comprises the following steps: an intelligent agent end executes a first-stage training operation on an intelligent agent end model to obtain a first intelligent agent end model parameter; the server side carries out parameter aggregation on the respective first agent side model parameters of the plurality of agent sides to obtain global model parameters; the multiple agent ends execute second-stage training operation on the global agent model to obtain second agent end model parameters of the agent ends; the server side clusters the respective second agent side model parameters of the plurality of agent sides to obtain a plurality of center model parameters; the agent end determines a target agent end model parameter from the plurality of center model parameters; and the agent side regulates and controls power equipment related to the agent side to execute power equipment actions by using a target agent side model determined based on the target agent side model parameters.

Description

Technical Field

[0001] The present invention relates to the fields of power engineering, artificial intelligence and federated learning, and more specifically, to an active substation power equipment control method and a power system based on federated learning. Background Art

[0002] In the power system, with the rapid development of distributed energy and smart grids, the power equipment in the power system's substations are gradually equipped with intelligent control and self-optimization capabilities. However, with the demand for real-time collection of power operation data and intelligent control, the traditional substation control model lacks sufficient intelligent means when dealing with complex energy transactions, carbon quota management, power equipment operation and other tasks, making it difficult to achieve efficient energy management and low-carbon optimization of the power system. Summary of the invention

[0003] In view of this, the present invention provides an active substation power equipment control method and a power system based on federated learning.

[0004] One aspect of the present invention provides an active substation power equipment control method based on federated learning, which is applied to a power grid system, wherein the power grid system includes a service end and multiple intelligent agent ends, and the active substation power equipment control method includes: the intelligent agent end performs a first stage training operation on the intelligent agent end model to obtain a first intelligent agent end model parameter, wherein the intelligent agent end model is used to obtain a power equipment action for controlling the power equipment in the target active substation by processing the substation operation status of the target active substation in the power grid system; the service end performs parameter aggregation on the first intelligent agent end model parameters of each of the multiple intelligent agent ends to obtain to the global model parameters; the above-mentioned multiple intelligent agent terminals perform the second stage training operation on their respective global intelligent agent models to obtain the second intelligent agent terminal model parameters of the above-mentioned intelligent agent terminals, and the above-mentioned global intelligent agent model is determined based on the above-mentioned global model parameters; the above-mentioned server terminal clusters the second intelligent agent terminal model parameters of the above-mentioned multiple intelligent agent terminals to obtain multiple central model parameters; the above-mentioned intelligent agent terminals determine the target intelligent agent terminal model parameters from the multiple central model parameters; and the above-mentioned intelligent agent terminals use the target intelligent agent terminal model determined based on the above-mentioned target intelligent agent terminal model parameters to regulate the power equipment related to the above-mentioned intelligent agent terminals to perform power equipment actions.

[0005] Another aspect of the present invention provides a power grid system, including: a service end and multiple intelligent agent ends; the intelligent agent end is configured to perform a first stage training operation on the intelligent agent end model to obtain a first intelligent agent end model parameter, wherein the intelligent agent end model is used to obtain a power equipment action for regulating the power equipment in the target active substation by processing the substation operation status of the target active substation in the above-mentioned power grid system; the service end is configured to perform parameter aggregation on the first intelligent agent end model parameters of each of the multiple intelligent agent ends to obtain a global model parameter; the multiple intelligent agent ends are configured to perform a second stage training operation on their respective global intelligent agent models to obtain a second intelligent agent end model parameter of the above-mentioned intelligent agent, and the global intelligent agent model is determined based on the global model parameter; the service end is configured to cluster the second intelligent agent end model parameters of each of the multiple intelligent agent ends to obtain a plurality of central model parameters; the intelligent agent end is configured to determine the target intelligent agent end model parameter from the plurality of central model parameters; and the target intelligent agent end model determined based on the target intelligent agent end model parameter is used to regulate the power equipment related to the intelligent agent end to perform the power equipment action.

[0006] According to an embodiment of the present invention, each intelligent end is enabled to train its own local intelligent end model based on a federated learning mechanism, so that the first intelligent end model parameter can initially learn the local power equipment control strategy of the intelligent end. The server aggregates the first intelligent end model parameters of multiple intelligent ends, so that the obtained global model parameters can fuse the power equipment control strategies of multiple intelligent ends, and then each intelligent end can accelerate the learning efficiency and learning speed of the power equipment control strategy by training the global intelligent end model. The server clusters the second intelligent end model parameters so that multiple central model parameters represent the center of different types of power equipment control strategies, and then the target intelligent end model parameters can be determined from the multiple central model parameters by the intelligent end, so that the target intelligent end model fuses the respective power equipment control strategies of multiple intelligent end models of the same type, so that the intelligent end can more efficiently and stably control the power equipment to perform power equipment actions based on the target intelligent end model, thereby improving the overall operation stability of the power grid system. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0008] Figure 1 An exemplary system architecture of an active area power equipment control method and a power system based on federated learning according to an embodiment of the present invention is shown;

[0009] Figure 2A flow chart of a method for controlling power equipment in an active area based on federated learning according to an embodiment of the present invention is shown;

[0010] Figure 3 A flow chart showing the process of determining a target agent terminal model parameter from a plurality of central model parameters by an agent terminal according to an embodiment of the present invention is shown;

[0011] Figure 4 A flow chart of a method for controlling power equipment in an active area based on federated learning according to another embodiment of the present invention is shown;

[0012] Figure 5 A comparison diagram of the effects of the active substation power equipment control method based on federated learning provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0013] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0014] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0015] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0016] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0017] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0018] In the embodiment of the present invention, before obtaining or collecting the user's personal information, the user's authorization or consent is obtained.

[0019] The inventors found that with the rapid development of artificial intelligence and blockchain technology, the device control method based on federated learning has gradually attracted attention. The active area power equipment control method based on federated learning provided by the embodiment of the present invention can realize cross-area data collaborative computing and power equipment control optimization.

[0020] The embodiments of the present invention provide an active substation power equipment control method and a power system based on federated learning. The active substation power equipment control method based on federated learning is applied to a power grid system, which includes a service end and multiple intelligent agent ends. The active substation power equipment control method includes: the intelligent agent end performs a first-stage training operation on the intelligent agent end model to obtain first intelligent agent end model parameters, wherein the intelligent agent end model is used to obtain power equipment actions for controlling power equipment in the target active substation by processing the substation operation status of the target active substation in the power grid system; the service end performs parameter aggregation on the first intelligent agent end model parameters of each of the multiple intelligent agent ends to obtain global model parameters; the multiple intelligent agent ends perform a second-stage training operation on their respective global intelligent agent models to obtain second intelligent agent end model parameters of the intelligent agent ends, and the global intelligent agent model is determined based on the global model parameters; the service end clusters the second intelligent agent end model parameters of each of the multiple intelligent agent ends to obtain multiple central model parameters; the intelligent agent end determines the target intelligent agent end model parameters from the multiple central model parameters; and the intelligent agent end uses the target intelligent agent end model determined based on the target intelligent agent end model parameters to control the power equipment related to the intelligent agent end to perform power equipment actions.

[0021] According to an embodiment of the present invention, each intelligent end is enabled to train its own local intelligent end model based on a federated learning mechanism, so that the first intelligent end model parameter can initially learn the local power equipment control strategy of the intelligent end. The server aggregates the first intelligent end model parameters of multiple intelligent ends, so that the obtained global model parameters can fuse the power equipment control strategies of multiple intelligent ends, and then each intelligent end can accelerate the learning efficiency and learning speed of the power equipment control strategy by training the global intelligent end model. The server clusters the second intelligent end model parameters so that multiple central model parameters represent the center of different types of power equipment control strategies, and then the target intelligent end model parameters can be determined from the multiple central model parameters by the intelligent end, so that the target intelligent end model fuses the respective power equipment control strategies of multiple intelligent end models of the same type, so that the intelligent end can more efficiently and stably control the power equipment to perform power equipment actions based on the target intelligent end model, thereby improving the overall operation stability of the power grid system.

[0022] Figure 1 The exemplary system architecture of the active area power equipment control method and the power system based on federated learning according to the embodiment of the present invention is shown. It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0023] like Figure 1 As shown, the system architecture of the power system 100 according to this embodiment may include: a first intelligent agent terminal 101 , a second intelligent agent terminal 102 , a third intelligent agent terminal 103 and a server 104 .

[0024] The user can use the first intelligent terminal 101, the second intelligent terminal 102, and the third intelligent terminal 103 to interact with the server 104 through the network 104 to receive or send messages, etc. The first intelligent terminal 101, the second intelligent terminal 102, and the third intelligent terminal 103 can be servers, computers, and other devices for performing data processing tasks. The server 104 can be any type of server such as a cloud server and a cluster server. The server 104 can be used as a service end to interact with the first intelligent terminal 101, the second intelligent terminal 102, and the third intelligent terminal 103 for data. The first intelligent terminal 101, the second intelligent terminal 102, and the third intelligent terminal 103 can respectively regulate the power equipment of the first target active area, the second target active area, and the third target active area. The power equipment may include power generation equipment such as gas generators and wind turbines, or may also include other types of power equipment such as transformers and secondary equipment.

[0025] It should be understood that Figure 1 The number of servers, agent terminals, and target active stations in the embodiment is only for illustration. Any number of servers, agent terminals, and target active stations may be provided according to the implementation requirements.

[0026] Figure 2 A flow chart of an active substation power equipment control method based on federated learning according to an embodiment of the present invention is shown.

[0027] like Figure 2 As shown, the method includes operations S210 to S260.

[0028] In operation S210, the agent performs a first-stage training operation on the agent model to obtain first agent model parameters, wherein the agent model is used to obtain a power equipment action for regulating the power equipment in the target active area by processing the area operation state of the target active area in the power grid system;

[0029] According to an embodiment of the present invention, the invention is applied to a power grid system, which includes a service end and multiple intelligent end ends. The service end may include any type of device capable of executing data cloud functions, such as a server and a distributed server. The intelligent end may include any type of device, such as a computer and a local server.

[0030] In one example, the intelligent terminal may be an edge device of the power grid system, for example, the intelligent terminal may be an intelligent fusion terminal in an active area. The edge device may communicate with terminal devices such as smart meter devices in the power grid system to form an edge device of the power grid system.

[0031] In one example, the agent-side model can be constructed based on a reinforcement learning model, and the agent-side can perform the first-stage training operation based on a deep deterministic policy gradient algorithm (DDPG).

[0032] According to an embodiment of the present invention, a target active area may be associated with an intelligent agent terminal, and the same intelligent agent terminal may be associated with one or more target active areas to adjust power devices of one or more target active areas to perform power device actions.

[0033] According to an embodiment of the present invention, the operation status of the substation may include the operation status of the power equipment in the target active substation, the power load on the user side, the operation status data of the power supply line, etc. The power equipment action may include control information for controlling the operation parameters of the power equipment, such as power control information for controlling the power generation parameters of the generator, etc.

[0034] In operation S220, the server aggregates the first agent model parameters of each of the plurality of agent terminals to obtain a global model parameter.

[0035] In operation S230, the plurality of agents perform a second stage training operation on their respective global agent models to obtain second agent model parameters of the agent.

[0036] According to an embodiment of the present invention, the global agent model is determined based on the global model parameters. The server can send the global model parameters to multiple agent terminals, and each agent terminal can migrate the global model parameters to the current first agent terminal model to obtain the global agent models of the multiple agent terminals.

[0037] According to an embodiment of the present invention, the first agent model parameters of each of the multiple agent terminals can be processed based on any type of algorithm, so that the obtained global model parameters can learn the control strategies of the multiple first agent model parameters for the local control of the power equipment by the agent terminal. In this way, the multiple agent terminals can perform the second stage training operation on their respective global agent models to speed up the local model training speed of the agent terminal and save the computational overhead and energy consumption required for training the model.

[0038] In one example, the global agent model can be constructed based on a reinforcement learning model, and the agent side can perform the second stage training operation based on a deep deterministic policy gradient algorithm (DDPG).

[0039] In operation S240, the server clusters the second agent model parameters of each of the plurality of agent terminals to obtain a plurality of central model parameters.

[0040] According to an embodiment of the present invention, the second agent end model parameters of each of the multiple agent ends can be processed based on a clustering algorithm to obtain multiple clusters. The same cluster can include second agent end model parameters representing the same type, and the cluster center of the cluster can be the central model parameter.

[0041] In operation S250, the agent determines a target agent model parameter from among a plurality of central model parameters.

[0042] According to an embodiment of the present invention, the target intelligent agent model parameter may be a model parameter that meets the power equipment control requirements of the intelligent agent. The intelligent agent may determine the candidate intelligent agent model parameters from a plurality of central model parameters, and perform model parameter migration on the current second intelligent agent model based on the candidate intelligent agent model parameters to obtain a candidate intelligent agent model. Under the condition that the candidate power equipment action output by the candidate intelligent agent model can meet the control requirements, the candidate intelligent agent model may be determined as the target intelligent agent model.

[0043] It should be noted that the embodiments of the present invention do not limit the specific method of determining the target intelligent agent model parameters from multiple central model parameters, as long as the intelligent agent's demand for power equipment control in the target active substation can be met.

[0044] In operation S260, the intelligent agent uses the target intelligent agent model determined based on the target intelligent agent model parameters to regulate the power equipment related to the intelligent agent to perform the power equipment action.

[0045] According to an embodiment of the present invention, the intelligent agent can use the target intelligent agent model to process the station operation status of the target active station to obtain the action of the power equipment. The power equipment can be operated based on the equipment control information represented by the action of the power equipment, so as to improve the operation efficiency and operation stability of the power equipment in each target active station in the power system.

[0046] According to an embodiment of the present invention, the global model parameters and the central model parameters are ciphertext data encrypted by the server based on a homomorphic encryption algorithm, and the first agent-side model parameters and the second agent-side model parameters are ciphertext data encrypted by the agent-side based on a homomorphic encryption algorithm.

[0047] According to an embodiment of the present invention, the server and the agent can perform data encryption based on a consensus homomorphic encryption algorithm, so that the model parameters transmitted between the server and the agent (such as global model parameters and central model parameters, as well as first agent model parameters and second agent model parameters) can be ciphertext parameters, thereby improving the security of information interaction.

[0048] In one example, the server aggregates the first agent model parameters of each of the plurality of agents to obtain the global model parameters, which may include: the plurality of agents homomorphically encrypt the first agent model parameters, and send the obtained first agent model parameters with ciphertext data attributes to the server. The server performs a weighted average calculation on the first agent model parameters with ciphertext data attributes to obtain the global model parameters after parameter aggregation.

[0049] (1);

[0050] In formula (1), It represents the first agent model parameter with ciphertext data attributes at the i-th agent end, and Enc[] represents the homomorphic encryption algorithm. represents the global model parameter with ciphertext data attributes. Ω represents the number of agent terminals of a specified type.

[0051] In one example, the server can broadcast the global model parameters with ciphertext data attributes to multiple agent terminals, and the agent terminals can perform homomorphic decryption on the global model parameters with ciphertext data attributes based on the homomorphic decryption algorithm to obtain the plaintext information of the global model parameters. Model parameter migration is performed based on the plaintext information of the global model parameters to obtain a global agent model.

[0052] According to an embodiment of the present invention, the agent-side model includes an action network and an evaluation network. The agent-side model may be a model constructed based on a reinforcement learning algorithm, the action network (or actor network) and the evaluation network (or critic network) may be constructed based on a deep learning algorithm, and the action network and the evaluation network may be isomorphic models with the same or similar network structures.

[0053] According to an embodiment of the present invention, the first stage training operation may include the following operations.

[0054] Use the action network to process the operating status of the t-th sample substation to obtain the t-th sample power equipment action; use the evaluation network to process the operating status of the t-th sample substation and the t-th sample power equipment action to obtain the t-th target evaluation value; use the action network to process the operating status of the t+1-th sample substation to obtain the t+1-th sample power equipment action; use the evaluation network to process the operating status of the t+1-th sample substation and the t+1-th sample power equipment action to obtain the t+1-th target evaluation value; process the t-th target evaluation value and the t+1-th target evaluation value based on the minimization loss function to obtain the strategy loss value; update the model parameters of the current intelligent agent-side model based on the strategy loss value until the training conditions are met.

[0055] In one example, the i-th agent terminal of the target active station obtains the operating status of the t-th sample station and converts the operating status of the t-th sample station into i,t Input the action network of the i-th agent and output the t-th sample power equipment action a i,t The tth target evaluation value can be the Q value obtained after processing based on the Q function, and the tth target evaluation value is expressed as . Set the t+1th sample area operation status s i,t+1 Input the action network of the ith agent and output the t+1th sample power equipment action a i,t+1 The t+1th initial evaluation value is expressed as , the t+1th target evaluation value is expressed as . Among them, r represents the reward function value, and the reward function value corresponding to the operating state of the t-th sample station can be obtained from the quadruple of the sample experience pool. Represents the preset coefficient. The t-th target evaluation value and the t+1-th target evaluation value are processed by minimizing the loss function to obtain the policy loss value. The model parameters of the action network are updated by processing the measured loss value based on the stochastic gradient descent algorithm. Among them, the value function and policy function of the stochastic gradient descent algorithm can be guided by the time difference (Temporal Difference, TD) signal to update the model parameters of the action network. γ represents the preset coefficient of the reward function.

[0056] According to an embodiment of the present invention, the server end performs parameter aggregation on the first intelligent-end model parameters of each of the multiple intelligent-ends, including: determining the aggregation weights of the multiple first intelligent-end model parameters based on the sample data volumes of each of the multiple intelligent-ends; and performing weighted calculation on the multiple first intelligent-end model parameters based on the aggregation weights of the multiple first intelligent-end model parameters to obtain global model parameters.

[0057] According to an embodiment of the present invention, the sample data volume can be the data volume of the sample substation operation status used to train the first intelligent agent terminal model parameters. The larger the data volume scale of the sample data volume, the better the learning effect of the first intelligent agent model parameters. Based on the data volume scale represented by the sample data volume of each of the multiple intelligent agent terminals, the first intelligent agent terminal model parameters of the multiple intelligent agent terminals are assigned aggregation weights, so that the first intelligent agent terminal model parameters corresponding to the sample data volume with a larger data volume scale are assigned with a larger aggregation weight, so that the weights can be aggregated, and weighted calculations can be performed on the multiple first intelligent agent terminal model parameters, so that the global model parameters can be fully learned to have a strong control capability for the power equipment, so as to improve the accuracy and adaptability of the global model parameters for the prediction of the power equipment action, and improve the training efficiency and training effect of the subsequent intelligent terminal equipment in the second stage training operation.

[0058] According to an embodiment of the present invention, the agent terminal determines the target agent terminal model parameters from the multiple central model parameters, including each agent terminal executing Figure 3 The operation shown.

[0059] Figure 3 A flow chart is shown in which an intelligent agent determines target intelligent agent model parameters from multiple central model parameters according to an embodiment of the present invention.

[0060] like Figure 3 As shown, determining the target agent-side model parameters from a plurality of central model parameters may include operations S310 to S340.

[0061] In operation S310, the global agent model is updated based on the target central model parameters among the plurality of central model parameters to obtain an initial central agent end model.

[0062] In operation S320, the sample substation operation status is processed based on the initial central agent-side model to obtain the empirical power equipment action.

[0063] In operation S330 , the empirical power device actions are processed based on the loss function to obtain an empirical loss value.

[0064] In operation S340, in response to the experience loss value satisfying a preset loss threshold condition, the target center model parameter is determined as the target agent-side model parameter.

[0065] According to an embodiment of the present invention, updating the global agent model based on the target central model parameters among multiple central model parameters may include performing model parameter transfer learning on the global agent model based on the target central model parameters to obtain an initial central agent end model.

[0066] According to an embodiment of the present invention, processing the sample substation operation state based on the initial central intelligent agent end model may include using the initial central intelligent agent end model to process the sample substation operation state of the second stage training operation, and outputting the empirical power equipment action. Using the loss function to process the empirical power equipment action and the label data corresponding to the empirical power equipment action, the empirical loss value can be obtained.

[0067] According to an embodiment of the present invention, the experience loss value satisfies the preset loss threshold condition, which may include that the experience loss value is less than or equal to the preset loss value threshold. Alternatively, it may also include that the gradient value determined based on the experience loss value converges within a preset time threshold. Alternatively, it may also include that the gradient value determined based on the experience loss value is less than or equal to the preset gradient value threshold. The embodiment of the present invention does not limit the design method of the preset loss threshold condition.

[0068] According to an embodiment of the present invention, after the server clusters the model parameters of the second intelligent agent, it verifies whether the target central model parameters meet the power equipment control requirements of the intelligent agent based on the verification loss function at the intelligent agent in the target active substation, and implements local secondary clustering analysis on the multiple central model parameters obtained after the preliminary clustering at each intelligent agent, so as to avoid the heavy computational overhead and communication burden caused by clustering the model parameters of each two of the multiple intelligent agents in the federated learning process. By implementing the dual-end clustering analysis of the model parameters of the second intelligent agent by the server and the intelligent agent, the intelligent agent can dynamically select multiple central model parameters, and the clustering results of the central model parameters of the server can be corrected in time, which can accelerate the learning efficiency and training efficiency of the model for multiple intelligent agents with the same type of power equipment control mode, and improve the prediction accuracy and robustness of the target intelligent agent model.

[0069] In one example, the cluster c can be expressed based on the following formula (2): j The empirical loss value corresponding to the central model parameter.

[0070] (2);

[0071] In formula (2), argminL() represents the empirical loss function, Represents cluster c j The central model parameters of is the t-th sample data of the i-th intelligent agent terminal, and the t-th sample data may include the operating status of the t-th sample substation and the corresponding sample power equipment action, etc.

[0072] In one example, multiple agent terminals can each perform collaborative training operations based on the following operations.

[0073] The intelligent agent end of the target active station area performs model initialization, sets the total communication rounds during the initialization training period to Nc, the server-side clustering operation start round to Ne=Nc+1, and sets the learning rate and the number of intelligent agent ends i.

[0074] During the first stage training operation, each intelligent agent terminal performs the first and second stage training operations locally. For example, Nc rounds of reinforcement learning training can be specified to obtain the second intelligent agent terminal model parameters. Before the start of Ne training, the second intelligent agent terminal model parameters encrypted based on the homomorphic encryption algorithm are sent to the server. The server clusters the second intelligent agent terminal model parameters of multiple intelligent agents based on the clustering algorithm to obtain multiple cluster clusters. The cluster center of the cluster cluster is the central model parameter. The intelligent agent terminal verifies the target central model parameter based on the empirical loss function and sample data. When the empirical loss value meets the preset loss threshold condition, the target central model parameter is determined as the target intelligent agent terminal model parameter. At the same time, multiple intelligent agents corresponding to the target central model parameter can also be divided into target cluster clusters c corresponding to the target central model parameter. j , target clusters with the same regulatory strategy type.

[0075] According to an embodiment of the present invention, the intelligent agent uses a target intelligent agent model determined based on target intelligent agent model parameters to regulate power equipment related to the intelligent agent to perform power equipment actions, including the intelligent agent performing the following operations.

[0076] The target intelligent agent-side model is used to process the operating state of the t-th substation in the t-th period to obtain the action of the t-th power equipment; the power equipment related to the intelligent agent is controlled to perform the action of the t-th power equipment to obtain the operating state of the t+1-th substation in the t+1-th period; the operating state of the t-th substation and the action of the t-th power equipment are processed based on the reward function to obtain the t-th control reward information; the target intelligent agent-side model parameters of the current target intelligent agent-side model are optimized based on the t-th control reward information to obtain a new target intelligent agent-side model; the operating state of the t+1-th substation is processed based on the new target intelligent agent-side model to obtain the action of the t+1-th power equipment; and the power equipment related to the intelligent agent is controlled to perform the t+1-th power equipment action.

[0077] According to an embodiment of the present invention, the operating status of the t-th substation includes at least one of the following status information generated by the target active substation in the t-th time period: output of thermal energy storage equipment, power storage information of electric energy storage equipment, user-side load, electric energy trading information, and output of new energy equipment.

[0078] According to an embodiment of the present invention, the power storage information may include the power stored in the power storage device. The power transaction information may include the power price, but is not limited thereto, and may also include information such as carbon trading price and gas purchase price. The output of new energy equipment may include the output of new energy equipment such as photovoltaic energy equipment and wind power equipment.

[0079] According to an embodiment of the present invention, the tth electric power equipment action includes at least one of the following: fuel cell operation power regulation information, gas turbine operation power regulation information, gas boiler operation power regulation information.

[0080] In one example, the power regulation process of the i-th target active area in the t-th period can be understood as a sequential Markov decision process. The t-th area operation status s, which includes the output of thermal energy storage equipment, the power storage information of electric energy storage equipment, the user side load, the power transaction information, and the output of new energy equipment, is calculated. i,t Input the target intelligent agent model and obtain the tth power equipment action a i,t The intelligent agent executes the tth power equipment action a i,t To control the power equipment of the i-th target active area to perform the t-th power equipment action, and obtain the operating state s of the t+1-th area i,t+1 .

[0081] Among them, the operating status of the t-th station area is s i,t It can be expressed based on the following formula (3).

[0082] (3);

[0083] Among them, p i,t Indicates the price signal for purchasing electricity, .in,( ) is the price signal for direct purchase of electricity, ( ) is the carbon quota trading price signal, It is the price signal for purchasing gas.

[0084] Carbon quota trading price signals and gas purchase price signals. Indicates the user side load, Indicates the power consumption on the user side, Indicates the output of photovoltaic equipment, Indicates the output of wind power generation equipment. Indicates the power storage information of the electric energy storage (EES) device. Indicates the output of thermal energy storage (TES) equipment. Ω R Indicates the residential target active area, Ω C Indicates the commercial target active area, Ω I Indicates the active area of ​​industrial targets.

[0085] Among them, the tth power equipment action a i,t It can be expressed based on the following formula (4).

[0086] (4)

[0087] in, They represent the proportion of the operating power output of the fuel cell (FC) equipment, microturbine (MT) equipment and gas boiler (GB) equipment to the total operating power output. A value of 0 means no output. Represents the electric energy storage (EES) / thermal energy storage (TES) energy storage stage, the output value is negative, It represents the energy release stage, and the output value is positive.

[0088] In this example, the target agent model of the agent can be trained and updated based on the double-delayed deep deterministic policy gradient (DDPG) to generate a new operating state of the target active substation by interacting with the real-time environment of the target active substation, and optimize the power equipment action strategy to obtain the most economical low-carbon regulation strategy.

[0089] The optimization goal of each intelligent agent is to reduce the economic cost and carbon dioxide emissions caused by regulation and carbon quota trading as much as possible within a certain time scale. Therefore, the regulation reward information can include the electricity carbon trading price, gas cost, power equipment operation cost and thermal energy supply and demand balance constraint penalty item in the tth target active station area. Different from the trading constraints of net electric energy demand, if the active area agent fails to meet the thermal energy supply and demand balance under the constraints of each component, it will be punished by formula (7) to optimize the target agent model parameters of the current target agent model.

[0090] In one example, the t-th regulation reward information can be expressed based on formula (5).

[0091] (5);

[0092] Among them, r i,t is the reward information for the tth regulation, The operating cost of the power equipment, Gas transaction costs; The net demand for electricity is given by the following formulas (6) and (7):

[0093] (6);

[0094] (7)

[0095] in, Reward information for heat energy supply and demand balance, Sighing for the output of the cogeneration generator, Output for gas boiler, Q and S is the penalty coefficient of the running constraint. i，t The amount of gas purchased externally. and To directly purchase electric power quantity signal. and It is a signal of carbon trading volume. is the gas purchase price signal, To purchase electricity price signals directly, It is the price signal for carbon quota trading.

[0096] In one example, cluster analysis can be performed on multiple second agent model parameters based on the Improved Mean of Absolute Differences of pairwise Cosine similarity (IMADC) algorithm to solve the distance clustering defect caused by high-dimensional calculation.

[0097] For example, based on formula (8)

[0098] (8);

[0099] Where k represents the number of model gradient update vectors, which can be used to update the gradient vector corresponding to the model parameters of the second agent. Truncated singular value decomposition is performed along the neural network dimension to obtain k principal component update vectors v. α(n, v) represents the cosine similarity between the n-th second agent-side model parameters and the principal component update vector v. α(m, v) represents the cosine similarity between the m-th second agent-side model parameters and the principal component update vector v.

[0100] α(n, v) can be determined based on the following formula (9).

[0101] (9);

[0102] Formula (5) can be used to determine the cosine similarity between the second agent model parameters and the principal component update vectors of multiple agent terminals, and then obtain a similarity matrix. The similarity matrix is ​​expressed as formula (10).

[0103] (10)

[0104] Based on the matrix elements of the similarity matrix S, the IMADC metric can be implemented, and then the clustering clusters of the model parameters of the second agent end of multiple agents can be obtained based on the clustering algorithm. , where c i is the i-th cluster. The cluster center of the cluster can be expressed as the central model parameter.

[0105] According to an embodiment of the present invention, multiple associated agent terminals are associated with the same target center model parameter, and the multiple agent terminals include an associated agent terminal. It can be understood that the multiple associated agent terminals can correspond to the same cluster cluster, and the multiple associated agent terminals can be understood as the same agent terminal cluster cluster, and the multiple associated agent terminals of the same agent terminal cluster cluster have similar control strategies.

[0106] Figure 4 A flow chart of a method for controlling active substation power equipment based on federated learning according to another embodiment of the present invention is shown.

[0107] like Figure 4 As shown, the method for controlling power equipment in an active area based on federated learning also includes performing operations S410 to S430 in a preset training phase.

[0108] In operation S410, the associated agent side transmits the associated target agent side model parameters of the associated target agent side model to the server side.

[0109] In operation S420, the server aggregates the model parameters of the multiple current associated target agents to obtain associated global model parameters; and

[0110] In operation S430, the associated agent uses the new target agent model determined based on the associated global model parameters to regulate the power equipment in the target active substation to perform power equipment actions.

[0111] According to an embodiment of the present invention, before a preset training phase, the associated intelligent agent terminals may perform training operations on their respective target intelligent agent terminal models based on a reinforcement learning mechanism to obtain associated target intelligent agent terminal model parameters related to the preset training phase.

[0112] According to an embodiment of the present invention, the model parameters of the associated target intelligent agent may be ciphertext data encrypted based on a homomorphic encryption algorithm.

[0113] According to an embodiment of the present invention, the server aggregates the parameters of multiple current associated target intelligent agent model parameters, and can aggregate the power equipment control strategies learned by multiple associated intelligent agent models with similar control strategies, so that multiple associated intelligent agents of the same intelligent agent cluster can be trained collaboratively, so that the associated global model parameters can more accurately represent the diversified power equipment control strategies. In this way, a new target intelligent agent model is given to control the power equipment in the target active substation to perform power equipment actions, which can achieve iterative updates of the power equipment control strategies of the target active substation, and improve the overall operating efficiency and stability of the distribution network.

[0114] Figure 5 A comparison diagram of the effects of the active substation power equipment control method based on federated learning provided according to an embodiment of the present invention is shown.

[0115] like Figure 5 As shown, the active area power equipment control method based on federated learning provided in the embodiment of the present invention performs power equipment control with the three control methods based on the Advantage Actor-Critic algorithm (Advantage Actor-Critic, or A2C reinforcement learning algorithm), the master-slave game algorithm, and the Deep Q-Network (DQN) algorithm. In the comparison of the operating income improvement results of the target active area and the power equipment execution efficiency improvement results, both show a relatively obvious advantage, which shows that the active area power equipment control method based on federated learning provided in the embodiment of the present invention can effectively optimize the operating cost of the power equipment in the active area, and reduce energy waste and carbon emissions. Through the collaborative training of federated learning and the intelligent agent terminals of multiple target active areas, it is also possible to better deal with equipment privacy protection and external attack problems, and realize an efficient economic and low-carbon control strategy. In addition, through dynamic clustering and optimized federated reinforcement learning mechanism, collaborative optimization can be achieved between multiple active area intelligent agents, which speeds up the solution speed.

[0116] Based on the active area power equipment control method based on federated learning provided in the above embodiment, the embodiment of the present invention also provides a power grid system,

[0117] The power grid system provided according to an embodiment of the present invention includes: a service end and a plurality of intelligent body ends.

[0118] According to an embodiment of the present invention, the intelligent agent side is configured to perform a first stage training operation on the intelligent agent side model to obtain first intelligent agent side model parameters, wherein the intelligent agent side model is used to obtain power equipment actions for regulating power equipment in the target active substation by processing the substation operating status of the target active substation in the power grid system.

[0119] According to an embodiment of the present invention, the server is configured to perform parameter aggregation on the first agent-end model parameters of each of the plurality of agent-ends to obtain the global model parameters.

[0120] According to an embodiment of the present invention, multiple agent terminals are configured to perform a second stage training operation on their respective global agent models to obtain second agent terminal model parameters of the agent terminals, and the global agent model is determined based on the global model parameters.

[0121] According to an embodiment of the present invention, the server is configured to cluster the second agent model parameters of each of the plurality of agent terminals to obtain a plurality of central model parameters.

[0122] According to an embodiment of the present invention, the intelligent agent terminal is configured to determine target intelligent agent terminal model parameters from multiple central model parameters; and use the target intelligent agent terminal model determined based on the target intelligent agent terminal model parameters to regulate the power equipment related to the intelligent agent terminal to perform power equipment actions.

[0123] The power grid system provided by the embodiment of the present invention can be used to execute the active area power equipment control method based on federated learning provided by the embodiment of the present invention. The active area power equipment control method based on federated learning provided by the embodiment of the present invention can be applied to the power grid system provided by the embodiment of the present invention, and the embodiment of the present invention is not repeated here.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0125] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for controlling power equipment in an active area based on federated learning, characterized in that: Applied to a power grid system, the power grid system includes a service end and multiple intelligent end, and the active area power equipment control method includes: The intelligent agent performs a first stage training operation on the intelligent agent model to obtain a first intelligent agent model parameter, wherein the intelligent agent model is used to obtain an electric power equipment action for regulating the electric power equipment in the target active area by processing the area operation state of the target active area in the power grid system; The server aggregates the first agent model parameters of each of the plurality of agents to obtain a global model parameter; The plurality of agent terminals perform a second-stage training operation on their respective global agent models to obtain second agent terminal model parameters of the agent terminals, and the global agent model is determined based on the global model parameters; The server clusters the second agent model parameters of each of the plurality of agents to obtain a plurality of central model parameters; The agent terminal determines a target agent terminal model parameter from a plurality of the central model parameters; and The intelligent agent terminal uses the target intelligent agent terminal model determined based on the target intelligent agent terminal model parameters to regulate the power equipment related to the intelligent agent terminal to perform power equipment actions.

2. The active area power equipment control method according to claim 1 is characterized in that: The intelligent agent determines the target intelligent agent model parameter from the plurality of central model parameters, including: The intelligent agent performs the following operations: Based on the target central model parameters among the plurality of central model parameters, the global agent model is updated to obtain an initial central agent end model; Processing the sample area operation status based on the initial central intelligent agent end model to obtain the empirical power equipment action; Processing the empirical power equipment action based on the loss function to obtain an empirical loss value; and In response to the empirical loss value satisfying a preset loss threshold condition, the target center model parameter is determined as the target agent-side model parameter.

3. The active area power equipment control method according to claim 2 is characterized in that: A plurality of associated agent terminals are associated with the same target-centric model parameters, the plurality of agent terminals including the associated agent terminal, The method further comprises: In a preset training phase, the associated agent terminal transmits the associated target agent terminal model parameters of the associated target agent terminal model to the server terminal; The server aggregates the model parameters of multiple current associated target agents to obtain associated global model parameters; and The associated agent terminal uses a new target agent model determined based on the associated global model parameters to regulate the power equipment in the target active area to perform power equipment actions; Wherein, before the preset training stage, the associated intelligent agent terminals perform training operations on respective target intelligent agent terminal models based on a reinforcement learning mechanism to obtain associated target intelligent agent terminal model parameters related to the preset training stage.

4. The active area power equipment control method according to claim 1, characterized in that: The intelligent agent terminal uses the target intelligent agent terminal model determined based on the target intelligent agent terminal model parameters to regulate the power equipment related to the intelligent agent terminal to perform the power equipment action, including: The intelligent agent performs the following operations: Using the target intelligent agent end model to process the operating state of the t-th substation in the t-th period, obtaining the t-th power equipment action; Control the power equipment related to the intelligent terminal to perform the tth power equipment action, and obtain the operating state of the t+1th area in the t+1th time period; Processing the operating state of the t-th substation and the action of the t-th power equipment based on the reward function to obtain the t-th control reward information; Optimize the target agent-side model parameters of the current target agent-side model based on the t-th control reward information to obtain a new target agent-side model; Processing the operating state of the t+1th substation based on the new target intelligent agent model to obtain the t+1th power equipment action; and Control the power equipment related to the intelligent body terminal to perform the t+1th power equipment action.

5. The active area power equipment control method according to claim 4 is characterized in that: The operation status of the t-th station area includes at least one of the following status information generated by the target active station area in the t-th time period: Output of thermal energy storage equipment, power storage information of electric energy storage equipment, user-side load, electricity trading information, and output of new energy equipment.

6. The active area power equipment control method according to claim 4 is characterized in that: The tth electric equipment action includes at least one of the following: Fuel cell operating power control information, gas turbine operating power control information, gas boiler operating power control information.

7. The active area power equipment control method according to claim 1, characterized in that: The intelligent agent model includes an action network and an evaluation network; The first stage training operation includes: Using the action network to process the operating status of the t-th sample area, obtaining the t-th sample power equipment action; Processing the t-th sample substation operation state and the t-th sample power equipment action by using the evaluation network to obtain a t-th target evaluation value; Using the action network to process the operating status of the t+1th sample area, obtaining the t+1th sample power equipment action; Processing the t+1th sample substation operation state and the t+1th sample power equipment action by using the evaluation network to obtain the t+1th target evaluation value; Processing the tth target evaluation value and the t+1th target evaluation value based on a minimization loss function to obtain a strategy loss value; The model parameters of the current agent-side model are updated based on the strategy loss value until the training conditions are met.

8. The active area power equipment control method according to claim 1, characterized in that: The global model parameters and the central model parameters are ciphertext data encrypted by the server based on the homomorphic encryption algorithm, and the first agent-side model parameters and the second agent-side model parameters are ciphertext data encrypted by the agent-side based on the homomorphic encryption algorithm.

9. The active area power equipment control method according to claim 1, characterized in that: The server aggregating the first agent model parameters of each of the plurality of agent terminals comprises: Determining the aggregate weights of the model parameters of the plurality of the first agent terminals based on the sample data volumes of the plurality of the agent terminals respectively; and Based on the aggregate weights of the plurality of first agent-side model parameters, weighted calculation is performed on the plurality of first agent-side model parameters to obtain the global model parameters.

10. A power grid system, characterized in that: include: Server and multiple agent terminals; The intelligent agent end is configured to perform a first stage training operation on the intelligent agent end model to obtain a first intelligent agent end model parameter, wherein the intelligent agent end model is used to obtain an electric power equipment action for regulating the electric power equipment in the target active area by processing the area operation state of the target active area in the power grid system; The server is configured to aggregate the first agent model parameters of each of the plurality of agents to obtain a global model parameter; The plurality of agent terminals are configured to perform a second stage training operation on respective global agent models to obtain second agent terminal model parameters of the agent terminals, and the global agent model is determined based on the global model parameters; The server is configured to cluster the second agent model parameters of each of the plurality of agent terminals to obtain a plurality of central model parameters; The intelligent agent terminal is configured to determine a target intelligent agent terminal model parameter from a plurality of the central model parameters; and to regulate the power equipment associated with the intelligent agent terminal to perform power equipment actions using a target intelligent agent terminal model determined based on the target intelligent agent terminal model parameters.

Images