Power distribution network intelligent decision-making method and system based on knowledge embedding and multi-agent system

Abstract

The invention relates to a power distribution network intelligent decision-making method and system based on knowledge embedding and a multi-agent system, and the method comprises the following steps: collecting the operation data of each distributed energy device in a power distribution network in real time, and carrying out the preprocessing; performing hierarchical division on the power distribution network, and constructing a distributed multi-layer energy control architecture based on a multi-agent system; and based on the preprocessed operation data and a distributed multi-layer energy control architecture based on a multi-agent system, performing intelligent energy scheduling decision making by adopting a method of combining expert knowledge embedding and reinforcement learning, and completing a decision making process. Compared with the prior art, efficient distributed energy scheduling can be realized in a complex and changeable energy scene, and the intelligence, the flexibility and the robustness of power distribution network scheduling are improved.

Description

Technical Field

[0001] The present invention relates to the field of smart grid technology, and in particular to a distribution network intelligent decision-making method and system based on knowledge embedding and multi-agent system. Background Art

[0002] With the continuous growth of energy demand and the rapid development of distributed energy resources (such as photovoltaic power generation, wind power, energy storage systems and electric vehicles), the traditional centralized energy dispatch model has been unable to cope with the complexity of modern power systems. Due to the highly decentralized, unpredictable and diverse characteristics of distributed energy resources, the existing energy dispatch system performs poorly in dealing with uncertainty, real-time load changes and dynamic demand.

[0003] Traditional dispatching systems often rely on a single centralized control architecture, which makes it difficult to give full play to the advantages of distributed energy resources and easily leads to problems such as energy waste, dispatching delays, and load imbalance. In addition, with the continuous development of technologies such as energy Internet and smart grid, the complexity of modern power systems continues to increase, and the difficulty of dispatching and management is also increasing. Therefore, there is an urgent need for a more intelligent, distributed and adaptive energy dispatching method to cope with the complex energy network dispatching needs and achieve efficient coordination and optimization of distributed energy resources. As an emerging dispatching mode, the multi-agent system can achieve the coordinated optimization of distributed energy systems at different levels through hierarchical control and agent collaboration, and improve the flexibility and response speed of the system.

[0004] Patent application CN114841448A discloses a hierarchical and partitioned load optimization and control method based on a multi-agent system. In this control method, a hierarchical and partitioned dynamic control architecture is designed, and the architecture is combined with a multi-agent system. Through the "cloud-edge collaboration + Internet of Things technology", distributed resources with many points, wide ranges, various types and different characteristics are gathered together, and a multi-agent system is used for distributed management and control to achieve efficient energy interaction. On this basis, a multi-objective optimization strategy that takes into account economic applicability, low environmental pollution and high safety performance is proposed to achieve hierarchical and partitioned optimization and control of demand-side resources. However, this method needs to rely on precise modeling, which leads to performance degradation when it comes to the complexity and uncertainty of distributed energy resource scheduling in complex energy systems.

[0005] Therefore, technicians in this field urgently need to develop a multi-agent system intelligent decision-making method that can cope with the complexity and uncertainty problems of distributed energy resource scheduling in complex energy systems. Summary of the invention

[0006] The purpose of the present invention is to provide a distribution network intelligent decision-making method and system based on knowledge embedding and multi-agent system for improving the utilization rate of distributed energy in a complex environment.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A distribution network intelligent decision-making method based on knowledge embedding and multi-agent system includes the following steps:

[0009] Collect the operating data of each distributed energy device in the distribution network in real time and perform pre-processing;

[0010] Divide the distribution network into layers and build a distributed multi-layer energy control architecture based on a multi-agent system;

[0011] Based on the pre-processed operation data and the distributed multi-layer energy control architecture based on the multi-agent system, a method combining expert knowledge embedding and reinforcement learning is adopted to make intelligent energy scheduling decisions and complete the decision-making process.

[0012] Furthermore, the operating data of each distributed energy device includes the voltage, current and power output of photovoltaic power generation, wind power generation, energy storage equipment, electric vehicles and power networks.

[0013] Furthermore, the preprocessing operation includes standardizing the operation data of each distributed energy device, and the expression of the standardization processing is:

[0014]

[0015] Where, X norm is the standardized data, X is the original acquired data, and X min , X max are the minimum and maximum values ​​in the data, respectively.

[0016] Furthermore, the steps of constructing a distributed multi-layer energy control architecture based on a multi-agent system include:

[0017] The distribution network is divided into three levels from top to bottom, namely the distribution network dominant layer, regional coordination layer and equipment unit layer, and a distributed multi-layer energy control architecture based on a multi-agent system is constructed, including the distribution network dominant layer agent, regional coordination layer agent and equipment unit layer agent;

[0018] The distribution network leading layer is dominated by the distribution network, and controls the entire distributed multi-layer energy control architecture through the distribution network leading layer agent, formulates incentive signals according to the scheduling optimization goals, and communicates with the regional coordination layer agent to achieve cross-regional collaborative scheduling;

[0019] The regional coordination layer is used for operation scheduling and control of each distributed energy device in each region. Each region is provided with a regional coordination layer agent, which communicates with the device unit layer agent and performs local optimization in combination with the energy consumption demand in the region and the scheduling optimization target.

[0020] The device unit layer includes a variety of distributed energy devices, each distributed energy device is provided with a device unit layer agent, and the device unit layer agent communicates with the regional coordination layer agent to obtain scheduling instructions to perform energy scheduling for each distributed energy device.

[0021] Furthermore, the steps of making intelligent energy dispatching decisions include:

[0022] According to the dispatch optimization target, a dispatch model of the distribution network is constructed, wherein the dispatch model of the distribution network aims to minimize the operating cost, and the expression is:

[0023]

[0024] In the formula, E DSO represents the operating cost, P t grid It represents the amount of electricity purchased by the distribution system operator from the transmission grid; It represents the price at which the distribution system operator purchases electricity from the transmission grid; They represent the amount of electricity sold and purchased by the distribution system operator to microgrid i respectively; They represent the electricity sales and purchase prices from the distribution system operator to the microgrid i, respectively; N is the number of MGs connected to the distribution network; is the branch power loss; M is the total number of branches;

[0025] Acquire expert knowledge from expert knowledge base;

[0026] Based on the preprocessed operating data, the expert knowledge provides scheduling guidance, the scheduling model is solved in combination with reinforcement learning, and energy scheduling decisions are made according to the solution results.

[0027] Furthermore, the reinforcement learning adopts a deep deterministic policy gradient algorithm.

[0028] Furthermore, the architecture of the deep deterministic policy gradient algorithm adopts an actor-critic dual neural network architecture, and the training steps of the actor-critic dual neural network architecture include:

[0029] 1) Initialization: Initialize the parameters θ of the actor network Q , the parameters of the critic network θ π, parameters θQ' of the target actor network, parameters θ of the target critic network π ' and the experience replay buffer;

[0030] 2) Exploration and interaction: The actor network selects actions based on the current strategy, the agent executes the selected actions and interacts with the environment to obtain the current state, reward, and next state;

[0031] 3) Experience replay: store the current state, action, reward and next state obtained from the interaction into the experience replay buffer;

[0032] 4) Critic network update: A batch of experiences are randomly extracted from the experience playback buffer. The critic network uses the extracted experiences to calculate the loss function of the critic network and updates the parameters θ of the critic network parameters by the gradient descent method. Q ;

[0033] 5) Actor network update: Calculate the policy gradient of the critic network based on the output of the critic network, and update the parameters θ of the actor network parameters by the gradient ascent method π , where the calculation expression of the policy gradient of the critic network is:

[0034]

[0035] In the formula, is the policy gradient of the critic network objective function, is the actor network strategy, are the Q-function value of the critic network and the gradient of the actor network, n is the number of agents, q is the number of sample experiences, and s i,t is the state of the ith agent at time t;

[0036] 6) Target network update: Regularly copy the parameters of the actor network and critic network to the corresponding actor network and target critic network respectively;

[0037] 7) Strategy evaluation and improvement: Define an evaluation function to calculate the expected value of the Q-function to measure the strategy effect of the agent. The expression of the evaluation function is:

[0038]

[0039] In the formula, J(π) is the evaluation function, which means that the state s obeys the probability distribution p πWhen the agent executes actions according to strategy π, the expected value of the Q-function Q(s,π(s)) is obtained. Q(s,π(s)) is the Q-function value obtained by the agent executing actions according to strategy π. π is the probability distribution function;

[0040] 8) Iterative execution: Repeat steps 2)-7) until the iteration stop condition is met.

[0041] Furthermore, the expression of the reward update is:

[0042]

[0043] In the formula, r i ' ,t is the target reward of the ith agent at time t, r i,t is the immediate reward of the ith agent at time t, γ is the discount factor used to weigh the relationship between current rewards and future rewards, is the target critic network Q-function value, is the target actor network strategy, err is the error between the predicted Q value and the target Q value, a i,t is the action of the ith agent at time t, and q is the number of sample experiences.

[0044] Furthermore, the method further includes the step of optimizing the dispatch model by using a closed-loop feedback mechanism to adjust the intelligent energy dispatch decision, including the following steps:

[0045] Dynamically compare the actual dispatch execution results of the distribution network energy dispatch decision with the expected dispatch plan to identify execution deviations;

[0046] Corrections are made based on the execution deviations, and the scheduling model is adaptively optimized to dynamically adjust energy scheduling decisions.

[0047] The present invention also provides a distribution network intelligent decision-making system based on knowledge embedding and multi-agent system, comprising:

[0048] Data acquisition module: used to collect the operating data of each distributed energy device in the distribution network in real time and perform preprocessing;

[0049] Control architecture building module: used to divide the distribution network into layers and build a distributed multi-layer energy control architecture based on a multi-agent system;

[0050] Scheduling decision module: It is used to make intelligent energy scheduling decisions based on the pre-processed operation data and the distributed multi-layer energy control architecture based on the multi-agent system, using a method combining expert knowledge embedding and reinforcement learning to complete the decision-making process.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The present invention constructs a distributed multi-layer energy control architecture based on a multi-agent system according to the distribution network, and combines expert knowledge embedding and reinforcement learning technology to dispatch the distributed energy resources of the distribution network. The present invention does not need to accurately model each distributed energy device, and can realize intelligent decision-making and adaptive scheduling in a complex environment, ensuring that clean energy is absorbed to the maximum extent while ensuring safe operation, and optimizing the economy and energy utilization efficiency of the power system.

[0053] (2) The present invention is based on a distributed multi-layer energy control architecture of a multi-agent system, which realizes refined management and coordinated optimization of energy resources at different levels and improves the flexibility and response speed of the system.

[0054] (3) The present invention embeds expert domain knowledge into dispatching decisions, which enhances the intelligence and interpretability of the distribution network, ensuring that decisions in complex scenarios are more reasonable and reliable. By combining reinforcement learning algorithms, it can adaptively optimize dispatching strategies, effectively respond to dynamic changes and uncertainties in energy systems, and achieve more efficient energy management and optimized control.

[0055] (4) The present invention sets up a closed-loop feedback mechanism to optimize the scheduling model, thereby continuously running experience to obtain the best scheduling path and strategy, thereby improving the overall efficiency and robustness of the scheduling model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0057] Figure 2 The distributed multi-layer energy control architecture based on the multi-agent system of the present invention;

[0058] Figure 3 It is the DDPG training framework of the present invention. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] Example 1

[0061] This embodiment provides a distribution network intelligent decision-making method based on knowledge embedding and multi-agent system. The method first obtains the real-time operation data of distributed energy resources through a distributed sensor network. The data includes parameters such as voltage, current, and power output of photovoltaic power generation, wind power generation, energy storage equipment, and electric vehicles, and standardizes the data. Then, based on the distributed multi-layer energy control architecture of the multi-agent system, the power grid is divided into a distribution network leading layer, a regional coordination layer, and an equipment unit layer. Each layer of agents is responsible for corresponding scheduling tasks to ensure the stability and flexibility of the system. Next, expert knowledge is combined with the deep deterministic policy gradient (DDPG) algorithm to make intelligent scheduling decisions, and adaptive scheduling strategies are generated through reinforcement learning to dynamically optimize energy distribution. Finally, the scheduling execution results are monitored and adjusted in real time through a closed-loop feedback mechanism, the difference between actual execution and expected results is analyzed, and the scheduling model parameters are optimized. Specifically, Figure 1 As shown, the method comprises the following steps:

[0062] S1. Data collection and preprocessing.

[0063] The present invention first collects the operating data of distributed energy resources in real time through a distributed sensor network. The collected data covers the key operating parameters of various distributed energy resources such as photovoltaic power generation, wind power generation, energy storage system, electric vehicles, etc., including voltage, current, power output, load change, charging status, etc. During the data collection process, the sensor transmits real-time data to the central database of the system through the Internet of Things device. In order to ensure the consistency and availability of the data, the present invention performs the following standardized processing on data from different sources, which provides a basis for subsequent optimization decisions.

[0064]

[0065] S2. Construct a distributed multi-layer energy control architecture based on a multi-agent system.

[0066] In the present invention, the power grid is divided into three levels from top to bottom: the distribution network leading layer, the regional coordination layer and the equipment unit layer, and distributed multi-level energy control is realized through a multi-agent system, such as Figure 2 In this architecture, each level is controlled by a corresponding agent, and the energy allocation and scheduling optimization of the entire system are achieved through coordinated optimization, ensuring the effective use of resources and stable operation of the system.

[0067] (1) Distribution network leading layer: With the distribution network as the leading layer, it realizes cross-regional coordinated dispatch by communicating with the agents of each regional coordination layer. The distribution network leading layer formulates incentive signals according to the optimization goals of the overall system and transmits them to the equipment units through the agent layers at all levels to ensure the stable and safe operation of the entire system. Its core function is to realize the global dispatch and management of multiple virtual power plants and their distributed resources.

[0068] The minimization of the operating cost of the distribution network is taken as the optimization goal, and a dispatching model of the distribution network is established to dispatch the energy of each distributed energy device. The dispatching model is:

[0069]

[0070] In the formula, E DSO represents the operating cost, P t grid It represents the amount of electricity purchased by the distribution system operator from the transmission grid; It represents the price at which the distribution system operator purchases electricity from the transmission grid; They represent the amount of electricity sold and purchased by the distribution system operator to microgrid i respectively; They represent the electricity sales and purchase prices from the distribution system operator to the microgrid i, respectively; N is the number of MGs connected to the distribution network; is the branch power loss; M is the total number of branches.

[0071] (2) Regional coordination layer: responsible for the operation scheduling and control of equipment in the region. The agents at this level communicate with the agents at the device unit layer to obtain real-time equipment status data, and perform local optimization based on the energy consumption demand in the region and the overall system goals. The regional coordination layer can respond to the incentive signals sent by the upper-level agents to achieve autonomous optimization of various energy equipment in the region and ensure energy balance in the local area.

[0072] (3) Equipment unit layer: controls each distributed energy resource unit, such as photovoltaic power generation, energy storage devices, wind power, flexible loads, etc. The equipment unit layer agent obtains dispatch instructions through communication with the regional coordination layer, and adjusts the power output and operating status of the equipment in real time. The equipment unit layer is not only responsible for the safe operation of the equipment, but also maximizes the consumption of clean energy during the regulation process and promotes the efficient use of energy.

[0073] Through this hierarchical control structure, collaborative optimization is achieved between each level, which effectively improves the system's response speed and scheduling flexibility, reduces energy waste within the system, and maximizes the efficient consumption of clean energy and economical operation of the system.

[0074] S3. Make intelligent decisions based on expert knowledge embedding and reinforcement learning.

[0075] The present invention realizes intelligent scheduling decision of distributed energy resources by combining expert knowledge embedding with Deep Deterministic Policy Gradient (DDPG) reinforcement learning algorithm. This step not only utilizes the domain experience of experts, but also continuously optimizes the scheduling strategy in a complex dynamic environment through reinforcement learning algorithm, ensuring that the system can make the best decision under different load and uncertainty scenarios.

[0076] The expert knowledge base refers to the dispatching experience and decision-making rules for various typical scenarios, including historical dispatching experience, load characteristics under different meteorological conditions, equipment operation constraints, and strategies for dealing with emergencies. By embedding this expert knowledge into the intelligent decision-making system, it can provide guidance in the reinforcement learning process, so that the dispatching decision is not only data-driven, but also enhanced by the actual experience of experts. For example, during a hot summer peak electricity consumption period, historical experience shows that the air conditioning load in the area has increased sharply, resulting in a sharp increase in electricity demand. Photovoltaic power generation provides sufficient electricity during the day, but at night, due to insufficient photovoltaic power generation capacity, the power supply is tight. Based on the historical electricity consumption during similar peak electricity consumption periods, the expert knowledge base provides dispatching guidance: during the day, energy storage equipment should be charged first to prepare for the evening peak, and gradually discharge according to load demand in the evening, while appropriately adjusting the charging time of dispatchable loads such as air conditioners and electric vehicles to reduce power peaks. By embedding this dispatching strategy into the intelligent decision-making system, the reinforcement learning process will not only generate dispatching strategies based on real-time data, but also refer to the empirical knowledge provided by experts to ensure that energy storage equipment can reasonably allocate energy in the case of a sharp increase in load, and the dispatch of the power grid is more forward-looking and safe.

[0077] Reinforcement learning uses the DDPG algorithm, such as Figure 3 As shown in the figure, the DDPG algorithm uses an actor-critic structure, uses a random strategy to explore the actions of the agent, and uses a deterministic gradient strategy to update the algorithm. Among them, the actor and the critic are two deep neural networks (DNN). The critic network is also called the Q network, which can obtain the global information and global actions of the system. The actor network is also called the π network, which only takes actions based on the local environment. The two networks are trained separately to minimize the loss function and realize network updates. To solve the problem of unstable updates, the algorithm will create a backup network for them, namely the target network.

[0078] In the critic network, the parameter vector θ Q Used to estimate the value of the Q-function Q * (s,a|θ Q), Q-function, also known as state-action value function, is used to evaluate the decision quality of the participants and provide gradient direction information for the algorithm. In the actor network, the parameter vector θ π To estimate the policy π * (s|θ π ), which maps the state to each agent, allowing it to make decisions and output continuous actions. The evaluation function J(π) is defined as the expected value of the Q-function to measure the effect of the agent strategy:

[0079]

[0080] Where: p π is the probability distribution function; Q(s,π(s)) represents the Q-function value obtained by the agent when it executes actions according to strategy π; J(π) means that the state s obeys the probability distribution p π When , the expected value of the Q-function Q(s,π(s)) obtained by executing actions according to the strategy π.

[0081] The agent accumulates experience through interaction with the environment and updates the network, which is essentially a strategy update. The actor network selects an action through an initial random strategy, and then the critic network evaluates the action and outputs the corresponding Q-function value, guiding the actor to update the strategy parameters according to the maximum evaluation value. Repeating the above steps gradually increases the cumulative reward obtained by the strategy. The above update process is shown in the following formula:

[0082]

[0083] Where: The following formula represents the reward update; r i ' ,t For the target reward, is the target critic network Q-function value, is the target actor network strategy. The following formula represents the critic network error, and the network update is achieved by minimizing the error; is the actual critic network Q-function value, q is the sample experience number. The following formula represents the network policy gradient to determine the update direction of the algorithm; are the gradients of the critic network and the actor network respectively.

[0084] Specifically, the training steps of the actor-critic dual neural network architecture include:

[0085] 1) Initialization: Initialize the parameters θ of the actor network Q , the parameters of the critic network θ π , parameters θQ' of the target actor network, parameters θ of the target critic network π' and the experience replay buffer;

[0086] 2) Exploration and interaction: The actor network selects actions based on the current strategy, the agent executes the selected actions and interacts with the environment to obtain the current state, reward, and next state;

[0087] 3) Experience replay: store the current state, action, reward and next state obtained from the interaction into the experience replay buffer;

[0088] 4) Critic network update: A batch of experiences are randomly extracted from the experience playback buffer. The critic network uses the extracted experiences to calculate the loss function of the critic network and updates the parameters θ of the critic network parameters by the gradient descent method. Q ;

[0089] 5) Actor network update: Calculate the policy gradient of the critic network based on the output of the critic network, and update the parameters θ of the actor network parameters by the gradient ascent method π , where the policy gradient of the critic network is calculated according to formula (5);

[0090] 6) Target network update: Regularly copy the parameters of the actor network and critic network to the corresponding actor network and target critic network respectively;

[0091] 7) Strategy evaluation and improvement: Define the evaluation function to calculate the expected value of the Q-function to measure the strategy effect of the agent. The evaluation function is as shown in formula (2);

[0092] 8) Iterative execution: Repeat steps 2)-7) until the iteration stop condition is met.

[0093] In the distributed multi-layer energy control architecture, the intelligent decisions obtained by reinforcement learning are executed at different levels to ensure that the system can operate efficiently and safely. First, the distribution leading layer communicates with the agents of the regional coordination layer, receives and integrates the status information from each region, and generates a scheduling strategy based on the global optimization goal. The main goal of this layer is to coordinate the scheduling of multiple virtual power plants and distributed resources to ensure the stability of the system. The intelligent scheduling strategy generated by the reinforcement learning algorithm at this layer will be transmitted to the regional level through the incentive mechanism to promote the coordination of each region and the reasonable allocation of resources. Secondly, the agent of the regional coordination layer is responsible for executing these intelligent decisions and scheduling them in combination with the actual status of the equipment in the region. The function of the intelligent agent at this layer is equivalent to the regional agent. The actions generated by the intelligent agent will be fine-tuned according to the regional load situation, equipment status and other information to achieve the overall energy balance of the system. For example, the intelligent agent can optimize the charging and discharging timing of the energy storage equipment in the region through reinforcement learning strategies, or coordinate the use of renewable energy to ensure the efficient use of energy. At the equipment unit layer, the intelligent agent corresponds to the agent of each distributed energy resource, such as photovoltaic power generation, energy storage device or wind power equipment. At this layer, the agent performs specific operations based on the real-time status of the equipment, such as adjusting the output of photovoltaic power generation, controlling the charging and discharging process of energy storage equipment, etc. The agent at the equipment unit layer directly executes the scheduling instructions from the upper layer, ensuring rapid response and safe operation of the equipment in actual operation.

[0094] Through this layered architecture design, the intelligent decision-making of reinforcement learning can not only achieve system optimization at the global level, but also be implemented layer by layer to the regional and device levels, ensuring efficient scheduling and resource optimization from the global to the local level.

[0095] S4. Closed-loop feedback mechanism optimizes the scheduling model.

[0096] In the present invention, the closed-loop feedback mechanism plays a key role in scheduling optimization. By real-time monitoring of the execution of the scheduling plan, the actual execution results are compared and analyzed with the expected scheduling plan, thereby continuously optimizing the scheduling model. The closed-loop feedback mechanism ensures that the system can adapt to changes under different scenarios and load conditions, and continuously improves the accuracy and efficiency of scheduling decisions based on the execution effect. Specifically, the feedback mechanism includes the following steps:

[0097] S4.1: Real-time monitoring and data feedback. During the execution of the dispatch plan, the system tracks the operating status of distributed energy resources in real time through the sensor network and monitoring module, including the charging and discharging of power generation and energy storage equipment, load changes, equipment operating conditions, etc. All real-time data will be transmitted to the central control system through the IoT device and dynamically compared with the previously generated dispatch plan to identify possible deviations during the execution process.

[0098] S4.2: Feedback data analysis and error correction. The system can identify anomalies or inconsistencies in scheduling by analyzing the deviation between the actual execution results and the expected plan. For example, if the actual output power of a distributed energy device is lower than expected, or the discharge time of the energy storage device does not meet the expected effect, the system will capture these anomalies through the feedback mechanism. Subsequently, the system performs error correction based on the deviation value and adjusts the parameters in the scheduling strategy to ensure that the next scheduling cycle can make adjustments and optimizations to the problems in the previous cycle.

[0099] S4.3: Dynamically adjust the dispatch model. Based on the feedback data, the dispatch model in the system will be adaptively optimized. Through the reinforcement learning algorithm (DDPG) combined with expert knowledge, the system can dynamically adjust the dispatch strategy of various distributed energy sources. For example, when the system recognizes a sudden change in electricity demand, the dispatch model will immediately adjust the charging and discharging strategy of the energy storage equipment and redistribute the load according to actual demand to ensure the stable operation of the entire power grid.

[0100] S4.4: Long-term optimization and strategy upgrade. In addition to short-term feedback tuning, the system will also upgrade strategies based on long-term scheduling data. Through closed-loop feedback of multiple scheduling cycles, the system continuously accumulates operating experience and gradually optimizes the scheduling rules in the expert knowledge base. In long-term operation, the system can summarize the best scheduling path and strategy through big data analysis, thereby improving the overall efficiency and robustness of the scheduling model.

[0101] This embodiment applies the above method in an actual scenario, which mainly includes the following steps:

[0102] Step 1): Obtain real-time operating data of distributed energy resources, including voltage, current, power output, charge and discharge status, load demand and other parameters of photovoltaic power generation, wind power generation and energy storage equipment. Through the distributed sensor network, the system can monitor these key data in real time. All collected data is standardized through the data preprocessing module to ensure that different types of energy resources can be processed uniformly and provide reliable data support for subsequent scheduling analysis;

[0103] Step 2): Distributed multi-layer energy control architecture based on multi-agent system. The power grid system is divided into three levels: distribution network dominant layer, regional coordination layer and equipment unit layer. Agents at each level are responsible for corresponding scheduling tasks. The distribution network dominant layer agent formulates a global scheduling strategy through collaboration with the regional coordination layers to ensure the overall stability and efficient operation of the system. The regional coordination layer agent integrates the operating status of each distributed energy source in the region and generates a local optimization plan. The equipment unit layer agent controls specific energy resource equipment to ensure that it operates within a safe range and maximizes the consumption of clean energy;

[0104] Step 3): Make intelligent scheduling decisions based on the expert knowledge embedding and reinforcement learning (DDPG) algorithm. Under the guidance of the expert knowledge base, the system combines historical data and real-time data to optimize the energy scheduling strategy through the DDPG algorithm. The DDPG algorithm generates the optimal scheduling action based on the current system status, such as adjusting the charging and discharging time of energy storage equipment or adjusting load distribution, to ensure that the system can adaptively respond to complex and changing demands and uncertainties. The expert knowledge base provides scheduling experience in special scenarios, so that the system can accelerate convergence in the process of reinforcement learning and make reasonable scheduling decisions;

[0105] Step 4): The execution effect of the scheduling plan is monitored in real time through a closed-loop feedback mechanism. The system compares the actual execution results with the expected scheduling plan, identifies possible deviations, analyzes the causes, and corrects the scheduling model through the feedback mechanism. The system will optimize the parameters of the scheduling model based on the feedback results to ensure that the scheduling strategy is more accurate and efficient in the next round of execution.

[0106] Example 2

[0107] This embodiment provides a distribution network intelligent decision-making system based on knowledge embedding and multi-agent system, including:

[0108] Data acquisition module: used to collect the operating data of each distributed energy device in the distribution network in real time and perform preprocessing;

[0109] Control architecture building module: used to divide the distribution network into layers and build a distributed multi-layer energy control architecture based on a multi-agent system;

[0110] Scheduling decision module: It is used to make intelligent energy scheduling decisions based on the pre-processed operation data and the distributed multi-layer energy control architecture based on the multi-agent system, using a method combining expert knowledge embedding and reinforcement learning to complete the decision-making process.

[0111] The rest is the same as in Example 1.

[0112] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0113] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A distribution network intelligent decision-making method based on knowledge embedding and multi-agent system, characterized in that: The following steps are involved: Collect the operating data of each distributed energy device in the distribution network in real time and perform pre-processing; Divide the distribution network into layers and build a distributed multi-layer energy control architecture based on a multi-agent system; Based on the pre-processed operation data and the distributed multi-layer energy control architecture based on the multi-agent system, a method combining expert knowledge embedding and reinforcement learning is adopted to make intelligent energy scheduling decisions and complete the decision-making process.

2. According to claim 1, a distribution network intelligent decision-making method based on knowledge embedding and multi-agent system is characterized in that: The operating data of each distributed energy device includes the voltage, current and power output of photovoltaic power generation, wind power generation, energy storage equipment, electric vehicles and power networks.

3. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 1 is characterized in that: The preprocessing operation includes standardizing the operation data of each distributed energy device, and the expression of the standardization process is: Where, X norm is the standardized data, X is the original acquired data, and X min , X max are the minimum and maximum values ​​in the data, respectively.

4. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 1 is characterized in that: The steps of constructing a distributed multi-layer energy control architecture based on a multi-agent system include: The distribution network is divided into three levels from top to bottom, namely the distribution network dominant layer, regional coordination layer and equipment unit layer, and a distributed multi-layer energy control architecture based on a multi-agent system is constructed, including the distribution network dominant layer agent, regional coordination layer agent and equipment unit layer agent; The distribution network leading layer is dominated by the distribution network, and controls the entire distributed multi-layer energy control architecture through the distribution network leading layer agent, formulates incentive signals according to the scheduling optimization goals, and communicates with the regional coordination layer agent to achieve cross-regional collaborative scheduling; The regional coordination layer is used for operation scheduling and control of each distributed energy device in each region. Each region is provided with a regional coordination layer agent, which communicates with the device unit layer agent and performs local optimization in combination with the energy consumption demand in the region and the scheduling optimization target. The device unit layer includes a variety of distributed energy devices, each distributed energy device is provided with a device unit layer agent, and the device unit layer agent communicates with the regional coordination layer agent to obtain scheduling instructions to perform energy scheduling for each distributed energy device.

5. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 4 is characterized in that: The steps of making intelligent energy dispatching decisions include: According to the dispatch optimization target, a dispatch model of the distribution network is constructed, wherein the dispatch model of the distribution network aims to minimize the operating cost, and the expression is: In the formula, E DSO represents the operating cost, P t grid It represents the amount of electricity purchased by the distribution system operator from the transmission grid; It represents the price at which the distribution system operator purchases electricity from the transmission grid; They represent the amount of electricity sold and purchased by the distribution system operator to microgrid i respectively; They represent the electricity sales and purchase prices from the distribution system operator to the microgrid i, respectively; N is the number of MGs connected to the distribution network; is the branch power loss; M is the total number of branches; Acquire expert knowledge from expert knowledge base; Based on the preprocessed operating data, the expert knowledge provides scheduling guidance, the scheduling model is solved in combination with reinforcement learning, and energy scheduling decisions are made according to the solution results.

6. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 5 is characterized in that: The reinforcement learning adopts a deep deterministic policy gradient algorithm.

7. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 6 is characterized in that: The architecture of the deep deterministic policy gradient algorithm adopts an actor-critic dual neural network architecture, and the training steps of the actor-critic dual neural network architecture include: 1) Initialization: Initialize the parameters θ of the actor network Q , the parameters of the critic network θ π , parameters θQ' of the target actor network, parameters θ of the target critic network π ' and the experience replay buffer; 2) Exploration and interaction: The actor network selects actions based on the current strategy, the agent executes the selected actions and interacts with the environment to obtain the current state, reward, and next state; 3) Experience replay: store the current state, action, reward and next state obtained from the interaction into the experience replay buffer; 4) Critic network update: A batch of experiences are randomly extracted from the experience playback buffer. The critic network uses the extracted experiences to calculate the loss function of the critic network and updates the parameters θ of the critic network parameters by the gradient descent method. Q ; 5) Actor network update: Calculate the policy gradient of the critic network based on the output of the critic network, and update the parameters θ of the actor network parameters by the gradient ascent method π , where the calculation expression of the policy gradient of the critic network is: In the formula, is the policy gradient of the critic network objective function, is the actor network strategy, are the Q-function value of the critic network and the gradient of the actor network, n is the number of agents, q is the number of sample experiences, and s i,t is the state of the ith agent at time t; 6) Target network update: Regularly copy the parameters of the actor network and critic network to the corresponding actor network and target critic network respectively; 7) Strategy evaluation and improvement: Define an evaluation function to calculate the expected value of the Q-function to measure the strategy effect of the agent. The expression of the evaluation function is: In the formula, J(π) is the evaluation function, which means that the state s obeys the probability distribution p π When the agent executes actions according to strategy π, the expected value of the Q-function Q(s,π(s)) is obtained. Q(s,π(s)) is the Q-function value obtained by the agent executing actions according to strategy π. π is the probability distribution function; 8) Iterative execution: Repeat steps 2)-7) until the iteration stop condition is met.

8. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 7 is characterized in that: The expression of the reward update is: In the formula, r i ' ,t is the target reward of the ith agent at time t, r i,t is the immediate reward of the ith agent at time t, γ is the discount factor used to weigh the relationship between current rewards and future rewards, is the target critic network Q-function value, is the target actor network strategy, err is the error between the predicted Q value and the target Q value, a i,t is the action of the ith agent at time t, and q is the number of sample experiences.

9. The distribution network intelligent decision-making method based on knowledge embedding and multi-agent system according to claim 1 is characterized in that: It also includes the steps of optimizing the dispatch model using a closed-loop feedback mechanism to adjust the intelligent energy dispatch decision, including the following steps: Dynamically compare the actual dispatch execution results of the distribution network energy dispatch decision with the expected dispatch plan to identify execution deviations; Corrections are made based on the execution deviations, and the scheduling model is adaptively optimized to dynamically adjust energy scheduling decisions.

10. A distribution network intelligent decision-making system based on knowledge embedding and multi-agent system, characterized in that: include: Data acquisition module: used to collect the operating data of each distributed energy device in the distribution network in real time and perform preprocessing; Control architecture building module: used to divide the distribution network into layers and build a distributed multi-layer energy control architecture based on a multi-agent system; Scheduling decision module: It is used to make intelligent energy scheduling decisions based on the pre-processed operation data and the distributed multi-layer energy control architecture based on the multi-agent system, using a method combining expert knowledge embedding and reinforcement learning to complete the decision-making process.

Images