A method for scheduling multiple agents of a heat and power pipe network system based on a fusion communication learning mechanism

Abstract

The application provides a kind of thermal power pipe network system multi-agent scheduling method of fusion communication learning mechanism, solved the technical problem that current thermal power pipe network multi-agent reinforcement learning scheduling method lacks the differentiated information interaction mechanism between agents, so that the communication process is easy to introduce redundant information, interfere with agent scheduling decision.It includes constructing a thermal power pipe network system and setting constraints to obtain a thermal power pipe network simulation environment;Based on the thermal power pipe network simulation environment, a multi-agent Markov game framework coupled with a communication learning mechanism is built to determine the observations and actions of each agent;The MAPPO algorithm with adaptive clipping and multi-dimensional exploration mechanism is used to collaboratively train the observations and actions of each agent, and the Actor-Critic network of each agent after training is obtained, and the network is used to perform optimization scheduling for each agent.The application can be widely applied to the field of power system resource scheduling technology.

Description

Technical Field

[0001] This application belongs to the field of power system resource scheduling technology, and more specifically, it relates to a multi-agent scheduling method for a thermal power pipeline system that integrates a communication learning mechanism. Background Technology

[0002] Thermal power (CHP) networks transmit heat generated by heat sources to heat exchange stations via a primary heating network. After heat exchange, the heat is supplied to end users via a secondary network. These systems often integrate various flexible devices such as batteries, heat pumps, and thermal storage devices. Such systems are characterized by strong equipment heterogeneity, tight electrothermal coupling, and complex operational constraints. Their optimal scheduling problem is essentially a high-dimensional, stochastic, and multi-constraint sequential decision-making problem. Traditional model-based control methods, such as model predictive control and mixed-integer programming, rely on accurate physical models and load forecasts, making it difficult to cope with the strong uncertainties in renewable energy output and user demand. In recent years, multi-agent reinforcement learning methods have attracted widespread attention due to their ability to learn model-free control strategies through interaction with the environment, providing a new approach to adaptive scheduling of complex energy systems.

[0003] In existing multi-agent reinforcement learning scheduling schemes, each agent typically relies solely on its own local observations for decision-making, making it difficult to acquire the crucial information needed for global coordination. Although some studies have introduced centralized training and distributed execution frameworks to improve policy coordination by sharing global information during the training phase, effective online information interaction mechanisms between agents are still lacking in the actual execution phase. For systems like thermal power networks, which are widely distributed in space and tightly coupled in time, different devices such as heating plants, heat exchange stations, and thermal storage units are physically interconnected. Decision-making errors in a single device can be transmitted to the entire system through the thermal network, leading to insufficient heating or decreased energy efficiency. Therefore, how to achieve efficient information sharing among agents during distributed execution has become a key technical bottleneck for improving the overall scheduling performance of the system.

[0004] Existing technologies attempt to introduce communication mechanisms into multi-agent systems, enabling agents to proactively send messages and receive information from other agents at each decision step. However, such communication methods often employ broadcast or fixed message fusion rules, failing to fully consider the heterogeneity of agent roles and the differences in information needs among different devices. Using a uniform communication vector or simple average pooling fails to provide each agent with personalized information most relevant to its local decision-making task, and may even introduce redundant information that interferes with decision-making. Therefore, there is an urgent need to provide a new scheduling method for thermal power pipeline networks to address these problems. Summary of the Invention

[0005] The purpose of this application is to provide a multi-agent scheduling method for a thermoelectric pipeline network system that integrates communication learning mechanisms, so as to solve the technical problem that the existing multi-agent reinforcement learning scheduling method for thermoelectric pipeline networks lacks a differentiated information interaction mechanism between agents, which makes it easy to introduce redundant information in the communication process and interfere with the agent scheduling decision.

[0006] To achieve the above objectives, a first aspect of this application provides a multi-agent scheduling method for a thermal power pipeline network system that integrates a communication learning mechanism, comprising the following steps: A thermal power pipe network system is constructed and constraints are set to obtain a thermal power pipe network simulation environment; Based on the thermal power pipe network simulation environment, a multi-agent Markov game framework with coupled communication learning mechanism is built to determine the observation and actions of each agent. The MAPPO algorithm, which integrates adaptive pruning and multidimensional exploration mechanisms, is used to collaboratively train the observations and actions of each agent, resulting in the trained Actor-Critic network for each agent. This network is then used to perform optimized scheduling for each agent.

[0007] Preferably, the process of building a multi-agent Markov game framework with a coupled communication learning mechanism includes: based on a thermal power pipe network simulation environment, setting up several agents, defining global common information shared by all agents, configuring communication observations for each agent according to the communication learning mechanism, and splicing together the global common information, exclusive observations, and local observations of each agent to obtain the observations of each agent; Based on the control tasks of each agent, physical control quantities are determined, and the observations and actions of each agent are formed by combining communication actions. A globally shared reward function is constructed, and a multi-agent Markov game framework with coupled communication learning mechanism is built.

[0008] Preferably, the collaborative training process includes: scaling the dot product attention of the communication actions of each agent to obtain an attention head, performing residual connections and layer normalization, generating global interaction information for each agent, and splicing the global interaction information, global common information and the device observations of each agent to obtain the observations at the next time step. Experience tuples are constructed based on the observations of the next time step, the observations and actions of each agent at the current time step, and the reward value generated by the action on the actual environment at the previous time step. These tuples are then stored in the experience replay buffer of each agent. An advantage function is constructed based on the experience tuples, and the advantage function value is calculated. A loss function is constructed by dynamically and adaptively pruning based on the absolute value of the advantage function value, and the policy network is updated. Multiple error indices are set, and an overall error loss function is constructed by combining an improved radar chart, which updates the value network. Combined with a two-stage composite policy, the entropy coefficient is dynamically adjusted through multi-dimensional exploration, and each agent is trained in multiple rounds of collaborative training to obtain the trained Actor-Critic network of each agent.

[0009] Preferably, the process of performing optimized scheduling includes: the agent integrates global public information, its own local observations and communication observations, and inputs them into the Actor-Critic network of each agent after training, and outputs physical control actions and communication actions at each time step; during the model training phase, a Gaussian distribution random sampling strategy is used to explore possible actions; during the actual scheduling operation, the mean of the Gaussian distribution is taken as the deterministic action output to complete the multi-agent collaborative optimized scheduling.

[0010] Preferably, the formula for scaling dot product attention is: ; In the formula, This is the attention weight matrix. For querying the matrix, The key matrix, Scaling factor For value matrices, For the number of attention heads, This is the matrix transpose symbol. As key, It is a normalized exponential function.

[0011] Preferably, the formula for dynamic adaptive pruning based on the absolute value of the dominant function is:

[0012] In the formula, This is a dynamic pruning threshold function. For dynamic, For time, For the clipping function, It is the original clipping constant. For adaptive coefficients, Let be the mean of the dominance function. This is the lower limit of the clipping threshold. This represents the upper limit of the clipping threshold.

[0013] Preferably, the two-stage composite strategy includes a preheating stage, a cosine annealing stage, and an adaptive boosting mechanism; During the warm-up phase, the entropy coefficient is maximized, forcing the agent to fully explore various actions; During the cosine annealing stage, the entropy coefficient is smoothly decayed, reducing the degree of exploration. An adaptive boosting mechanism increases the entropy coefficient when performance stagnation is observed, helping the agent escape local optima; as performance continues to improve, the entropy coefficient continues to be smoothly decayed until it returns to the baseline.

[0014] Preferably, the formula for the change of the entropy coefficient is: ; In the formula, To train steps The entropy coefficient under the condition, The minimum entropy coefficient, The maximum entropy coefficient, This represents the total number of steps in the preheating phase. This represents the total number of training steps.

[0015] Preferably, the process of constructing the overall error loss function by combining the improved radar image includes: determining four evaluation quantification indicators, and mapping the four evaluation quantification indicators to the radar. Figure 4 Each coordinate axis dimension determines the angle of the sector area occupied by each optimized target in the radar image according to preset weight values. Based on the angle of the sector area occupied by each optimized target, the sector area and radar area are calculated. Figure 4 The total perimeter of the polygon, based on the sector area and radar. Figure 4 The total perimeter of the polygon is used to calculate the overall error loss function.

[0016] Preferably, the process of constructing a thermoelectric pipeline system includes: modeling the batteries, heat storage devices, heat pump stations, heat exchange stations, and heating pipelines in the thermoelectric pipeline system to obtain the thermoelectric pipeline system.

[0017] The beneficial effects of this application are as follows: This application provides a multi-agent scheduling method for a thermoelectric pipe network system that integrates a communication learning mechanism. It constructs a thermoelectric pipe network system and builds a thermoelectric pipe network simulation environment. Based on the thermoelectric pipe network simulation environment, a multi-agent Markov game framework with coupled communication learning mechanism is built to determine the observations and actions of each agent. Here, by constructing a communication learning mechanism coupled with a transformer, and using scaled dot product attention to perform self-attention transformation on the messages sent by each agent, each agent obtains personalized information containing only the most relevant information to its local decision-making task. This avoids information redundancy and decision interference caused by broadcast communication, significantly improving the collaborative decision-making efficiency among heterogeneous multi-agents.

[0018] Next, the MAPPO algorithm, which integrates adaptive pruning and multi-dimensional exploration mechanisms, is used to collaboratively train the observations and actions of each agent, resulting in the trained Actor-Critic network for each agent. This application introduces an adaptive pruning scheme based on the absolute advantage value into the policy network update of the MAPPO algorithm. When the absolute advantage value is large, the pruning range is appropriately widened to allow for larger policy adjustments; when the absolute advantage value is small, the pruning is tightened to avoid meaningless large updates, effectively balancing the policy update magnitude and training stability, and accelerating algorithm convergence.

[0019] Meanwhile, a two-stage composite exploration strategy was designed during the training process, which includes a warm-up stage, a cosine annealing stage, and an adaptive boosting mechanism. In the early stage of training, full exploration is forced, the exploration level is smoothly reduced in the middle stage, and the exploration coefficient is automatically increased when performance stagnation is detected. This effectively helps the algorithm escape local optima and improves the global optimality of the final strategy.

[0020] Furthermore, a multivariate loss function evaluation system was constructed, which includes mean absolute percentage error, mean absolute error, root mean square error, and decision coefficient. An improved radar chart method was introduced to standardize and visualize each error index, enabling the value network to learn the state value function more comprehensively, improving the accuracy of advantage estimation, and thus guiding the policy network to optimize in a better direction.

[0021] Finally, the trained Actor-Critic network for each agent is obtained, and the network is used to perform optimized scheduling on each agent to achieve optimized scheduling of agents. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of the overall process of a multi-agent scheduling method for a thermoelectric pipeline network system with a fusion communication learning mechanism provided in an embodiment of this application; Figure 2 An overall structural diagram of optimized scheduling of a thermal power pipeline network system provided in an embodiment of this application; Figure 3 A bidirectional stacked bar chart of electrical balance and thermal balance provided in an embodiment of this application; Figure 4 A comparison chart of reward function values ​​provided in an embodiment of this application. Detailed Implementation

[0024] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0025] Please see Figure 1-2 An embodiment of this application provides a multi-agent scheduling method for a thermoelectric pipeline network system that incorporates a fusion communication learning mechanism, comprising: S1: Construct a thermoelectric pipe network system and set constraints to obtain a thermoelectric pipe network simulation environment.

[0026] First, the charging and discharging power of the battery, the heat absorption and release power of the heat storage device, and the heat loss of the heat pump station, heat exchange station and heating network in the thermoelectric pipeline system are modeled to construct the thermoelectric pipeline system.

[0027] Specifically, the charging and discharging power of the battery is modeled using the following formula:

[0028] In the formula, , They represent the first , At the end of the period, the SOC (State of Charge) of the energy storage system. This indicates the charging and discharging efficiency of the energy storage system. and They represent the first The power of the time-limited energy storage system during charging and discharging processes. Indicates the capacity of the energy storage system. express Duration of the period.

[0029] Next, the heat absorption and release power of the heat storage device is modeled, and the formula is: ; In the formula, express The amount of heat stored in the thermal energy storage device at any given time is measured in MWh. express The heat storage capacity of the thermal storage device is constantly being monitored. The thermal storage efficiency of the thermal storage device is expressed as a percentage (%). , They represent The amount of heat stored and released at any given time is measured in MWh.

[0030] Then, the heat pump station is modeled, and the heat pump is in... Power consumed at all times Generates heat power The relationship between the two is determined by the energy efficiency ratio. The decision is made using the following formula: ; Subject to part Load rate at any time The effect is determined using the following nonlinear model, with the formula as follows: ; In the formula, The coefficient represents the energy efficiency ratio under rated operating conditions. Reflects the partial load characteristics. This is the partial load loss factor.

[0031] Furthermore, the heat exchange station is modeled. Plate heat exchangers are used as the main heat transfer structure in the heat exchange station. The heat exchange efficiency can be indirectly changed by altering the flow rates on the primary and secondary sides. The modeling formula is as follows: ; ; ; In the formula, The overall heat transfer coefficient is... The number of heat transfer units, For heat exchange efficiency, , These are constants related to structure and physical properties. , These represent the flow rates on the primary and secondary sides, respectively. For effective heat transfer area, This refers to the specific heat capacity of a heat transfer fluid that was circulated online. This refers to the specific heat capacity of the heat transfer fluid in the secondary network.

[0032] Finally, the heat loss of the heating network is modeled. In the heating network, the heat loss between the primary side pipes is significant; therefore, it is modeled using the following formula: ; ; ; In the formula, This indicates the heat loss on the primary water supply side of the network. This indicates the heat loss on the return water side of the primary network. This represents the heat loss correction factor. This represents the flow rate on the primary side. This refers to the specific heat capacity of a heat transfer fluid that was circulated online. It is the length of the pipe. It is the total thermal resistance. This is a network approaching the water supply temperature of the heating plant. To ensure the primary network is close to the return water temperature of the heat exchange station, The ambient temperature.

[0033] Furthermore, constraints are set for the thermal power network system. The constraints for the battery, heat storage tank, and heat pump station are constructed as follows: The charging and discharging power constraints of the battery are as follows: .in, The upper limit of battery charging power. This represents the current operating power of the battery. This is the upper limit of the battery's discharge power.

[0034] The energy storage state constraints of the thermal energy storage device are: .in, Let t be the heat capacity of the thermal storage device. This represents the minimum heat capacity of the thermal storage device. This represents the maximum heat capacity of the thermal storage device.

[0035] The ramp-up constraints for power output equipment, including heat pumps, batteries, and thermal storage devices, are as follows: .in, This represents the capacity value of the output equipment at time t; This represents the capacity value of the output equipment at time t-1; This indicates the maximum gradeability; j represents the equipment set, including heat pumps, batteries, and thermal storage devices.

[0036] S2: Based on the thermal power pipe network simulation environment, a multi-agent Markov game framework with coupled communication learning mechanism is built to determine the observations and actions of each agent.

[0037] The process of constructing a multi-agent Markov game framework with coupled transformer communication and learning mechanisms based on a thermoelectric pipe network simulation environment is as follows: In a simulation environment based on a thermal power network, four intelligent agents are set up: an electrical balance dispatcher, a heating station (with built-in heat pump clusters), a heat exchange station (with built-in heat transfer structures and flow valves), and a thermal storage unit. The state space of each intelligent agent is defined as follows: First, establish global public information for each agent to observe at every moment, obtaining state parameters including the time scale and the grid electricity price, wind turbine output, photovoltaic equipment output, electrical load, heat load, and external ambient temperature at each moment. The formula is: ; In the formula, For global public information, For time scale, For the grid electricity price at each moment, To provide power to the wind turbines at every moment. To contribute to photovoltaic equipment at every moment. For the electrical load at each moment, For the magnitude of the heat load at each moment, The external ambient temperature at each moment.

[0038] The communication learning mechanism combines the communication actions of each agent in the previous time step, and outputs the communication observations of each agent in the next time step after aggregation through a transformer. This application concatenates the personalized information returned by the transformer aggregator into the local observations of each agent, resulting in communication observations for the electrobalancing agent. Intelligent communication observation of heating stations Intelligent communication and observation of heat exchange stations Thermal storage intelligent agent communication observation .

[0039] The power balance scheduling agent can observe the state values ​​of electrical-related equipment at any given moment (its own local observation), including the grid output value at the previous moment, the power consumption value of the heat pump at the previous moment, the power loss value at the previous moment, and the battery charge value at the current moment. This is combined with global public information. and dedicated observation The observations obtained from the power balance scheduling agent are specifically represented as follows: ; In the formula, For observation of electric balance intelligent agents, The battery output power at time t-1 Let t be the battery capacity. The output power of the heat pump at time t-1 Let be the power wasted at time t-1. For communication observation of the electric balance intelligent agent.

[0040] The heating plant dispatching agent estimates the current thermal output power of its equipment by observing the electrical power of the heat pump equipment at the previous moment, the water temperature on the supply side at the previous moment, and the water temperature on the return side at the current moment. This results in its own local observations, which are then combined with global public information. and dedicated observation The observations obtained from the heating plant's dispatching agent are specifically represented as follows: ; In the formula, For intelligent monitoring of heating stations, For global public information, Let t be the return water temperature. Let t-1 be the water temperature on the supply side. The output power of the heat pump at time t-1 For intelligent agent communication observation of heating stations.

[0041] The heat exchange station scheduling agent adjusts its heat transfer efficiency by observing the supply and return water temperatures on both sides of the network at the current moment and the flow rates on both sides of the network at the previous moment. This allows it to obtain its own local observations and combine them with global public information. and dedicated observation The observations obtained from the heat exchange station scheduling agent are specifically represented as follows: ; In the formula, For intelligent monitoring of heat exchange stations, For a single network flow rate at time t-1, The secondary network fluid flow rate at time t-1 The water supply temperature at the heat exchange station side. This refers to the return water temperature of the secondary network. The return water temperature of the heat exchange station at time t-1. The secondary network water supply temperature at time t-1 For intelligent agent communication observation in heat exchange stations.

[0042] The thermal storage tank scheduling agent estimates the current thermal output by observing the heat storage and primary water supply temperature of the previous moment, obtaining its own local observations and combining them with global public information. and dedicated observation The observations obtained from the heat exchange station scheduling agent are specifically represented as follows: ; In the formula, For intelligent observation of thermal storage devices, For global public information, The water supply temperature at the primary heating station side at time t-1. Let be the heat capacity of the thermal storage device at time t-1. For intelligent agent communication observation of thermal storage tanks.

[0043] The action decisions of each agent are shown in the following formula: The power balance scheduling agent is responsible for scheduling the battery charging and discharging power at each moment, as shown in the formula: ; In the formula, For the action of the electrically balanced intelligent agent, Let be the battery power at time t. The communication action of the electric balance agent at time t.

[0044] The intelligent agent in the heating plant is responsible for scheduling the heat pump power, determining the electrical power required by the heat pump equipment at each moment, using the following formula: ; In the formula, For the heating station to take action, Let be the heat pump power at time t. The communication action of the heating station at time t.

[0045] The intelligent system of the heat exchange station changes the heat transfer efficiency by altering the heat transfer area of ​​the plate heat transfer structure; it also changes the overall liquid flow rate of the primary and secondary piping networks through flow valves, using the following formula: ; In the formula, For the operation of the heat exchange station, For heat exchange area, For a single network traffic event at time t, The secondary network throughput at time t is... The communication action of the heat exchange station at time t.

[0046] The thermal storage agent participates in system scheduling by deciding the amount of heat stored and released at each moment, as shown in the formula: ; In the formula, For the operation of the heat storage device, Let t be the heat capacity of the thermal storage device. The communication action of the thermal storage device at time t.

[0047] In a thermal power pipeline system, all agents share a global reward function, driving all devices to collaboratively optimize the global objective and guiding the agents towards optimization in one direction. The globally shared reward function can be set to a negative value of the overall system operating cost.

[0048] ; ; ; ; ; ; ; In the formula, For the overall operating cost of the system, This refers to the cost of purchasing electricity from an external power grid; It is the cost of equipment communication; This is a cost item for storage batteries; This is a cost item for the thermal storage unit; This is a cost item for heat pumps; This indicates the actual number of observations broadcast. The regularization coefficient; This represents the penalty cost for violating the constraints. It is a very large penalty cost constant term. This is the unit power cost coefficient for batteries. To charge the battery, This refers to the battery discharge power. This is the unit power cost coefficient for the thermal energy storage device. The heat storage device has a heat absorption capacity of , The heat release power of the heat storage device. This is the unit power cost coefficient of the heat pump. This refers to the output power of the heat pump. This is the start-stop cost coefficient. The start / stop flag is set at time t. This is the start / stop flag at time t-1. Let be the power consumption at time t. Let be the electricity price at time t. This is a state space quantity.

[0049] S3: The MAPPO algorithm, which integrates adaptive pruning and multi-dimensional exploration mechanisms, is used to collaboratively train the observations and actions of each agent to obtain the trained Actor-Critic network of each agent. The trained Actor-Critic network of each agent is then used to perform optimized scheduling of the agents.

[0050] The MAPPO (Multi-Agent Proximal Policy Optimization) algorithm, employing coupled adaptive pruning and multi-dimensional exploration mechanisms, is used to collaboratively train the observations and actions of each agent, as detailed below: Experience collection and advantage function calculation are performed for each agent in each training epoch. Each entity, through its own observations, samples actions according to the current strategy and executes joint actions. The environment returns to the next observation. and global rewards In this application, in addition to interaction with the environment, the communication information of each agent's actions is aggregated through a transformer architecture to obtain the current communication observation as a supplement to the state. The specific implementation is as follows: First, experience is collected, and the communication action outputs of the four agents are used to form a message matrix. The attention head is obtained by scaling the dot product attention calculation, and the formula is: ; In the formula, This is the attention weight matrix. For querying the matrix, The key matrix, Scaling factor Let be the value matrix. Here, scaling dot product attention is used to perform self-attention transformation on the messages sent by each agent, so that each agent obtains personalized information containing only the information most relevant to its local decision-making task. This avoids information redundancy and interference caused by broadcast communication and significantly improves the efficiency of collaborative decision-making among heterogeneous multi-agents.

[0051] After residual connection and layer normalization, global interaction information containing only the most relevant local decision-making tasks is generated for each agent. The local decision-making tasks refer to the physical control tasks decided by each agent mentioned above. In this embodiment, the power balance scheduling agent is responsible for deciding the charging and discharging power of the battery at each moment; the heating plant scheduling agent is responsible for deciding the power consumed by the heat pump equipment at each moment; the heat exchange station scheduling agent is responsible for deciding the heat transfer area of ​​the plate heat exchanger and the working fluid flow rates of the primary and secondary networks; and the heat storage tank scheduling agent is responsible for deciding the heat storage or heat release (i.e., changes in heat storage capacity) of the heat storage tank at each moment.

[0052] Finally, the communication observations of each agent are combined with global public information and the device observations of each agent to form the observation for the next time step. .

[0053] Next, based on the observations at the next moment... Observation at that moment Actions at the current moment The reward value generated by the previous action on the actual environment at this moment. Construct experience tuples, and put experience tuples Stored in the experience replay buffer of each agent. (Collection complete) After one step, the generalized advantage estimate is calculated. For each agent... The advantage function is calculated using generalized advantage estimation. The TD (Time Differential) error is defined as follows: ; In the formula, Let TD be the error term at time t. Provide timely rewards at time t. As a discount factor, Let be the state value at time t+1. Let be the state value at time t.

[0054] Calculate the current state using a value network ( ) and the previous state ( The value function at each time step is calculated by adding the reward (the negative of the cost) to the total value that could be gained in future time steps, and then subtracting the total value that could be gained from the previous time step to the end. This determines the "goodness" or "badness" of the current action. The advantage function is then calculated recursively. The formula is: ; In the formula, The dominant function value, As a smoothing factor, As a discount factor, This is the timing difference error term.

[0055] The policy network (Actor) is updated as follows: This application introduces a pruning scheme based on dynamic adaptive absolute advantage value during the update process of the policy network (Actor), the formula of which is:

[0056] In the formula, This is a dynamic pruning threshold function. For adaptive coefficients, This is the lower limit of the clipping threshold. This is the upper limit of the cropping threshold. It is the original clipping constant. It is the mean of the advantage function. When the absolute value of the advantage at a certain time step is significantly greater than the batch average, the pruning range should be appropriately widened to allow the strategy to make larger adjustments at that decision point; conversely, the pruning should be tightened to avoid meaningless large updates in low advantage regions, which helps to accelerate convergence and improve sample efficiency.

[0057] Then, the loss function is constructed using the improved upper and lower limits of pruning as follows: ; In the formula, Let be the policy loss function. For mathematical expectation, In the state Use actions The advantage value gained, For the clipping function, Let t be the action taken at time t. The state at time t, This indicates the magnitude of the policy update.

[0058] Then, the gradient of each network parameter is calculated for the loss function to complete the policy network parameter update: ; In the formula, For policy network parameters, For the policy network learning rate, For gradient operators, Let be the loss function of the policy network.

[0059] The updates to the value network are as follows: First, four quantitative indicators were selected: MAPE (Mean Absolute Percentage Error), MAE (Mean Absolute Error), and RMSE (Root Mean Square Error). The determination coefficient.

[0060] Specifically, an improved radar chart method is introduced, mapping these four quantitative indicators to the four dimensions of the radar chart. The area and side length of the quadrilateral enclosed by the radar chart are calculated and combined into an overall error loss function, used to construct a holistic visual representation of the error loss. The angle of the sector area occupied by each optimization target in the radar chart is determined according to preset weight values, and the area of ​​the quadrilateral can be obtained graphically. and side length The calculation formula is as follows: ; ; In the formula, For the i-th optimization target value, For the (i+1)th adjacent optimization target value, Let the angle of the sector region corresponding to the i-th optimized target in the radar image be [the angle of the sector region]. Let be the angle of the sector region corresponding to the (i+1)th optimization objective.

[0061] Based on this, the overall error loss function is defined as follows: ; The error level of the value network is quantified using the overall error loss function, and the gradient is used to update the value network parameters as follows: ; In the formula, For value network parameters, For the value network learning rate, For gradient operators, This is the loss function for the value network.

[0062] Furthermore, a two-stage composite strategy was designed to enable the agent to fully explore multi-dimensional features during the training phase and avoid getting trapped in local optima, as follows: The warm-up phase, in the early stages of training (before...) (Step), the entropy coefficient remains at its maximum value. This forces the agent to fully explore various actions, avoiding premature convergence to the optimal strategy. During the cosine annealing stage, after the preheating is complete ( The entropy coefficient follows the cosine curve from its maximum value. Smooth decay to minimum value This gradually reduces the level of exploration and avoids mutations.

[0063] The adaptive improvement mechanism dynamically adjusts the entropy coefficient during the policy update phase by monitoring recent improvements in the reward function. If long-term performance stagnation is observed (i.e., no significant reward improvement), the entropy coefficient is temporarily increased to re-inject exploration and help the agent escape local optima; if performance continues to improve, the entropy coefficient is maintained or slowly decreased to gradually restore it to the baseline. The specific entropy coefficient in the exploration strategy... The changes are as follows: ; In the formula, To train steps The entropy coefficient under the condition, The minimum entropy coefficient, The maximum entropy coefficient, For the number of preheating steps, This represents the total number of training steps.

[0064] Finally, the training is repeated until the specified rounds are reached to obtain the Actor-Critic network for each agent after training. The Actor-Critic network for each agent after training is then used to perform optimized scheduling.

[0065] The distributed execution optimization scheduling based on the fully trained Actor-Critic network is as follows: The agent integrates global public information, its own local observations, and communication observations, and inputs them into the fully trained Actor-Critic network, outputting physical control actions and communication actions at each time step; during the model training phase, a Gaussian distribution random sampling strategy is adopted. Explore actionable actions; during actual scheduling, take the mean of the Gaussian distribution as the deterministic action output. It completes multi-agent collaborative optimization scheduling, realizing random exploration and safe execution of continuous actions.

[0066] Example: Actual environmental measurements of a thermal power pipeline network.

[0067] In a typical thermal power pipeline network environment, the multi-agent scheduling method for thermal power pipeline network systems, which incorporates a communication learning mechanism as provided in this application, was tested. The optimization scheduling period was set to a 24-hour period per day, with the electricity price fluctuating between 300-800 yuan / MWh.

[0068] The system is equipped with large-capacity batteries and thermal storage devices to ensure the stability of the electrothermal balance. The heating system mainly uses heat pump units as the main device for electrothermal conversion to heat the primary network return water. In addition, the system also has non-controllable wind power and photovoltaic units as new energy output power. The output curves are all set according to typical intraday distribution and superimposed with a certain disturbance to reflect volatility and randomness.

[0069] During system initialization, the SOC of all energy storage units was set to 50% to ensure that the energy storage units have a certain adjustment margin under different intraday conditions. This configuration ensures both the representativeness of the thermal power network system and provides it with a certain degree of flexible adjustment capability, serving to verify the validity of this application.

[0070] Please see Figure 2 This architecture comprises two subsystems: a thermal power pipeline network environment system and a communication environment. Four agents interact collaboratively within these two environments, inputting three types of observations into their respective actor and value networks. Based on the policies output by the networks, they select actions and apply these actions to the system environment. The system model then calculates and outputs the system environment observations for the next time step. The communication messages generated by the agents are processed by the Transformer architecture, outputting differentiated communication messages as communication observations. These are then concatenated with global environment variables to form the overall state space, which serves as the observation input for the next time step.

[0071] During network updates, the policy (Actor) network employs a pruning strategy based on the ratio of the dominance function: when the absolute value of the dominance function is large, the pruning interval is widened to allow for a larger policy update; conversely, the pruning interval is narrowed to prevent drastic changes in the policy in low-dominance regions, thereby ensuring stable system performance. The value network, on the other hand, uses a comprehensive error loss function composed of four types of loss functions instead of the mean squared error to calculate the gradient and update the parameters in reverse.

[0072] Please see Figure 3 The power balance dispatch diagram shows the time-series variation curves of power-related variables during the dispatch period, including renewable energy output, power load, grid interaction power, energy storage charging and discharging power, heat pump consumption power, and abandoned power.

[0073] Depend on Figure 3 It can be seen that the power purchased from the external power grid is positive during periods of high load and insufficient renewable energy, while the power sold is positive during periods of renewable energy surplus, proving that the system has the ability to exchange power in both directions and to connect surplus power to the grid. The batteries are charged during periods of renewable energy surplus and discharged during periods of peak load, and their state of charge is maintained within the safe threshold throughout the process, thus achieving peak shaving and valley filling.

[0074] The power curtailment only shows a small positive value at the peak output and generally approaches zero, while the power consumption of heat pumps increases during the same period, indicating that excess electrical energy is effectively converted into heat energy for consumption. The heat balance scheduling diagram shows the temporal changes of relevant variables of the thermal system within the same scheduling period. As can be seen from the diagram, the heat generation power of heat pumps and the power consumption power of heat pumps in the power scheduling are highly synchronized in time, indicating that they utilize surplus electricity from renewable energy sources for heating.

[0075] The thermal storage unit absorbs heat during peak heat pump production periods and releases heat at night or during periods of higher heat demand, achieving intertemporal heat transfer. The primary network supply water temperature and the secondary return water temperature fluctuate smoothly throughout the entire cycle, remaining stable within their set ranges without overheating or underheating, thus ensuring heating quality. The battery state of charge is sequentially related to the operation of the heat pump and thermal storage unit. When energy storage is sufficient and renewable energy sources are abundant, electric heating and thermal storage are enhanced; conversely, the thermal storage unit partially replaces electric heating, demonstrating complementary thermoelectric coupling operation.

[0076] according to Figure 3 The scheduling results shown demonstrate that the integrated thermal and power energy load prediction and scheduling strategy based on deep reinforcement learning proposed in this application can achieve the maximum absorption of renewable energy, the rational charging and discharging of energy storage systems, the economic efficiency of grid interaction, and the stability of heating temperature, all while meeting the electricity and heat load demands and equipment operation constraints. This has significant technical effects.

[0077] Please see Figure 4 In the initial training rounds (round 0), the total rewards of both algorithms were below -100,000, indicating a low level. From round 2000 to round 8000, the total rewards of both algorithms converged steadily, with the improved algorithm accumulating slightly greater rewards than the original algorithm after sufficient training. However, it should be noted that the Com-Clip-MAPPO algorithm proposed in this application exhibits superior performance in the convergence process compared to the benchmark MAPPO algorithm.

[0078] Specifically, the baseline MAPPO algorithm exhibits significant reward oscillations throughout the training period. In contrast, the Com-Clip-MAPPO algorithm of this application introduces a communication learning mechanism and a pruning policy update, allowing the reward function to be fully explored in the early stages, resulting in more pronounced reward function oscillations. In the middle stages, cosine annealing is used for exploration, which significantly reduces the fluctuation amplitude of the reward curve after reaching a stable value. In the later stages, an adaptive collaborative mechanism is used to avoid getting trapped in local optima. Although the reward function fluctuates, the policy can escape local optima and slowly increase the reward value.

[0079] In summary, this application proposes a multi-agent scheduling method for a thermal power pipeline network system that integrates a communication learning mechanism. By utilizing the Transformer to achieve personalized information sharing among agents, and combining adaptive pruning and multi-level exploration strategies, the method effectively improves the collaborative efficiency and training stability of the multi-agent system. Simulation results demonstrate that the proposed method has significant advantages in renewable energy consumption, thermal power balance control, and economical system operation, exhibiting good technological advancement and engineering application value.

[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0081] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-agent scheduling method for a thermal power pipeline network system integrating a communication learning mechanism, characterized in that, Includes the following steps: A thermal power pipe network system is constructed and constraints are set to obtain a thermal power pipe network simulation environment; Based on the aforementioned thermoelectric pipe network simulation environment, a multi-agent Markov game framework with coupled communication learning mechanism is built to determine the observations and actions of each agent. The MAPPO algorithm, which integrates adaptive pruning and multidimensional exploration mechanisms, is used to collaboratively train the observations and actions of each agent, resulting in an Actor-Critic network for each agent after training. This network is then used to perform optimized scheduling for each agent.

2. The multi-agent scheduling method for a thermal power pipeline system with a fusion communication learning mechanism as described in claim 1, characterized in that, The process of building the multi-agent Markov game framework with coupled communication learning mechanism includes: based on the thermoelectric pipe network simulation environment, setting up several agents, defining global common information shared by all agents, configuring communication observations for each agent according to the communication learning mechanism, and splicing the global common information, exclusive observations and local observations of each agent to obtain the observations of each agent; Based on the control tasks of each agent, physical control quantities are determined, and the observations and actions of each agent are formed by combining communication actions. A globally shared reward function is constructed, and a multi-agent Markov game framework with coupled communication learning mechanism is built.

3. The multi-agent scheduling method for a thermal power pipeline network system with a fusion communication learning mechanism as described in claim 2, characterized in that, The process of performing the collaborative training includes: scaling the dot product attention of the communication actions of each agent to obtain an attention head, performing residual connections and layer normalization, generating global interaction information for each agent, and splicing the global interaction information, global common information and the device observations of each agent to obtain the observations at the next moment. Based on the observations of the next time step, the observations and actions of each agent at the current time step, and the reward value generated by the previous time step on the actual environment at the current time step, an experience tuple is constructed and stored in the experience replay buffer of each agent. Based on the experience tuple, an advantage function is constructed and the advantage function value is calculated. Based on the absolute value of the advantage function value, a loss function is constructed through dynamic adaptive pruning and the policy network is updated. Multiple error indicators are set, and an overall error loss function is constructed in combination with an improved radar chart to update the value network. Combined with a two-stage composite strategy, the entropy coefficient is dynamically adjusted through multi-dimensional exploration, and each agent is trained in multiple rounds of collaborative training to obtain the Actor-Critic network of each agent after training.

4. The multi-agent scheduling method for a thermal power pipeline network system with a fusion communication learning mechanism as described in claim 2, characterized in that, The process of performing the optimized scheduling includes: the agent integrates the global public information, its own local observations and communication observations, and inputs them into the Actor-Critic network of each trained agent, and outputs the physical control action and communication action at each time step; during the model training phase, a Gaussian distribution random sampling strategy is used to explore possible actions; during the actual scheduling operation, the mean of the Gaussian distribution is taken as the deterministic action output to complete the multi-agent collaborative optimized scheduling.

5. The multi-agent scheduling method for a thermal power pipeline system with a fusion communication learning mechanism as described in claim 3, characterized in that, The formula for scaling dot product attention is: ； In the formula, This is the attention weight matrix. For querying the matrix, The key matrix, Scaling factor For value matrices, For the number of attention heads, This is the matrix transpose symbol. As key, It is a normalized exponential function.

6. The multi-agent scheduling method for a thermal power pipeline system with a fusion communication learning mechanism as described in claim 3, characterized in that, The formula for dynamic adaptive pruning based on the absolute value of the dominant function is as follows: In the formula, This is a dynamic pruning threshold function. For dynamic, For time, For the clipping function, It is the original clipping constant. For adaptive coefficients, Let be the mean of the dominance function. This is the lower limit of the clipping threshold. This represents the upper limit of the clipping threshold.

7. The multi-agent scheduling method for a thermal power pipeline network system with a fusion communication learning mechanism as described in claim 3, characterized in that, The two-stage composite strategy includes a preheating stage, a cosine annealing stage, and an adaptive boosting mechanism. During the preheating phase, the entropy coefficient is maximized, forcing the agent to fully explore various actions. During the cosine annealing stage, the entropy coefficient is smoothly decayed, reducing the degree of exploration. The adaptive boosting mechanism increases the entropy coefficient when performance stagnation is observed, helping the agent escape local optima; and when performance continues to improve, it continues to smoothly decrease the entropy coefficient until the entropy coefficient returns to the baseline.

8. The multi-agent scheduling method for a thermal power pipeline system with a fusion communication learning mechanism as described in claim 7, characterized in that, The formula for the change of the entropy coefficient is: ； In the formula, To train steps The entropy coefficient under the condition, The minimum entropy coefficient, The maximum entropy coefficient, This represents the total number of steps in the preheating phase. This represents the total number of training steps.

9. The multi-agent scheduling method for a thermal power pipeline network system with a fusion communication learning mechanism as described in claim 3, characterized in that, The process of constructing an overall error loss function by combining an improved radar chart includes: determining four evaluation quantification indicators; mapping the four evaluation quantification indicators to the four coordinate axes of the radar chart; determining the angle of the sector area occupied by each optimized target in the radar chart according to preset weight values; calculating the sector area and the total perimeter of the radar chart quadrilateral based on the angle of the sector area occupied by each optimized target; and calculating the overall error loss function based on the sector area and the total perimeter of the radar chart quadrilateral.

10. The multi-agent scheduling method for a thermal power pipeline network system with a fusion communication learning mechanism as described in claim 1, characterized in that, The process of constructing the thermoelectric pipeline system includes: modeling the batteries, heat storage devices, heat pump stations, heat exchange stations and heating pipelines in the thermoelectric pipeline system to obtain the thermoelectric pipeline system.

Images