Urban pipeline flow optimization control system based on multi-agent reinforcement learning

Abstract

The present invention belongs to the field of automatic control technology, and specifically relates to an urban pipeline flow optimization control system based on multi-agent reinforcement learning. It includes: a pipeline network multi-agent model construction part, a multi-agent reinforcement learning part and a multi-agent optimization control part; the pipeline network multi-agent model construction part is used to assign an agent to each pipeline segment, and each agent is described by a set consisting of a state space, an action space, a reward function and a strategy; the multi-agent reinforcement learning part is used to continuously learn and update the Q value of each agent based on a nonlinear flow dynamics equation; the multi-agent optimization control part is used to continuously optimize the strategy of each agent through a policy gradient method, so that each agent approaches the optimal control strategy. The present invention uses a reinforcement learning algorithm to enable the system to adaptively adjust the control strategy in a complex and changeable environment.

Description

Technical Field

[0001] The present invention belongs to the field of automatic control technology, and in particular relates to an urban pipeline flow optimization control system based on multi-agent reinforcement learning. Background Art

[0002] Urban pipeline flow control systems are an important part of modern urban infrastructure and are widely used in water supply, drainage, natural gas transmission and other fields. With the expansion of urban scale and the increase of population, the complexity of pipeline systems and the difficulty of flow control are increasing. In order to ensure the efficient and safe operation of urban pipeline systems, the research and application of flow control technology is particularly important. However, the existing technology still has many shortcomings in flow control and cannot meet the needs of increasingly complex urban pipeline systems.

[0003] PID (proportional-integral-differential) controller is one of the most widely used control technologies in pipeline flow control systems. By adjusting the proportional, integral and differential parameters, the PID controller can adjust the pipeline flow. However, when facing complex and changeable pipeline environments, the PID controller has problems such as slow response speed and poor robustness. Especially when facing nonlinear and strongly coupled multivariable systems, the performance of the PID controller is greatly limited, and it is difficult to achieve precise control. Fuzzy control is a control method based on fuzzy logic, which is suitable for dealing with nonlinear, time-varying and uncertain control problems. Fuzzy controllers can handle complex flow control problems by fuzzifying control rules. However, fuzzy control relies on expert experience in the design and adjustment of rules and is difficult to adapt to dynamically changing environments. In addition, the performance of fuzzy controllers is highly dependent on the design of fuzzy rules and is difficult to automatically optimize and adjust. Neural network control uses the powerful learning and generalization capabilities of artificial neural networks to achieve pipeline flow control by learning historical data. Neural network control has advantages in dealing with complex nonlinear problems and can adaptively adjust control strategies. However, the training process of neural networks is complex, the amount of calculation is large, and there is a certain degree of application difficulty for pipeline flow control systems with high real-time requirements. In addition, the black box characteristics of neural networks make them less interpretable and controllable, and there are certain risks in practical applications. Multi-agent systems (MAS) achieve global optimization by dividing the pipeline system into multiple agents, each of which independently performs flow control and decision-making. The advantage of multi-agent systems lies in their distributed control and coordination capabilities, which can effectively cope with the complexity of large-scale pipeline networks. However, existing multi-agent systems still face challenges in collaborative control, information sharing, and conflict resolution, making it difficult to achieve efficient collaboration in practical applications. Summary of the invention

[0004] In view of this, the main purpose of the present invention is to provide an urban pipeline flow optimization control system based on multi-agent reinforcement learning. The present invention uses a reinforcement learning algorithm to enable the system to adaptively adjust the control strategy in a complex and changing environment. Through continuous learning and optimization, the agent can quickly adapt to changes in parameters such as flow, pressure, and energy consumption to ensure the stable operation of the system.

[0005] The technical solution adopted by the present invention is as follows:

[0006] An urban pipeline flow optimization control system based on multi-agent reinforcement learning, comprising: a pipeline network multi-agent model construction part, a multi-agent reinforcement learning part and a multi-agent optimization control part; the pipeline network multi-agent model construction part is used to divide the urban pipeline network into multiple pipeline sections, assign an agent to each pipeline section, and each agent is described by a set consisting of a state space, an action space, a reward function and a strategy; define the state space, action space, strategy and reward function of each agent; define the nonlinear flow dynamics equation of each pipeline section; the multi-agent reinforcement learning part is used to continuously learn and update the Q value of each agent based on the nonlinear flow dynamics equation; the multi-agent optimization control part is used to continuously optimize the strategy of each agent through a policy gradient method, so that each agent approaches the optimal control strategy.

[0007] The pipeline network multi-agent model construction part allocates the agent to each pipeline segment as follows: ;in, It is An intelligent agent, is the state space, is the action space, is the reward function, is the discount factor, is the strategy; state space ;in, It's traffic. It's pressure. is the fluid level, Base flow, is energy consumption; action space ;in, is the valve opening adjustment, is the pump frequency adjustment.

[0008] Furthermore, the strategy of each agent constructed in the pipeline network multi-agent model construction part Use the following formula to express it:

[0009] ;

[0010] in, are the preset strategy parameters; is the preset bias vector; Represents all possible actions and is used to normalize the probability distribution; Indicates in status Next select action The probability of is the first feature mapping function; is the second feature mapping function.

[0011] Furthermore, the first feature mapping function Use the following formula to express it:

[0012] ;

[0013] The second feature mapping function Use the following formula to express it:

[0014] ;

[0015] in, is the transpose operation of the vector; is the L2 norm operation; is the L1 norm operation; is a symbolic function;

[0016] Furthermore, the reward function is expressed using the following formula:

[0017] ;

[0018] in, Indicates The set of neighbors of an agent; is the maximum energy consumption of all agents; Indicates The mutual information between the states and actions of each agent; For the The amount of change in the actions of each agent; Represents calculation strategy Entropy of

[0019] Furthermore, the pipeline network multi-agent model construction part defines the nonlinear flow dynamics equation of each pipeline segment using the following formula:

[0020] ;

[0021] in, For the The agent and The flow difference of each agent; For time; is the distance along the pipeline; For the The agent and The difference in the cross-sectional area of ​​the pipelines of the agents; For the The agent and The pressure difference of each agent; is the liquid density; is the acceleration due to gravity; For the The agent and The pipeline inclination difference of each agent; For the The agent and The friction coefficient of each agent is different; For the The agent and The baseline flow difference of each agent; is the dynamic viscosity of the liquid; is the elastic coefficient of the pipeline corresponding to all agents; is the vibration frequency of the pipeline corresponding to all intelligent agents;

[0022] Furthermore, the multi-agent reinforcement learning part continuously learns and updates the Q value of each agent based on the nonlinear flow dynamics equation through the following formula:

[0023] ;

[0024] in, For learning and updating The Q value of an agent represents Agents in state Take action The value of action; is the learning rate; For the Discount factor for each agent; To perform actions The next state after Possible actions in the next state; It is the preset neighbor influence factor, which controls the influence of the neighboring agent's Q value on the current agent's Q value update; For the Agents in state Current strategy under For the Agents are in state The target strategy below; Indicates the calculation of KL divergence; For the first The nonlinear flow dynamics equations of the pipeline section corresponding to each intelligent agent; For the Agents in state Take action The value of action;

[0025] Furthermore, the multi-agent optimization control part uses the following formula to continuously optimize the strategy of each agent through the policy gradient method, so that each agent approaches the optimal control strategy:

[0026] ;

[0027] in, is the policy objective function About strategy parameters The gradient of Indicated in strategy Expected value under represents the discounted accumulation over infinite time steps, is the discount factor, satisfying ; is the state action value function, which means that in the state Take action Expected cumulative rewards; is the state value function, which means that in the state Expected cumulative rewards under For strategy Entropy of Indicates the current strategy With the prior strategy The KL divergence between is the entropy regularization coefficient; is the KL divergence regularization coefficient, which controls the closeness between the current strategy and the prior strategy.

[0028] Further, Agents at time step The learning rate Update using the following formula:

[0029] ;

[0030] in, For the updated The learning rate for each time step.

[0031] By adopting the above technical scheme, the present invention produces the following beneficial effects: by introducing a multi-agent reinforcement learning algorithm, the present invention significantly improves the system's response speed and control accuracy to flow changes. The multi-agent system divides the complex pipeline network into multiple independent agents, each of which can dynamically adjust the control strategy based on its own state and environment. This distributed control method enables the system to respond to local flow changes more quickly, and at the same time, achieve global optimization control through the collaborative cooperation between agents. The reinforcement learning algorithm enables the agent to adaptively optimize the control strategy in the continuous learning process, improve the robustness of the system, and ensure that it can still maintain stable operation in the face of emergencies and system failures. The present invention adopts a reinforcement learning algorithm, so that the agent can continuously learn and optimize its control strategy based on real-time data. Unlike traditional PID control and fuzzy control that rely on preset rules, the reinforcement learning algorithm guides the agent to autonomously adjust the strategy through a reward mechanism to maximize the cumulative reward. Specifically, the agent dynamically adjusts the valve opening and pump frequency by real-time monitoring of parameters such as flow, pressure, and energy consumption, thereby achieving precise control of the flow. In addition, the introduction of the policy gradient method and adaptive learning rate enables the agent to automatically adjust the learning pace according to environmental changes during the learning process, thereby improving learning efficiency and control effects. In a multi-agent system, each agent not only operates independently, but can also achieve global optimal control through information sharing and collaborative decision-making. The present invention designs a complex reward function and introduces neighbor influence factors, so that each agent can comprehensively consider the behavior and state of neighboring agents while optimizing its own control strategy, thereby achieving collaborative optimization of the overall system. For example, at a key node in a pipeline network, when an agent detects a flow anomaly, it can quickly adjust its own strategy and coordinate adjacent agents through neighbor influence factors to jointly optimize flow distribution, avoid local flow overload or shortage, and improve the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of an urban pipeline flow optimization control system based on multi-agent reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0034] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other alternative features that are equivalent or have similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0035] Example 1: Reference Figure 1 , an urban pipeline flow optimization control system based on multi-agent reinforcement learning, which includes: a pipeline network multi-agent model construction part, a multi-agent reinforcement learning part and a multi-agent optimization control part; the pipeline network multi-agent model construction part is used to divide the urban pipeline network into multiple pipeline segments, assign an agent to each pipeline segment, and each agent is described by a set consisting of a state space, an action space, a reward function and a strategy; define the state space, action space, strategy and reward function of each agent; define the nonlinear flow dynamics equation of each pipeline segment; the multi-agent reinforcement learning part is used to continuously learn and update the Q value of each agent based on the nonlinear flow dynamics equation; the multi-agent optimization control part is used to continuously optimize the strategy of each agent through the policy gradient method, so that each agent approaches the optimal control strategy.

[0036] Specifically, the complex urban pipeline network is first divided into multiple manageable pipeline segments, and each pipeline segment is managed and optimized by an intelligent agent. Through this division, the system can reduce the overall complexity, so that each intelligent agent only needs to focus on the flow control of the local pipeline segment, thereby achieving refined management. In the specific implementation, each pipeline segment is regarded as an independent intelligent agent, and the intelligent agent describes and controls the flow of the corresponding pipeline segment through its state space, action space, reward function and strategy set. The state space is an important way for the intelligent agent to perceive the external environment. For the intelligent agent of the pipeline segment, the state space can include the current flow, pressure, flow rate and other related parameters, which can fully describe the current working state of the pipeline segment. Through real-time monitoring and analysis of these state parameters, the intelligent agent can fully grasp the operation of the current pipeline segment and provide a basis for subsequent action selection. The action space is a collection of control measures that the intelligent agent can take. For the pipeline flow control system, the action space of the intelligent agent can include operations such as adjusting the opening of the pipeline valve, starting or shutting down the pump, etc. These actions directly affect the flow change of the pipeline segment. The intelligent agent achieves optimal control of the pipeline flow by selecting appropriate actions. The reward function plays a vital role in the learning process of the agent. The design of the reward function directly affects the learning effect of the agent. In this system, the design goal of the reward function is to reflect the optimization goal of the system, such as minimizing energy consumption, avoiding overpressure or underpressure, etc. Specifically, the reward function can comprehensively consider factors such as energy consumption and pressure state, so that the agent not only considers the current flow demand when choosing actions, but also takes into account the overall operation efficiency and safety of the system. The strategy is the probability distribution of the agent's choice of action under a specific state. The optimization of the strategy is the key to achieving effective control of the agent. In this system, the agent continuously optimizes its strategy through the multi-agent reinforcement learning algorithm, so that it can select the optimal control action under different states. In the specific implementation process, the agent continuously interacts with the environment, and based on the feedback of the reward function, it continuously adjusts and optimizes its strategy, so as to gradually approach the optimal control strategy. In the construction of the multi-agent model of the pipeline network, the nonlinear flow dynamics equation plays a key role. This equation can accurately describe the dynamic changes of the flow in the pipeline section. By establishing the nonlinear flow dynamics equation for each pipeline section, the agent can predict the change trend of the flow, so that it can be more accurate and efficient when selecting control actions. The nonlinear flow dynamics equation comprehensively considers factors such as flow, pressure, and pipe cross-sectional area, so that the model can more realistically reflect the actual situation. Through the above method, this system realizes the refined management and optimization control of complex urban pipeline networks. Each agent realizes intelligent control of pipeline section flow by perceiving its state space, selecting appropriate actions, and continuously optimizing its strategy based on the feedback of the reward function. The multi-agent system achieves the global optimal flow control effect through coordination and collaboration.This system not only improves the operating efficiency of the urban pipeline network, but also effectively reduces energy consumption and operating risks. It has high practical application value and innovation.

[0037] Reinforcement learning is a method to obtain the best decision-making strategy through interaction with the environment. In this system, each agent interacts independently with its corresponding pipeline segment environment. By sensing the current state, selecting actions and receiving environmental feedback, the agent can continuously accumulate experience and improve its decision-making strategy. In each interaction step, the agent determines the action it should take at the current moment through the perception of the state. This process not only considers the current flow demand, but also comprehensively considers the long-term return in the future. This method enables the agent to gradually approach the optimal control strategy in a dynamic and uncertain environment. In the specific implementation, the learning process of each agent relies on the nonlinear flow dynamics equation, which can accurately describe the dynamic changes of the flow in the pipeline segment. Through this dynamics equation, the agent can predict the impact of its control action on the flow change, so as to be more accurate and efficient in selecting actions. Each agent updates its value function based on the nonlinear flow dynamics equation through continuous interaction with the environment. The value function reflects the cumulative reward that can be obtained by taking a specific action in a specific state. This process enables the agent to continuously evaluate and improve its control strategy to achieve optimal control of the overall system. In the reinforcement learning process, the agent updates its strategy through the Q-learning algorithm. The Q-learning algorithm is a model-free reinforcement learning algorithm. By continuously updating the state-action value pair (Q value), the agent can evaluate the goodness of taking a certain action in a specific state. Specifically, the agent continuously updates its Q value in each state by selecting an action and observing the immediate reward and the next state, thereby gradually learning the optimal control strategy. The update of the Q value is based on the feedback of the reward function. The reward function design reflects the optimization goal of the system, such as minimizing energy consumption, avoiding overvoltage or undervoltage, etc. By comprehensively considering these factors, the agent can select the optimal control action in different states. In order to further optimize the agent's strategy, this system adopts the policy gradient method. The policy gradient method updates the policy parameters by maximizing the expected reward, so that the agent can continuously improve its decision-making strategy in a dynamic environment. Specifically, the agent gradually optimizes its strategy by making small adjustments to the policy parameters during each interaction, so that the optimal control action can be selected in different states. The introduction of the policy gradient method enables the agent to make decisions not only based on historical experience, but also to make adaptive adjustments in a constantly changing environment, thereby achieving more efficient and robust control effects. In addition, the multi-agent reinforcement learning part realizes global optimization control through collaboration and information sharing between agents. Each agent not only learns and makes decisions independently, but also shares its learning experience and strategy information through collaboration with other agents, thereby achieving optimal flow control on a global scale. This collaborative mechanism enables the system to better cope with complex and dynamic pipeline network environments and improves the overall optimization effect of the system.

[0038] The policy gradient method is an optimization algorithm based on gradient ascent, which updates the policy parameters by maximizing the cumulative expected reward of the agent in the environment. Each agent in its local environment continuously samples and evaluates the effects of the actions it takes, and gradually adjusts its policy parameters so that it can select the optimal control action under different states. This process relies on the experience data obtained by the agent in continuous interaction. By analyzing and processing these data, the agent can identify the optimal action selection under a specific state, thereby gradually approaching the optimal control strategy. In the specific implementation, the policy update process of each agent is based on its perceived state, the action selected, and the immediate reward and future expected reward obtained from the environment. The agent accumulates a large amount of state-action-reward data by continuously interacting with its pipeline segment environment, and calculates the policy gradient based on this data to optimize its control strategy. The calculation of the policy gradient involves estimating the expected value of the cumulative reward. By making small adjustments to the policy parameters, the agent can gradually improve its strategy so that the probability of selecting the optimal action under different states continues to increase. In this way, the agent can not only adapt to the current environmental changes, but also maintain a high control effect in a dynamic environment. The multi-agent optimization control part also emphasizes collaboration and information sharing between agents. While each agent is optimizing in its local environment, it also needs to communicate and collaborate with other agents to achieve global optimal flow control. This collaboration mechanism can be achieved through sharing state information, action selection, and reward feedback, so that each agent can take into account the global state and goals of the entire pipeline network when performing local optimization, thereby achieving global coordination and efficient control. In this way, the system can better cope with complex and dynamic pipeline network environments and improve the overall optimization effect. In the multi-agent optimization control part, the application of nonlinear flow dynamics equations is an important foundation. Based on the nonlinear flow dynamics equations, each agent can accurately predict the impact of its control actions on the flow changes in the pipeline section, so that it can be more accurate and efficient when selecting actions. By continuously learning and updating the nonlinear flow dynamics equations, the agent can better adapt to environmental changes and achieve efficient flow control in a dynamic environment. This process not only relies on the autonomous learning ability of the agent, but also requires the system to continuously correct and optimize the nonlinear flow dynamics equations to ensure the accuracy and robustness of its model. In addition, the multi-agent optimization control part also needs to consider various constraints and limiting factors in practical applications. For example, in an urban pipeline network, there may be pipeline capacity limitations, pressure limitations, and energy consumption limitations. In the process of optimizing control, the intelligent agent must comprehensively consider these constraints to ensure that the selected control strategy can not only achieve the optimization goal but also meet various constraints and limitations in practical applications.This process requires a reasonable design of the reward function so that when the intelligent agent chooses an action, it not only considers the maximization of the cumulative reward, but also the satisfaction of the constraints, so as to achieve the optimal comprehensive control effect.

[0039] Embodiment 2: The pipeline network multi-agent model construction part allocates an agent to each pipeline segment: ;in, It is An intelligent agent, is the state space, is the action space, is the reward function, is the discount factor, is the strategy; state space ;in, It's traffic. It's pressure. is the fluid level, Base flow, is energy consumption; action space ;in, is the valve opening adjustment, is the pump frequency adjustment.

[0040] Specifically, the definition of the state space enables the agent to fully perceive the current state of its pipeline segment, including parameters such as flow, pressure, liquid level, reference flow and energy consumption. Flow is a key indicator of liquid flow in the pipeline, pressure directly affects the change of flow, liquid level reflects the height change of liquid in the pipeline, reference flow provides a reference standard, and energy consumption is one of the goals that need to be considered in optimization control. Through real-time monitoring of these state variables, the agent can fully grasp the operation of the pipeline segment and provide a basis for subsequent action selection. The action space defines the control measures that the agent can take, mainly including valve opening adjustment and pump frequency adjustment. These actions directly affect the flow change of the pipeline segment. The agent achieves precise control of the flow by selecting a suitable combination of actions. The actions selected by the agent in different states should not only consider the current flow demand, but also comprehensively consider energy consumption and the overall operating efficiency of the system. The reward function is a crucial component in the learning process of the agent, and its design reflects the optimization goal of the system. In this system, the reward function comprehensively considers multiple goals of flow control, such as minimizing energy consumption, maintaining stable flow and pressure, etc. The agent gradually adjusts and optimizes its control strategy based on the feedback of the reward function by continuously interacting with the environment. Specifically, when the action taken by the agent can achieve the expected flow rate with the minimum energy consumption, the system gives a high reward; on the contrary, when the actual flow rate is greatly different from the benchmark flow rate or the energy consumption is too high, the system gives a penalty. Through this mechanism, the agent can continuously improve its control strategy during the learning process to achieve the global optimization of the system. The introduction of the discount factor enables the agent to take into account the importance of both immediate rewards and future rewards when optimizing the strategy. A larger discount factor means that the agent pays more attention to long-term returns, making it more robust and balanced when choosing actions. The strategy is the probability distribution of the agent's choice of action under a specific state. Through the reinforcement learning algorithm, the agent can continuously optimize its strategy so that it can choose the optimal action under different states. In the process of constructing the agent model, the nonlinear flow dynamics equation plays a key role. This equation can accurately describe the dynamic changes of the flow rate in the pipeline section, so that the agent can predict its impact on the flow rate when selecting the control action. By continuously learning and updating the nonlinear flow dynamics equation, the agent can better adapt to environmental changes and achieve efficient flow control in a dynamic environment. This process not only relies on the autonomous learning ability of the intelligent agent, but also requires the system to continuously correct and optimize the nonlinear flow dynamics equations to ensure the accuracy and robustness of its model.

[0041] Embodiment 3: Strategy of each agent constructed in the pipeline network multi-agent model construction part Use the following formula to express it:

[0042] ;

[0043] in, are the preset strategy parameters; is the preset bias vector; Represents all possible actions and is used to normalize the probability distribution; Indicates in status Next select action The probability of is the first feature mapping function; is the second feature mapping function.

[0044] Specifically, this policy model transforms the state and actions Mapped into a high-dimensional feature space to capture the complex relationship between states and actions. Feature mapping function The joint features of state and action are extracted. Through this feature extraction, the characteristics of state-action pairs can be better described, thus providing rich information for strategy optimization. In addition, the feature mapping function is specifically used to extract state features, which help the agent perceive the current environment more accurately and make more reasonable action choices based on this. Indicates in status Next select action Here, the policy parameter and the bias vector It is the key parameter optimized by the reinforcement learning algorithm. Strategy parameters Determines the impact of state and action features on the strategy, while the bias vector In the process of policy optimization, the agent continuously updates the policy through continuous interaction with the environment based on the policy gradient method. and The value of makes the probability of the agent choosing the best action in different states increase continuously. is a valid probability distribution and needs to be normalized to the probabilities of all possible actions. It is used in this normalization process, which ensures that the sum of the probabilities of the agent choosing an action in each state is 1. This normalization process not only conforms to the basic properties of probability distribution, but also ensures that the agent makes reasonable decisions in the actual control process. In this policy model, the feature mapping function and The introduction of is to better capture the complex relationship between state and action, so that the intelligent agent can optimize the strategy in the high-dimensional feature space. Through these feature mapping functions, the intelligent agent can extract and process a large amount of state and action information, thereby improving its perception and decision-making ability. This method allows the intelligent agent to not only rely on the current state information, but also combine historical experience and the optimization results of policy parameters to make more intelligent and precise control decisions. The policy gradient method plays a vital role in the optimization process. By maximizing the expected cumulative reward, the policy gradient method can guide the intelligent agent to gradually adjust its policy parameters, so that the probability of selecting the optimal action in different states continues to increase. In the continuous interaction with the environment, the intelligent agent adjusts the policy parameters and the bias vector The optimization enables it to adaptively adjust the control strategy in a dynamic environment, thereby achieving efficient and robust flow control. Through the construction and optimization of this policy model, the intelligent agent can achieve more accurate and efficient flow control in complex urban pipeline networks. The intelligent agent selects the optimal action based on the optimized strategy in each state, thereby achieving precise control of the flow of the pipeline section, and through collaboration with other intelligent agents, achieves the global optimal flow control effect. The introduction of feature mapping functions and policy gradient methods enables the system to achieve efficient and reliable flow control in complex and dynamic environments, and has high application potential and practical value.

[0045] Embodiment 4: First feature mapping function Use the following formula to express it:

[0046] ;

[0047] The second feature mapping function Use the following formula to express it:

[0048] ;

[0049] in, is the transpose operation of the vector; is the L2 norm operation; is the L1 norm operation; is a symbolic function.

[0050] Specifically, the first feature mapping function The role of is to change the state and actions The feature mapping function is composed of three parts: , and .first, Using the cosine function to calculate flow and valve opening adjustment This term can capture the periodic relationship between flow rate and valve opening through the periodicity of the cosine function. In fluid dynamics, the change of flow rate is often periodic. The introduction of the cosine function can well reflect this characteristic, allowing the intelligent agent to understand the periodic behavior in flow control and thus consider these factors when selecting a control strategy. Secondly, The pressure is calculated by the sine function and pump frequency adjustment The sine function also has periodic characteristics and can capture the nonlinear effect of pressure changes on flow control. The interaction between pressure and pump frequency is very important in pipeline flow control. Through the processing of sine functions, the intelligent agent can better understand the impact of pressure changes on flow at different frequencies, thereby optimizing its control strategy. Part III, The liquid level is calculated by the hyperbolic tangent function and baseline flow The hyperbolic tangent function has the characteristic of compressing the input value into the range of [-1,1], which enables it to enhance the robustness of features when processing data with a large range of variations and prevent the influence of extreme values ​​on feature extraction. The relationship between liquid level and reference flow is also a key factor in flow control. Through this feature extraction, the intelligent agent can better balance flow control and liquid level management. Next, the second feature mapping function Mainly used to extract status The state variables are processed by logarithmic, square root and sigmoid functions. The norm is logarithmic, and the flow information is compressed and smoothed. This processing method not only enhances the robustness of the feature, but also effectively reduces the impact of extreme values ​​on the feature. The logarithmic function can compress a large range of values, making the data fluctuate within a smaller range, so that the agent can still maintain a stable feature extraction effect when facing extreme flow values. Secondly, Through the pressure The square root of the norm can smooth the impact of pressure changes. The use of the square root function helps smooth the process, allowing the agent to maintain stable control effects when facing drastic pressure fluctuations. Pressure changes directly affect flow. Through this feature processing, the agent can more accurately perceive and respond to pressure changes, thereby optimizing its control strategy. Finally, The inner product of the liquid level and the reference flow is processed by the sigmoid function. The sigmoid function has the characteristic of compressing the input value to the range of [0,1], so that the feature can reflect the probability distribution of the state variable, thereby providing more information for the agent to refer to when making decisions. This processing method enables the agent to better understand the relationship between the liquid level and the flow, and ensure the stability of the liquid level while controlling the flow.

[0051] Through these two feature mapping functions, the agent can map complex state and action information to a high-dimensional feature space, so that in the process of policy optimization, these rich feature information can be used to further improve the accuracy and robustness of the control strategy. The design of the feature mapping function fully considers the nonlinear relationship between state and action variables. Through nonlinear transformation, the agent can better capture and understand these relationships, thereby achieving more accurate and efficient flow control in a complex and dynamic pipeline network environment. In the urban pipeline flow optimization control system, the accuracy and robustness of flow control are key indicators of system performance. By introducing the above feature mapping functions, the agent can better understand and process the complex relationship between state and action, thereby improving the accuracy and robustness of flow control. For example, in the face of drastic flow fluctuations or pressure changes, the agent can quickly adjust the control strategy through the high-dimensional features extracted by the feature mapping function to ensure the stable operation of the system. In addition, this feature mapping method is also highly scalable and adaptable. The design of the feature mapping function can be adjusted according to specific application requirements to adapt to different types of pipeline networks and control tasks. For example, for pipeline networks with more complex dynamic behaviors, more nonlinear transformations can be introduced or the dimension of the feature mapping function can be increased to capture more detailed state and action relationships.

[0052] Example 5: The reward function is expressed using the following formula:

[0053] ;

[0054] in, Indicates The set of neighbors of an agent; is the maximum energy consumption of all agents; Indicates The mutual information between the states and actions of each agent; For the The amount of change in the actions of each agent; Represents calculation strategy The entropy of .

[0055] Specifically, the core of the reward function is to measure the difference between the current flow of the agent and the benchmark flow. The combination of the norm can accurately reflect the impact of flow error on rewards. Specifically, when the actual flow of the agent Close to the preset reference flow When the flow rate is large, the value of the exponential function is large, so the reward value is high; conversely, when the flow rate difference is large, the value of the exponential function decreases rapidly, resulting in a decrease in the reward. This design can encourage the agent to continuously adjust its control strategy so that the actual flow rate is close to the benchmark flow rate, thereby achieving precise control. In addition, by calculating the sum of the squares of the flow errors of other agents in the neighbor set, the differences between individuals are further smoothed, promoting the collaborative optimization of the entire system. Secondly, energy consumption is one of the key factors in flow control. The present invention achieves this by combining the maximum energy consumption of all agents The energy consumption of the current agent By comparison, an energy consumption-related reward item is designed. The reward item in this part is realized by taking the logarithm of the maximum energy consumption and performing a ratio operation with the current energy consumption. In this way, when the energy consumption of the intelligent agent is low, the denominator is small, so the reward is higher; on the contrary, when the energy consumption is high, the denominator increases and the reward decreases. This design effectively guides the intelligent agent to optimize energy consumption while controlling the flow, thereby achieving the purpose of energy saving. Third, pressure control is also crucial in pipeline flow control. The present invention is achieved by The norm measures the difference between the agent's current pressure and the target pressure, and adds one to it and takes the inverse to calculate the reward. This method can intuitively reflect the impact of pressure error on the reward. When the difference between the pressure and the target pressure is small, The smaller the norm, the higher the reward value; conversely, when the difference is large, The norm increases and the reward value decreases. This part of the design enables the agent to optimize the flow while ensuring that the pressure in the pipeline is maintained within the preset target range, thereby ensuring the safety and stability of the system. In addition, the smoothness of action changes is also an important aspect of agent optimization. By introducing the action change of norm, and combined with policy entropy , the reward function can evaluate the change of the agent's strategy. Specifically, when the agent's action changes slightly, the amount of action change The norm is small, thereby increasing the reward value; conversely, when the action changes too much, the norm increases and the reward decreases. The introduction of policy entropy is processed by the hyperbolic tangent function and combined with the norm of the action change, so that the agent can smoothly adjust the action when making decisions, avoid drastic fluctuations, and improve the robustness and stability of the system. Finally, the mutual information measurement between the state and action of the agent is also an important part of the reward function. Mutual Information It represents the information gain of the agent taking a specific action in the current state. By comparing it with the maximum mutual information value in the neighbor set, the effectiveness of the agent's decision can be effectively evaluated. Through this information gain comparison, the reward function encourages the agent to choose the actions with the highest information gain in the current state in each decision-making process, thereby improving the overall decision-making efficiency of the entire system.

[0056] Embodiment 6: The pipeline network multi-agent model construction part defines the nonlinear flow dynamics equation of each pipeline segment using the following formula:

[0057] ;

[0058] in, For the The agent and The flow difference of each agent; For time; is the distance along the pipeline; For the The agent and The difference in the cross-sectional area of ​​the pipelines of the agents; For the The agent and The pressure difference of each agent; is the liquid density; is the acceleration due to gravity; For the The agent and The pipeline inclination difference of each agent; For the The agent and The friction coefficient of each agent is different; For the The agent and The baseline flow difference of each agent; is the dynamic viscosity of the liquid; is the elastic coefficient of the pipeline corresponding to all agents; is the vibration frequency of the pipeline corresponding to all intelligent agents.

[0059] Specifically, the equation describes the change of flow rate through partial derivatives in time and space. Over time Rate of change and distance along the route Rate of change It comprehensively reflects the changes in kinetic energy and potential energy of the fluid in the pipeline. Kinetic energy part It describes the relationship between the square of velocity and cross-sectional area during fluid flow, indicating the effect of velocity change on flow rate, and the potential energy part is related to pressure and density, reflecting the behavior of the fluid under different pressure conditions. These two parts work together to provide the agent with a comprehensive description of flow changes, enabling it to dynamically adjust the control strategy based on real-time flow and pressure data. The effect of gravity on flow is The expression reflects that is the acceleration due to gravity, is the pipe inclination. This term describes the effect of gravity on the flow at different inclination angles, which is particularly important for pipe systems with large height differences. By taking gravity into account, the agent can more accurately predict and control the flow, especially in urban pipe networks with complex terrain. The effect of friction on flow is Here, the friction coefficient and baseline flow The friction between the fluid and the pipe wall will cause energy loss, thus affecting the change of flow rate. Therefore, considering the friction factor in the intelligent control strategy can effectively reduce energy loss and improve the accuracy and efficiency of flow control. The dynamic viscosity of the liquid is calculated by The expression reflects the viscous diffusion effect of the fluid. This term describes the second-order derivative of the flow rate with space, which represents the diffusion characteristics of the viscous fluid in the pipe. Dynamic viscosity is an important parameter in fluid mechanics, which directly affects the flow characteristics and stability of the fluid. By considering dynamic viscosity, the intelligent agent can better predict the change of flow under different working conditions, thereby optimizing the control strategy. Finally, the elasticity and vibration effects of the pipeline are calculated by The expression shows that the elastic coefficient and vibration frequency The effect of pipeline on flow under vibration is described. In practical applications, pipelines will produce elastic deformation due to external vibration or internal pressure fluctuations, thereby affecting the flow characteristics of the fluid. By incorporating these factors into the flow dynamics equation, the agent can effectively adjust the control strategy under vibration environment to ensure the stability and accuracy of the flow. The multi-agent reinforcement learning system of the present invention can fully capture the dynamic characteristics of pipeline flow through the above-mentioned nonlinear flow dynamics equation, and provide a detailed and accurate flow prediction model for the agent. In practical applications, the agent uses this equation to quickly adapt to environmental changes and dynamically adjust its control strategy through real-time monitoring and data feedback, thereby realizing efficient and intelligent management of urban pipeline networks. For example, in actual operation, when the flow of a certain pipeline section suddenly increases, the agent can predict the future flow change trend through the flow dynamics equation, combined with the current flow, pressure and pipeline inclination data, and timely adjust the valve opening and pump speed to ensure the stability of the flow. At the same time, the agent can also evaluate the energy loss through friction and dynamic viscosity terms, optimize the control strategy, reduce unnecessary energy consumption, and improve the overall efficiency of the system.

[0060] Embodiment 7: The multi-agent reinforcement learning part continuously learns and updates the Q value of each agent based on the nonlinear flow dynamics equation through the following formula:

[0061] ;

[0062] in, For learning and updating The Q value of an agent represents Agents in state Take action The value of action; is the learning rate; For the Discount factor for each agent; To perform actions The next state after Possible actions in the next state; It is the preset neighbor influence factor, which controls the influence of the neighboring agent's Q value on the current agent's Q value update; For the Agents in state Current strategy under For the Agents are in state The target strategy below; Indicates the calculation of KL divergence; For the first The nonlinear flow dynamics equations of the pipeline section corresponding to each intelligent agent; For the Agents in state Take action action value.

[0063] Specifically, The value represents the expected reward of the agent taking an action in a specific state, and it is a key variable in reinforcement learning. The update formula of the value ensures that the agent can gradually optimize its control strategy in a dynamic environment by combining multiple factors such as immediate rewards, expected future returns, neighbor influence, and policy optimization. The various parts of the formula work together to enable each agent to balance current and long-term benefits while taking into account the behavior of neighboring agents and the overall operation of the system. In the update formula, Representative Agents in state Take action new This new The value is changed from old The learning rate consists of a value and a learning update based on the current state and action. Determine the new and old The weight distribution of the value in the update. A larger learning rate allows the agent to adapt to the new environment faster, while a smaller learning rate means a greater reliance on historical data and is suitable for running in a more stable environment. The direct reward obtained by the agent for taking an action in the current state. This reward function combines multiple factors such as the difference between the flow rate and the baseline flow rate, energy consumption, pressure difference, action change and smoothness. For example, when the action taken by the agent can make the current flow rate close to the baseline flow rate, the reward value is high; when the action causes the flow rate to deviate from the baseline flow rate, the reward value is low. Through this design, the agent can adjust its control strategy according to the immediate reward, so that it gradually approaches the optimal control target. Discount factor Used to balance the importance of immediate rewards and future rewards. By considering the maximum expected return of future states , the agent is able to choose actions that have high rewards not only in the current state, but also in the future state. The introduction of the discount factor enables the agent to balance short-term and long-term benefits when making decisions, thereby formulating a more globally optimal strategy.

[0064] Neighborhood influence factor It reflects the influence of the behavior of neighboring agents on the current agent. In practical applications, each agent in the pipeline system does not operate in isolation, and their behaviors and states will affect each other. The current agent can take into account the behavior of neighboring agents while optimizing its own control strategy, thus achieving better collaborative control effect. This collaborative mechanism not only improves the control accuracy of a single agent, but also improves the overall performance of the system globally. The introduction of nonlinear flow dynamics equations makes The value updating process can accurately reflect the dynamic changes of the actual pipeline flow. By considering the feedback of the nonlinear flow dynamics equation, the intelligent agent can update The difference between the current strategy and the target strategy can be comprehensively considered when the value is . It is used to measure the distance between the current strategy and the target strategy. When the current strategy of the agent is close to the target strategy, the KL divergence value is small, indicating that the strategy optimization effect is good; conversely, when the difference is large, the KL divergence value is large, indicating that the strategy needs to be further optimized. In the learning process of the agent, the introduction of KL divergence can guide the agent to continuously adjust its strategy so that it gradually approaches the optimal strategy. Nonlinear flow dynamics equation and target strategy value The difference between the two is measured by KL divergence, so that the agent can obtain important information about the direction of policy improvement at each step. Under the guidance of this feedback mechanism, the agent can continuously optimize its control strategy to make it more accurate and efficient. For example, in actual applications, when an agent detects that the difference between the pipeline flow and the baseline flow is large, it will The value update formula comprehensively considers the immediate return, future expected return, the influence of neighboring agents, and strategy optimization information, and dynamically adjusts its control strategy to reduce the flow error. At the same time, by considering the maximum value of neighboring agents By using the feedback of the value and nonlinear flow dynamics equations, the agents can achieve collaborative optimization in the local scope, thereby improving the overall performance of the system in the global scope. This multi-agent reinforcement learning method, through complex The value update mechanism enables each agent to continuously learn and adapt in a dynamic environment, thereby optimizing its control strategy. The combination of immediate rewards and future expected returns ensures that the agent can take into account long-term interests while pursuing short-term gains. The introduction of neighbor influence factors allows the agent to consider the behavior of other agents while optimizing its own behavior, thereby achieving better global collaborative control. The role of KL divergence in strategy optimization enables the agent to continuously adjust and optimize its strategy, gradually approaching the optimal control strategy.

[0065] Embodiment 8: The multi-agent optimization control part uses the following formula to continuously optimize the strategy of each agent through the policy gradient method, so that each agent approaches the optimal control strategy:

[0066] ;

[0067] in, is the policy objective function About strategy parameters The gradient of Indicated in strategy Expected value under represents the discounted accumulation over infinite time steps, is the discount factor, satisfying ; is the state action value function, which means that in the state Take action Expected cumulative rewards; is the state value function, which means that in the state Expected cumulative rewards under For strategy Entropy of Indicates the current strategy With the prior strategy The KL divergence between is the entropy regularization coefficient; is the KL divergence regularization coefficient, which controls the closeness between the current strategy and the prior strategy.

[0068] Specifically, in Example 8, the present invention continuously optimizes the strategy of each agent through the policy gradient method, so that it gradually approaches the optimal control strategy, thereby achieving efficient control of urban pipeline flow. The core of the policy gradient method is to calculate the policy objective function About strategy parameters Gradient , and use this gradient to update the policy parameters , thereby maximizing the expected cumulative reward. The application of this method in multi-agent reinforcement learning enables each agent to continuously adjust and optimize its control strategy based on its state to achieve the global optimal flow control effect. First, the policy gradient method determines the direction of the probability change of the agent taking a specific action in different states by calculating the gradient of the policy. Indicated in strategy The expected value under the strategy reflects the overall performance of the strategy under different scenarios by taking a weighted average of all possible states and actions under the strategy. By calculating the expected value, the agent can understand the average effect of its strategy in the long term, thus providing a basis for strategy optimization. In the formula, represents the discounted accumulation over infinite time steps, where is the discount factor, satisfying The role of the discount factor is to balance the importance of immediate rewards and future rewards, ensuring that the agent not only focuses on current benefits but also considers potential future benefits when optimizing the control strategy. Through discount accumulation, the agent can evaluate the cumulative rewards of each future time step under the current strategy, and thus select actions that can bring high rewards in the present and high returns in the future. Represents the policy function About strategy parameters By calculating the logarithmic gradient, the agent can determine that at a particular state Next, take action The probability change direction and magnitude of . This part of the calculation ensures that the direction of the strategy update is towards increasing the expected cumulative reward, thereby gradually approaching the optimal strategy. In the formula, and They represent the state action value function and the state value function respectively. Reflects the status Take action The expected cumulative reward that can be obtained, and the state value function It means that in the state By calculating and The agent can evaluate the pros and cons of the current action relative to the average strategy and adjust the strategy to make it more in line with the direction of the optimal strategy. Representation strategy The gradient of the entropy, is the entropy regularization coefficient. Entropy reflects the randomness of the strategy. By increasing the entropy of the strategy, the agent can be prevented from falling into a suboptimal solution. The introduction of entropy regularization allows the strategy update process to maintain a certain degree of exploration, ensuring that the agent can fully explore the state space during the optimization process and avoid local optimality. In addition, Indicates the current strategy With the prior strategy The gradient of the KL divergence between is the KL divergence regularization coefficient. KL divergence measures the difference between the current strategy and the prior strategy. Through the regularization term, the amplitude of the strategy update can be controlled to avoid drastic changes in the strategy. The introduction of KL divergence regularization helps the strategy to remain smooth and stable during the update process and gradually approach the optimal strategy. In practical applications, multi-agent optimization control systems continuously adjust and optimize the control strategy of each agent through the above-mentioned policy gradient method. For example, when the flow in a certain pipeline section fluctuates abnormally, the agent can use the policy gradient method to adjust the valve opening or pump speed according to real-time monitoring data to stabilize the flow. At the same time, by considering the behavior of other agents and the overall operating status of the system, the agent can achieve collaborative control on a global scale and improve the overall performance of the system. Through the application of the policy gradient method, the agent can not only make the best decision in the current state, but also consider the potential benefits of the future state, thereby achieving global optimization in the long run. By continuously updating the policy parameters , the intelligent agent can adapt to the dynamically changing environment, optimize its control strategy, and achieve efficient and accurate flow control.

[0069] Embodiment 9: Agents at time step The learning rate Update using the following formula:

[0070] ;

[0071] in, For the updated The learning rate for each time step.

[0072] Specifically, the update formula of the adaptive learning rate dynamically adjusts the learning rate by combining the gradient norm of the current policy parameter and the norm of the policy parameter itself. The core idea of ​​this formula is: when the policy optimization progresses quickly, increase the learning rate to accelerate convergence; when the policy optimization progresses slowly or the policy tends to be stable, reduce the learning rate to avoid over-adjustment, thereby maintaining the stability of the system. Represents the gradient of the policy objective function with respect to the policy parameters The square of the norm reflects the degree of change in the current policy optimization. A large gradient norm means that the policy parameters are undergoing significant adjustments. In this case, increasing the learning rate can accelerate the optimization of the policy. On the contrary, a small gradient norm means that the policy adjustment tends to be smooth. In this case, reducing the learning rate can help prevent the system from falling into oscillation, thereby ensuring the stability of the optimization process. In addition, in the formula Represents the policy parameters Norm squared, this term is used to standardize the gradient norm in order to adjust the learning rate more reasonably. Through this standardization, the adjustment of the learning rate can more accurately reflect the changes in the policy parameters and avoid improper adjustment of the learning rate due to different parameter scales. The formula also contains a constant term , which is used to control the sensitivity of learning rate adjustment. By setting the appropriate The value can balance the dynamic adjustment of the learning rate so that it can respond to the progress of strategy optimization without causing frequent learning rate fluctuations due to excessive sensitivity. Specifically, The selection of the value should be adjusted according to the dynamic characteristics of the system in practical applications to ensure the stability and adaptability of the learning rate adjustment. In practical applications, this adaptive learning rate update mechanism can significantly improve the optimization efficiency and stability of the multi-agent reinforcement learning system. For example, in the urban pipeline flow control system, each agent needs to continuously adjust its control strategy to cope with complex flow changes and dynamic environments. By introducing an adaptive learning rate, each agent can respond to environmental changes more quickly and dynamically adjust the learning rate according to the progress of the current policy optimization, thereby achieving faster policy optimization and more robust system control. The advantage of the adaptive learning rate update mechanism lies in its flexibility and adaptability. In a multi-agent system, different agents may face different environmental changes and optimization requirements. By dynamically adjusting the learning rate, the agents can adaptively adjust the optimization pace according to their respective situations, thereby improving the coordination and optimization effect of the overall system. Especially when facing a complex urban pipeline network, this mechanism can effectively balance the optimization speed of each agent, avoiding system instability caused by too fast adjustment of some agents, or inefficiency in optimization caused by too slow adjustment of some agents.

[0073] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that these specific embodiments are only illustrative, and those skilled in the art may omit, replace, and change the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, merging the above method steps so as to perform substantially the same functions in substantially the same manner to achieve substantially the same results is within the scope of the present invention. Therefore, the scope of the present invention is limited only by the appended claims.

Claims

1. Urban pipeline flow optimization control system based on multi-agent reinforcement learning, characterized by: It includes: Pipeline network multi-agent model construction part, multi-agent reinforcement learning part and multi-agent optimization control part; The pipeline network multi-agent model construction part is used to divide the urban pipeline network into multiple pipeline segments, assign an agent to each pipeline segment, and each agent is described by a set consisting of a state space, an action space, a reward function and a strategy; define the state space, action space, strategy and reward function of each agent; define the nonlinear flow dynamics equation of each pipeline segment; The multi-agent reinforcement learning part is used to continuously learn and update the Q value of each agent based on the nonlinear flow dynamics equation; The multi-agent optimization control part is used to continuously optimize the strategy of each agent through a policy gradient method so that each agent approaches the optimal control strategy; The pipeline network multi-agent model construction part allocates the agent to each pipeline segment as follows: ;in, It is An intelligent agent, is the state space, is the action space, is the reward function, is the discount factor, is the strategy; the state space ;in, It's traffic. It's pressure. is the fluid level, Base flow, is energy consumption; action space ;in, is the valve opening adjustment, is the pump frequency adjustment amount; The strategy of each agent in the pipeline network multi-agent model construction part is constructed Use the following formula to express it: ； in, are the preset strategy parameters; is the preset bias vector; Represents all possible actions and is used to normalize the probability distribution; Indicates in status Next select action probability; is the first feature mapping function; is the second feature mapping function; The first feature mapping function Use the following formula to express it: ； The second feature mapping function Use the following formula to express it: ； in, is the transpose operation of the vector; is the L2 norm operation; is the L1 norm operation; is a symbolic function; The reward function is expressed as follows: ； in, Indicates The set of neighbors of an agent; is the maximum energy consumption of all agents; Indicates The mutual information between the states and actions of each agent; For the The amount of change in the actions of each agent; Represents calculation strategy Entropy of The pipeline network multi-agent model construction part defines the nonlinear flow dynamics equation of each pipeline segment using the following formula: ； in, For the The agent and The flow difference of each agent; For time; is the distance along the pipeline; For the The agent and The difference in the cross-sectional area of ​​the pipelines of the agents; For the The agent and The pressure difference of each agent; is the liquid density; is the acceleration due to gravity; For the The agent and The pipeline inclination difference of each agent; For the The agent and The friction coefficient of each agent is different; For the The agent and The baseline flow difference of each agent; is the dynamic viscosity of the liquid; is the elastic coefficient of the pipeline corresponding to all agents; is the vibration frequency of the pipeline corresponding to all intelligent agents; The multi-agent reinforcement learning part continuously learns and updates the Q value of each agent based on the nonlinear flow dynamics equation through the following formula: ； in, For learning and updating The Q value of an agent represents Agents in state Take action The value of action; is the learning rate; For the Discount factor for each agent; To perform actions The next state after Possible actions in the next state; It is the preset neighbor influence factor, which controls the influence of the neighboring agent's Q value on the current agent's Q value update; For the Agents in state Current strategy under For the Agents are in state The target strategy below; Indicates the calculation of KL divergence; For the first The nonlinear flow dynamics equations of the pipeline section corresponding to each intelligent agent; For the Agents in state Take action The value of action; The multi-agent optimization control part uses the following formula to continuously optimize the strategy of each agent through the policy gradient method, so that each agent approaches the optimal control strategy: ； in, is the policy objective function About Strategy Parameters The gradient of Indicated in strategy Expected value under represents the discounted accumulation over infinite time steps, is the discount factor, satisfying ; is the state action value function, which means that in the state Take action Expected cumulative rewards; is the state value function, which means that in the state Expected cumulative rewards under For strategy Entropy of Indicates the current strategy With the prior strategy The KL divergence between is the entropy regularization coefficient; is the KL divergence regularization coefficient, which controls the closeness between the current strategy and the prior strategy. Agents at time step The learning rate Update using the following formula: ； in, For the updated The learning rate for each time step; is a constant term used to control the sensitivity of learning rate adjustment.

Images