Sluice group scheduling method based on QMIX framework

Abstract

The invention discloses a QMIX framework-based sluice group scheduling method. The method comprises the following steps of: constructing a simulation basin environment by utilizing a historical environment data set; constructing a multi-agent reinforcement learning method for cooperative sluices in the watershed based on a QMIX framework to realize dynamic cooperation and global optimization of the sluices in the watershed; designing a reinforcement learning single-agent DQN model of a single sluice, and supporting autonomous perception and decision of the single sluice; designing a multi-agent global reward function and a single-agent independent reward function, and balancing a group collaboration target and individual local requirements through a layered reward mechanism; a mixed experience playback mechanism based on high-priority experience similarity matching is introduced, and the problems that a traditional experience playback method is low in sample utilization efficiency and weak in key event generalization ability in a complex dynamic environment are solved. According to the method, the water gate intelligent agent can fully explore the scheduling action space in the compliance framework, and the robustness and the control accuracy of the water gate scheduling strategy under the QMIX framework are improved.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning and sluice gate scheduling, specifically involving a sluice gate group scheduling method based on the QMIX (Q-value Mixing Network, monotonic value function decomposition algorithm) framework. Background Technology

[0002] Water resources management refers to comprehensive activities aimed at flood control and disaster reduction, water supply security, ecological maintenance, and sustainable development through scientific planning, rational scheduling, and effective monitoring of water resources in river basins. Against the backdrop of intensifying climate change and frequent human activities, watershed hydrological processes are increasingly exhibiting non-stationarity and uncertainty. Frequent heavy rainfall and significant seasonal fluctuations often lead to rapid increases in upstream runoff, resulting in strong non-linear coupling between factors such as water level and flow, posing significant challenges to scheduling decisions. Currently, sluice gate control in most river basins still relies on manual experience or fixed scheduling rules, with operators making gate opening and closing decisions based on historical data or instructions from the control center. Faced with sudden changes in hydrological processes, this model is slow to respond, struggles to balance flood control safety and water supply needs, and fails to effectively integrate multi-source heterogeneous data such as meteorological forecasts and real-time monitoring, thus restricting the timeliness and adaptability of scheduling.

[0003] In recent years, with the development of artificial intelligence technology, reinforcement learning has shown significant advantages in decision optimization of complex systems, providing a new approach for the adaptive scheduling of sluice gates. Through continuous interaction with the environment, reinforcement learning models can integrate multi-source information in real time and autonomously optimize gate control strategies, demonstrating greater flexibility and robustness in dealing with non-stationary hydrological conditions. Summary of the Invention

[0004] Purpose of the invention: To address the shortcomings of existing technologies, this invention proposes a sluice gate group scheduling method based on the QMIX framework, which can more intelligently find the optimal decision for sluice gate scheduling and adapt quickly to different environments.

[0005] Technical solution: The present invention provides a sluice gate group scheduling method based on the QMIX framework, comprising the following steps:

[0006] (1) Using hydrological prediction models, a simulated watershed environment that conforms to flow scheduling is constructed based on historical datasets;

[0007] (2) Based on the simulation environment, the QMIX multi-agent reinforcement learning method is designed. By combining local Q network and centralized hybrid network, a centralized training and decentralized execution framework that supports individualized reward modeling, global collaborative optimization and high-value experience matching is constructed.

[0008] (3) Within the QMIX framework, based on the standard specifications and actual needs of watershed scheduling, a single agent network structure conforming to the scheduling standard and an independent reward function conforming to the scheduling standard are built to support the autonomous perception and decision-making of a single gate.

[0009] (4) A multi-agent structure is built using a hybrid network, and a global reward function that conforms to scheduling standards is used;

[0010] (5) Design a hybrid experience replay mechanism based on high-priority experience similarity matching to deal with rare sudden rainfall events, so that the sluice gates in the basin can be scheduled more adaptively.

[0011] Furthermore, the implementation process of step (1) is as follows:

[0012] Based on a unified timestamp, and combined with the station number and sluice gate number, spatiotemporal pairing of water condition, rainfall condition and engineering condition data is realized;

[0013] Constructing multidimensional state feature vectors of the dataset , For the water level at the monitoring station, The water level upstream of the sluice gate. The water level downstream of the sluice gate. The number of openings in the sluice gate. The height of the sluice gate opening. The flow rate of the sluice gate. It is raining;

[0014] Using a pre-trained hydrological prediction model, based on the current time State symbol vectors and scheduling actions and the resulting sluice gate flow Predicting the next moment Key water level variables , , ;

[0015] Combining the predicted output with the actual rainfall at the next moment Construct the next state Forming state transition samples Used to construct the state transition matrix of the environment This completes the formal modeling of the transition from physical processes to Markov decision processes.

[0016] Furthermore, the flow rate of the sluice gate for:

[0017]

[0018] in, This is the current sluice gate flow rate. It is an empirical constant. and It is an empirical index used to correct for the impact of flow under different conditions. and It refers to the water levels upstream and downstream of the sluice gate. It refers to the number of openings in the sluice gate. It is the height of the sluice gate opening.

[0019] Furthermore, the implementation process of step (2) is as follows:

[0020] Within the basin A sluice gate with scheduling function is modeled as follows: Individual intelligent agents constitute a multi-agent system. Each intelligent agent At each time step The system receives a hybrid input consisting of local private observations and shared global observations, where the local private observations include upstream water levels. and downstream water level Shared global observations include water levels at target stations. and average rainfall across the entire basin intelligent agent At any moment The complete observation vector is defined as:

[0021]

[0022] intelligent agent Execute action Action space For discrete sets, , For the first The number of openings in each sluice gate is taken as a value. to non-negative integers, For the first The opening height of each opening of a sluice gate takes values ​​from a finite discrete set. ,and ;

[0023] Establish global state Used for joint strategy optimization; during the execution phase, each agent... Relying solely on its locally observable information Independent decision-making, requiring no real-time communication or centralized coordination; among which, global state The observations and environmental variables encompassing all agents are defined as follows:

[0024]

[0025] in, They represent the first Each sluice gate at any time The upstream and downstream water levels; They are time points The target station water level and the average precipitation in the basin;

[0026] For each sluice gate intelligent agent Construct a deep Q network As its local policy network, the parameters are The network relies on local observations by the agents. The input is the corresponding optional scheduling action, and the output is the corresponding action. Action value estimation;

[0027] Design a centralized QMIX hybrid network This is used during the training phase to fuse the local Q-values ​​of each agent into a global Q-value for joint actions; the hybrid network receives the local Q-values ​​of all agents. As input, and with global state Given the condition, a global action value is generated through a learnable nonlinear transformation:

[0028]

[0029] in, For joint operations, For hybrid network parameters;

[0030] A temporal difference learning method based on deep Q-networks is adopted, combined with the joint value function of the QMIX framework, to construct a multi-agent collaborative optimization objective. By minimizing the error between the predicted global Q-value and the target Q-value, the parameters of the local Q-network and the QMIX hybrid network of each agent are jointly updated to achieve continuous policy optimization.

[0031] Building a shared experience pool It is used to store transfer samples generated by the interaction between the agent and the environment.

[0032] Furthermore, the implementation process of step (3) is as follows:

[0033] Based on the operation and management documents of sluice gates in different river basins, obtain the first... The following parameters for each sluice gate during the current scheduling cycle include: the target water level range. , To allow the minimum water level, Maximum allowable water level; current actual water level Current number of openings Number of holes opened at the previous time point Current single-hole opening height Started at a higher altitude than the previous moment ;

[0034] For each sluice gate intelligent agent Design independent and collaborative independent reward functions:

[0035]

[0036] in, Indicates the first Each sluice gate at any time The partial reward obtained Indicates the water level deviation of the corresponding sluice gate Changes in the number of open gates and changes in gate opening height The weighting coefficient; the current actual water level is defined as... According to the scheduling document, the sluice gate controls the target water level range. The water level deviation penalty is as follows:

[0037]

[0038] in, It can be dynamically adjusted according to different scheduling standards; and These are the current and previous time steps, respectively. The number of openings of each sluice gate, then Represented as:

[0039]

[0040] Simultaneously, a penalty item for adjusting the gate opening height is introduced. :

[0041]

[0042] in, These are the current and previous time steps, respectively. The single-span opening height of a sluice gate;

[0043] Using local deep Q-networks Combined with the independent rewards currently obtained Calculate the time-difference objective:

[0044]

[0045] in, As a discount factor, For target network parameters;

[0046] By minimizing the loss function:

[0047]

[0048] For local Q network parameters Gradient descent updates are performed to gradually optimize the local scheduling strategies of each agent.

[0049] Furthermore, the aforementioned It is a positive value. and Negative values ​​constitute a penalty-based reward mechanism, with weights ordered by magnitude. This reflects the control hierarchy; it prioritizes water level adjustment, minimizes gate switching, and allows for smooth opening and adjustment.

[0050] Furthermore, the implementation process of step (4) is as follows:

[0051] Value of global actions Apply monotonicity constraints:

[0052]

[0053] Ensure that an increase in the local Q-value of any agent does not lead to a decrease in the global Q-value;

[0054] Construct a system-level global reward function :

[0055]

[0056] in, Indicates at time The global reward value; For the first Local rewards for individual sluice gate intelligent agents; The total number of sluice gates participating in the coordinated scheduling;

[0057] A batch of transfer samples is sampled from the experience replay pool, based on the global reward. Constructing a temporal difference objective:

[0058]

[0059] in, As a discount factor, Target hybrid network parameters;

[0060] Minimize the QMIX loss function:

[0061]

[0062] The parameters of the QMIX hybrid network are jointly updated using gradient descent. and the local Q-network parameters of all agents Every Step 1: Softly update the current network parameters to the target network.

[0063] .

[0064] Furthermore, the implementation process of step (5) is as follows:

[0065] During the interaction between the intelligent agent and the simulated watershed environment, the experience tuples generated at each time step Stored in the shared experience pool ; for each new experience tuple that interacts with the environment Initialize a base priority The calculation is based on the absolute value of the TD-error corresponding to this experience:

[0066]

[0067] in, As a discount factor, For the target network parameters, It is a very small constant used to ensure that all experiences have a non-zero sampling probability;

[0068] Dynamically maintain a fixed capacity High-priority experience reference set When from the experience pool Sampling an experience At that time, if its Ranked first in the current sampling batch or historical records If the experience is deemed "high-value experience", it will be added to the reference set. Reference set Updates are performed using a first-in-first-out strategy or a mechanism based on the least priority elimination to ensure that they always contain the most recent key events with the greatest learning value.

[0069] For any empirical tuple Define its state feature vector To be from its state Numerical vectors composed of key hydrological features extracted and normalized; in the scenario of watershed sluice gate regulation. This includes the water level at the monitoring station, the water level upstream of the sluice gate, the water level downstream of the sluice gate, and the rainfall; calculate its state feature vector. With reference set Each high-priority experience similarity The similarity metric uses a Gaussian kernel function.

[0070]

[0071] in, The bandwidth parameter is used to adjust the sensitivity of similarity to changes in state differences; finally, the maximum value among all similarity values ​​is taken as the bandwidth parameter. Similarity measurement results:

[0072]

[0073] Based on basic priority and similarity gain Calculation experience Mixed priority :

[0074]

[0075] in, This is the similarity gain coefficient, used to adjust the strength of the influence of similarity on priority.

[0076] During the training phase of the DQN network, from the experience pool Sampling a size of Small batch data; sampling probability With mixed priority Proportional:

[0077]

[0078] in, The importance sampling coefficients are used; simultaneously, the importance sampling weights are calculated for each sampling experience. Used in the weighted loss function to correct sampling bias:

[0079]

[0080] The agent uses this small batch of data and weights. Update its Q network parameters After multiple rounds of training and convergence, the QMIX algorithm outputs an optimal cooperative policy network.

[0081] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: This invention utilizes a support vector regression model to construct a simulated watershed environment that conforms to flow scheduling based on historical datasets, thereby significantly shortening the training interaction time of the intelligent agent; Based on the specifications and actual needs of watershed scheduling, this invention designs a global reward function for multiple agents and an independent reward function for a single agent that conforms to scheduling standards, clarifies the reward and punishment rules when the agent performs different scheduling actions, and adjusts the parameter combination of the agent to guide the agent to learn the optimal scheduling strategy; This reward mechanism comprehensively considers multiple dimensions such as water level deviation control, gate operation frequency suppression, and operation stability constraints, prioritizing ensuring that the water level of key sections is within the safe threshold, while encouraging the reduction of unnecessary gate opening and closing actions, avoiding the impact of frequent adjustments on structural safety and water flow stability, thereby achieving the scheduling goal of "precise adjustment and stable operation"; and it can adapt to the scheduling priorities and operational constraints of different watersheds; This invention designs a hybrid experience replay mechanism based on high-priority experience similarity matching to cope with rare sudden rainfall events, enabling the sluice gates in the watershed to be scheduled more adaptively, thereby avoiding model convergence problems caused by sparse time and enhancing the generalization ability of traditional priority-based experience replay mechanisms. Attached Figure Description

[0082] Figure 1 This is a flowchart of the present invention;

[0083] Figure 2 This is a schematic diagram of the single sluice gate intelligent agent structure proposed in this invention;

[0084] Figure 3 This is a schematic diagram of the multi-sluice gate intelligent agent structure proposed in this invention. Detailed Implementation

[0085] The present invention will now be described in further detail with reference to the accompanying drawings.

[0086] like Figure 1 As shown, this invention proposes a sluice gate group scheduling method based on the QMIX framework, which mainly includes the following steps:

[0087] Step 1: Using a hydrological prediction model (support vector regression model), construct a simulated watershed environment that conforms to flow scheduling based on the historical dataset.

[0088] Based on a unified timestamp, and combined with station numbers and sluice gate numbers, spatiotemporal alignment and correlation matching of water conditions, rainfall and engineering data are achieved, and a dataset for the Qinhuai River Basin is constructed.

[0089] Utilizing the water level upstream of the sluice gates in the Qinhuai River basin ( ), downstream water level of the sluice gate ( ), Number of sluice gate openings ( ), sluice gate opening height ( ) and other calculation flow As shown below:

[0090]

[0091] in, This is the current sluice gate flow rate. It is an empirical constant. and It is an empirical index used to correct for the impact of flow under different conditions. and It refers to the water levels upstream and downstream of the sluice gate. It refers to the number of openings in the sluice gate. It is the height of the sluice gate opening.

[0092] Constructing multidimensional state feature vectors of the dataset ,in For the water level at the monitoring station, The water level upstream of the sluice gate. The water level downstream of the sluice gate. The number of openings in the sluice gate. The height of the sluice gate opening. The flow rate of the sluice gate. It is due to rainfall.

[0093] With the current data in the dataset Water level at the station at that time ( ), upstream water level of the sluice gate ( ), downstream water level of the sluice gate ( Rainfall during the period () ) and sluice gate flow ( Using the input features, a prediction model is used to predict the next time step. Upstream water level of the sluice gate ( ), downstream water level of the sluice gate ( ) and station water level ( The prediction model employs Support Vector Regression (SVR) for each scheduling cycle. Predicting the next moment Key hydrological variables. Specifically, this includes the following sub-steps:

[0094] The SVR basic model is constructed by mapping input features to a high-dimensional feature space to create an optimal regression hyperplane that minimizes prediction error. Specifically, the SVR model solves the following optimization problem:

[0095]

[0096]

[0097] in, This is the weight vector; For bias terms; This is the total number of training samples; It is a nonlinear mapping function; This is a regularization parameter that controls the trade-off between model complexity and training error. To avoid insensitive bandwidth loss; These are slack variables.

[0098] To capture the nonlinear dynamic relationships in the hydrological system, radial basis functions (RBF) are selected as the kernel function:

[0099]

[0100] in, Control the model's sensitivity to local changes; They are the first The input feature vector of each training sample.

[0101] At the current moment water level at the monitoring station Downstream water level of the sluice gate Rainfall during a given period and current scheduling actions and sluice gate flow Using these input features, the first SVR prediction sub-model is constructed to predict the next time step. Upstream water level of the sluice gate .

[0102] At the current moment water level at the monitoring station Upstream water level of the sluice gate Rainfall during a given period and current scheduling actions and sluice gate flow Using these features as input, a second SVR prediction sub-model is constructed to predict the next time step. Downstream water level of the sluice gate .

[0103] At the current moment Upstream water level of the sluice gate Downstream water level of the sluice gate Rainfall during a given period and current scheduling actions and sluice gate flow Using these input features, a third SVR prediction sub-model is constructed to predict the next time step. water level at the monitoring station .

[0104] Combining the predicted output with the actual rainfall at the next moment Construct the next state Forming state transition samples Used to construct the state transition matrix of the environment This completes the formal modeling of the physical process into a Markov decision process (MDP).

[0105] Step 2: Based on the simulation environment, design the QMIX multi-agent reinforcement learning method. By combining local Q-networks and centralized hybrid networks, construct a centralized training and decentralized execution framework that supports individualized reward modeling, global collaborative optimization and high-value experience matching, so as to realize intelligent collaborative scheduling of sluice gate groups.

[0106] Within the Qinhuai River Basin A water gate with discretization scheduling function is modeled as follows: Individual intelligent agents constitute a multi-agent system. Each intelligent agent At each time step The system receives a hybrid input consisting of local private observations and shared global observations, where the local private observations include upstream water levels. and downstream water level Shared global observations include water levels at target stations. and average rainfall across the entire basin Therefore, intelligent agents At any moment The complete observation vector is defined as:

[0107]

[0108] intelligent agent Execute action The action space For discrete sets, ,in For the first The number of openings in each sluice gate is taken as a value. to non-negative integers, For the first The opening height (in meters) of each gate opening of a sluice gate represents the opening height of each gate opening. To simplify the motion space, it is discretized, with the discrete value set to a step size of . The gate opening height value.

[0109] Adopting a learning paradigm of "centralized training and decentralized execution," the global state is accessed during the training phase. Used for joint strategy optimization; during the execution phase, each agent... Relying solely on its locally observable information Independent decision-making, requiring no real-time communication or centralized coordination. This includes global state. The observations and environmental variables encompassing all agents are defined as follows:

[0110]

[0111] in, They represent the first Each sluice gate at any time The upstream and downstream water levels; They are time points The target station water level and the average precipitation in the basin.

[0112] For each sluice gate intelligent agent Construct a deep Q network As its local policy network, the parameters are The network relies on local observations by the agents. The input is the corresponding optional scheduling action, and the output is the corresponding action. The value of the action is estimated.

[0113] Design a centralized QMIX hybrid network This is used during the training phase to fuse the local Q-values ​​of each agent into a global Q-value for joint actions. The hybrid network receives the local Q-values ​​of all agents. As input, and with global state Given the condition, a global action value is generated through a learnable nonlinear transformation:

[0114]

[0115] in For joint operations, These are mixed network parameters.

[0116] A temporal difference learning method based on deep Q-networks (DQN) is employed, combined with the joint value function of the QMIX framework, to construct a multi-agent collaborative optimization objective. By minimizing the error between the predicted global Q-value and the target Q-value, the parameters of the local Q-networks and the QMIX hybrid network of each agent are jointly updated to achieve continuous policy optimization.

[0117] Building a shared experience pool It is used to store transfer samples generated by the interaction between the agent and the environment.

[0118] Step 3: As Figure 2As shown, within the QMIX framework, a single-agent network structure conforming to the scheduling standard and actual needs is constructed, along with a corresponding independent reward function conforming to the scheduling standard, based on the specifications and actual requirements of watershed scheduling.

[0119] Based on the operation and management documents of sluice gates in different river basins (such as the Wudingmen Sluice Gate in the Qinhuai River Basin), obtain the first... The following parameters for each sluice gate during the current scheduling cycle include: the target water level range. , To allow the minimum water level, To allow the highest water level, in accordance with the Qinhuai River Basin dispatching documents, the water level at Dongshan Station is controlled within the prescribed range under three standards: flood season / plum rain season, flood season / non-plum rain season, and non-flood season; the current actual water level... The current number of openings is obtained in real time by sensors or hydrological models. Number of holes opened at the previous time point Current single-hole opening height Started at a higher altitude than the previous moment Among them, the target water level range The function can be dynamically adjusted according to scheduling standards such as flood season / non-flood season, flood control / water storage mode, and ecological flow requirements to ensure that the reward function adapts to different operating conditions.

[0120] For each sluice gate intelligent agent Design independent and collaborative independent reward functions:

[0121]

[0122] in, Indicates the first Each sluice gate at any time The partial reward obtained Indicates the water level deviation of the corresponding sluice gate ( ), Changes in the number of open gates ( ) and changes in gate opening height ( The weighting coefficients of ). It is a positive value. and Negative values ​​constitute a penalty-based reward mechanism, where weights are ranked in descending order. This reflects the control hierarchy: prioritizing water level adjustment, minimizing gate switching, and allowing for smooth opening and adjustment. The current actual water level is defined as... According to the scheduling document, the sluice gate controls the target water level range. The water level deviation penalty is as follows:

[0123]

[0124] in, It can be dynamically adjusted according to different scheduling standards such as flood season, non-flood season, flood control and drainage, or water storage and conservation; and These are the current and previous time steps, respectively. The number of open gates of a sluice gate is represented as follows:

[0125]

[0126] Simultaneously, a penalty item for adjusting the gate opening height is introduced. :

[0127]

[0128] in, These are the current and previous time steps, respectively. The single-span opening height of the sluice gate. (This is achieved by introducing...) and The reward function penalizes unnecessary gate opening adjustments and suppresses sudden or large-scale operations, thereby promoting smoother control behavior while reducing mechanical stress and maintenance requirements.

[0129] At each training step, the local deep Q-network constructed in step 2 is utilized. Combined with the independent rewards currently obtained Calculate the time-difference objective:

[0130]

[0131] in, As a discount factor, These are the target network parameters.

[0132] By minimizing the loss function:

[0133]

[0134] For local Q network parameters Gradient descent updates are performed to gradually optimize the local scheduling strategies of each agent.

[0135] Step 4: As Figure 3 As shown, a multi-agent structure is built using a hybrid network, along with a corresponding global reward function that conforms to the scheduling criteria.

[0136] Value of global actions Apply monotonicity constraints:

[0137]

[0138] That is, an increase in the local Q value of any agent should not lead to a decrease in the global Q value.

[0139] To guide all agents in jointly optimizing the overall scheduling performance of the watershed, a system-level global reward function is constructed. Linear aggregation is performed based on the independent rewards of each agent in step 3:

[0140]

[0141] in, Indicates at time The global reward value; For the first The individual rewards for each sluice gate agent include tiered penalties for water level deviation, changes in the number of openings, and adjustments to the opening height; The total number of sluice gates participating in the coordinated scheduling.

[0142] In each training step, a batch of transfer samples is sampled from the experience replay pool, based on the global reward. Constructing a temporal difference objective:

[0143]

[0144] in, As a discount factor, The target is to mix network parameters.

[0145] Minimize the QMIX loss function:

[0146]

[0147] The parameters of the QMIX hybrid network are jointly updated using gradient descent. and the local Q-network parameters of all agents Every Step 1: Softly update the current network parameters to the target network.

[0148] .

[0149] Step 5: Design a hybrid experience replay mechanism based on high-priority experience similarity matching to cope with rare and sudden rainfall events, so that the sluice gates in the basin can be scheduled more adaptively.

[0150] During the interaction between the intelligent agent and the simulated watershed environment, the experience tuples generated at each time step Stored in the shared experience pool Each empirical tuple Initialize a base priority The calculation is based on the absolute value of the TD-error corresponding to this experience:

[0151]

[0152] in, As a discount factor, For the target network parameters, It is a very small constant used to ensure that all experiences have a non-zero sampling probability.

[0153] Dynamically maintain a fixed capacity High-priority experience reference set When from the experience pool Sampling an experience At that time, if its Ranked first in the current sampling batch or historical records (For example If the experience is deemed "high-value experience," it will be added to the reference set. Reference set Updates are performed using a first-in, first-out (FIFO) strategy or a mechanism based on least priority eviction to ensure that they always contain the most recent key events that are of greatest learning value.

[0154] For any empirical tuple Define its state feature vector To be from its state Numerical vectors composed of key hydrological features extracted and normalized. In the scenario of watershed sluice gate regulation, This includes the water level at the monitoring station, the water level upstream of the sluice gate, the water level downstream of the sluice gate, and rainfall. Calculate its state feature vector. With reference set Each high-priority experience similarity The similarity metric uses a Gaussian kernel function:

[0155]

[0156] in, This is a bandwidth parameter used to adjust the sensitivity of similarity to changes in state differences. Finally, the maximum value among all similarity values ​​is taken as the bandwidth parameter. Similarity measurement results:

[0157]

[0158] Based on basic priority and similarity gain Calculation experience Mixed priority :

[0159]

[0160] in, This is the similarity gain coefficient, used to adjust the strength of the influence of similarity on priority.

[0161] During the training phase of the DQN network, from the experience pool Sampling a size of Small batches of data. Sampling probability. With mixed priority Proportional:

[0162]

[0163] in, These are the importance sampling coefficients. Simultaneously, importance sampling weights are calculated for each sampling experience. Used in the weighted loss function to correct sampling bias:

[0164]

[0165] The agent uses this small batch of data and weights. Update its Q network parameters .

[0166] After multiple rounds of training and convergence, the QMIX algorithm outputs an optimal cooperative policy network, which is represented as follows: For any given global state:

[0167] This network is able to generate a set of optimal joint action instructions:

[0168]

[0169] Each sub-action .

[0170] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A water gate group scheduling method based on a QMIX framework, characterized in that, The method comprises the following steps: (1) using a hydrological prediction model, constructing a simulated river basin environment conforming to flow regulation for a historical data set; (2) designing a QMIX multi-agent reinforcement learning method based on the simulated environment, combining a local Q network with a centralized hybrid network to construct a centralized training and decentralized execution framework supporting individualized reward modeling, global collaborative optimization and high-value experience matching; (3) in the QMIX framework, according to the specification standards and actual requirements of river basin regulation, building a single-agent network structure conforming to the regulation standards and an independent reward function conforming to the regulation standards to support the autonomous perception and decision-making of a single gate; (4) using a hybrid network to build a multi-agent structure and a global reward function conforming to the regulation standards; (5) designing a hybrid experience replay mechanism based on high-priority experience similarity matching to cope with rare sudden rainfall events, so that the water gates in the river basin can adaptively regulate.

2. The water gate group scheduling method based on the QMIX framework according to claim 1, wherein, The implementation process of step (1) is as follows: Based on a unified timestamp, combining station numbers and gate numbers to realize the spatio-temporal matching of water regime, rainfall regime and work regime data; Constructing a multi-dimensional state feature vector for a dataset , For the water level of the station, For the water level upstream of the sluice, For the water level downstream of the sluice, For the number of openings of the sluice, For the height of the openings of the sluice, For the flow of the sluice, For the rainfall; Using a pre-trained hydrological prediction model, based on the state symbol vector at the current time and the scheduled action and resulting gate flow , the key water level variable at the next time is predicted , , ; combining the predicted output with the actual rainfall at the next time instant , constructing the next state , forming state transition samples , for constructing the state transition matrix of the environment , completing the formalization of the modeling from the physical process to the Markov decision process.

3. The water gate group scheduling method based on the QMIX framework according to claim 2, characterized in that, The sluice flow Is: wherein, is the current sluice flow, is an empirical constant, and is an empirical exponent for correcting the flow influence under different conditions, and are the water levels upstream and downstream of the sluice, is the number of openings of the sluice, is the height of the sluice openings.

4. The water gate group scheduling method based on the QMIX framework according to claim 1, characterized in that, The implementation process of step (2) is as follows: The watersheds are modeled as a number of sluices with scheduling functions as independent agents, constituting a multi-agent system; each agent receives a hybrid input at each time step , which is composed of local private observations and shared global observations, where the local private observations include the upstream water level and the downstream water level , and the shared global observations include the water level at the target station and the average rainfall over the whole watershed ; the complete observation vector of the agent at time is defined as: Agent Performing an action , action space is a discrete set, , is the number of openings of the nth gate, which is a non-negative integer, is the opening height of each opening of the nth gate, which is a finite discrete set ;​​​​​ Establish global state for federated policy optimization; in the execution phase, each agent only relies on its local observable information decides independently without real-time communication or central coordination; wherein the global state contains all agents' observations and environment variables is defined as: in, They represent the first Each sluice gate at any time The upstream and downstream water levels; They are time points The target station water level and the average precipitation in the basin; for each water gate agent build a deep Q network , as its local policy network, parameterized by ; this network takes the agent's local observation as input and outputs action value estimates for each of the available scheduling actions ; design a centralized QMIX mixing network for fusing local Q-values of individual agents into global Q-values for joint actions during training; the mixing network receives local Q-values of all agents as input and generates global action values through a learnable nonlinear transformation conditioned on the global state : wherein, is a joint action, is a mixed network parameter; Using a deep Q network-based time series difference learning method, combining the joint value function of the QMIX framework to construct a multi-agent collaborative optimization goal; by minimizing the error between the predicted global Q value and the target Q value, jointly updating the parameters of the local Q network and the QMIX hybrid network of each agent to realize continuous optimization of the strategy; Building a shared experience pool for storing transition samples resulting from the interaction of agents with the environment.

5. The water gate group scheduling method based on the QMIX framework according to claim 1, characterized in that, The implementation process of step (3) is as follows: According to the operation and management files of the water locks in different river basins, the following parameters of the first water lock in the current dispatching period are obtained, including: a dispatching target water level interval , , allowing a minimum water level, allowing a maximum water level; a current actual water level ; a current number of openings and a number of openings at a previous time; a current single opening opening height and an opening height at a previous time; for each water gate agent design independent and synergistic independent reward functions: in, Indicates the first Each sluice gate at any time The partial reward obtained Indicates the water level deviation of the corresponding sluice gate Changes in the number of open gates and changes in gate opening height The weighting coefficient; the current actual water level is defined as... According to the scheduling document, the sluice gate controls the target water level range. The water level deviation penalty is as follows: Wherein, It can be dynamically adjusted according to different scheduling standards; And N and N-1 respectively represent the number of open holes of the current and last time of the first gate, and It is represented as: At the same time, the opening height adjustment penalty term is introduced : in, These are the current and previous time steps, respectively. The single-span opening height of a sluice gate; Utilizing local deep q-networks in combination with the currently obtained independent reward computing a time-difference target: wherein, is a discount factor, is a target network parameter; By minimizing the loss function: Local q network parameters Gradient descent updates are performed to iteratively optimize the local scheduling policy of each agent.

6. The water gate group scheduling method based on the QMIX framework according to claim 5, characterized in that, The is positive, with is negative, constitutes a reward mechanism based on punishment, and the weight size is sorted , which embodies the control level; the water level is adjusted preferentially, the gate switching is minimized, and smooth opening adjustment is allowed.

7. The water gate group scheduling method based on the QMIX framework according to claim 1, characterized in that, The implementation process of step (4) is as follows: For global action values Apply monotonicity constraints: Ensure that the increase of the local Q value of any agent will not cause the decrease of the global Q value; Constructing system-level global reward functions : wherein, represents a global reward value at time ; is a local reward of the th sluice agent; is the total number of sluices participating in collaborative scheduling; sample a batch of transitions from the experience replay pool, based on global rewards constructing a temporal difference target: wherein, is a discount factor, is a target mixing network parameter; Minimizing the QMIX loss function: And through the gradient descent method combined with the update of QMIX mixed network parameters And all the local Q network parameters of the agents ; every step, the current network parameters are soft-updated to the target network: 。 8. The water gate group scheduling method based on the QMIX framework according to claim 1, characterized in that, The implementation process of step (5) is as follows: In the interaction process of the agent and the simulation river basin environment, the experience tuple generated at each time step is stored in the shared experience pool ; for each new experience tuple interacting with the environment , an initial priority is initialized , and the calculation basis is the absolute value of the TD-error corresponding to the experience wherein, is a discount factor, is a target network parameter, is a small constant to ensure that all experiences have a non-zero sampling probability; A high-priority empirical reference set of fixed capacity is maintained dynamically ; when sampling an experience from the empirical pool , if it ranks top in the current sampling batch or history, it is determined as a "high-value experience" and added to the reference set ; the reference set is updated with a first-in-first-out strategy or a mechanism based on the minimum priority elimination to ensure that it always contains the most recent key events with the most learning value; For any experience tuple , define its state feature vector as a numerical vector composed of key hydrological features extracted from its state and normalized; in the context of watershed gate regulation, including station water level, gate upstream water level, gate downstream water level, rainfall; calculate the similarity of its state feature vector with each high-priority experience in the reference set ; the similarity measure adopts Gaussian kernel function: wherein, is a bandwidth parameter, used to adjust the sensitivity of the similarity to the state difference; finally, the maximum value among all similarity values is taken as the similarity measure result of Based on base priority and similarity gain , compute experience of mixed priority : wherein, is a similarity gain coefficient, used to adjust the strength of the influence of similarity on priority; During the DQN network training phase, a mini-batch of size is sampled from the experience pool ; the sampling probability is proportional to the mix priority : where, is the importance sampling coefficient; meanwhile, importance sampling weights are calculated for each sampled experience for weighting the loss function to correct sampling bias: Wherein the agent uses the small batch data and the weight updates its Q network parameters After multiple rounds of training converge, the QMIX algorithm outputs an optimal cooperative strategy network.

Images