A step-by-step pump station optimal scheduling method based on deep reinforcement learning

Abstract

The application discloses a kind of based on deep reinforcement learning's cascade pump station optimization scheduling method, belongs to pump station scheduling technical field, specific steps include: constructing cascade pump station optimization scheduling system mathematical model;Collecting three levels of pump station, mixing tank, pipe section, ore pulp parameter, time-of-use electricity price and daily transportation task and other operation data, form the scheduling state vector including time, electricity price, transportation progress, liquid level, pump starting number, flow, pressure and flow rate;Parameterized hybrid action space is defined by pump starting number combination and single pump continuous flow regulation parameter;The optimization scheduling model is solved by multi-channel deep Q network (MP-DQN), and low-cost scheduling strategy is obtained by combining experience playback and target network soft update.The application comprehensively considers ore pulp transportation safety constraint, transportation task completion rate and time-of-use electricity price and other factors, can reduce cascade pump station operation electricity fee on the basis of guaranteeing pipeline safe operation and transportation target completion, and improve optimization scheduling efficiency.

Description

Technical Field

[0001] This invention belongs to the field of pump station scheduling technology, and in particular relates to an optimized scheduling method for cascade pump stations based on deep reinforcement learning. Background Technology

[0002] Slurry pipeline transportation systems typically consist of multiple pumping stations cascaded along a route with varying elevations, with transportation between stations facilitated by mixing tanks or buffer facilities. For long-distance, high-lift slurry pipelines, pumping station scheduling must not only meet daily dry ore transportation requirements but also consider constraints such as minimum safe flow rate, maximum permissible flow rate, upper limit of outlet pressure, single pump flow range, mixing tank level range, and time-of-use electricity pricing.

[0003] Existing scheduling methods for cascade pumping stations mainly include manual experience-based scheduling, fixed-rule scheduling, mathematical programming methods, and intelligent optimization methods such as Particle Swarm Optimization (PSO), Grey Wolf Optimization (GWO), and Deep Deterministic Policy Gradient (DDPG). Manual experience-based or fixed-rule scheduling typically relies on pre-set start-stop schedules, making it difficult to dynamically adjust according to time-of-use electricity prices, inter-stage inventory, and transportation task progress. Mathematical programming methods have high modeling and solution complexity when there are many constraints. Traditional swarm intelligence optimization methods are prone to problems such as search space expansion, long computation time, sensitivity to initial parameters, and unexecutable candidate actions when simultaneously handling discrete pump start-up numbers and continuous single pump flow allocation.

[0004] Therefore, it is necessary to propose an optimized scheduling method for cascade pumping stations that can uniformly handle discrete start-up and shutdown and continuous flow regulation, and can adaptively learn scheduling strategies under operational constraints. Summary of the Invention

[0005] To overcome the above-mentioned technical problems, this invention provides an optimized scheduling method for cascade pumping stations based on deep reinforcement learning.

[0006] To achieve the above technology, the following steps are included: S1. Based on the target three-level pump station, construct a scheduling state and parameterized action model oriented to MP-DQN to obtain the state vector and define the hybrid scheduling action; S2. Based on the results of S1, construct the motion projection and slurry transportation safety constraint model to obtain the executable scheduling instructions of the pumping station, the update status of the regulating tank of the pumping station in the next time period, and perform safety verification. S3. Based on the output of S2, construct the operating cost, scheduling reward and MP-DQN learning evaluation model to calculate the electricity cost, transportation task completion status and scheduling evaluation value. Through S1 to S3, a closed-loop design of the optimized scheduling method for cascade pumping stations is completed.

[0007] Specifically, the state vector includes: collection and scheduling time period, time-of-use electricity price, cumulative transportation volume, regulating tank inventory, pump start status, station-level flow rate, outlet pressure, and pipeline flow velocity.

[0008] Specifically, the steps for defining hybrid scheduling actions include: Define the combination of the number of pumps to be started in a tertiary pumping station: ,in, For the first The combination of the number of pumps started at the three-stage pumping stations during different time periods. These represent the number of pumps started at the first, second, and third pumping stations, respectively. Define the continuous flow regulation parameter vector: , For continuous flow rate regulation parameter vector, to These correspond to the flow regulation parameters of all 10 pumps in the system. Define hybrid scheduling actions: ,in, For the first Parameterized scheduling actions for time periods.

[0009] Specifically, step S2 includes: based on the combination of pump start-up numbers and the continuous flow parameters output by the continuous parameter network obtained in S1, combined with the upper and lower limits of inventory, cascade inflow relationship, pump capacity, hydraulic conditions, pipeline flow velocity and outlet pressure constraints, determining the executable flow range, and modifying the candidate actions output by the intelligent agent into executable scheduling instructions for the project through interval truncation and safety verification.

[0010] Specifically, step S2 includes: Based on the start / stop status and continuous flow regulation parameters, the flow rate of a single pump is mapped to the allowable operating range, and the expression is as follows: In the formula, For the first The actual candidate flow rate of the pump. For the first The start / stop status of the pump. and These are the minimum and maximum allowable flow rates of the pump, respectively; when The pump stops when the value is 0. When taking 1, according to The mapping yields the single-pump flow rate within the permissible operating conditions; The first output of the MP-DQN continuous parameter network Flow rate adjustment parameters for the trolley pump; The candidate flow rates of all pumps in operation within the same pumping station are summarized to obtain the station-level candidate flow rate, which is expressed as follows: In the formula, For the first Candidate station-level flow rates for pumping stations. For the first The collection of pumps included in a pumping station; Determine the operational flow range of the pumping station, including: The lower limit of the flow rate that the pumping station can operate at is expressed as: ;in, For the first The lower limit of the flow rate that can be implemented for the pumping station; Indicates the first The number of pumps in operation at the pumping station. This indicates the minimum operating flow rate of a single pump; This indicates the lower limit of the flow rate required to meet the minimum safe flow velocity in the pipeline. This indicates the minimum safe flow rate allowed in the pipeline. Indicates the cross-sectional area of ​​the pipe; The maximum flow rate that a pumping station can operate at is expressed as: ;in, For the first The maximum flow rate that a pumping station can operate at; This indicates the maximum flow rate limited by hydraulic conditions; Indicates the first The current equivalent liquid level of the inventory in the regulating tank of the primary pump station; Indicates the first Inflow rate of the pumping station; Indicates the first Minimum allowable inventory or liquid level in the primary regulating tank; This indicates the maximum operating flow rate of a single pump; The station-level candidate traffic is restricted to the executable traffic range using an interval truncation function to obtain the actual executed traffic. The expression is as follows: In the formula, For the first The actual executable flow rate of the pumping station; For candidate site-level traffic; It is an interval cutoff function; when Less than When the pumping station stops pumping, Not less than At that time, candidate traffic Limit the flow to an executable range to generate executable scheduling instructions for the project; The inventory of each level of regulating tank is updated according to the inflow and outflow rates, as expressed below: In the formula, For the first The inventory or equivalent liquid level in the regulating tank of the pumping station for the next period; Furthermore, in a three-stage cascade pumping station, the outflow from the previous stage pumping station becomes the inflow from the next stage pumping station. The expression for the cascaded inflow relationship is as follows: In the formula, The inflow rate of the first pumping station is equal to the external inflow. ; The inflow rate of the second pumping station is equal to the outflow rate of the first pumping station. ; The inflow rate of the third pumping station is equal to the outflow rate of the second pumping station. ; Perform hydraulic condition and transport safety checks, including: The expression for the inventory constraint of the regulating tank is as follows: In the formula, and The first Minimum and maximum allowable inventory or liquid level in the primary regulating tank; The flow velocity in the pipe section is calculated by converting the station-level flow rate and the pipe cross-sectional area, and its expression is as follows: In the formula, For the first Flow velocity in the first-stage pipe section; The required head of the pumping station is determined by both the static head and the friction head, and its expression is as follows: In the formula, For the first The required head for the pumping station; Indicates the first Elevation of downstream control point of the pumping station; Indicates the first Elevation of the upstream control point of the pumping station; Indicates the difference in control elevation between upstream and downstream; This is the drag coefficient; For the length of the pipe section, D For pipe diameter, g The acceleration due to gravity is used; this formula determines the hydraulic requirements of the pumping station by combining the static head and the friction head. The pump station outlet pressure is calculated based on the hydraulic head, and its expression is as follows: In the formula, For the first The outlet pressure of the pumping station The density of the slurry is given by this formula, which is used to convert hydraulic head into outlet pressure. To simultaneously meet the requirements of preventing slurry sedimentation, preventing high-velocity abrasion, and ensuring pipeline pressure resistance, a safety constraint for slurry transportation is set, the expression of which is as follows: In the formula, and These are the lower and upper limits of the allowed flow rate, respectively. This is the upper limit of the outlet pressure; this formula is used to simultaneously constrain slurry deposition prevention, high-velocity abrasion prevention, and pipeline pressure safety.

[0011] Specifically, step S3 includes: Based on the executable scheduling instructions obtained from S2, as well as the single pump power fitting function, time-of-use electricity price and cumulative transportation volume, the hourly electricity cost, the total daily electricity cost and the transportation task completion status are calculated, and the electricity cost, safety constraint deviation, transportation target deviation and time-of-use electricity price guidance term are combined to form the scheduling evaluation value. An MP-DQN network structure, comprising a multi-path Q-network, a continuous parameter network, an experience replay buffer, a target Q-network, and a target continuous parameter network, is established for training. The scheduling strategy is updated through experience replay sampling, temporal difference target calculation, and target network soft update to obtain a low-cost scheduling scheme that satisfies safety constraints and transportation tasks.

[0012] Specifically, the steps for calculating hourly electricity costs, total daily electricity costs, and the completion status of the transportation task include: After obtaining the execution flow rate that meets safety constraints, the operating cost and scheduling evaluation value are calculated based on the pump power fitting function, time-of-use electricity price, and cumulative dry ore transportation volume; the expression for the single pump power fitting function is as follows: In the formula, This is a single-pump power fitting function. This refers to the flow rate of a single pump. According to the The hourly electricity cost is calculated based on the pump's start / stop status, single pump power, and time-of-use pricing. The total electricity cost is then summed over a 24-hour scheduling cycle, as shown in the following expression: In the formula, For the first Electricity charges for specific time periods; For the first Time-of-use electricity pricing; For the first Pump start / stop status; For the first Operating power of the pump; This represents the total electricity cost within a 24-hour dispatch cycle.

[0013] Specifically, the expression for the scheduling evaluation value is as follows: In the formula, For the first Time-of-use scheduling evaluation value; to Weights for each evaluation item; Penalties are imposed for liquid level, flow rate, pressure, and cascade constraints. For deviation from transportation targets; Rewards for achieving the goal; This is a time-of-use electricity price adjustment item that accounts for more electricity usage during off-peak hours and less during peak hours; By maximizing This allows for the acquisition of low-cost scheduling strategies that meet both safety constraints and transportation tasks.

[0014] Specifically, the steps for establishing and training an MP-DQN network structure comprising a multi-path Q-network, a continuous parameter network, an experience replay buffer, a target Q-network, and a target continuous parameter network include: After obtaining the scheduling evaluation value, MP-DQN is used to train the parameterized hybrid scheduling actions; First, calculate the continuous parameters and Q-value, and select the combination of pump start-up units. The expression for the continuous parameter network is as follows: In the formula, For a continuous parameter network, where, These are continuous parameter network parameters; The combination of pumps to be started is evaluated and selected by a Q-value network, and its expression is as follows: In the formula, For Q-value networks, where, These are the Q-value network parameters; For the candidate combinations of pump start-up numbers; Secondly, a temporal difference objective is constructed and the Q-value network is updated. The temporal difference objective is composed of the current scheduling evaluation value, the discount factor, and the maximum Q-value for the next time period, and its expression is as follows: In the formula, For time-series difference objectives; Discount factor; For the target Q-value network, where, The target Q-value network parameters; This is the state vector for the next time period; Parameters for continuous flow regulation in the next time period; The Q-value network loss function is used to update the Q-value network, and its expression is as follows: In the formula, B is the Q-value network loss function; B is the batch size. The current Q-value is the network's predicted value; the Q-value network parameters are updated by minimizing this loss function. Next, the continuous parameter network is optimized and a scheduling strategy is output. The optimization objective of the continuous parameter network is to increase the Q value corresponding to the selected pump start-up combination and continuous flow parameters, and its expression is as follows: In the formula, The optimization objective is for continuous parameter networks; By maximizing this objective, the continuous flow parameters are updated in the direction of higher scheduling evaluation values, ultimately outputting an optimized scheduling scheme for the entire day.

[0015] Specifically, S3 further includes: calculating the completed amount of transportation tasks and constructing a scheduling reward to limit the completion range of transportation tasks and avoid under-transportation or over-transportation, the expression of which is as follows: , In the formula, This represents the cumulative equivalent transportation volume completed within the scheduling cycle. Mass concentration; For the first The output flow rate of the third-level pumping station during the period; The target dry ore quantity.

[0016] The beneficial effects of this invention are: This invention comprehensively considers parameters such as the combination of pump start-up numbers, pump flow rate adjustment, regulating tank inventory, cascade inflow, pipeline flow velocity, outlet pressure, time-of-use electricity price, and transportation targets. By using motion projection and safety constraint verification to correct unexecutable scheduling instructions, it improves the engineering feasibility of the scheduling scheme. Compared with existing optimization methods, this invention can achieve lower operating electricity costs and transportation costs while meeting transportation tasks and slurry transportation safety constraints, thereby improving the economy, reliability, and optimization efficiency of cascade pump station scheduling. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall process of the optimized scheduling method for cascade pumping stations according to the present invention; Figure 2 This is a diagram showing the hourly outflow of the three-stage pumping station according to the present invention; Figure 3 This is a comparison chart of transportation costs between the optimized model of this invention and other optimized models. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention.

[0019] This embodiment provides an optimized scheduling method for cascade pumping stations based on deep reinforcement learning. This method focuses on the 24-hour scheduling of three-stage cascade pumping stations, incorporating pump station start-up and shutdown, pump flow allocation, inter-stage inventory changes, slurry safety transport constraints, and time-of-use electricity costs into the unified scheduling environment.

[0020] S1. Construct a scheduling state and parameterized action model for MP-DQN; By collecting data on scheduling periods, time-of-use electricity prices, cumulative transport volume, regulating tank inventory, pump start-up status, station-level flow rate, outlet pressure, and pipeline flow velocity, a state vector is formed to meet the needs of the intelligent agent for decision-making. At the same time, the three-level pump station start-up array is treated as a discrete action, and the continuous flow regulation parameters of each pump are used as continuous action parameters. A continuous parameter network is set up to generate the flow regulation parameters of each pump in the entire system based on the current state, thereby uniformly describing start-up and shutdown decisions and flow allocation decisions.

[0021] like Figure 1 As shown, data on the three-stage pumping station, mixing tank, pipe section, slurry density, mass concentration, pipe cross-sectional area, time-of-use electricity price, and daily transportation tasks are first collected. A state vector is then constructed for each scheduling period. The specific steps are as follows: S1.1 Constructing the state vector; Data collection includes scheduling periods, time-of-use electricity prices, cumulative transport volume, regulating tank inventory, pump start-up status, station-level flow rate, outlet pressure, and pipeline flow velocity. This data forms the state vector required for agent decision-making, represented as follows: In the formula, For the first The system state vector for a given time period; The time period number; This is the current time-of-use electricity price; This represents the cumulative transport volume. For the inventory or liquid level of each level of regulating tank; Pump is in start-up state; The flow rate of the pumping station; To alleviate export pressure; The flow velocity in the pipe; This formula is used to provide the environmental information needed for scheduling decisions by the MP-DQN agent.

[0022] S1.2 Construct the combination of pump start-up numbers and the continuous flow regulation parameter vector; The combination of pump start-up numbers in a tertiary pumping station is defined as a discrete action, and the flow rate adjustment parameter for each pump is defined as a continuous action parameter, with the following expression: In the formula, For the first The combination of the number of pumps started at the three-stage pumping stations during different time periods. These represent the number of pumps started at the first, second, and third pumping stations, respectively. For continuous flow rate regulation parameter vector, to These correspond to the flow regulation parameters of all 10 pumps in the system.

[0023] S1.3 Construct parameterized hybrid scheduling actions; The combination of the number of pumps started and the continuous flow rate regulation parameter together constitute the agent action, and its expression is as follows: In the formula, For the first Parameterized scheduling actions for specific time periods; This formula is used to uniformly describe start / stop decisions and traffic allocation decisions.

[0024] S2. Construct a motion projection and slurry transport safety constraint model; After obtaining the discrete pump start-up combination and the continuous flow parameters output by the continuous parameter network in S1, the executable flow range is determined by combining the upper and lower limits of inventory, cascade inflow relationship, pump capacity, hydraulic conditions, pipeline flow velocity and outlet pressure constraints. Then, the candidate actions output by the agent are modified into executable scheduling instructions for the project through interval truncation and safety verification. The specific steps are as follows: S2.1, Single pump flow mapping; Based on the start / stop status and continuous flow regulation parameters, the flow rate of a single pump is mapped to the allowable operating range, and the expression is as follows: In the formula, For the first The actual candidate flow rate of the pump. For the first The start / stop status of the pump. and These are the minimum and maximum allowable flow rates of the pump, respectively; when The pump stops when the value is 0. When taking 1, according to The mapping yields the single-pump flow rate within the permissible operating conditions; The first output of the MP-DQN continuous parameter network Flow rate adjustment parameters for the trolley pump.

[0025] S2.2, Station-level candidate traffic summary; Furthermore, the candidate flow rates of all activated pumps within the same pumping station are aggregated to obtain the station-level candidate flow rate, expressed as follows: In the formula, For the first Candidate station-level flow rates for pumping stations. For the first The collection of pumps included in a pumping station; This formula obtains the station-level candidate outflow rate by summing the flow rates of all activated pumps within the same pumping station. See [link to relevant documentation]. Figure 2 .

[0026] S2.3 Determine the flow range that the pumping station can operate within; The expression for the lower limit of the flow rate that the pumping station can operate on is: ;in, For the first The lower limit of the flow rate that can be implemented for the pumping station; This indicates the minimum operating flow rate requirement per pump corresponding to the number of pumps in operation. Indicates the first The number of pumps in operation at the pumping station. This indicates the minimum operating flow rate of a single pump; This indicates the lower limit of the flow rate required to meet the minimum safe flow velocity in the pipeline. This indicates the minimum safe flow rate allowed in the pipeline. This indicates the cross-sectional area of ​​the pipe; this formula is used to avoid sedimentation at low flow rates and low-flow operation of a single pump.

[0027] The maximum flow rate that a pumping station can operate at, taking into account the available pumping capacity, pump start-up capacity, and hydraulic constraints, is expressed as follows: ;in, For the first The maximum flow rate that a pumping station can operate at; This indicates the maximum flow rate that can be drawn while maintaining or exceeding the minimum inventory level. This indicates the maximum capacity corresponding to the number of pumps in operation; This indicates the maximum flow rate limited by hydraulic conditions; Indicates the first The current equivalent liquid level of the inventory in the regulating tank of the primary pump station; Indicates the first Inflow rate of the pumping station; Indicates the first Minimum allowable inventory or liquid level in the primary regulating tank; This indicates the maximum operating flow rate of a single pump.

[0028] S2.4, Project the motion; The station-level candidate traffic is restricted to the executable traffic range using an interval truncation function to obtain the actual executed traffic. The expression is as follows: In the formula, For the first The actual executable flow rate of the pumping station; For candidate site-level traffic; It is an interval cutoff function; when Less than At that time, the pumping station stopped pumping. Not less than At that time, candidate traffic The execution flow is restricted to a range of executable traffic to generate executable scheduling instructions for the project.

[0029] S2.5, Perform cascading inventory recursion; The inventory of each level of regulating tank is updated according to the inflow and outflow rates, as expressed below: In the formula, For the first The inventory or equivalent liquid level in the regulating tank of the pumping station for the next period; This formula is used to describe the dynamic changes in inventory among cascade pumping stations; Furthermore, in a three-stage cascade pumping station, the outflow from the previous stage pumping station becomes the inflow from the next stage pumping station. The expression for the cascaded inflow relationship is as follows: In the formula, The inflow rate of the first pumping station is equal to the external inflow. ; The inflow rate of the second pumping station is equal to the outflow rate of the first pumping station. ; The inflow rate of the third pumping station is equal to the outflow rate of the second pumping station. This formula is used to describe the cascaded transport relationship of a three-stage pumping station.

[0030] S2.6. Perform hydraulic condition and transportation safety checks, including: The expression for the inventory constraint of the regulating tank is as follows: In the formula, and The first Minimum and maximum allowable inventory or liquid level in the primary regulating tank; The flow velocity in the pipe section is calculated by converting the station-level flow rate and the pipe cross-sectional area, and its expression is as follows: In the formula, For the first Flow velocity in the first-stage pipe section; The required head of the pumping station is determined by both the static head and the friction head, and its expression is as follows: In the formula, For the first The required head for the pumping station; Indicates the first Elevation of downstream control point of the pumping station; Indicates the first Elevation of the upstream control point of the pumping station; Indicates the difference in control elevation between upstream and downstream; This is the drag coefficient; For the length of the pipe section, D For pipe diameter, g The acceleration due to gravity is used; this formula determines the hydraulic requirements of the pumping station by combining the static head and the friction head. The pump station outlet pressure is calculated based on the hydraulic head, and its expression is as follows: In the formula, For the first The outlet pressure of the pumping station The density of the slurry is given by this formula, which is used to convert hydraulic head into outlet pressure. To simultaneously meet the requirements of preventing slurry sedimentation, preventing high-velocity abrasion, and ensuring pipeline pressure resistance, a safety constraint for slurry transportation is set, the expression of which is as follows: In the formula, and These are the lower and upper limits of the allowed flow rate, respectively. This is the upper limit of the outlet pressure; this formula is used to simultaneously constrain slurry deposition prevention, high-velocity abrasion prevention, and pipeline pressure safety.

[0031] S3. Construct an evaluation model for operating costs, scheduling rewards, and MP-DQN learning; Based on the executable scheduling instructions obtained in S2, the hourly electricity cost, total daily electricity cost, and transportation task completion status are calculated according to the single-pump power fitting function, time-of-use electricity price, and cumulative transportation volume. The electricity cost, safety constraint deviation, transportation target deviation, and time-of-use electricity price guidance term are combined to form the scheduling evaluation value. An MP-DQN training structure is established, including a multi-path Q-network, a continuous parameter network, an experience replay buffer, a target Q-network, and a target continuous parameter network. The scheduling strategy is updated through experience replay sampling, temporal difference target calculation, and soft update of the target network to obtain a low-cost scheduling scheme that satisfies both safety constraints and transportation tasks. The specific steps are as follows: S3.1 Calculate the operating power and electricity cost of a single pump; After obtaining the execution flow rate that meets safety constraints, the operating cost and scheduling evaluation value are calculated based on the pump power fitting function, time-of-use electricity price, and cumulative dry ore transportation volume. The expression for the single pump power fitting function is as follows: In the formula, This is a single-pump power fitting function. This is the flow rate of a single pump; this formula is used to calculate the operating power under different flow conditions based on pump performance data. Furthermore, according to the first The hourly electricity cost is calculated based on the pump's start / stop status, single pump power, and time-of-use pricing. The total electricity cost is then summed over a 24-hour scheduling cycle, as shown in the following expression: In the formula, For the first Electricity charges for specific time periods; For the first Time-of-use electricity pricing; For the first Pump start / stop status; For the first Operating power of the pump; This represents the total electricity cost within a 24-hour scheduling cycle; this formula is used to convert pump power and time-of-use electricity prices into operating costs.

[0032] S3.2 Calculate the amount of transportation tasks completed and construct scheduling rewards; The cumulative equivalent transport volume is used to constrain the transport task completion interval, and the expression is as follows: , In the formula, This represents the cumulative equivalent transportation volume completed within the scheduling cycle. Mass concentration; For the first The output flow rate of the third-level pumping station during the period; The target dry ore quantity is defined by this formula, which is used to limit the range of transportation task completion to avoid under-transportation or over-transportation.

[0033] S3.3, Construct scheduling evaluation values; Based on electricity costs, deviations from safety constraints, deviations from transportation targets, rewards for achieving targets, and time-of-use pricing, a dispatch evaluation value is constructed, the expression of which is as follows: In the formula, For the first Time-of-use scheduling evaluation value; to Weights for each evaluation item; Penalties are imposed for liquid level, flow rate, pressure, and cascade constraints. For deviation from transportation targets; Rewards for achieving the goal; This is a time-of-use pricing adjustment item that allows for more electricity to be used during off-peak hours and less during peak hours; by maximizing... This allows for the acquisition of low-cost scheduling strategies that meet both safety constraints and transportation tasks.

[0034] S3.4 Calculate the continuous parameters and Q value and select the combination of pump start-up numbers; After obtaining the scheduling evaluation value, MP-DQN is used to train the parameterized hybrid scheduling actions.

[0035] MP-DQN includes: a multi-path Q-network, a continuous parameter network, an experience replay buffer, a target Q-network, and a target continuous parameter network. The continuous parameter network is used to generate pump flow regulation parameters for each pump. The multi-path Q-network is used to evaluate the scheduling value of candidate combinations of pump start-up numbers. The experience replay buffer is used to store states, actions, scheduling evaluation values, and next state samples. The target Q-network improves the stability of temporal difference learning through soft updates.

[0036] The expression for a continuous parameter network is as follows: In the formula, For a continuous parameter network, where, These are continuous parameter network parameters; this formula is used to generate the continuous part of the parameterized action. The combination of pumps to be started is evaluated and selected by a Q-value network, and its expression is as follows: In the formula, For Q-value networks, where, These are the Q-value network parameters; This represents the combination of candidate pump start-up numbers.

[0037] S3.5, Construct the temporal difference objective and update the Q-value network; The temporal differential objective is composed of the current scheduling evaluation value, the discount factor, and the maximum Q value for the next time period, and its expression is as follows: In the formula, For time-series difference objectives; Discount factor; For the target Q-value network, where, The target Q-value network parameters; This is the state vector for the next time period; Parameters for continuous flow regulation in the next time period; The Q-value network loss function is used to measure the error between the target value and the predicted Q-value. Its expression is as follows: In the formula, B is the Q-value network loss function; B is the batch size. The current Q-value is the network prediction value; the Q-value network parameters are updated by minimizing this loss function.

[0038] S3.6 Optimize the continuous parameter network and output the scheduling strategy; The optimization objective of the continuous parameter network is to increase the Q value corresponding to the selected pump start-up combination and continuous flow parameters, and its expression is as follows: In the formula, The optimization objective is to optimize the continuous parameter network. By maximizing this objective, the continuous flow parameters are updated in the direction of higher scheduling evaluation values, and finally, an optimized scheduling scheme for the whole day is output.

[0039] This invention was applied to the 24-hour scheduling of a three-stage cascade pumping station. The scheduling objective was to reduce operating electricity costs while meeting transportation tasks, inventory constraints, flow rate constraints, and outlet pressure constraints. The optimization results are shown in Table 1. Table 1: Comparison of the present invention with existing technologies The results above show that the present invention can form a low-cost scheduling scheme under the premise that the transportation completion volume falls within the target range, indicating that the parametric motion modeling, motion projection constraint and MP-DQN training mechanism can work together to achieve cost optimization of cascade pumping stations.

[0040] As a comparative example, under the same transportation task, safety constraints, time-of-use electricity pricing, and number of iterations, the transportation costs of the MP-DQN method used in this invention are compared with those of DDPG, GWO, and PSO. The results are as follows: Figure 3 As shown.

[0041] Depend on Figure 3 It can be seen that the MP-DQN curve converges to a relatively low cost level of approximately 48,853.18 yuan in the later stages of iteration, which is lower than the convergence cost of DDPG, GWO and PSO. This indicates that the present invention, by combining parameterized hybrid actions, action projection constraints and deep reinforcement learning evaluation mechanisms, can further reduce the transportation cost of cascade pumping stations while meeting engineering constraints.

[0042] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for optimal scheduling of cascade pumping stations based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Based on the target three-level pump station, construct a scheduling state and parameterized action model oriented to MP-DQN to obtain the state vector and define the hybrid scheduling action; S2. Based on the results of S1, construct the motion projection and slurry transportation safety constraint model to obtain the executable scheduling instructions of the pumping station, the update status of the regulating tank of the pumping station in the next time period, and perform safety verification. S3. Based on the output of S2, construct the operating cost, scheduling reward and MP-DQN learning evaluation model to calculate the electricity cost, transportation task completion status and scheduling evaluation value. Through S1 to S3, a closed-loop design of the optimized scheduling method for cascade pumping stations is completed.

2. The method for optimizing the scheduling of cascade pumping stations based on deep reinforcement learning according to claim 1, characterized in that: The state vector includes: collection and scheduling time period, time-of-use electricity price, cumulative transportation volume, regulating tank inventory, pump start status, station-level flow rate, outlet pressure, and pipeline flow velocity.

3. The method for optimizing the scheduling of cascade pumping stations based on deep reinforcement learning according to claim 2, characterized in that: The steps for defining hybrid scheduling actions include: Define the combination of the number of pumps to be started in a tertiary pumping station: ,in, For the first The combination of the number of pumps started at the three-stage pumping stations during different time periods. These represent the number of pumps started at the first, second, and third pumping stations, respectively. Define the continuous flow regulation parameter vector: , For continuous flow rate regulation parameter vector, to These correspond to the flow regulation parameters of all 10 pumps in the system. Define hybrid scheduling actions: ,in, For the first Parameterized scheduling actions for time periods.

4. The method for optimizing the scheduling of cascade pumping stations based on deep reinforcement learning according to claim 3, characterized in that: The steps in S2 include: based on the combination of pump start-up numbers and the continuous flow parameters output by the continuous parameter network obtained in S1, combined with the upper and lower limits of inventory, cascade inflow relationship, pump capacity, hydraulic conditions, pipeline flow velocity and outlet pressure constraints, determining the executable flow range, and modifying the candidate actions output by the intelligent agent into executable scheduling instructions for the project through range truncation and safety verification.

5. The method for optimizing the scheduling of cascade pumping stations based on deep reinforcement learning according to claim 4, characterized in that: Step S2 includes: Based on the start / stop status and continuous flow regulation parameters, the flow rate of a single pump is mapped to the allowable operating range, and the expression is as follows: In the formula, For the first The actual candidate flow rate of the pump. For the first The start / stop status of the pump. and These are the minimum and maximum allowable flow rates of the pump, respectively; when The pump stops when the value is 0. When taking 1, according to The mapping yields the single-pump flow rate within the permissible operating conditions; The first output of the MP-DQN continuous parameter network Flow rate adjustment parameters for the trolley pump; The candidate flow rates of all pumps in operation within the same pumping station are summarized to obtain the station-level candidate flow rate, which is expressed as follows: In the formula, For the first Candidate station-level flow rates for pumping stations. For the first The pump set included in a pumping station; Determine the operational flow range of the pumping station, including: The lower limit of the flow rate that the pumping station can operate at is expressed as: ;in, For the first The lower limit of the flow rate that can be implemented for the pumping station; Indicates the first The number of pumps in operation at the pumping station. This indicates the minimum operating flow rate of a single pump; This indicates the lower limit of the flow rate required to meet the minimum safe flow velocity in the pipeline. This indicates the minimum safe flow rate allowed in the pipeline. Indicates the cross-sectional area of ​​the pipe; The maximum flow rate that a pumping station can operate at is expressed as: ;in, For the first The maximum flow rate that a pumping station can operate at; This indicates the maximum flow rate limited by hydraulic conditions; Indicates the first The current equivalent liquid level of the inventory in the regulating tank of the primary pump station; Indicates the first Inflow rate of the pumping station; Indicates the first Minimum allowable inventory or liquid level in the primary regulating tank; This indicates the maximum operating flow rate of a single pump; The station-level candidate traffic is restricted to the executable traffic range using an interval truncation function to obtain the actual executed traffic. The expression is as follows: In the formula, For the first The actual executable flow rate of the pumping station; For candidate site-level traffic; It is an interval cutoff function; when Less than When the pumping station stops pumping, Not less than At that time, candidate traffic Limit the flow to an executable range to generate executable scheduling instructions for the project; The inventory of each level of regulating tank is updated according to the inflow and outflow rates, as expressed below: In the formula, For the first The inventory or equivalent liquid level in the regulating tank of the pumping station for the next period; Furthermore, in a three-stage cascade pumping station, the outflow from the previous stage pumping station becomes the inflow from the next stage pumping station. The expression for the cascaded inflow relationship is as follows: In the formula, The inflow rate of the first pumping station is equal to the external inflow. ; The inflow rate of the second pumping station is equal to the outflow rate of the first pumping station. ; The inflow rate of the third pumping station is equal to the outflow rate of the second pumping station. ; Perform hydraulic condition and transport safety checks, including: The expression for the inventory constraint of the regulating tank is as follows: In the formula, and The first Minimum and maximum allowable inventory or liquid level in the primary regulating tank; The flow velocity in the pipe section is calculated by converting the station-level flow rate and the pipe cross-sectional area, and its expression is as follows: In the formula, For the first Flow velocity in the first-stage pipe section; The required head of the pumping station is determined by both the static head and the friction head, and its expression is as follows: In the formula, For the first The required head for the pumping station; Indicates the first Elevation of the downstream control point of the pumping station; Indicates the first Elevation of the upstream control point of the pumping station; This indicates the difference in elevation between the upstream and downstream control points; This is the drag coefficient; For the length of the pipe section, D For pipe diameter, g The acceleration due to gravity is used; this formula determines the hydraulic requirements of the pumping station by combining the static head and the friction head. The pump station outlet pressure is calculated based on the hydraulic head, and its expression is as follows: In the formula, For the first The outlet pressure of the pumping station The density of the slurry is given by this formula, which is used to convert hydraulic head into outlet pressure. To simultaneously meet the requirements of preventing slurry sedimentation, preventing high-velocity abrasion, and ensuring pipeline pressure resistance, a safety constraint for slurry transportation is set, the expression of which is as follows: In the formula, and These are the lower and upper limits of the allowed flow rate, respectively. This is the upper limit of the outlet pressure; this formula is used to simultaneously constrain slurry deposition prevention, high-velocity abrasion prevention, and pipeline pressure safety.

6. The method for optimal scheduling of cascade pumping stations based on deep reinforcement learning according to claim 5, characterized in that: The steps in S3 include: Based on the executable scheduling instructions obtained from S2, as well as the single pump power fitting function, time-of-use electricity price and cumulative transportation volume, the hourly electricity cost, the total daily electricity cost and the transportation task completion status are calculated, and the electricity cost, safety constraint deviation, transportation target deviation and time-of-use electricity price guidance term are combined to form the scheduling evaluation value. An MP-DQN network structure, comprising a multi-path Q-network, a continuous parameter network, an experience replay buffer, a target Q-network, and a target continuous parameter network, is established for training. The scheduling strategy is updated through experience replay sampling, temporal difference target calculation, and target network soft update to obtain a low-cost scheduling scheme that satisfies safety constraints and transportation tasks.

7. The method for optimal scheduling of cascade pumping stations based on deep reinforcement learning according to claim 6, characterized in that: The steps for calculating hourly electricity costs, total daily electricity costs, and the completion status of transportation tasks include: After obtaining the execution flow rate that meets safety constraints, the operating cost and scheduling evaluation value are calculated based on the pump power fitting function, time-of-use electricity price, and cumulative dry ore transportation volume; the expression for the single pump power fitting function is as follows: In the formula, This is a single-pump power fitting function. This refers to the flow rate of a single pump. According to the The hourly electricity cost is calculated based on the pump's start / stop status, single pump power, and time-of-use pricing. The total electricity cost is then summed over a 24-hour scheduling cycle, as shown in the following expression: In the formula, For the first Electricity charges for specific time periods; For the first Time-of-use electricity pricing; For the first Pump start / stop status; For the first Operating power of the pump; This represents the total electricity cost within a 24-hour dispatch cycle.

8. The method for optimal scheduling of cascade pumping stations based on deep reinforcement learning according to claim 7, characterized in that: The expression for the scheduling evaluation value is as follows: In the formula, For the first Time-of-use scheduling evaluation value; to Weights for each evaluation item; Penalties are imposed for liquid level, flow rate, pressure, and cascade constraints. For deviation from transportation targets; Rewards for achieving the goal; This is a time-of-use electricity price adjustment item that accounts for more electricity usage during off-peak hours and less during peak hours; By maximizing This allows for the acquisition of low-cost scheduling strategies that meet both safety constraints and transportation tasks.

9. The method for optimal scheduling of cascade pumping stations based on deep reinforcement learning according to claim 8, characterized in that: The steps for establishing and training an MP-DQN network structure that includes a multi-path Q-network, a continuous parameter network, an empirical replay buffer, a target Q-network, and a target continuous parameter network include: After obtaining the scheduling evaluation value, MP-DQN is used to train the parameterized hybrid scheduling actions; First, calculate the continuous parameters and Q-value, and select the combination of pump start-up units. The expression for the continuous parameter network is as follows: In the formula, For a continuous parameter network, where, These are continuous parameter network parameters; The combination of pumps to be started is evaluated and selected by a Q-value network, and its expression is as follows: In the formula, For Q-value networks, where, These are the Q-value network parameters; For the candidate combinations of the number of pumps to start; Secondly, a temporal difference objective is constructed and the Q-value network is updated. The temporal difference objective is composed of the current scheduling evaluation value, the discount factor, and the maximum Q-value for the next time period, and its expression is as follows: In the formula, For time-series difference objectives; Discount factor; For the target Q-value network, where, The target Q-value network parameters; This is the state vector for the next time period; Parameters for continuous flow regulation in the next time period; The Q-value network loss function is used to update the Q-value network, and its expression is as follows: In the formula, B is the Q-value network loss function; B is the batch size. The current Q-value is the network's predicted value; the Q-value network parameters are updated by minimizing this loss function. Next, the continuous parameter network is optimized and a scheduling strategy is output. The optimization objective of the continuous parameter network is to increase the Q value corresponding to the selected pump start-up combination and continuous flow parameters, and its expression is as follows: In the formula, The optimization objective is for continuous parameter networks; By maximizing this objective, the continuous flow parameters are updated in the direction of higher scheduling evaluation values, ultimately outputting an optimized scheduling scheme for the entire day.

10. The method for optimal scheduling of cascade pumping stations based on deep reinforcement learning according to claim 9, characterized in that: S3 further includes: calculating the completed amount of transportation tasks and constructing a scheduling reward to limit the completion range of transportation tasks and avoid under-transportation or over-transportation, the expression of which is as follows: ， In the formula, This represents the cumulative equivalent transportation volume completed within the scheduling cycle. Mass concentration; For the first The output flow rate of the third-level pumping station during the period; The target dry ore quantity.

Images