A method and system for controlling dosing of a strong chemical phosphorus removal agent based on a tree model simulation environment

Abstract

The present application relates to sewage treatment technical field, especially a kind of based on tree model simulation environment's strong chemical removal phosphorus reagent dosing control method and system.It includes data acquisition, the multi-dimensional operation data in sewage treatment process is collected;Simulation environment construction, based on historical operation data, train gradient boosting decision tree model, construct the off-line simulation environment for simulating the dynamic response of sewage treatment;Reinforcement learning model training, in off-line simulation environment, train reinforcement learning agent, to inlet condition as state space, with dosing quantity as action space, with multi-objective compound reward function guide agent optimization dosing strategy;Control strategy deployment, the reinforcement learning model trained is deployed in water plant control system, and the optimal dosing quantity is output according to inlet condition in real time, realizes closed-loop automatic control.The present application uses real historical data to construct GBDT tree model as the simulation environment of reinforcement learning, can accurately capture the complex nonlinear logic in sewage treatment process.The environment model can be more consistent with the actual operation condition of water plant.

Description

Technical Field

[0001] This invention relates to the field of wastewater treatment technology, and in particular to a reinforcement learning-based method for controlling the dosing of phosphorus removal agents in a tree model simulation environment. Background Technology

[0002] In wastewater treatment, chemical phosphorus removal is a crucial step in ensuring that total phosphorus (TP) in effluent meets standards. Current phosphorus removal agent dosing control primarily relies on manual experience or traditional PID control strategies. Existing technologies suffer from the following drawbacks: 1. Low control accuracy and slow response. Wastewater treatment processes exhibit highly nonlinear and time-delay characteristics. Traditional PID control struggles to dynamically adjust to real-time fluctuations in influent flow and water quality parameters, resulting in significant fluctuations in total phosphorus in the effluent and difficulty in consistently meeting standards.

[0003] 2. The simulation environment is disconnected from actual operating conditions. Existing mechanistic models, such as the baseline simulation (BSM1) model and the activated sludge (ASM series) model, cannot accurately reproduce the actual operating conditions of a specific water plant when constructing a simulation environment for control strategies due to low data utilization or poor model generalization ability. This results in poor control performance of control algorithms trained in the simulation environment after being transferred to the field.

[0004] 3. The conflict between economic efficiency and stability. To mitigate the risk of failing to meet standards, excessive dosage is often employed in practice. This not only increases the economic cost of chemicals for water plants but may also lead to secondary pollution.

[0005] Therefore, how to build an interactive environment that closely matches actual working conditions, and on this basis, develop an intelligent dosing strategy that can both ensure that the effluent meets the standards and optimize the cost of chemical consumption, is a technical challenge that needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to address the problems existing in the background technology by proposing a reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment.

[0007] The technical solution of the present invention, in its first aspect, provides a reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment, comprising the following specific steps: S1. Data Acquisition: Collect multi-dimensional operational data during the wastewater treatment process, including influent flow rate, total phosphorus in the influent, orthophosphate in the influent, pH in the influent, influent temperature, historical total phosphorus in the effluent, and historical recommended dosage. S2. Simulation environment construction: Based on historical operating data, a gradient boosting decision tree model is trained to construct an offline simulation environment for simulating the dynamic response of wastewater treatment. The simulation environment takes the current influent conditions and dosing actions as inputs and outputs the predicted total phosphorus concentration in the effluent. S3. Reinforcement learning model training: In an offline simulation environment, a reinforcement learning agent is trained, with the water inflow condition as the state space, the dosage as the action space, and a multi-objective composite reward function to guide the agent to optimize the dosage strategy. S4. Control strategy deployment: The trained reinforcement learning model is deployed in the water plant control system to output the optimal dosage in real time according to the water intake conditions, so as to realize closed-loop automatic control.

[0008] Preferably, the simulation environment construction in step S2 further includes the following steps: S21. Construct an expert model M1, whose inputs are the influent flow rate, total phosphorus, orthophosphate, pH, temperature and total phosphorus in the effluent at historical times, and whose output is the expert-recommended dosage. S22. Construct an environmental model M2, whose inputs are historical influent flow rate, total phosphorus influent, orthophosphate influent, pH influent, influent temperature, and historical recommended dosage, and whose output is the predicted total phosphorus concentration in effluent. The environment model M2 serves as the interactive environment for the reinforcement learning agent, while the expert model M1 is used to guide the agent's exploration direction in the early stages of training.

[0009] Preferably, the reinforcement learning model in step S3 is a dual-delay deep deterministic policy gradient algorithm, and its network architecture includes: A current policy network and a target policy network are used to output the drug delivery action; Two current valuation networks and two target valuation networks are used to evaluate the expected return of state-action pairs; Both the policy network and the valuation network are fully connected, with a hidden layer size of 128-64-64, using ReLU activation function, and the output layer uses Tanh activation function.

[0010] Preferably, the multi-objective composite reward function in step S3 is composed of the following weighted combinations of components: Water Quality Assessment Index (EQI): Based on a piecewise nonlinear mapping of total phosphorus concentration in effluent, it is used to incentivize compliance and penalize exceedance. Efficiency score (ES) and cost score (CS): Based on phosphorus removal efficiency per unit of pesticide consumption and dosage, used to optimize pesticide costs; Collaborative Score (MDS) and Constraint Range Score (MRS): Recommended values ​​based on expert model M1, used to guide the agent in learning basic control logic; Security constraints: Used to ensure the security of system operation.

[0011] Preferably, the safety constraints include: The basic safety margin (BS) is used to evaluate the continuous quantification of safety margin. Even if the effluent meets the standards, different scores will be given according to the distance from the boundary of exceeding the standard, leaving a safety buffer. The safety multiplier SM is dynamically adjusted based on historical compliance rates. When the compliance rate is low, the BS weight is automatically increased, forcing the system to adopt a more conservative control strategy. Strict constraints and penalties are imposed on CPs (Consumer Protection Centers), with substantial negative rewards set for extreme drug administration or serious water quality exceeding standards. This serves as the system's safety baseline to prevent substantial risks during the training and exploration phase.

[0012] Preferably, the weights of each component in the multi-objective composite reward function are dynamically adjusted with each training round: In the early stages of training, the weight of expert guidance terms is increased to accelerate policy convergence; In the later stages of training, the weight of efficiency terms is increased to encourage the agent to autonomously explore better strategies. When the total phosphorus in the effluent exceeds the standard, the weight of water quality and safety items will be automatically adjusted, while the weight of other items will be reduced to ensure that the system prioritizes meeting the standard requirements.

[0013] A second aspect of the present invention provides a reinforcement learning-based phosphorus removal agent dosing control system based on a tree model simulation environment, using the above-described method for control, including: The data acquisition unit is used to collect multidimensional data from the wastewater treatment site and output the real-time status St. Historical database, used to store historical operational data; The intelligent control unit communicates with the data acquisition unit and the historical database, specifically including: The simulation environment construction module is pre-trained based on historical operating data in the historical database and is used to simulate the dynamic response of the wastewater treatment process. The input of the simulation environment module receives the real-time status St and the dosing action At, and the output outputs the predicted total phosphorus concentration of the effluent at the current moment. The expert experience module is pre-trained based on historical operational data in the historical database and is used to provide expert experience references. The input of the expert experience module receives the real-time status St, and the output outputs the expert-recommended dosage At_exp. The intelligent agent module is connected to the data acquisition unit, the simulation environment construction module and the expert experience module, respectively. It is used to output the drug delivery action At according to the real-time state St, and to receive the reward signal R during the training phase to optimize the control strategy. The calculation module is connected to the simulation environment construction module, the expert experience module, and the intelligent agent module respectively. It is used to calculate the multi-objective composite reward R based on the total phosphorus concentration in the effluent output by the simulation environment construction module, the expert-recommended dosage At_exp output by the expert experience module, and the dosing action At output by the intelligent agent module, and then feed it back to the L intelligent agent module. The actuator, connected to the intelligent agent module, is set up at the wastewater treatment site to receive the optimal dosing action At output by the intelligent agent module and control the variable frequency dosing pump to dosing the agent accordingly.

[0014] Preferably, the simulation environment construction module includes: The expert model unit M1 is used to output the expert-recommended dosage based on the influent conditions and historical total phosphorus in the effluent. The environmental model unit M2 is used to predict the total phosphorus concentration in the effluent based on the influent conditions and chemical dosing actions. The M2 unit serves as the interactive environment for the reinforcement learning agent, while the M1 unit is used for policy guidance during the initial training phase.

[0015] Preferably, the reward function in the calculation module is a multi-objective composite reward function Rtotal, which includes water quality evaluation index, efficiency score, cost score, expert collaboration score, constraint range score and safety constraint term, and the weights of each are dynamically adjusted with each training round.

[0016] Preferably, the actuator reads the parameters of the inlet water sensor in real time, calculates the optimal dosage through a reinforcement learning model, and outputs the result to the dosing actuator to achieve automatic control.

[0017] Compared with the prior art, the present invention has the following beneficial technical effects: 1. High realism and strong generalization ability of environmental modeling: This invention utilizes real historical data to construct a GBDT tree model as a simulation environment for reinforcement learning, which can accurately capture the complex nonlinear logic in the wastewater treatment process. Compared with traditional mechanistic models, this environmental model can better reflect the actual operating conditions of water plants. Experimental verification shows that the environmental model constructed by this invention performs excellently in predicting total phosphorus in effluent, with a determination coefficient of... The results showed an accuracy of 0.9662, a root mean square error (RMSE) of 0.0300, and a mean absolute error (MAE) of 0.0177. This demonstrates that the current simulation environment effectively addresses the gap between training in a virtual environment and the actual working conditions.

[0018] 2. Excellent economic efficiency and phosphorus removal efficiency: Analysis of experimental results shows that, under steady-state conditions, the reinforcement learning strategy adopted in this invention reduces the average dosage by 13% to 25% compared to the traditional GBDT model and manual experience while ensuring the compliance rate of total phosphorus in the effluent. This is due to the dual constraints of efficiency score (ES) and cost score (CS) in the reward function, which enables refined control of reagent costs.

[0019] 3. Strong resistance to disturbances and control stability; Under the disturbance conditions of fluctuating influent water quality, the present invention exhibits extremely high robustness: In terms of dosing smoothness, experiments have shown that the dosing coefficient of variation (CV value) of this method is lower, avoiding drastic fluctuations in reagent addition; In terms of effluent stability, the growth rate of the standard deviation of total phosphorus in the effluent is effectively suppressed, proving that the system can still maintain stable effluent water quality when facing extreme fluctuations.

[0020] 4. Rapid Convergence and Safety Assurance Driven by Expert Knowledge: By introducing MDS and MRS, this invention successfully embeds expert experience into the training logic of the intelligent agent, accelerating training and shortening the training cycle. The TD3 agent showed an initial convergence trend by the 200th iteration, while the TD3 agent without expert model guidance only gradually showed convergence signs around the 370th iteration. Furthermore, regarding operational safety, in conjunction with SM and CP, the system proactively reserves a safety margin during continuous operation, effectively preventing the risk of excessive water discharge caused by blind algorithm exploration.

[0021] 5. High potential for automation and unattended operation; experimental comparisons during continuous operation of this invention show that its control strategy outperforms historical manual experience in terms of effluent total phosphorus stability, dosing frequency variation, and system stability indicators. This demonstrates that this solution possesses the technical capability to replace manual decision-making and achieve fully automated closed-loop control of the water plant dosing process, significantly reducing manual operation and maintenance costs. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall system architecture according to an embodiment of the present invention; Figure 2 This is a logic diagram for calculating the reward function in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the algorithm training process in an embodiment of the present invention. Detailed Implementation

[0023] Example 1 This invention proposes a reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment. Its core logic lies in constructing a highly realistic simulation environment using historical data and optimizing the reinforcement learning strategy within this environment. The specific steps are as follows: 1. Data Collection Multidimensional data from the wastewater treatment process are collected as input features, and are divided into three categories of model inputs: The expert model (GBDT, named M1) inputs include influent flow rate, total phosphorus (TP), orthophosphate, pH, and temperature at historical times, as well as the corresponding historical effluent total phosphorus.

[0024] Environmental model (GBDT, named M2) inputs include influent flow rate, total phosphorus (TP), orthophosphate, pH, and temperature at historical times, as well as the corresponding historical recommended dosage.

[0025] Reinforcement learning (RL) state space: real-time influent flow rate, total phosphorus influent, orthophosphate influent, pH influent, and influent temperature are selected as the agent's observed state.

[0026] 2. Construction of an interactive environment based on a tree model Using the environmental characteristics from step 1 as input and total phosphorus in the effluent as the output target, a gradient boosting decision tree (GBDT) model is trained using historical operational data to simulate the dynamic response of the water plant. An offline simulation interactive environment for the reinforcement learning agent is constructed, with the dosage of chemicals used as the output indicator for the expert model. The specific construction process and logical connections are as follows: Data preprocessing After data cleaning, the data is divided into training, validation, and test sets in a 6:2:2 ratio. Based on the training set, all values ​​are standardized, and the standardization process is saved.

[0027] For each dataset, the input feature vector of M1 is determined as follows: This includes the influent flow rate, total phosphorus, orthophosphate, pH, and temperature at the same time t, as well as the historical effluent total phosphorus; the input feature vector of M2 is determined as follows. This includes the influent flow rate, total phosphorus, orthophosphate, pH, temperature, and historical dosage at the same time t.

[0028] Model architecture and connection relationships This strategy employs an ensemble learning architecture, utilizing the summation of multiple regression decision trees to approximate a nonlinear function. The core logic is that each new tree is trained based on the residuals of the previous tree. For the input features... The final output of the model is:

[0029] in, The total number of trees, Representing the An independent decision tree.

[0030] Based on this, the strategy constructs a positive and negative mapping relationship between influent operating conditions, historical effluent total phosphorus, and dosing decisions and influent operating conditions, dosing decisions, and effluent total phosphorus.

[0031] Training steps and hyperparameter configuration Mean squared error (MSE) was used as the objective optimization loss function to accurately fit the continuous changes in effluent water quality. The automated tuning tool Optuna was used to search for core parameters. The parameter range included: Number of trees (n_estimators): 30–300; Learning rate: 0.001–0.2; Maximum tree depth (max_depth): 3–50; Minimum number of sample node splits (min_samples_split): 3-50; Minimum number of leaf node samples (min_samples_leaf): 3-50.

[0032] Interactive environment operation logic After the GBDT model is trained, it is encapsulated as an interactive environment interface for reinforcement learning. For the input response, the expert model at each training step... Receive current operating conditions as well as Based on the corresponding historical total phosphorus in the effluent, expert-recommended pesticide application actions are provided. Used to guide the agent; each training step In the simulation environment, the drug delivery action is received from the reinforcement learning agent. and current working conditions .

[0033] In terms of output feedback, the GBDT model outputs the predicted total phosphorus concentration in the effluent as part of the state transition. This output is then passed to the multi-objective reward calculation module to calculate the immediate reward. This completes the closed-loop interaction between the intelligent agent and the simulated environment.

[0034] Through training, the GBDT model establishes a nonlinear mapping from influent operating conditions to dosing decisions to effluent effects, thereby replacing the traditional mechanistic model and constructing an offline simulation environment that closely matches the actual operating conditions of water plants.

[0035] 3. Construction of Reinforcement Learning Control Model Here, the dual-delay deep deterministic policy gradient (TD3) algorithm is used as the core decision-making architecture, and the policy is iteratively optimized in a pre-built GBDT simulation environment.

[0036] (1) Network architecture and connectivity This invention constructs an intelligent agent logic architecture consisting of six deep neural networks: Policy Network (Actor): Contains a current policy network ( ) and a target policy network ( It employs a fully connected (FC) architecture with a 128-64-64 layer design. The hidden layers use the ReLU activation function, and the output layer uses the Tanh activation function to map the output to the normalized action space.

[0037] Valuation Network (Critic): Contains two independent current valuation networks ( ) and the corresponding two target valuation networks ( This forms a dual Critic structure. The hierarchical design is also 128-64-64, and all fully connected layers use ReLU activation.

[0038] The overall connection logic is that the Actor network receives the current state. Output drug delivery action ; Critic network reception status With action The combination of factors is used to evaluate the expected return of the current decision. value.

[0039] (2) Definition of state space and action space The state space contains only five influent operating conditions: [influent flow rate, total phosphorus, orthophosphate, pH, and temperature]. This model does not rely on temporal features or action feedback, aiming to improve its sensitivity to instantaneous operating conditions. The action space consists of the continuous numerical values ​​output by the agent, mapped through a linear transformation to obtain the pesticide dosage instructions.

[0040] (3) Dynamic guidance mechanism of expert model M1 This invention introduces parallel guidance logic for the expert model M1 during the training phase, and the specific steps are as follows: Parallel decision-making: at each sampling time step The intelligent agent provides the dosage. Simultaneously, the system calls expert model M1 in real time to calculate the expert recommendation value under the current working condition. .

[0041] Reward Adjustment: Calculation and The bias ratio. In the early stages of training, the bias adjustment term is given a very high weight, and the agent is guided to quickly master the basic drug administration logic through MDS (cooperation score) and MRS (constraint range score).

[0042] Weight decay: As the number of training rounds increases, the guiding weights gradually decrease according to a preset curve, enabling the agent to break away from expert experience in the mid-to-late stages and autonomously explore a better emission reduction drug consumption ratio through interaction with the GBDT environment.

[0043] (4) Strategy iteration and environment interaction process The agent undergoes closed-loop training at each time step according to the following process. During the environment interaction phase, the agent executes actions. The GBDT simulation environment receives the action and current operating conditions. Simulate and calculate the total phosphorus concentration in the effluent at the next time step. Calculate the immediate reward using a composite reward function. Next, during the target value estimation phase, target policy smoothing is performed. Policy noise with a standard deviation of 0.1 is added to the actions output by the target Actor, and then pruned by 0.2 to increase the policy's robustness. Simultaneously, the TD3 algorithm incorporates a double-truncation estimation method to calculate the target... When calculating the value, the minimum value output of the two target Critic networks is taken to alleviate the overestimation problem. Then, the system randomly selects samples of batch size 64 from the experience replay pool, with a discount factor of 0.99. and The learning rate is optimized using gradient descent; finally, a policy update is performed with delay, and the number of rounds is [number missing]. After every two updates to the Critic network, the Actor network and each target network are updated once, using a soft update coefficient. =0.005 achieves smooth parameter fusion, ultimately leading to continuous optimization of the phosphorus removal control strategy.

[0044] 4. Reward function design for multi-objective optimization To balance water quality compliance with economic costs, this invention designs a multi-objective composite reward function. The aim is to achieve a balance between four dimensions: optimized dosage, effluent quality compliance, collaboration of expert experience, and operational safety.

[0045] Overall architecture of the reward function

[0046] in, These are the weights of each component. A key feature of this invention is the dynamic adjustment mechanism for the weights; in the early stages of training, an expert guidance term is assigned (…). Higher weights are used to accelerate convergence in the early stages of the strategy; as training progresses, efficiency terms are gradually increased. The weights guide the agent from simple imitation learning to autonomously exploring optimal control. Specific weight changes are shown in Table 1. Table 1. Numerical summary of weight changes with Iteration

[0047] Setting boundary conditions for total phosphorus in effluent When >0.3mg / L, and The proportions are adjusted by parameter factors, while the remaining weights are compressed.

[0048] Parameter factor The settings are as follows:

[0049] when When >0.3mg / L,

[0050] At this point, other weights need to be compressed simultaneously. Specifically, the system first determines the remaining available weights. For all other issues besides water quality and safety... There are 1 target item, and their corresponding benchmark weights are defined as follows: Then the actual weights allocated to each objective item The calculation formula is:

[0051] Water Quality Assessment Index (EQI) based on piecewise nonlinear mapping To strictly control the total phosphorus in the effluent ( Regarding concentration, this invention designs EQI as a three-segment function: Excellent range ( ): Provide exponential positive rewards to incentivize agents to pursue higher processing standards.

[0052] Qualified range ( ): It decreases linearly with increasing concentration, guiding the strategy to optimize towards lower concentrations.

[0053] Exceeding the standard range ( ): Imposing high-intensity, exponential penalties to fundamentally curb illegal emissions.

[0054] Economic dual constraint design (ES and CS) This invention introduces an efficiency score (ES) and a cost score (CS) by comparing the unit drug consumption efficiency of agent decision-making and expert models (M1): ES is the ratio of phosphorus removal efficiency per unit drug consumption. The system employs segmented scoring, providing positive incentives when the agent achieves the same phosphorus removal effect with less pesticide consumption. CS directly quantifies the dosage, implementing progressively decreasing penalties for dosages exceeding expert experience benchmarks, thus achieving deep constraints on economic efficiency.

[0055] Expert knowledge fusion and collaboration mechanisms (MDS and MRS) Collaborative Score (MDS) and Constraint Range Score (MRS) embed the prior knowledge of M1 into the reward structure: MDS measures the similarity between the agent's decision and the expert's recommended solution, utilizing the bias ratio. The agent is guided to master basic control laws; the MRS establishes a tolerance range centered on expert-recommended values, and this is further refined through training rounds. Dynamically adjust tolerance radius and the intensity of punishment This ensures that the space for movement is within a safe range while preserving space for exploration.

[0056] Hard constraints (BS, SM, and CP) To ensure the system's robustness under complex operating conditions, this invention designs a multi-layered security defense: Basic safety margin (BS): Focuses on the continuous quantification of safety margin. Even if the water meets the standards, different scores will be given according to the distance from the boundary of exceeding the standard, encouraging the agent to leave a safety buffer.

[0057] Safety Multiplier (SM): Based on historical compliance rates Dynamic adjustment automatically increases the BS weight when the compliance rate is low, forcing the system to adopt a more conservative control strategy.

[0058] Hard Constraints and Penalties (CP): Set huge negative rewards for extreme drug administration or serious water quality exceeding standards as the safety bottom line of the system to prevent substantial risks from occurring during the training and exploration phase.

[0059] 5. Strategy Deployment and Real-time Control The trained RL model is deployed in the water plant control system. The system reads the parameters from the influent sensors in real time, calculates the optimal dosage using the model, and achieves closed-loop automatic control.

[0060] Taking the control test of a wastewater treatment plant in South China as an example, the test results after 4 months of continuous operation show that the TD3 model has significant advantages over the expert GBDT model in terms of cost per ton of water treatment and resistance to disturbances, while ensuring that the total phosphorus in the effluent meets the standard. See Tables 2, 3 and 4 for details.

[0061] Table 2 Comparison of TD3 strategy and Baseline performance under steady-state conditions

[0062] The effluent compliance rate was 100%. Table 3 Comparison of TD3 Strategy and Baseline Operating Costs

[0063] As shown in Tables 2 and 3, the TD3 model achieved a 100% effluent compliance rate throughout the entire testing period and significantly outperformed the GBDT model in terms of chemical cost control. Regarding chemical consumption, the TD3 model maintained a consistently low cost per ton of water treated from May to August. For example, in June, the monthly cost per ton of water for the GBDT model was 0.0617 yuan / ton, while the TD3 model's was only 0.0451 yuan / ton, resulting in a monthly chemical saving rate of 26.88%. Data shows that the TD3 model significantly reduced the average chemical dosage while maintaining a similar total phosphorus level in the effluent as the benchmark model, demonstrating the successful optimization of the dosing logic by the efficiency score (ES) in its reward function.

[0064] Table 4 (Comparison of TD3 strategy with Baseline under different disturbance conditions in simulation environment)

[0065] To address the common influent water quality fluctuations in wastewater treatment, this invention conducted perturbation tests with different proportions in a simulation environment, and the results are summarized in Table 4. The results show that the TD3 strategy exhibits superior robustness in dealing with system fluctuations. Under a high perturbation ratio of 20%, the dosing variation coefficient (CV) of the GBDT model significantly increased to 0.288, while the TD3 model still managed to control the CV at 0.215, effectively avoiding drastic fluctuations in the dosing pump and extending the equipment's service life. Furthermore, under all perturbation ratios, the growth rate of the standard deviation of total phosphorus in the effluent from the TD3 scheme was lower than that of the baseline model, demonstrating the system's stability advantage under extreme environments.

[0066] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A reinforcement learning-based method for controlling phosphorus removal agent dosing in a tree-model simulation environment, characterized in that, The specific steps include the following: S1. Data Acquisition: Collect multi-dimensional operational data during the wastewater treatment process, including influent flow rate, total phosphorus in the influent, orthophosphate in the influent, pH in the influent, influent temperature, historical total phosphorus in the effluent, and historical recommended dosage. S2. Simulation environment construction: Based on historical operating data, a gradient boosting decision tree model is trained to construct an offline simulation environment for simulating the dynamic response of wastewater treatment. The simulation environment takes the current influent conditions and dosing actions as inputs and outputs the predicted total phosphorus concentration in the effluent. S3. Reinforcement learning model training: In an offline simulation environment, a reinforcement learning agent is trained, with the water inflow condition as the state space, the dosage as the action space, and a multi-objective composite reward function to guide the agent to optimize the dosage strategy. S4. Control strategy deployment: The trained reinforcement learning model is deployed in the water plant control system to output the optimal dosage in real time according to the water intake conditions, so as to realize closed-loop automatic control.

2. The reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment according to claim 1, characterized in that, Step S2, the simulation environment construction, also includes the following steps: S21. Construct an expert model M1, whose inputs are the influent flow rate, total phosphorus, orthophosphate, pH, temperature and total phosphorus in the effluent at historical times, and whose output is the expert-recommended dosage. S22. Construct an environmental model M2, whose inputs are historical influent flow rate, total phosphorus influent, orthophosphate influent, pH influent, influent temperature, and historical recommended dosage, and whose output is the predicted total phosphorus concentration in effluent. The environment model M2 serves as the interactive environment for the reinforcement learning agent, while the expert model M1 is used to guide the agent's exploration direction in the early stages of training.

3. The reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment according to claim 1, characterized in that, In step S3, the reinforcement learning model is a dual-delay deep deterministic policy gradient algorithm, and its network architecture includes: A current policy network and a target policy network are used to output the drug delivery action; Two current valuation networks and two target valuation networks are used to evaluate the expected return of state-action pairs; Both the policy network and the valuation network are fully connected, with a hidden layer size of 128-64-64, using ReLU activation function, and the output layer uses Tanh activation function.

4. The reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment according to claim 1, characterized in that, The multi-objective composite reward function in step S3 is composed of the following weighted components: Water Quality Assessment Index (EQI): Based on a piecewise nonlinear mapping of total phosphorus concentration in effluent, it is used to incentivize compliance and penalize exceedance. Efficiency score (ES) and cost score (CS): Based on phosphorus removal efficiency per unit of pesticide consumption and dosage, used to optimize pesticide costs; Collaborative Score (MDS) and Constraint Range Score (MRS): Recommended values ​​based on expert model M1, used to guide the agent in learning basic control logic; Security constraints: Used to ensure the security of system operation.

5. The reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment according to claim 4, characterized in that, Safety constraints include: The basic safety margin (BS) is used to evaluate the continuous quantification of safety margin. Even if the effluent meets the standards, different scores will be given according to the distance from the boundary of exceeding the standard, leaving a safety buffer. The safety multiplier SM is dynamically adjusted based on historical compliance rates. When the compliance rate is low, the BS weight is automatically increased, forcing the system to adopt a more conservative control strategy. Strict constraints and penalties are imposed on CPs (Consumer Protection Centers), with substantial negative rewards set for extreme drug administration or serious water quality exceeding standards. This serves as the system's safety baseline to prevent substantial risks during the training and exploration phase.

6. The reinforcement learning-based phosphorus removal agent dosing control method based on a tree model simulation environment according to claim 4, characterized in that, The weights of each component in the multi-objective composite reward function are dynamically adjusted with each training epoch: In the early stages of training, the weight of expert guidance terms is increased to accelerate policy convergence; In the later stages of training, the weight of efficiency terms is increased to encourage the agent to autonomously explore better strategies. When the total phosphorus in the effluent exceeds the standard, the weight of water quality and safety items will be automatically adjusted, while the weight of other items will be reduced to ensure that the system prioritizes meeting the standard requirements.

7. A reinforcement learning-based phosphorus removal agent dosing control system based on a tree model simulation environment, wherein the control is performed using the method described in any one of claims 1-6, characterized in that, include: The data acquisition unit is used to collect multidimensional data from the wastewater treatment site and output the real-time status St. Historical database, used to store historical operational data; The intelligent control unit communicates with the data acquisition unit and the historical database, specifically including: The simulation environment construction module is pre-trained based on historical operating data in the historical database and is used to simulate the dynamic response of the wastewater treatment process. The input of the simulation environment module receives the real-time status St and the dosing action At, and the output outputs the predicted total phosphorus concentration of the effluent at the current moment. The expert experience module is pre-trained based on historical operational data in the historical database and is used to provide expert experience references. The input of the expert experience module receives the real-time status St, and the output outputs the expert-recommended dosage At_exp. The intelligent agent module is connected to the data acquisition unit, the simulation environment construction module and the expert experience module, respectively. It is used to output the drug delivery action At according to the real-time state St, and to receive the reward signal R during the training phase to optimize the control strategy. The calculation module is connected to the simulation environment construction module, the expert experience module, and the intelligent agent module respectively. It is used to calculate the multi-objective composite reward R based on the total phosphorus concentration in the effluent output by the simulation environment construction module, the expert-recommended dosage At_exp output by the expert experience module, and the dosing action At output by the intelligent agent module, and then feed it back to the L intelligent agent module. The actuator, connected to the intelligent agent module, is set up at the wastewater treatment site to receive the optimal dosing action At output by the intelligent agent module and control the variable frequency dosing pump to dosing the agent accordingly.

8. The reinforcement learning-based phosphorus removal agent dosing control system based on a tree model simulation environment according to claim 7, characterized in that, The simulation environment building module includes: The expert model unit M1 is used to output the expert-recommended dosage based on the influent conditions and historical total phosphorus in the effluent. The environmental model unit M2 is used to predict the total phosphorus concentration in the effluent based on the influent conditions and chemical dosing actions. The M2 unit serves as the interactive environment for the reinforcement learning agent, while the M1 unit is used for policy guidance during the initial training phase.

9. The reinforcement learning-based phosphorus removal agent dosing control system based on a tree model simulation environment according to claim 7, characterized in that, The reward function in the calculation module is a multi-objective composite reward function Rtotal, which includes water quality evaluation index, efficiency score, cost score, expert collaboration score, constraint range score and safety constraint term, and the weights of each are dynamically adjusted with each training round.

10. The reinforcement learning-based phosphorus removal agent dosing control system based on a tree model simulation environment according to claim 7, characterized in that, The actuator reads the parameters of the inlet water sensor in real time, calculates the optimal dosage through a reinforcement learning model, and outputs the result to the dosing actuator to achieve automatic control.

Images