Intelligent control method and system for wastewater treatment plants based on Bayesian optimization reinforcement learning

Abstract

The present invention discloses an intelligent control method and system for a wastewater treatment plant based on Bayesian optimization reinforcement learning. The present invention uses professional simulation software to model an operating wastewater treatment plant, and the model uses actual influent data of the wastewater treatment plant; uses reinforcement learning to perform multi-objective optimization, and interacts the reinforcement learning algorithm with the professional simulation software. At each time step, the intelligent body receives state information and takes actions to adjust operating parameters; uses Bayesian optimization to select the initial action point of reinforcement learning, sets an objective function with Bayesian optimization, receives control points and influent data, and obtains effluent water quality indicators by interacting with professional simulation software, and outputs target values; the optimization process uses a Gaussian process to identify the best parameter combination that optimizes the target value. The present invention has universality and practicality, enhances the adaptability to complex and real-time changing influent indicators in the WWTP process, and improves the reliability and stability of the processing performance.

Description

Technical Field

[0001] The invention relates to an intelligent control method and system for a wastewater treatment plant based on Bayesian optimization reinforcement learning, and belongs to the technical field of environmental protection. Background Art

[0002] The use of intelligent control in wastewater treatment plants (WWTPs) is becoming increasingly popular. From an environmental sustainability perspective, efficient control of wastewater treatment plants ensures the reduction of pollutants and harmful substances in the discharged water, thereby protecting aquatic ecosystems and human health. Strict environmental regulations require wastewater treatment plants to meet specific discharge standards, and intelligent control systems help to maintain these standards continuously. In terms of operational efficiency, intelligent control systems enable real-time monitoring and control of treatment processes, thereby improving performance and responding quickly to changes in influent characteristics. Advanced control strategies can optimize the use of energy-intensive processes, thereby reducing overall energy consumption throughout the plant. The economic benefits of intelligent control are also significant. By optimizing chemical use and energy consumption, intelligent control systems can significantly reduce operating costs. Predictive maintenance supported by intelligent control systems can prevent equipment failures and reduce downtime, ensuring continuous operation and extending the service life of plant infrastructure.

[0003] Existing reinforcement learning (RL) algorithms for wastewater treatment plant control are mainly focused on the BSM1 (Benchmark Simulation Model no.1) benchmark scenario, with limited testing on more realistic scenarios, which limits the demonstration of wider applicability of these control algorithms. In addition, existing studies often simplify the simulation of wastewater treatment plants, so their operating strategies may have limitations in actual control. Therefore, it is necessary to test the versatility of RL algorithms in wastewater treatment plant control in a wider range of scenarios and merge these algorithms with professional simulation software to develop control strategies that better meet actual operational needs.

[0004] Parameter optimization is a critical step in reinforcement learning because it affects the performance and convergence of the learning algorithm. Properly optimized parameters can speed up the convergence of RL algorithms and improve performance, thereby enhancing the stability and robustness of RL models and reducing the possibility of divergence or suboptimal learning. Bayesian optimization (BO) is able to efficiently search complex high-dimensional spaces in minimal evaluations, achieving this goal by balancing exploration and exploitation. It does this by using probabilistic models to predict the performance of different parameter settings. In wastewater treatment engineering research, several studies have successfully applied Bayesian optimization (BO) for hyperparameter tuning of machine learning models. However, most studies that apply BO to optimize RL focus on common parameters such as learning rate, discount factor, and exploration rate. In the context of optimizing wastewater treatment plants, choosing the right action starting point may be more beneficial. This is because optimizing the best initial action point through BO will prioritize exploring action areas associated with high potential benefits, which will make the optimization process more efficient and more in line with the actual needs of the application. Therefore, the current study will focus on using Bayesian optimization to find the action starting point instead of optimizing common parameters in RL. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a wastewater treatment plant intelligent control method and system based on Bayesian optimization reinforcement learning, to improve the universality and practicality of the RL algorithm for WWTP control, to enhance the robustness to the inherent uncertainty and variability in the WWTP process, and to improve the reliability and stability of the treatment performance.

[0006] Technical solution: In order to achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0007] An intelligent control method for a wastewater treatment plant based on Bayesian optimization reinforcement learning comprises the following steps:

[0008] Use professional simulation software to model the operating wastewater treatment plant. The simulation model uses the actual influent data of the wastewater treatment plant, including the time-varying data of flow rate, chemical oxygen demand and total Kjeldahl nitrogen;

[0009] Use reinforcement learning for multi-objective optimization, interact the reinforcement learning algorithm with professional simulation software, integrate the simulation environment with the reinforcement learning algorithm agent, and at each time step, the agent receives state information, including flow, chemical oxygen demand, and total Kjeldahl nitrogen, and takes actions to adjust operating parameters; the optimization goals of the reinforcement learning algorithm include reducing operating costs, improving processing efficiency, and minimizing energy consumption.

[0010] The Bayesian optimization is used to select the initial action point of reinforcement learning. The Bayesian optimization sets an objective function, which receives the control point and the water inlet data, obtains the effluent water quality index by interacting with the professional simulation software, and calculates according to the specific optimization target set to output the target value; the optimization process adopts the Gaussian process to identify the best parameter combination that makes the target value optimal, and the control point in the best parameter combination is used as the initial action point of reinforcement learning; the optimal target value is the one that meets the effluent water quality conditions with the minimum operating cost, the highest processing efficiency or the minimum energy consumption.

[0011] In some embodiments, the wastewater treatment plant adopts an anaerobic-anoxic-aerobic treatment process, and the reinforcement learning actions include: internal circulation liquid return, dissolved oxygen set point in the aerobic reactor, and carbon additive dosage in the anoxic reactor; the action range of dissolved oxygen is set between 0.5 and 2 mg / L, and is adjusted by 0.2 mg / L each time, the action range of internal circulation liquid return is from 0 to 90,000 m³ / day, and is adjusted by 10,000 m³ / day each time, and the carbon dosage is set between 0 and 5 m³ / day, and is adjusted by 0.5 m³ / day each time.

[0012] In some embodiments, the wastewater treatment plant adopts a three-stage anoxic-aerobic treatment process, and the reinforcement learning actions include the dissolved oxygen set point in the aerobic reactor and the carbon additive dosage in the anoxic reactor. The action range of dissolved oxygen is set between 0.5 and 2 mg / L, and is adjusted by 0.5 mg / L each time. The carbon dosage is set between 0 and 5 m³ / day, and is adjusted by 0.5 m³ / day each time.

[0013] Further preferably, the professional simulation software adopts Dynamita Sumo, the water inflow data of the simulation model is monitored daily, and the rainfall pattern and storm pattern of BSM1 are incorporated into the water inflow.

[0014] Further preferably, the operating cost is evaluated by a cost index provided by professional simulation software, taking into account costs related to energy consumption, carbon addition and sludge treatment; the treatment efficiency is evaluated by an effluent quality index based on a BSM1 evaluation algorithm; and the energy consumption is evaluated by an energy index, taking into account the energy consumed in each wastewater treatment process.

[0015] Further preferably, the reinforcement learning algorithm adopts a deep Q-network algorithm to support extended optimization objectives.

[0016] Further preferably, the reward of the reinforcement learning algorithm is set according to the following method:

[0017] A basic reward is allocated for reaching the threshold of effluent quality standards;

[0018] When the NOx concentration in the anaerobic reactor exceeds the set value, additional rewards are allocated;

[0019] The cost index, water quality index and energy index used to evaluate operating costs, treatment efficiency and energy consumption allocate rewards inversely proportional to the index. The lower the index, the higher the reward. The total reward summary value of the index is fixed through the inverse proportionality coefficient.

[0020] If effluent quality regulations are not met, penalties are imposed to discourage operations that do not meet established environmental standards.

[0021] Further preferably, the output of the objective function of the Bayesian optimization is the sum of the cost index and the generated penalty; if the effluent water quality exceeds the specified discharge limit, the penalty is imposed.

[0022] Based on the same inventive concept, the present invention provides an intelligent control system for a wastewater treatment plant based on Bayesian optimization reinforcement learning, comprising:

[0023] Professional simulation module, used to model the operating wastewater treatment plant using professional simulation software. The simulation model uses the actual influent data of the wastewater treatment plant, including the time-varying data of flow rate, chemical oxygen demand and total Kjeldahl nitrogen;

[0024] A reinforcement learning module is used to perform multi-objective optimization using reinforcement learning. The reinforcement learning algorithm interacts with professional simulation software. The simulation environment is integrated with the reinforcement learning algorithm agent. At each time step, the agent receives state information, including flow rate, chemical oxygen demand, and total Kjeldahl nitrogen, and takes actions to adjust operating parameters. The optimization goals of the reinforcement learning algorithm include reducing operating costs, improving processing efficiency, and minimizing energy consumption.

[0025] And an initial action point optimization module, which is used to select the initial action point of reinforcement learning using Bayesian optimization. The Bayesian optimization sets an objective function, which receives control points and water inlet data, obtains effluent water quality indicators by interacting with professional simulation software, and calculates according to the specific optimization goals set, and outputs the target value; the optimization process uses a Gaussian process to identify the best parameter combination that makes the target value optimal, and uses the control point in the best parameter combination as the initial action point of reinforcement learning; the optimal target value is the one that meets the effluent water quality conditions with the lowest operating cost, the highest processing efficiency or the lowest energy consumption.

[0026] Based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning.

[0027] Beneficial effects: The present invention provides a wastewater treatment plant intelligent control method and system based on Bayesian optimization reinforcement learning, which further explores the application of RL to achieve more advanced control of WWTP operations. Compared with the prior art, it has the following advantages: 1) The present invention tests the versatility of RL algorithms in WWTP control in a wider range of scenarios, and derives more practical strategies through interaction with professional simulation software, with specific universality and practicality. 2) Explore the potential of integrating BO with RL for WWTP optimization, focusing on how to select the initial action point will help multi-objective optimization. This knowledge will help develop more adaptive, efficient and cost-effective WWTP control systems that can dynamically respond to different operating conditions and compliance requirements. Experiments show that integrating BO with RL will enhance the robustness of the control solution to the inherent uncertainty and variability in the WWTP process, thereby improving the reliability and stability of treatment performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The figure is a schematic diagram of the overall process of an embodiment of the present invention.

[0029] Figure 2 A schematic diagram of reward settings for a reinforcement learning algorithm in an embodiment of the present invention.

[0030] Figure 3 Schematic diagram of the process of Bayesian optimization of initial action points in an embodiment of the present invention.

[0031] Figure 4 Figure 1 shows a comparison of knowledge-based control and RL control strategies for Plant R and Plant Y during a 15-day test period. (A) and (C) show the effluent COD, ammonia nitrogen, and total nitrogen levels over time, with threshold lines used to indicate regulatory limits for effluent quality. Comparative analysis of EQI, CI, and PI using RL control strategy and knowledge-based strategy for Plant R (B) and Plant Y (D). (B) and (D) show the percentage changes in these indices achieved by RL control.

[0032] Figure 5 This is a comparison chart of the RL control strategy and the RL+BO control strategy during the test period of the R plant. (A) shows the effluent COD, ammonia nitrogen and total nitrogen levels over time, and (B) shows the comparative analysis results of the EQI, CI and PI of the R plant using the RL control strategy and the RL+BO strategy. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0034] like Figure 1As shown, an intelligent control method for a wastewater treatment plant based on Bayesian optimization reinforcement learning disclosed in an embodiment of the present invention mainly includes: using professional simulation software to model an operating wastewater treatment plant, the simulation model uses actual influent data of the wastewater treatment plant, including time-varying data of flow rate, chemical oxygen demand and total Kjeldahl nitrogen; using reinforcement learning for multi-objective optimization, interacting the reinforcement learning algorithm with the professional simulation software, integrating the simulation environment with the reinforcement learning algorithm agent, and at each time step, the agent receives state information, including flow rate, chemical oxygen demand and total Kjeldahl nitrogen, and takes actions to adjust operating parameters; using Bayesian optimization to select the initial action point of reinforcement learning, Bayesian optimization sets an objective function, the objective function receives control points and influent data, and obtains effluent water quality indicators by interacting with professional simulation software, and calculates according to the set specific optimization objectives to output the target value; the optimization process adopts a Gaussian process to identify the best parameter combination that optimizes the target value, and uses the control point in the best parameter combination as the initial action point of reinforcement learning; the optimal target value is to minimize the operating cost, maximize the treatment efficiency or minimize the energy consumption under the condition of satisfying the effluent water quality.

[0035] The optimization goal of the reinforcement learning algorithm may include one or more of reducing operating costs, improving processing efficiency, and minimizing energy consumption. The professional simulation software may adopt Dynamita Sumo, and the reinforcement learning algorithm may adopt a deep Q network algorithm, or other tools or algorithms with specific similar functions.

[0036] Operating costs can be evaluated through a cost index provided by professional simulation software, taking into account costs associated with energy consumption, carbon addition and sludge treatment; treatment efficiency can be evaluated through an effluent quality index based on the BSM1 evaluation algorithm; and energy consumption can be evaluated through an energy index, taking into account the energy consumed in each wastewater treatment process.

[0037] The rewards for the reinforcement learning algorithm can be set as follows:

[0038] A basic reward is allocated for reaching the threshold of effluent quality standards;

[0039] When the NOx concentration in the anaerobic reactor exceeds the set value, additional rewards are allocated;

[0040] The cost index, water quality index and energy index used to evaluate operating costs, treatment efficiency and energy consumption allocate rewards inversely proportional to the index. The lower the index, the higher the reward. The total reward summary value of the index is fixed through the inverse proportionality coefficient.

[0041] Imposing penalties if effluent quality regulations are not met to discourage operations that do not meet established environmental standards

[0042] The output of the objective function of the Bayesian optimization can be the sum of the cost index and the penalty imposed; if the effluent water quality exceeds the specified discharge limit, the penalty is imposed.

[0043] Taking two actually operating wastewater treatment plants as examples, the process of the wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning described in the present invention is described in detail.

[0044] 1. Sewage Treatment Plant Simulation

[0045] Two operational wastewater treatment plants managed by China Water Management, Plant R and Plant Y, were modeled using Dynamita Sumo. Plant R uses an anaerobic-anoxic-aerobic (AAO) treatment process, where wastewater is first introduced into an anaerobic reactor to decompose organic matter. To enhance denitrification, internal liquid mixing and recirculation are implemented to increase the concentration of nitrogen oxides (NOx) in the anoxic reactor. In addition, volatile fatty acids (VFA) are introduced as an external carbon source to further support the denitrification process. In contrast, Plant Y uses a three-stage anoxic-aerobic treatment process, where wastewater undergoes anoxic and aerobic conditions in sequence. This approach promotes the growth of specific microbial communities. It is worth noting that Plant Y does not use internal mixing, as this may disrupt the specific conditions required for optimal microbial function, which may affect the overall performance of the treatment process. Similar to Plant R, VFA carbon sources are also added to the anoxic reactor 3 of Plant Y to promote denitrification.

[0046] The simulation model uses actual influent data from the two plants, including time-varying data on flow (Q), chemical oxygen demand (COD), and total Kjeldahl nitrogen (TKN). These monitoring data were recorded daily from 2023 to the first half of 2024. In order to take into account daily variations and evaluate the algorithm's response performance to extreme scenarios, the rainfall pattern and storm pattern of BSM1 were incorporated into the influent of R and Y plants, respectively.

[0047] 2. Multi-objective optimization using reinforcement learning

[0048] This example uses a reinforcement learning deep Q network (DQN) algorithm to interact with Dynamita Sumo, using a python-based digital twin toolkit as an interface. The DQN architecture consists of a feedforward neural network with one hidden layer, designed to approximate the value of various actions based on the current state of the system. The network is trained using experience replay, storing transitions consisting of the current state, the action taken, the reward received, and the subsequent state in a memory buffer. The agent selects actions using an epsilon-greedy strategy that balances exploration and exploitation to maximize the cumulative reward over a series of episodes. The simulation environment is integrated with the DQN agent, allowing real-time interaction with the wastewater treatment model. At each time step, the agent receives state information, including flow, chemical oxygen demand, and total Kjeldahl nitrogen, and takes actions to adjust operating parameters.

[0049] For Plant R, the action range of the dissolved oxygen (DO) set point in the aerobic reactor was set between 0.5 and 2 mg / L in steps of 0.2 mg / L. The action range of the IMLR was set from 0 to 90,000 m³ / day in steps of 10,000 m³ / day, reflecting the historical operating record of the internal cycle of Plant R, ensuring that the upper limit exceeded the historical maximum. The carbon dose was set between 0 and 5 m³ / day in steps of 0.5 m³ / day, thus covering the maximum carbon dose calculated from domain knowledge and control strategy. Similarly, for Plant Y, the action range of DO was set from 0.5 to 2 mg / L in steps of 0.5 mg / L. The carbon dose parameters were the same as those for Plant R, set between 0 and 5 m³ / day in steps of 0.5 m³ / day, thus following the maximum carbon dose for Plant Y determined through domain knowledge and control strategy analysis.

[0050] In this example, the development and evaluation of a reinforcement learning algorithm aims to optimize three interrelated objectives: reducing costs, improving treatment efficiency, and minimizing energy consumption. Operating costs are evaluated by the cost index (CI) provided by Sumo software, which reflects the total operating costs of the plant, including costs related to energy consumption, carbon addition, sludge treatment, and others. Treatment efficiency is evaluated by the effluent quality index (EQI) based on the BSM1 evaluation algorithm. This comprehensive index includes various parameters such as effluent suspended solids (TSS), COD, TKN, NOx, BOD5, and effluent flow rate. The lower the EQI value, the better the treatment performance. Energy consumption is evaluated by the energy index (EI), which takes into account the energy consumed in processes such as internal mixing flow, return activated sludge rate, and waste energy rate. The calculation of EQI and EI follows the calculation rules of BSM1. The balance of these objectives is an example of multi-objective optimization achieved through the algorithm. The algorithm is designed to be extensible to allow the inclusion of other indicators such as greenhouse gas (GHG) emissions to promote more sustainable management practices within the WWTP.

[0051] During the training phase, the reinforcement learning algorithm utilized the annual (365 records) water inflow data for plants R and Y in 2022. After interpolation according to the BSM1 model, each training dataset contained one record every 6 hours, for a total of 1,460 records in a year. The records used for testing were from January 1 to January 14, 2024, providing 56 water inflow records for each plant. These 14 days included weather-related spikes according to the BSM1 rainfall and storm patterns, providing a basis for testing the adaptability of various control strategies under dynamic environmental conditions.

[0052] The rewards of reinforcement learning algorithms are as follows Figure 2 As shown. Rewards are allocated based on achieving effluent quality standards: effluent COD ≤ 40 mg / L, effluent ammonia ≤ 3 mgN / L, and effluent TN ≤ 15 mgN / L. Achieving these thresholds results in a base reward of 0.6. An additional reward of 0.1 is awarded when the NOx concentration in the anaerobic reactor exceeds 1 mg / L, indicating improved nitrification efficiency. Further incentives are also incorporated into the reward function to promote cost-effective and environmentally sustainable operations. CI, EQI, and EI are inversely proportional to the rewards awarded (the lower these indices, the higher the rewards), thereby encouraging cost reduction, improved treatment efficiency, and reduced energy consumption. The inverse proportionality coefficient ensures that each index generates a reward of approximately 0.33, and the total reward for all three indices is summed to 1. For example, when the EQI is approximately equal to 3000, setting the inverse coefficient to 1000 means that when the discharge regulations are met, the associated reward from the EQI will be approximately 0.33. Conversely, if the plant fails to meet the effluent quality regulations, a penalty of -1 is imposed, effectively discouraging operation that does not meet the established environmental standards. The design of this reward function gives a reward of around 1 when the emission standards are met and a penalty of -1 when they are not, maximizing the learning efficiency of the algorithm agent.

[0053] 3. Bayesian optimization is used to find the starting point of the action

[0054] The process of BO for selecting the initial action point of reinforcement learning application is as follows Figure 3 As shown. By using BO to select the initial action point, the speed at which the agent reaches the optimal control strategy can be accelerated, which may increase the efficiency of the agent in discovering the optimal strategy, or conversely, mitigate the risk of performance degradation. This embodiment was tested on the R plant, which represents a typical AAO process widely used in wastewater treatment plants, and can evaluate the effectiveness of the algorithm within a mature operating framework.

[0055] The core of Bayesian optimization is to formulate an objective function containing three control points: (i) internal liquid return (IMLR), (ii) dissolved oxygen (DO) set point in the aerobic reactor, and (iii) carbon additive dosage in the anoxic reactor. In addition, the objective function also receives influent characteristic data, including flow (Q), chemical oxygen demand (COD), and total Kjeldahl nitrogen (TKN). The objective function interacts with the SUMO simulation model and outputs a cost index and effluent water quality indicators, such as effluent COD, effluent ammonia nitrogen, and effluent TN. If the effluent water quality exceeds the specified discharge limit, the objective function will impose a significant penalty. The output of the objective function is the sum of the cost index and any penalties generated. In this example, the ability of BO was tested in the context of optimizing the total cost. It will be appreciated by those skilled in the art that the objective function can be adjusted to optimize other objectives (for example, the output of the objective function is the sum of the effluent quality index and any penalties generated, or the sum of the energy index and any penalties generated), thereby enhancing its applicability in different operating scenarios and objectives.

[0056] After the objective function is developed, the optimization process uses Gaussian processes (GPs) as the underlying mechanism for parameter tuning. Gaussian processes assume that the values ​​of different inputs are random variables with a joint Gaussian distribution, which enables output predictions for unseen inputs based on this probabilistic model. Using the gp_minimize function in the scikit-optimize library, the Gaussian process can identify the best combination of parameters that minimizes the cost (including any penalties), which is determined by the objective function.

[0057] During the optimization process, the selection of the next parameter evaluation point is guided by a low confidence bound (LCB) acquisition function. This acquisition function balances between exploring new areas and exploiting known good areas, effectively making a trade-off between exploring unknown parameter space and refining known solutions. This process continues until a preset convergence criterion is reached, ensuring that the optimization process is both comprehensive and targeted in achieving the most cost-effective and satisfactory action starting point for subsequent reinforcement learning.

[0058] The following experimental results illustrate the universality of the reinforcement learning algorithm in the present invention and the advantages of using Bayesian optimization.

[0059] Figure 4(A) and (B) show the comparative performance of the knowledge-based and reinforcement learning (RL) control strategies at the R plant. Both strategies were very effective in keeping the effluent COD level below the discharge limit, indicating that each method excels in chemical pollutant management. Overall, the RL strategy showed lower volatility and greater stability in controlling effluent water quality compared to the knowledge-based strategy, as evidenced by the tighter clustering of its data points and fewer spikes exceeding the discharge threshold. Although the knowledge-based strategy was effective to some extent, it showed higher volatility, especially in controlling ammonia nitrogen and total nitrogen levels. In terms of ammonia nitrogen control, the RL strategy consistently maintained ammonia nitrogen levels significantly below the threshold of 3 mg / L throughout the test period, showing strong performance. In contrast, the knowledge-based strategy was more volatile, with frequent spikes approaching or exceeding the 3 mg / L limit. The comparison of the two algorithms in controlling effluent ammonia nitrogen highlights the superior capabilities of the RL algorithm, especially in terms of adaptability and reliability, and better ability to meet stringent environmental standards. In terms of total nitrogen control, the RL strategy performed well, maintaining total nitrogen levels below the limit of 15 mg / L throughout the test period. In contrast, while the knowledge-based strategy was generally effective, it more frequently experienced spikes that approached or occasionally exceeded the 15 mg / L threshold. This performance fluctuation may be due to the knowledge-based strategy’s inability to provide the most reasonable carbon dosing strategy when the influent carbon source is insufficient. This deficiency weakens the effectiveness of the knowledge-based strategy, especially when the influent conditions fluctuate, affecting its ability to consistently meet regulatory limits.

[0060] like Figure 4 As shown in (B), under the RL strategy, the effluent quality index (EQI) was reduced by 15.07%. This improvement is attributed to the occasional effluent ammonia nitrogen peaks under the knowledge-based control strategy, which have a significant impact on the EQI calculation according to the BSM1 formula. Under RL control, the power index was reduced by 9.72%, although the cost index (CI) increased slightly by 2.08%, which is negligible. Therefore, the RL strategy significantly enhances the adaptability of the algorithm and reduces costs, while achieving better environmental compliance at almost the same operating expenses.

[0061] like Figure 4 As shown in (C), when controlling Plant Y, both strategies performed similarly in managing COD; however, the RL strategy reduced the number of spikes and showed a more stable overall trend. In terms of total nitrogen (TN) control, the RL strategy maintained higher TN levels, likely due to reduced carbon dosage, resulting in higher effluent TN, but still within regulatory limits. Both strategies successfully kept ammonia nitrogen levels within emission standards, but the RL strategy showed a clear advantage in reducing the occurrence of spikes, highlighting its superior performance in control. As Figure 4As shown in (D), for Plant Y, the RL strategy increased the effluent quality index (EQI) by 1.25% compared to the knowledge-based control, but significantly reduced the power index and cost index by 4.21% and 15.07%, respectively, indicating that the RL strategy performed better in terms of cost effectiveness and energy efficiency. This indicates that the RL strategy optimizes resource allocation and strikes a balance between operating costs and effluent quality while still maintaining regulatory compliance.

[0062] In summary, the comprehensive analysis of the performance of the RL strategy in WWTPs R and Y shows that the RL strategy is more versatile in different operating scenarios and can provide better adaptability, cost savings, and energy efficiency, especially in complex treatment processes with changing influent conditions.

[0063] Figure 5 (A) shows the comparative analysis of effluent water quality parameters between RL and RL+BO over the 15-day test period. In terms of effluent total nitrogen (Total-N) control, while both strategies successfully kept total nitrogen levels below the regulatory threshold of 15 mg / L, RL+BO showed more effective control, as evidenced by fewer peaks and lower overall volatility. Higher average total nitrogen levels were observed for RL+BO, although still within regulatory standards. This suggests that RL+BO may have reduced carbon dosing in relevant process steps to save costs while ensuring regulatory compliance, as the nitrogen removal process is energy-intensive and has a high carbon demand. Regarding effluent ammonia-N (Ammonia-N), only the RL strategy showed higher volatility, with some values ​​approaching or exceeding the ammonia emission limit. In contrast, RL+BO maintained more stable ammonia control, always keeping ammonia nitrogen levels below the threshold of 4 mg / L. This shows that integrating Bayesian optimization (BO) into the RL strategy effectively mitigated the ammonia nitrogen spikes that had previously occurred in knowledge-based control, providing a more reliable solution for ammonia control. In addition, Figure 5 The standard deviation of each point in (A) across three repeated runs is smaller under the RL+BO strategy than under the RL-only strategy. This reduction in standard deviation indicates that the RL+BO strategy improves repeatability and consistency, further demonstrating the additional advantage of Bayesian optimization in enhancing the robustness and reliability of RL control strategies.

[0064] Figure 5(B) compares the key performance indicators - effluent quality index (EQI), power index (PI) and cost index (CI) under RL and RL+BO strategies. The results clearly show that after Bayesian optimization, the cost index was significantly reduced by 45.94%. This reduction is attributed to the fact that Bayesian optimization helps the algorithm identify a more appropriate starting point for action, effectively reducing aeration and internal recirculation, thereby reducing operating costs. In terms of power index (PI), RL+BO reduces energy consumption by 12.02%. This reduction in energy consumption not only reduces operating costs, but also brings environmental benefits, because the reduction in energy consumption leads to lower greenhouse gas emissions, thereby reducing the environmental footprint of the wastewater treatment process.

[0065] Although RL+BO resulted in a slight increase in the effluent quality index (EQI) of 3.47%, the effluent quality still complies with all regulatory limits. Therefore, compliant effluent quality and a slightly increased EQI were achieved with overall cost savings, which provides a better balance between the objectives. Figure 5 As shown in (B), the error bars of the RL+BO strategy are significantly smaller, indicating that the robustness and repeatability of the strategy have been improved. In summary, the RL strategy enhanced by Bayesian optimization to select the best action starting point achieved excellent multi-objective optimization, effectively reduced costs, avoided excessive effluent quality, saved energy, and improved the overall repeatability and robustness of the control strategy.

[0066] Based on the same inventive concept, the embodiment of the present invention also discloses an intelligent control system for a wastewater treatment plant based on Bayesian optimization reinforcement learning, including:

[0067] Professional simulation module, used to model the operating wastewater treatment plant using professional simulation software. The simulation model uses the actual influent data of the wastewater treatment plant, including the time-varying data of flow rate, chemical oxygen demand and total Kjeldahl nitrogen;

[0068] A reinforcement learning module is used to perform multi-objective optimization using reinforcement learning. The reinforcement learning algorithm interacts with professional simulation software. The simulation environment is integrated with the reinforcement learning algorithm agent. At each time step, the agent receives state information, including flow rate, chemical oxygen demand, and total Kjeldahl nitrogen, and takes actions to adjust operating parameters. The optimization goals of the reinforcement learning algorithm include reducing operating costs, improving processing efficiency, and minimizing energy consumption.

[0069] And an initial action point optimization module, which is used to select the initial action point of reinforcement learning using Bayesian optimization. The Bayesian optimization sets an objective function, which receives control points and water inlet data, obtains effluent water quality indicators by interacting with professional simulation software, and calculates according to the specific optimization goals set, and outputs the target value; the optimization process uses a Gaussian process to identify the best parameter combination that makes the target value optimal, and uses the control point in the best parameter combination as the initial action point of reinforcement learning; the optimal target value is the one that meets the effluent water quality conditions with the lowest operating cost, the highest processing efficiency or the lowest energy consumption.

[0070] The embodiment of the present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning.

[0071] The program / instruction code for implementing the method of the present invention can be written in any combination of one or more programming languages. These programs / instruction codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program / instruction code, when executed by the processor or controller, enables the steps of the method of the present invention to be implemented. The program / instruction code can be executed completely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine or completely on a remote machine or server. The parts not described in detail in the present invention are all known technologies of those skilled in the art.

Claims

1. A wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning, characterized in that: The steps include: Use professional simulation software to model the operating wastewater treatment plant. The simulation model uses the actual influent data of the wastewater treatment plant, including the time-varying data of flow rate, chemical oxygen demand and total Kjeldahl nitrogen; Use reinforcement learning for multi-objective optimization, interact with the reinforcement learning algorithm and professional simulation software, integrate the simulation environment with the reinforcement learning algorithm agent, and at each time step, the agent receives state information, including flow, chemical oxygen demand, and total Kjeldahl nitrogen, and takes actions to adjust operating parameters; The initial action point of reinforcement learning is selected using Bayesian optimization. The Bayesian optimization sets an objective function, which receives control points and inlet data, obtains effluent water quality indicators by interacting with professional simulation software, and calculates according to the specific optimization goals set to output the target value. The optimization process uses a Gaussian process to identify the best parameter combination that optimizes the target value, and uses the control point in the best parameter combination as the initial action point of reinforcement learning. The optimal target value is the one that satisfies the effluent water quality conditions with the lowest operating cost, the highest processing efficiency or the lowest energy consumption. During the optimization process, the next parameter evaluation point is selected to be guided by the low confidence bound acquisition function. The process continues until the preset convergence standard is reached, thereby outputting the optimized action starting point.

2. The intelligent control method for wastewater treatment plant based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The wastewater treatment plant adopts an anaerobic-anoxic-aerobic treatment process, and the reinforcement learning actions include: internal circulation liquid return, dissolved oxygen set point in the aerobic reactor, and carbon additive dosage in the anoxic reactor; the action range of dissolved oxygen is set between 0.5 and 2 mg / L, and is adjusted by 0.2 mg / L each time, the action range of internal circulation liquid return is from 0 to 90,000 m³ / day, and is adjusted by 10,000 m³ / day each time, and the carbon dosage is set between 0 and 5 m³ / day, and is adjusted by 0.5 m³ / day each time.

3. The intelligent control method for wastewater treatment plant based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The wastewater treatment plant adopts a three-stage anoxic-aerobic treatment process. The actions of reinforcement learning include the dissolved oxygen set point in the aerobic reactor and the carbon additive dosage in the anoxic reactor. The action range of dissolved oxygen is set between 0.5 and 2 mg / L, and is adjusted by 0.5 mg / L each time. The carbon dosage is set between 0 and 5 m³ / day, and is adjusted by 0.5 m³ / day each time.

4. The wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The professional simulation software adopts Dynamita Sumo, the water inflow data of the simulation model is monitored on a daily basis, and the rainfall pattern and storm pattern of BSM1 are incorporated into the water inflow.

5. The intelligent control method for wastewater treatment plant based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The operating cost is evaluated by the cost index provided by professional simulation software, taking into account the costs related to energy consumption, carbon addition and sludge treatment; the treatment efficiency is evaluated by the effluent quality index based on the BSM1 evaluation algorithm; the energy consumption is evaluated by the energy index, taking into account the energy consumed in each wastewater treatment process.

6. The wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The reinforcement learning algorithm adopts a deep Q-network algorithm and supports extended optimization objectives.

7. The intelligent control method for wastewater treatment plant based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The rewards of the reinforcement learning algorithm are set as follows: A basic reward is allocated for reaching the threshold of effluent quality standards; When the NOx concentration in the anaerobic reactor exceeds the set value, additional rewards are allocated; The cost index, water quality index and energy index used to evaluate operating costs, treatment efficiency and energy consumption allocate rewards inversely proportional to the index. The lower the index, the higher the reward. The total reward summary value of the index is fixed through the inverse proportionality coefficient. If effluent quality regulations are not met, penalties are imposed to discourage operations that do not meet established environmental standards.

8. The wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning according to claim 1 is characterized in that: The output of the objective function of the Bayesian optimization is the sum of the cost index and the penalty generated; if the effluent water quality exceeds the specified discharge limit, the penalty is imposed.

9. An intelligent control system for a wastewater treatment plant based on Bayesian optimization reinforcement learning, characterized in that: include: Professional simulation module, used to model the operating wastewater treatment plant using professional simulation software. The simulation model uses the actual influent data of the wastewater treatment plant, including the time-varying data of flow rate, chemical oxygen demand and total Kjeldahl nitrogen; Reinforcement learning module, used for multi-objective optimization using reinforcement learning, interacting reinforcement learning algorithms with professional simulation software, integrating the simulation environment with the reinforcement learning algorithm agent, and at each time step, the agent receives state information, including flow, chemical oxygen demand, and total Kjeldahl nitrogen, and takes actions to adjust operating parameters; And an initial action point optimization module, which is used to select the initial action point of reinforcement learning using Bayesian optimization. The Bayesian optimization sets an objective function, which receives the control point and the water inlet data, obtains the effluent water quality index by interacting with the professional simulation software, and calculates according to the specific optimization target set to output the target value; the optimization process adopts the Gaussian process to identify the best parameter combination that makes the target value optimal, and uses the control point in the best parameter combination as the initial action point of reinforcement learning; the optimal target value is to meet the minimum operating cost, the highest processing efficiency or the minimum energy consumption under the condition of effluent water quality; in the optimization process, the next parameter evaluation point is selected to be guided by the low confidence bound acquisition function; this process will continue until the preset convergence standard is reached, thereby outputting the optimized action starting point.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the steps of the wastewater treatment plant intelligent control method based on Bayesian optimization reinforcement learning according to any one of claims 1 to 8 are implemented.

Images