Data-driven increment self-adaptive evaluation control method for sewage treatment process

Abstract

According to the sewage treatment process-oriented data-driven incremental adaptive evaluation control method provided by the invention, a model-free adaptive evaluation design framework is adopted, an optimal control strategy is directly learned online from real-time operation data, and the dynamic state of a system is identified by automatically adjusting network parameters; and the risk of performance deterioration caused by model mismatch is effectively eliminated. An increment control strategy is adopted, and an online controller is designed by utilizing increment information of tracking errors and control input, so that the approximation pressure of a neural network is reduced, and the learning efficiency and the anti-interference stability are improved. Meanwhile, a composite utility function fusing a tracking error absolute value and a control increment is designed to replace a traditional fixed quadratic function form, so that the controller can better adapt to dynamic changes in the sewage treatment process, and better adaptive capacity, online stability and control precision are obtained; therefore, the online optimal tracking control of the dissolved oxygen concentration in the sewage treatment process is efficiently and stably realized.

Description

Technical Field

[0001] This application relates to the field of wastewater treatment control technology, and in particular to a data-driven incremental adaptive evaluation control method for wastewater treatment processes. Background Technology

[0002] Wastewater treatment is a core component of modern urban infrastructure and ecological civilization construction, and a crucial cornerstone for achieving water resource recycling, protecting public health, and promoting sustainable development. Globally, freshwater resources are becoming increasingly scarce, especially for some megacities facing water shortages. Wastewater must undergo efficient purification to effectively prevent water pollution, eutrophication, and other environmental problems, thus protecting the ecological environment.

[0003] The activated sludge process is the most widely used wastewater treatment technology in the world today. Its core principle is the removal of organic pollutants and nutrients such as nitrogen and phosphorus from wastewater through the metabolism of microorganisms. In this process, the dissolved oxygen concentration in the aeration tank is a crucial control variable, directly affecting the efficiency of the biochemical reaction, effluent quality, energy consumption, and operating costs. Dissolved oxygen concentration determines the activity of aerobic and nitrifying bacteria. Too low a concentration leads to insufficient microbial activity, inhibiting the degradation of organic matter, resulting in incomplete decomposition of organic matter and hindered ammonia nitrogen conversion, causing effluent quality to exceed standards. Conversely, too high a concentration accelerates the consumption of organic matter in wastewater, not only wasting significant aeration energy but also potentially causing sludge bulking and hindered denitrification.

[0004] However, the complex biochemical reactions and unknown system parameters in wastewater treatment pose significant challenges to controller design. Multiple interfering factors, significant nonlinear characteristics, and unknown models impose stringent requirements on system stability and safety. Therefore, maintaining dissolved oxygen concentration at an appropriate level has become one of the core challenges for wastewater treatment plants to achieve stable compliance, energy conservation, and intelligent operation.

[0005] Currently, most wastewater treatment plants use traditional control methods, such as proportional-integral-derivative (PID) control, which is simple in structure and easy to implement. However, as a linear controller, PID struggles to handle the nonlinear characteristics present in wastewater treatment processes. To maintain system stability, PID parameters are typically set conservatively, resulting in poor control performance, an inability to effectively handle unknown disturbances, and a lack of adaptive capability.

[0006] In recent years, with the development of artificial intelligence technology, several intelligent control methods have been proposed to address the limitations of traditional control methods, such as model predictive control (MMC) and neural network control. MMC relies on precise mathematical models; however, the numerous parameters in wastewater treatment systems that are difficult to measure online make establishing their mechanistic models extremely challenging. Furthermore, model mismatch can lead to a significant decline in control performance or even system instability. Neural network control can circumvent modeling difficulties using a data-driven approach, but these methods primarily focus on current tracking performance while neglecting long-term returns and the optimality of the control process.

[0007] Subsequently, adaptive dynamic programming and reinforcement learning methods provided a new approach to the intelligent operation of wastewater treatment systems: adaptive evaluative control. This method integrates reinforcement learning, optimal control, and neural networks. It evaluates the long-term performance of the current control strategy through an evaluator network and performs self-optimization through an execution network, ultimately approximating the optimal control law of the dynamic system. Unlike other methods, adaptive evaluative control does not minimize the tracking error at the current moment, but rather optimizes the accumulation of all expected costs from the current moment to the future through an approximation and evaluation cost function, maximizing the long-term benefits of the control strategy. Simultaneously, the neural network it employs can update its own parameters through training, effectively addressing challenges such as nonlinearity in wastewater treatment systems. Although adaptive evaluative control possesses strong optimization and adaptive capabilities, for wastewater treatment systems operating in complex environments with external disturbances and unknown dynamics, the basic adaptive evaluative design still lacks in terms of learning efficiency, accuracy, and stability.

[0008] In summary, current technologies for tracking and controlling dissolved oxygen concentration in wastewater treatment processes suffer from the following problems: some methods rely on precise mathematical models, which are difficult to model for complex nonlinear wastewater treatment systems; designs based on fixed parameters or models struggle to maintain optimal performance over long periods, resulting in insufficient adaptive capability of the controller; and most methods achieve tracking control by directly calculating the control quantity, but exhibit poor disturbance resistance in engineering practice, affecting control effectiveness. Therefore, there is an urgent need to design intelligent control methods that simultaneously achieve model-free operation, optimality, and adaptability, enabling efficient and stable online optimal tracking and control of dissolved oxygen concentration. Summary of the Invention

[0009] The purpose of this application is to address the problems of strong model dependence, insufficient adaptive capability, and poor control stability in existing wastewater treatment process control, and to provide a data-driven incremental adaptive evaluation control method for wastewater treatment processes, so as to achieve optimal tracking control of dissolved oxygen concentration, thereby improving control accuracy and system stability.

[0010] The adaptive evaluation control method of this application includes: In the biochemical reactor based on activated sludge technology, dissolved oxygen concentration data of aerobic zone unit 5 is collected as the system status, and the corresponding oxygen transfer coefficient data is collected as the control input.

[0011] A composite utility function is constructed based on the tracking error and the control input increment, which includes a tracking error term and a control increment term: a quadratic form of the tracking error is constructed as the tracking error term, and a quadratic form of the control input increment is constructed as the control increment term; the weight of the control increment term is designed as a function related to the absolute value of the tracking error based on utility reshaping, resulting in an adaptively adjusted control increment term; the tracking error term and the adaptively adjusted control increment term are combined to form the composite utility function.

[0012] An adaptive evaluation framework based on an evaluation network and an execution network is established. The evaluation network is used to approximate the cost function, taking the tracking error and control input increment as inputs. The approximate cost function value is calculated through forward propagation of the evaluation network, and the weights of the evaluation network are updated by minimizing the prediction error using gradient descent, thus completing the policy evaluation. The execution network is used to approximate the incremental control policy, taking the tracking error as inputs. The approximate incremental control policy is calculated through forward propagation of the execution network, and the weights of the execution network are updated by minimizing the approximate cost function value using gradient descent, thus completing the policy improvement.

[0013] Based on the weights of the trained execution network, an approximately optimal incremental control strategy is calculated, thereby obtaining an approximately optimal tracking control strategy. This strategy is used as the setpoint for the oxygen transfer coefficient to adjust the aeration intensity of unit 5 in the wastewater treatment system. Through closed-loop feedback control, real-time tracking control of dissolved oxygen concentration is achieved, keeping the dissolved oxygen concentration within the preset range of the target setpoint, thus realizing optimal tracking control of dissolved oxygen concentration.

[0014] Compared with the prior art, the technical solution of this application has the following advantages: 1. The data-driven incremental adaptive evaluation control method for wastewater treatment processes provided in this application does not require an accurate mathematical model of the system. It directly learns the optimal control strategy online from the system operation data, which overcomes the limitations of traditional model-dependent methods and improves the reliability and adaptability of the control system.

[0015] 2. This application adopts an incremental control strategy instead of directly calculating the control quantity, which significantly improves the system's ability to cope with large-amplitude disturbances and nonlinear characteristics, reduces the approximation pressure on the neural network, and improves the system's learning efficiency and control stability.

[0016] 3. The composite utility function designed in this application integrates the absolute value of the tracking error and the control increment, and introduces an adaptive weighting coefficient, enabling the controller to exhibit better adaptive capability and control accuracy during dynamic processes. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the system structure in this embodiment; Figure 2 This is a flowchart of the incremental adaptive evaluation and control method for wastewater treatment in this embodiment; Figure 3 A schematic diagram showing the flow rate of a wastewater treatment system under three weather conditions; Figure 4 A schematic diagram illustrating the tracking control effect under clear weather conditions; Figure 5 A schematic diagram illustrating the tracking control effect under rainy conditions; Figure 6 This is a schematic diagram illustrating the tracking and control effect under heavy rain conditions. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Throughout, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0020] like Figure 1 As shown, Figure 1 This embodiment demonstrates the system structure, specifically the process of adjusting dissolved oxygen concentration in the wastewater treatment system. The chemical reaction tank in the activated sludge process consists of five units: Units 1 and 2 are designated as anaerobic zones, while Units 3, 4, and 5 are designated as aerobic zones. After entering the biological reaction tank, wastewater flows sequentially through the five units. In the anoxic environment of the anaerobic zone, denitrification occurs using nitrates from the returned sludge. Subsequently, the mixed liquor enters the aerobic zone, where the oxidation of organic matter and nitrification of ammonia nitrogen are completed through precise control of dissolved oxygen concentration. The treated mixed liquor undergoes solid-liquid separation in the secondary sedimentation tank. The supernatant is discharged as compliant effluent, while part of the settled sludge is returned to the anaerobic zone to maintain biomass, and the remainder is discharged as excess sludge.

[0021] Dissolved oxygen concentration S in Unit 5 O,5Dissolved oxygen is a key factor affecting effluent water quality and is the main research object of this embodiment. Dissolved oxygen concentration is determined by the oxygen transfer coefficient K. La,5 The system adjusts the oxygen transfer coefficient by inputting the current dissolved oxygen concentration and the desired setpoint into the adaptive evaluation controller. Through online learning and training, the controller obtains the optimal control strategy to regulate the dissolved oxygen concentration at the expected level, thereby ensuring the efficient operation of the wastewater treatment process.

[0022] Example 1: like Figure 2 As shown, this embodiment provides a data-driven incremental adaptive evaluation and control method for wastewater treatment processes, including: Step S1: Obtain the dissolved oxygen concentration in the wastewater treatment system as the system state and the oxygen transfer coefficient as the control input; Step S2: Calculate the tracking error based on the system state and target setpoint, and calculate the control input increment using an incremental control strategy; Step S3: Construct a composite utility function based on the tracking error and control input increment. The composite utility function includes a tracking error term and a control increment term. Step S4: Establish an adaptive evaluation framework based on an evaluation network and an execution network, wherein the evaluation network is used to approximate the cost function and the execution network is used to approximate the incremental control strategy; Step S5: Calculate the approximate cost function value using the evaluation network based on the tracking error and control input increment. Update the weights of the evaluation network by minimizing the prediction error to complete the policy evaluation. Step S6: Calculate an approximate incremental control strategy based on the tracking error using the execution network, and update the weights of the execution network by minimizing the approximate cost function value to complete the strategy improvement; Step S7: Calculate the approximate optimal incremental control strategy based on the weights of the trained execution network to obtain the approximate optimal tracking control strategy and achieve optimal tracking control of dissolved oxygen concentration.

[0023] In this embodiment, the first step is to acquire key parameter data from the wastewater treatment system. Dissolved oxygen concentration is a crucial indicator reflecting the effectiveness of aerobic biological treatment during wastewater treatment, while the oxygen transfer coefficient is an important parameter for measuring aeration efficiency. Therefore, this embodiment defines dissolved oxygen concentration as a system state variable and the oxygen transfer coefficient as a control input variable. In practical applications, dissolved oxygen concentration data can be collected in real time using a dissolved oxygen sensor installed in the aeration tank, and the oxygen transfer coefficient can be calculated based on the aeration rate and aeration efficiency.

[0024] Next, the tracking error is calculated based on the system state and the target setpoint, and the incremental control input increment is calculated using an incremental control strategy. The tracking error is defined as the difference between the current dissolved oxygen concentration and the target setpoint, directly reflecting the performance of the control system. Unlike traditional absolute control strategies, this embodiment uses an incremental control strategy, which calculates the amount by which the oxygen transfer coefficient should increase or decrease in each control cycle. The incremental control strategy has several advantages: First, it reduces abrupt changes in the control signal, avoids frequent start-ups and shutdowns of the aeration system, and extends the equipment's lifespan; second, it helps suppress the impact of system noise and improves the stability of the control system; third, it gives the control system a certain "memory" capability, enabling it to optimize current decisions using historical information. The magnitude and direction of the control input increment depend on the magnitude and trend of the tracking error. By accurately calculating the control input increment, rapid and stable tracking of the dissolved oxygen concentration to the target value can be achieved.

[0025] Secondly, a composite utility function is constructed based on the tracking error and control input increment, which includes a tracking error term and a control increment term. The composite utility function is the objective function of the optimization problem, comprehensively considering both tracking performance and control cost. This function contains two key terms: the tracking error term and the control increment term. The tracking error term measures the deviation between the dissolved oxygen concentration and the setpoint, and its weighting coefficients are fixed positive definite matrices. Q (In this embodiment) Q =1), ensuring that error correction is always addressed with stable weights; the control increment term is used to limit drastic changes in the control input, and its weight is dynamically correlated with the tracking error, which is one of the core innovations of this embodiment. This dynamic weight design has clear engineering significance: when the tracking error is large, it is necessary to prioritize rapidly reducing the deviation. At this time, the weight of the control increment term will adaptively decrease, allowing for a larger adjustment range of the control input to accelerate error correction; while when the tracking error is close to zero, the focus shifts to maintaining stable operation. At this time, the weight of the control increment term will adaptively increase, strictly limiting the change range of the control input and avoiding system oscillations caused by frequent adjustments. This adaptive weight mechanism applied to the control increment term can achieve a precise balance between rapid response error and stable control process, which is particularly suitable for industrial processes such as wastewater treatment, which need to ensure water quality meets standards through timely adjustments while avoiding frequent adjustments to aeration equipment to reduce energy consumption and extend equipment life.

[0026] Secondly, an adaptive evaluation framework based on an evaluation network and an execution network is established. Adaptive evaluation control is an advanced control method combining reinforcement learning and adaptive control theory. Its core is the construction of two functional modules: an evaluation network and an execution network. The evaluation network is responsible for policy evaluation, receiving tracking error and control input increment as input, and outputting an approximate cost function value to evaluate the merits of the current control policy. The evaluation network typically adopts a multi-layer neural network structure, possessing strong nonlinear mapping capabilities and accurately approximating the cost function. The execution network is responsible for policy improvement, receiving tracking error as input, and outputting an approximately optimal control increment to directly guide the operation of the aeration system. The execution network also adopts a multi-layer neural network structure and is capable of learning complex nonlinear control policies. These two networks cooperate and promote each other, forming a complete adaptive evaluation control framework that can achieve optimal control of complex nonlinear systems without relying on precise mathematical models.

[0027] Then, a judgment network is used for policy evaluation. Policy evaluation is a crucial step in adaptive judgment control, aiming to assess the performance of the current control policy. The judgment network calculates an approximate cost function value based on the current tracking error and control input increment; the smaller this value, the closer the current policy is to the optimum. To ensure the judgment network accurately approximates the true cost function, the network weights need to be continuously updated through online learning. The learning process adopts the principle of minimizing prediction error, i.e., minimizing the difference between the judgment network output and the true cost. The prediction error is calculated based on temporal difference learning theory, combining the current composite utility function value and the predicted cost for the next time step to form the learning objective. The gradient is calculated using the backpropagation algorithm, and the network weights are updated using gradient descent. The judgment network gradually learns accurate cost evaluation capabilities, providing reliable guidance for policy improvement in the execution network.

[0028] Secondly, policy improvement is achieved using an execution network. Policy improvement is the process of further optimizing the control policy based on policy evaluation. The execution network receives the tracking error as input and outputs an approximate incremental control policy. To enable the execution network to generate a near-optimal control policy, the weights of the execution network need to be updated by minimizing the approximate cost function value output by the evaluation network. This process employs the policy gradient method, calculating the gradient of the cost function value with respect to the execution network weights and updating the weights along the negative gradient direction, thereby minimizing the long-term cumulative cost of the control policy output by the execution network. The policy improvement process is an iterative optimization process; as learning progresses, the execution network gradually approaches the optimal policy, and the control performance continuously improves. Unlike traditional model-based control methods, this data-driven policy optimization method does not rely on an accurate system model and has stronger adaptability and robustness.

[0029] Finally, based on the trained execution network weights, an approximately optimal incremental control strategy is calculated to obtain an approximately optimal tracking control strategy, achieving optimal tracking control of dissolved oxygen concentration. During the actual operation of the wastewater treatment system, the controller acquires dissolved oxygen concentration data in real time, calculates the optimal control increment through the trained execution network, and combines it with the control input from the previous moment to form a new control command, which is then sent to the aeration system. Since the execution network has mastered the dynamic characteristics and optimal control laws of the system through online learning, the generated control strategy enables the dissolved oxygen concentration to quickly and accurately track the target setpoint while maintaining stable changes in the control signal, avoiding frequent start-ups and shutdowns of the aeration system. This optimal tracking control strategy not only improves wastewater treatment efficiency and ensures that effluent quality meets standards, but also reduces energy consumption and operating costs, resulting in significant environmental and economic benefits.

[0030] In one implementation method, obtaining the dissolved oxygen concentration in the wastewater treatment system as the system state and the oxygen transfer coefficient as the control input includes: collecting dissolved oxygen concentration data of the aerobic zone unit 5 in the biochemical reactor based on activated sludge process as the system state, and the corresponding oxygen transfer coefficient data as the control input; setting the data sampling time interval to 45 seconds, continuously collecting wastewater treatment process data for multiple days to obtain a system operation dataset containing system state and control input; wherein the system operation dataset serves as the data basis for subsequent adaptive evaluation control, realizing data-driven model-free control.

[0031] In this embodiment, firstly, based on the activated sludge process, the biochemical reactor is divided into units 1 and 2 in the anaerobic zone, and units 3, 4, and 5 in the aerobic zone. The activated sludge process is currently the most mature and widely used biological treatment process in the wastewater treatment field. Its core is to utilize the microbial community in activated sludge to degrade and transform organic matter and pollutants such as nitrogen and phosphorus in wastewater. Activated sludge is essentially a highly complex microbial ecosystem, mainly composed of various microorganisms such as bacteria, fungi, protozoa, and metazoa. These microorganisms transform pollutants in the water into harmless substances or their own cellular material through metabolic activities. The biochemical reactor is the core structure of the activated sludge process, and its spatial division reflects the different biochemical stages of wastewater treatment. The anaerobic zone refers to an area with extremely low dissolved oxygen concentration (usually below 0.2 mg / L) and no oxidizing substances such as nitrates and nitrites. In this embodiment, units 1 and 2 constitute the anaerobic zone, laying the foundation for the subsequent biological phosphorus removal process. In this embodiment, units 3, 4, and 5 constitute the aerobic zone, which refers to an area with sufficient dissolved oxygen concentration (usually greater than 2 mg / L). This zone is the primary site for organic matter degradation, ammonia nitrification, and phosphorus uptake by polyphosphate-accumulating bacteria, and is the core area of ​​the entire treatment process. This "anaerobic-aerobic" combined process can simultaneously achieve organic matter removal and biological phosphorus removal, offering advantages such as good treatment effect and stable operation. It is a commonly used process configuration in modern wastewater treatment plants.

[0032] Next, the dissolved oxygen concentration data of Unit 5 is collected as the system status, and the corresponding oxygen transfer coefficient data is collected as the control input. Dissolved oxygen concentration refers to the content of dissolved oxygen in water, usually expressed in milligrams per liter (mg / L), and is one of the most important operating parameters in aerobic biological treatment processes. Dissolved oxygen concentration directly affects the metabolic activity and treatment efficiency of microorganisms: too low a concentration will lead to reduced activity of aerobic microorganisms, incomplete degradation of organic matter, and insufficient nitrification of ammonia nitrogen; too high a concentration will increase energy consumption and even cause operational problems such as sludge bulking. In the "anaerobic-aerobic" process, Unit 5, as the last unit in the aerobic zone, has a dissolved oxygen concentration that not only affects the treatment effect of this unit but also affects the performance of the entire process system through the reflux system. Therefore, selecting it as the system status monitoring point is of great significance. Dissolved oxygen concentration is usually measured using an electrochemical dissolved oxygen meter (such as a Clark electrode) or an optical dissolved oxygen meter (such as a fluorescence quenching method). Oxygen transfer coefficient ( ) is an important parameter for measuring the mass transfer efficiency of oxygen from the gas phase to the liquid phase, and its unit is usually the negative first power of hours ( The oxygen transfer coefficient (OCC) comprehensively reflects the performance of the aeration system and is influenced by various factors, including aerator type, aeration intensity, water temperature, mixing conditions, and sludge concentration. In activated sludge systems, the OCC is typically represented indirectly by the blower frequency, valve opening, or aeration rate output by the control system. These parameters directly determine the rate at which the system delivers oxygen to the water, thus affecting the dynamic changes in dissolved oxygen concentration. By precisely controlling these parameters, the goals of stabilizing dissolved oxygen concentration and saving energy can be achieved. In this embodiment, the OCC data and dissolved oxygen concentration data of Unit 5 are collected in pairs, providing the necessary input-output data pairs for subsequent model-free control.

[0033] Then, a data sampling interval of 45 seconds is set to continuously collect wastewater treatment process data for multiple days, obtaining a system operation dataset containing system status and control inputs. The selection of the data sampling interval needs to comprehensively consider factors such as system dynamic characteristics, control accuracy requirements, data storage capacity, and communication load. Wastewater treatment systems, especially dissolved oxygen control systems, typically have a large time constant, and dissolved oxygen concentration changes relatively slowly, generally on the order of minutes. This implementation chooses a 45-second sampling interval, which can fully capture the dynamic characteristics of the system, including the rise and fall of dissolved oxygen concentration, while avoiding data redundancy and storage pressure caused by excessively high sampling frequencies. This moderate sampling strategy ensures the temporal resolution of the data while also taking into account the actual needs of long-term operation. In the actual data collection process, a data acquisition system needs to be established, including components such as sensors, signal converters, communication networks, and data storage servers, to ensure the continuity and reliability of data acquisition. Data preprocessing functions should also be provided to detect and mark abnormal data in real time, such as sensor failures, communication interruptions, or extreme values, to support subsequent data analysis. Continuous data collection over multiple days is to obtain a sufficiently long time series to cover the operating status of the wastewater treatment system under different operating conditions, time periods, and environmental conditions. Wastewater treatment systems are affected by various factors, including daily fluctuations in water quality and quantity, temperature changes, and differences between weekdays and weekends. Only through data collection over a sufficiently long period can these patterns of change be captured, providing a comprehensive data foundation for model-free control. In practice, "several days" typically refers to 7-30 days, with the specific duration determined comprehensively based on factors such as the scale of the treatment plant, the complexity of the treatment process, and seasonal variations. This long-term, high-frequency data collection yields a system operation dataset rich in spatiotemporal characteristics, providing a solid foundation for subsequent development of model-free control strategies.

[0034] The system's operational dataset serves as the data foundation for subsequent adaptive evaluative control, enabling data-driven model-free control. Model-free control is a method that designs and implements control strategies directly based on system operational data, without relying on explicit mathematical models. Compared to traditional model-based control methods, model-free control skips the complex modeling step, directly extracting the necessary control information from input-output data, and is characterized by its simplicity and adaptability. In highly nonlinear, time-varying, and multi-factor-affected complex systems such as wastewater treatment systems, traditional mechanistic model-based control methods often face challenges such as modeling difficulties, complex parameter identification, and limited model accuracy. Model-free control, by directly analyzing the statistical characteristics and dynamic behavior of system operational data, can better adapt to the nonlinear and time-varying characteristics of the system, achieving more precise and efficient control.

[0035] The adaptive evaluation control method employed in this embodiment is a model-free control approach. It continuously optimizes system performance by dynamically adjusting the control strategy through real-time evaluation of control decisions. In its implementation, the collected system operating data first undergoes preprocessing, including outlier detection, missing value imputation, and data standardization, ensuring data quality and usability. Then, based on the processed data, a mapping relationship between control decisions and system responses is established without the need for explicit mathematical models. During the control execution phase, the system predicts the possible effects of different control decisions based on the current state and historical data, selects the optimal decision for implementation, and collects new operating data in real time to update the decision strategy. This data-driven learning approach automatically adapts to changes in system characteristics, such as water temperature fluctuations, sludge characteristics changes, and influent water quality fluctuations, without requiring manual adjustment of control parameters, significantly improving the adaptability and robustness of the control system.

[0036] In this embodiment, adaptive evaluative control fully utilizes the collected system operation dataset to achieve precise control of dissolved oxygen concentration. When dissolved oxygen concentration needs to be controlled, the possible effects of different control decisions are evaluated based on the difference between the current dissolved oxygen concentration and the target value, combined with similar situations in historical data, to select the optimal oxygen transfer coefficient adjustment strategy. Simultaneously, factors such as energy consumption and equipment wear are considered to find the best balance between control accuracy and economy. After control execution, new system response data is added to the dataset to further optimize future control decisions. This model-free control method avoids the difficulty of building complex mathematical models, can adaptively learn system characteristics, and achieves efficient and energy-saving dissolved oxygen control. Compared with traditional PID control or model predictive control, model-free control has advantages such as simple implementation, low maintenance costs, and strong adaptability, making it particularly suitable for the control needs of complex industrial processes such as wastewater treatment. With the increase in data volume and the accumulation of control experience, the performance of the model-free control system will continuously improve, achieving self-evolution and optimization, providing strong technical support for the long-term stable operation of wastewater treatment plants.

[0037] The model-free method used in this embodiment is particularly suitable for complex systems like wastewater treatment systems, which are difficult to model accurately. Traditional control methods typically require the establishment of precise mathematical models, which poses a significant challenge in wastewater treatment systems: First, activated sludge is a complex biological system involving the metabolic processes of various microorganisms, making it difficult to describe with simple mathematical equations; second, system characteristics change with external factors such as temperature, pH, and influent water quality, leading to uncertainties in model parameters; finally, the modeling process is time-consuming and costly, and the obtained models are often only effective under specific conditions.

[0038] In contrast, the model-free method in this embodiment learns control strategies directly from data, eliminating the intermediate step of modeling and greatly simplifying the implementation process. Simultaneously, it can continuously learn and optimize control strategies from accumulated operational data, adapting to changes in the environment and system characteristics, achieving true adaptive control. In practical applications, this method can significantly improve control accuracy, reduce energy consumption, and decrease maintenance costs, bringing considerable economic and environmental benefits to wastewater treatment plants. By collecting operational data from actual systems and directly applying it to adaptive evaluation control, this application provides a practical and efficient solution for the intelligent upgrading of wastewater treatment systems, with broad application prospects.

[0039] Wastewater treatment systems can be viewed as an unknown discrete-time nonlinear system, namely: Among them, system status x k Indicating the wastewater treatment process kDissolved oxygen concentration at time S O,5 System control input u k express k oxygen transfer coefficient at time K La,5 ; This represents an unknown wastewater treatment system function.

[0040] As one implementation method, the tracking error is calculated based on the system state and the target setpoint, including: based on the number of days. T Different target dissolved oxygen concentrations are set to design the target setpoint function; the target setpoint for the current time is calculated based on the number of days corresponding to the current time; the tracking error is obtained based on the difference between the current system state and the target setpoint.

[0041] In this embodiment, firstly, based on the number of days T The target setpoint function is designed by setting different target values ​​for dissolved oxygen concentration. This dynamic target setting scheme reflects an important innovation of this embodiment: the dissolved oxygen control target is no longer a fixed value in the traditional sense, but a dynamic value that changes over time. This design fully considers various factors such as the seasonal changes of the wastewater treatment system, the evolution of microbial populations, and energy consumption optimization. Specifically, the target setpoint function can be determined based on various factors, including historical operating data analysis, the balance optimization between energy consumption and treatment effect, and seasonal temperature variation patterns. For example, during the high-temperature period in summer, the increased water temperature leads to enhanced microbial activity and reduced oxygen solubility. At this time, the dissolved oxygen target value can be appropriately increased to ensure the biological treatment effect. Conversely, during the low-temperature period in winter, the reduced microbial activity allows for an appropriate decrease in the dissolved oxygen target value to reduce energy consumption. In addition, the target setpoint function can also be set according to the variation pattern of different times of day. For example, a relatively higher target value can be set during the daytime when the influent water quality fluctuates greatly, and a lower target value can be set during the nighttime when the water quality is relatively stable, further optimizing the system's energy efficiency ratio. This dynamic target setting strategy can not only adapt to the long-term changing trends of the system, but also minimize energy consumption while ensuring the quality of the effluent, resulting in significant economic and environmental benefits.

[0042] Next, the target setpoint for the current moment is calculated based on the number of days corresponding to the current moment. This step transforms time information into specific control target values ​​and is the core implementation step of the dynamic target strategy. In practice, an accurate time tracking mechanism must first be established to ensure that the system can accurately identify the current date and time. This can be achieved through time synchronization with the wastewater treatment plant's central control system or through a separate time server. After obtaining the current time, the system converts the current date into the number of days since the control system started or from a specific reference date. T The number of days hereT It can be a series of consecutive natural numbers, or it can be periodic, such as a cycle of 365 days per year. Then, the number of days... T The target dissolved oxygen concentration is calculated by substituting the values ​​into a predefined target setpoint function. During function evaluation, additional information such as the specific time of day, recent rainfall, and influent water quality monitoring data may need to be considered to further refine and optimize the target value. This time-based dynamic target value calculation method enables the control system to proactively adapt to periodic changes (such as seasonal variations and weekend / workday changes), improving the system's foresight and adaptability. During system operation, the target setpoint function itself can also be updated and optimized based on actual operating results and changes in the external environment, forming an adaptive target setting mechanism that further enhances the system's intelligence level.

[0043] Then, the difference between the current system state and the target setpoint is calculated to obtain the tracking error. This is the final and most direct step in calculating the tracking error. The system state, i.e., the current dissolved oxygen concentration, is obtained in real time through online sensors. In modern wastewater treatment plants, commonly used dissolved oxygen measurement devices include optical dissolved oxygen meters and electrochemical dissolved oxygen meters, which can provide high-precision and high-reliability continuous monitoring data. The tracking error, based on the difference between the current dissolved oxygen concentration and the target setpoint, reflects the deviation between the current system state and the desired state, and is an important basis for subsequent control decisions. A positive error indicates that the current dissolved oxygen concentration is lower than the target value, requiring an increase in aeration; a negative error indicates that the current dissolved oxygen concentration is higher than the target value, requiring a decrease in aeration. The absolute value of the error reflects the urgency of the adjustment; the larger the error, the greater the adjustment is usually required. In actual control, the error may also be normalized and mapped to a standard range (such as [-1,1]) to facilitate subsequent neural network processing.

[0044] Through the above steps, this embodiment achieves time-dynamic target value setting and accurate tracking error calculation, providing a reliable basis for subsequent control strategy evaluation and optimization. Compared with traditional fixed target value control, this dynamic target control strategy is more flexible and can better adapt to the long-term changes and periodic fluctuations of wastewater treatment systems. It minimizes energy consumption while ensuring treatment effectiveness, achieving a win-win situation for both economic and environmental benefits. In practical applications, the parameter settings and optimization of this method can be personalized based on historical operating data and expert experience of specific wastewater treatment plants to obtain the best control effect and economic benefits. With the accumulation of system operating data and the advancement of analysis technology, the target setpoint function can be further optimized to achieve more refined and intelligent control target management.

[0045] The target setting for tracking dissolved oxygen concentration is: in, This represents the setpoint function obtained through performance optimization. Therefore, the tracking error between the dissolved oxygen concentration and the target setpoint can be obtained as: Specifically, to verify the adaptability and stability of the proposed method to dynamic changes, the desired target tracking trajectory is set in a dynamic form, and its expression is: in, T This represents the current number of days.

[0046] In one implementation method, constructing a composite utility function based on tracking error and control input increment includes: obtaining the tracking error and control input increment at the current moment as input parameters of the utility function; constructing a quadratic term based on the tracking error as the tracking error term; constructing a quadratic term based on the control input increment as the control increment term; by setting adaptive adjustment parameters and calculating the absolute value of the tracking error, designing the weight of the control increment term as a function related to the absolute value of the tracking error, reshaping the control utility, and obtaining the adaptively adjusted control increment term; combining the tracking error term and the adaptively adjusted control increment term to form a composite utility function.

[0047] The key innovation of this implementation lies in introducing an adaptive adjustment mechanism: by setting adaptive adjustment parameters and calculating the absolute value of the tracking error, the weight of the control increment term is designed as a function related to the absolute value of the tracking error. Specifically, the quadratic term of the control increment term is divided by a factor related to the magnitude of the error, which makes the effective weight of the control increment term decrease as the tracking error increases, and vice versa. This design ensures that the system prioritizes fast tracking when the tracking error is large, while focusing more on control stability when the tracking error is small.

[0048] In this embodiment, the tracking error is the difference between the current dissolved oxygen concentration and the target setpoint, directly reflecting the quality of control performance. The control input increment is the change in the oxygen transfer coefficient between two adjacent control cycles, reflecting the intensity of the control action. In practical applications, the tracking error can be obtained by measuring the difference between the dissolved oxygen concentration and the setpoint in real time, while the control input increment is the direct output of the control algorithm. These two parameters together constitute the basic indicators for evaluating the performance of the control system and are also the core inputs for constructing the composite utility function. When obtaining these parameters, it is necessary to ensure measurement accuracy and real-time performance to guarantee the accuracy of subsequent control decisions. In wastewater treatment, dissolved oxygen concentration is usually measured in milligrams per liter (mg / L), and its measurement accuracy should generally reach ±0.1 mg / L. The control input increment needs to be appropriately standardized according to the characteristics of the specific aeration equipment to ensure that this control method is applicable to treatment systems of different scales.

[0049] Secondly, a key feature is the differentiated processing strategy employed for the tracking error term and the control increment term when constructing the composite utility function. The tracking error term uses a quadratic form to significantly penalize larger deviations, ensuring the system's tracking accuracy. Simultaneously, the absolute value of the tracking error is innovatively introduced into the control increment term as an adaptive adjustment factor. This design means that when the tracking error is large, the equivalent weight of the control increment term decreases, allowing the system to prioritize and quickly eliminate the error; when the tracking error is small, the equivalent weight of the control increment term increases, and the system focuses on maintaining control stability. This adaptive mechanism based on the absolute value of the error effectively avoids drastic fluctuations in the control input while ensuring accuracy, making it particularly suitable for industrial processes such as wastewater treatment, which involve uncertainty and disturbances.

[0050] Secondly, the introduction of the control input increment term is primarily to limit drastic changes in the control signal. In aeration systems, frequent and significant adjustments to the aeration rate can lead to increased equipment wear, higher energy consumption, and potential instability in the activated sludge system. By constructing a quadratic term regarding the control increment, a greater penalty can be imposed on large control changes, encouraging the control system to generate smoother, asymptotic control sequences. The mathematical form of the quadratic form implies that the penalty intensity increases quadratically with the absolute value of the control increment; this nonlinear characteristic effectively suppresses abrupt changes in the control input. In practical implementation, the coefficient of the quadratic term can be adjusted according to the physical characteristics and dynamic response capability of the aeration equipment to balance the relationship between control response speed and stability. For example, for mechanical aeration systems with slower responses, the weight of the quadratic term can be appropriately increased to avoid generating control commands that exceed the equipment's response capability; while for blower systems with rapid responses, the weight of the quadratic term can be appropriately decreased to improve the response speed of the control system.

[0051] Then, adaptive adjustment parameters are set to introduce a dynamic weighting mechanism for the control increment term. This is a key innovation of this embodiment; by introducing adaptive adjustment parameters, the control system can automatically adjust the weight of the control increment according to the magnitude of the tracking error. Specifically, the equivalent weight coefficient of the control increment term becomes... This design has profound implications for control theory: when the tracking error is large, the denominator increases, the equivalent weight of the control increment term decreases, and the system focuses more on reducing the error, allowing for larger changes in the control input; when the tracking error approaches zero, the denominator approaches 1, and the equivalent weight of the control increment term returns to its original value. The system prioritizes the stability of the control signal, avoiding unnecessary adjustments. The adaptive adjustment parameter is a key design parameter, determining the balance between tracking performance and control stability. In practical applications, an appropriate adaptive adjustment parameter value can be selected based on the specific characteristics and control requirements of the wastewater treatment system. Generally, for fluctuating influent conditions, a smaller adaptive adjustment parameter can be chosen to improve the system's tracking capability; while for relatively stable operating conditions, a larger adaptive adjustment parameter can be selected to emphasize the stability of the control signal.

[0052] Finally, the tracking error term and the adaptively adjusted control increment term are combined to form a composite utility function. This composite utility function is the objective function for subsequent optimization processes, and its design directly affects the performance of the control strategy. An ideal composite utility function should achieve a good balance between tracking performance and control stability, responding quickly to changes in the setpoint while avoiding drastic fluctuations in the control signal. It is worth noting that because the weights of the control increment term are adaptive, even its baseline weights... Even when kept constant, the composite utility function still has the ability to automatically adjust according to the magnitude of the error, which greatly enhances the adaptability of the control system to various operating conditions.

[0053] The composite utility function constructed through the above steps considers not only the tracking performance of the control system but also the stability of the control signal, achieving a dynamic balance between the two through an adaptive weighting mechanism. This design is particularly suitable for dissolved oxygen control in wastewater treatment systems, enabling the reduction of energy consumption, extension of equipment lifespan, and improvement of overall system operating efficiency while ensuring treatment effectiveness. In practical applications, the parameters of the composite utility function can be fine-tuned according to specific engineering requirements to achieve optimal control performance.

[0054] Specifically, the composite utility function designed in this embodiment can be expressed as: in, It is an adaptive adjustment parameter. Based on experimental tests and experience, this embodiment sets the parameter in the utility function to... =100, Q =1, R =0.01.

[0055] As one implementation method, an adaptive evaluation framework based on an evaluation network and an execution network is established, including: constructing a three-layer evaluation network and an execution network; setting fixed weight vectors from the input layer to the hidden layer as two-dimensional to fifteen-dimensional and one-dimensional to fifteen-dimensional vectors respectively; setting adjustable weight vectors from the hidden layer to the output layer as fifteen-dimensional to one-dimensional vectors; setting the number of neurons in the hidden layer to fifteen; selecting the hyperbolic tangent function as the activation function; initializing the network weights; and establishing the forward propagation computation framework of the neural network.

[0056] In this embodiment, firstly, a three-layer structure of the evaluation network is constructed. The evaluation network is a key component responsible for policy evaluation in the adaptive evaluation control framework, and its structural design directly affects the performance of the control system. This embodiment adopts a classic three-layer neural network structure, including an input layer, a hidden layer, and an output layer. The input layer of the evaluation network receives tracking errors and control input increments, therefore, the input layer dimension is two-dimensional. The hidden layer is the core processing unit of the neural network, responsible for extracting features from the input data and performing nonlinear mapping. The connection from the input layer to the hidden layer uses a fixed weight vector, i.e., a matrix of two to fifteen dimensions. This design avoids over-parameterization and simplifies the learning process. The output layer has only one neuron, used to output an approximation of the cost function. The connection from the hidden layer to the output layer uses an adjustable weight vector, set as a vector of fifteen to one dimension. These weights will be continuously adjusted during the learning process, enabling the evaluation network to accurately approximate the true cost function.

[0057] Next, we construct the three-layer structure of the execution network. The execution network is a key component in the adaptive evaluative control framework responsible for generating the control strategy, and its design also employs a three-layer neural network structure. Unlike the evaluative network, the input of the execution network only includes the tracking error, therefore the input layer is one-dimensional. The connection from the input layer to the hidden layer of the execution network uses a fixed weight vector, set as a matrix from one to fifteen dimensions; the connection from the hidden layer to the output layer uses an adjustable weight vector, set as a vector from fifteen to one dimension; the output layer has only one neuron, used to generate the control input increment. Through this design, the execution network can directly calculate the optimal or near-optimal control strategy based on the tracking error, without the need for complex intermediate inference processes, which is beneficial for the realization of real-time control.

[0058] Then, the activation function is a key element in introducing nonlinear characteristics into neural networks, significantly impacting the network's expressive power and learning performance. This embodiment chooses the hyperbolic tangent function as the activation function, a commonly used nonlinear activation function with favorable mathematical properties. The hyperbolic tangent function's output ranges from -1 to 1, it is centrally symmetric about the origin, its derivative reaches its maximum value of 1 at the origin, and its derivative is easy to calculate; these characteristics make it perform well in neural network training. In control problems involving multiple nonlinear dynamics, such as wastewater treatment systems, the hyperbolic tangent function can provide sufficient nonlinear expressive power to capture the complex dynamic characteristics of the system. Furthermore, this embodiment sets the number of hidden layer neurons in both the evaluation network and the execution network to fifteen, a number verified through theoretical analysis and experiments. Too few neurons would limit the network's expressive power and fail to accurately approximate complex functions; too many would increase the computational burden and potentially lead to overfitting. The fifteen-neuron setting achieves a good balance between expressive power and computational efficiency, making it particularly suitable for moderately complex control problems such as dissolved oxygen control.

[0059] Finally, the weight parameters of the evaluation network and the execution network are initialized, establishing the forward propagation computation framework of the neural network. Network initialization is a crucial step in learning, affecting the convergence speed and final performance of training. This embodiment employs a classic random initialization method, where the weight vectors from the hidden layer to the output layer are randomly initialized according to a uniform or normal distribution, with the initial values ​​controlled within a small range (typically ±0.1 or ±0.05). This helps prevent neuron saturation and accelerates training convergence. For the fixed weights, relatively dispersed values ​​are used to ensure that neurons in different hidden layers can capture different features of the input data. After initialization, the forward propagation computation framework of the neural network is established, including input data preprocessing, linear combination and nonlinear activation of the hidden layers, and linear combination of the output layer. Forward propagation is the fundamental process for the neural network to perform predictions and is also a prerequisite for subsequent backpropagation to update weights. In practical applications, the forward propagation computation framework is usually implemented using matrix operations to improve computational efficiency. By setting appropriate weight initialization methods and establishing an efficient computation framework, a solid foundation can be laid for subsequent online learning and control strategy optimization.

[0060] Through the above steps, an adaptive evaluation framework for the evaluation network and execution network was established. This framework fully combines the powerful expressive ability of neural networks with the superior performance of adaptive control, enabling precise control of dissolved oxygen in wastewater treatment systems without relying on precise mathematical models. The evaluation network is responsible for assessing the long-term performance of the control strategy, while the execution network is responsible for generating near-optimal control strategies. Working together, they continuously optimize through online learning, ultimately achieving optimal tracking control of dissolved oxygen concentration. This data-driven control framework exhibits strong adaptability and robustness, capable of adapting to various uncertainties and changes in wastewater treatment systems, improving energy efficiency while ensuring treatment effectiveness.

[0061] Specifically, the evaluation network is used to approximate the cost function, and can be expressed as: in, This represents the fixed weight vector from the input layer to the hidden layer of the evaluation network. This represents the adjustable weight vector from the hidden layer to the output layer of the evaluation network. To evaluate the activation function of the network, the hyperbolic tangent function was chosen.

[0062] The execution network used to approximate an incremental control strategy can be represented as: in, This represents the fixed weight vector from the input layer to the hidden layer of the execution network. This represents the adjustable weight vector from the hidden layer to the output layer of the network. To implement the activation function for the network, the hyperbolic tangent function is chosen.

[0063] In one implementation, a policy evaluation is completed by using an evaluation network to calculate an approximate cost function value based on the tracking error and control input increment, and updating the evaluation network weights by minimizing the prediction error. This includes: using the tracking error and control input increment as input to the evaluation network, and calculating the approximate cost function value through forward propagation of the evaluation network; calculating the composite utility function value at the current time step based on the tracking error and control input increment using a composite utility function; calculating the prediction error of the evaluation network based on the approximate cost function value and the composite utility function value; setting the performance index function of the evaluation network to half the square of its prediction error, and calculating the weight update amount of the evaluation network using gradient descent; updating the adjustable weight vector from the hidden layer to the output layer of the evaluation network online according to a preset learning rate, and repeating training until the weights converge, thus completing the policy evaluation.

[0064] In this embodiment, the tracking error and control input increment are first used as inputs to the evaluation network, and an approximate cost function value is calculated through forward propagation. The tracking error reflects the deviation between the current system state and the desired state, while the control input increment represents the magnitude of the control action change; together, they form the basis for evaluating the current system performance and control strategy. The evaluation network typically employs a multi-layer feedforward neural network structure, including an input layer, hidden layers, and an output layer. The input layer receives the tracking error and control input increment, the hidden layers transform and extract the input information using a nonlinear activation function, and the output layer generates a scalar value, i.e., the approximate cost function value. Forward propagation is the basic operation process of a neural network; it passes the input signal sequentially from the input layer through each hidden layer, ultimately producing the output value. In practical applications of wastewater treatment systems, the tracking error may have a large range; therefore, before inputting it into the evaluation network, it usually needs to be normalized, mapping it to the [-1,1] or [0,1] interval to improve the network's learning efficiency and generalization ability. Similarly, the control input increment also needs to be appropriately normalized to ensure it is on the same order of magnitude as the tracking error.

[0065] Secondly, based on the approximate cost function value and the composite utility function value, the prediction error of the evaluation network is calculated. This step is the core of the policy evaluation process, directly measuring the accuracy of the evaluation network's output. The calculation of the prediction error is based on temporal difference learning theory, combining the composite utility function value at the current time step with the predicted cost at the next time step to form the learning objective. Specifically, the prediction error equals the difference between the composite utility function value and the predicted cost at the current time step and the predicted cost at the previous time step. This calculation method reflects the Bellman optimality principle in reinforcement learning, that is, the cost function under the optimal policy should satisfy a certain recursive relationship. The magnitude of the prediction error directly reflects how accurately the evaluation network approximates the true cost function; the smaller the error, the better the performance of the evaluation network.

[0066] Next, the performance index function of the evaluation network is set to half of the squared prediction error of the evaluation network, and the weight update of the evaluation network is calculated using the gradient descent method. The performance index function is the objective function that guides the neural network learning process, quantitatively measuring the difference between the network output and the expected output. Half of the squared prediction error is used as the performance index function because this form has good mathematical properties, facilitating differentiation and optimization. Gradient descent is the most commonly used neural network training method. It calculates the partial derivative of the performance index function with respect to the network weights (i.e., the gradient), and adjusts the weights in the opposite direction of the gradient, thereby continuously decreasing the performance index function and continuously improving the network performance. In actual calculation, the gradient information of each layer of the network needs to be calculated using the backpropagation algorithm, and then the network weights are updated according to the gradient information. For the training of networks like the evaluation network, which is close to a regression problem, the classic BP (backpropagation) algorithm is used here.

[0067] Then, based on the preset learning rate of the evaluation network, the adjustable weight vector from the hidden layer to the output layer of the evaluation network is updated online. The learning rate is a key hyperparameter in the neural network training process, determining the magnitude of weight adjustment in each iteration. An excessively large learning rate can lead to instability in the training process, potentially causing oscillations or divergence; an excessively small learning rate can result in slow training speeds, making it difficult to achieve the desired effect within a limited time. In practical engineering applications such as wastewater treatment systems, a smaller learning rate is typically used to ensure training stability. It is worth noting that this embodiment emphasizes "online update," meaning that the weights of the evaluation network are continuously adjusted during the actual operation of the system, rather than being fixed after pre-training. This online learning mechanism enables the control system to adapt to environmental changes and system characteristic drift, maintaining long-term control performance. Specifically, regarding the weight update method, this embodiment particularly emphasizes updating the "adjustable weight vector from the hidden layer to the output layer." This indicates that in a multi-layer network structure, only some weights may be updated. This strategy can reduce computational burden, improve real-time performance, and still maintain sufficient learning capacity.

[0068] Finally, training is repeated until the weights converge, completing the policy evaluation. Training a neural network is an iterative process, requiring multiple repetitions of the preceding steps until the network weights reach a certain convergence state, i.e., the change in weights is less than a preset threshold, or the performance index function drops to an acceptable range. The choice of convergence criteria needs to balance training effectiveness and computational efficiency, ensuring network performance meets standards while avoiding the waste of computational resources caused by overtraining. In practical applications, various convergence conditions can be set, such as the maximum number of iterations, the minimum performance index value, and insignificant performance improvement after multiple consecutive iterations. Training can be stopped once any one of these conditions is met. After weight convergence, the evaluation network possesses the ability to accurately assess the performance of the current control policy, providing reliable guidance for subsequent policy improvements.

[0069] Through the above steps, the evaluation network can accurately assess the performance of the current control strategy, providing a reliable foundation for strategy improvement. This data-driven online learning mechanism enables the control system to continuously adapt to environmental changes and system characteristic drift, maintaining excellent long-term control performance. In practical applications, the design parameters of the evaluation network (such as network structure, activation function, learning rate, etc.) can be optimized and adjusted according to specific engineering needs to achieve the best control effect. For example, for large-scale wastewater treatment plants, a more complex network structure can be used to enhance the network's expressive power; while for small systems with limited computing resources, a simpler network configuration with high computational efficiency can be selected.

[0070] Specifically, the prediction error of the evaluation network can be expressed as: in, This represents the composite utility function value. The performance metric function for evaluating the network is set as follows: According to the gradient descent principle, the rules for evaluating the update of network weights during training are as follows: in, η c To evaluate the learning rate during network training, it is set to... η c = 0.1.

[0071] In one implementation method, an approximate incremental control policy is calculated using an execution network based on the tracking error. The policy is then improved by updating the weights of the execution network by minimizing the cost function. This includes: using the tracking error as input to the execution network and calculating the approximate incremental control policy through forward propagation; calculating the prediction error of the execution network based on the objective of minimizing the approximate cost function value; setting the performance index function of the execution network to half the square of its prediction error and calculating the weight update amount of the execution network using gradient descent; updating the adjustable weight vector from the hidden layer to the output layer of the execution network online according to a preset learning rate, and repeating the training until the weights converge, thus improving the policy.

[0072] In this embodiment, the tracking error is first used as the input to the execution network, and an approximate incremental control strategy is calculated through the forward propagation of the execution network. Unlike the evaluation network, the input to the execution network only contains the tracking error, which is the most direct and core information, reflecting the deviation between the current state and the desired state of the system. The structure of the execution network is usually a multi-layer feedforward neural network, including an input layer, a hidden layer, and an output layer. The input layer receives the tracking error, the hidden layer extracts the control-related information contained in the tracking error through nonlinear transformation, and the output layer generates an approximate incremental control strategy, namely the adjustment amount of the oxygen transfer coefficient. In practical applications of wastewater treatment systems, the design of the execution network needs to fully consider the dynamic characteristics of dissolved oxygen and the physical constraints of the aeration system. For example, the activation function of the hidden layer can be a hyperbolic tangent function (tanh), whose output range is [-1, 1], which can be mapped to the control increment range of the aeration system after appropriate scaling. At the same time, the output layer can use a linear activation function or a bounded nonlinear function to ensure that the generated control increment is within a reasonable range. Forward propagation is the computational process by which the neural network transforms input information into output control signals, and its computational efficiency directly affects the real-time performance of the control system. Therefore, when designing the execution network structure, it is necessary to balance the network's expressive power and computational complexity to ensure that the computation can be completed within the control cycle while meeting the control requirements.

[0073] Secondly, based on the objective of minimizing the approximate cost function value, the prediction error of the execution network is calculated. This step is the core of policy improvement; it uses the result of policy evaluation (the approximate cost function value) as guidance to lead the execution network to generate a better control policy. Unlike evaluating the prediction error during network training, the prediction error of the execution network focuses more on the long-term benefits of the control policy, reflecting the impact of the current control decision on the future state of the system. The calculation process is based on policy gradient theory, evaluating the quality of the control policy by calculating the gradient of the cost function with respect to the control policy.

[0074] Next, the performance index function of the execution network is set to half of the squared prediction error of the execution network, and the weight update of the execution network is calculated using the gradient descent method. The performance index function is the objective function that guides the learning process of the execution network, quantifying the gap between the current control policy and the optimal policy. Using half of the squared prediction error as the performance index function not only has good differentiability mathematically, facilitating gradient calculation, but also imposes a larger penalty on large errors, prompting the network to learn quickly. Gradient descent is a classic method for optimizing the performance index function. It calculates the gradient of the performance index function with respect to the network weights and adjusts the weights along the negative gradient direction, causing the performance index function to continuously decrease. In the training of the execution network, gradient calculation usually uses stochastic gradient descent, which can improve training speed and stability while ensuring convergence. It should be noted that the gradient calculation of the execution network is more complex than that of the evaluation network because it requires backpropagation of gradients through the evaluation network. Specifically, first, the gradient of the cost function with respect to the control increment is calculated, then the gradient of the control increment with respect to the execution network weights is calculated using the chain rule, and finally the gradient of the cost function with respect to the execution network weights is obtained. This "backpropagation of gradients through evaluation of the network" is a characteristic of network training and the core mechanism of the entire adaptive evaluation control framework. In practical applications, to improve the accuracy and stability of gradient calculation, experience replay technology can be used. This involves storing historical state-action-reward-next state samples and randomly selecting samples for batch training. This method can break the correlation between samples and improve training efficiency.

[0075] Then, based on the preset learning rate of the execution network, the adjustable weight vectors from the hidden layers to the output layer of the execution network are updated online. The learning rate is a hyperparameter that controls the magnitude of weight adjustment in each iteration, and it has a significant impact on the training process and the final result.

[0076] It is worth noting that this embodiment specifically emphasizes "online update" and "adjustable weight vectors from hidden layers to output layers," indicating that the system employs a partial network update strategy, i.e., updating only the weights of specific layers in the network, rather than the entire network. This strategy helps improve training efficiency while maintaining the network's expressive power.

[0077] Finally, training is repeated until the weights converge, completing the policy improvement. Similar to the evaluator network, the training of the execution network is also an iterative process, requiring multiple repetitions of the preceding steps until the network weights reach a certain convergence state. Convergence criteria can include weight changes being less than a preset threshold, insignificant performance improvement after multiple consecutive iterations, or reaching the maximum number of iterations. Unlike the evaluator network, the convergence of the execution network typically requires more iterations because its learning task is more complex; it needs to learn a strategy that maps states to optimal control decisions. For long-running wastewater treatment systems, the training of the execution network does not truly "end," but rather enters a "continuous learning" state. The system continuously fine-tunes its control strategy based on environmental changes and performance feedback to maintain optimal performance. This continuous learning mechanism is a key feature of this embodiment and crucial for its long-term high-efficiency control. After weight convergence, the execution network possesses the ability to generate near-optimal control strategies, achieving precise tracking and control of dissolved oxygen concentration.

[0078] Through the above steps, the execution network can learn a near-optimal incremental control strategy, achieving efficient tracking control of dissolved oxygen concentration. Compared with traditional model-based control methods, this data-driven adaptive control method does not rely on a precise mathematical model. It can adaptively learn the dynamic characteristics of the system and adjust the control strategy according to environmental changes, exhibiting stronger adaptability and robustness. In practical applications, the parameters of the execution network (such as network structure, learning rate, convergence conditions, etc.) can be optimized and adjusted according to the specific characteristics and control requirements of the wastewater treatment system to obtain the best control effect. This adaptive evaluative control framework is not only applicable to dissolved oxygen control but can also be extended to other control problems in wastewater treatment, such as sludge concentration control and recirculation ratio control, showing broad application prospects.

[0079] Specifically, the prediction error of the execution network is calculated according to the following formula: in, The desired training objective is to perform network performance metrics. The network's performance metrics function is set to... According to the gradient descent principle, the rule for updating network weights during training is as follows: in, η a The learning rate for performing the network training process is set to... η a =0.15.

[0080] As one implementation method, the incremental control strategy is used to calculate the control input increment, which includes: obtaining the incremental control strategy based on the execution network learning, i.e., a function of the tracking error; using the strategy to calculate the control input increment, avoiding the impact of large-amplitude control output on system stability, and passing the tracking error and control input increment to the composite utility function construction process.

[0081] First, in wastewater treatment systems, the increment of control input actually reflects the magnitude of change in aeration intensity. A positive increment indicates an increase in aeration intensity, while a negative increment indicates a decrease in aeration intensity; the absolute value of the increment reflects the magnitude of the adjustment. The calculated increment of control input is not only a direct reflection of the current system control changes but also an important reference for subsequent control strategy design.

[0082] Secondly, an incremental control strategy, a function of the tracking error, is obtained through online learning via the execution network. The incremental control strategy is essentially a nonlinear mapping implemented by the execution network, dynamically generating control increments based on the current tracking error to minimize the long-term cost function. In designing this strategy, the execution network calculates the control increments through forward propagation and updates network weights using the policy gradient method, thereby continuously optimizing the strategy. In nonlinear systems such as wastewater treatment systems, the execution network can automatically capture the dynamic characteristics of the system without pre-assuming a control structure (such as PID). The execution network typically employs a multi-layer feedforward neural network, handling complex mappings through nonlinear activation functions in the hidden layers. Furthermore, the adaptive mechanism in adaptive dynamic programming allows the control strategy to automatically adjust according to the system's operating state, improving the system's robustness and adaptability. In practical implementation, the control strategy is parameterized by the execution network, and the network weights are updated through online learning, ensuring the strategy adapts to process changes. Its core advantage lies in its data-driven nature; it does not rely on a precise system model but learns the optimal strategy through interactive learning. The ultimate goal of control strategy design is to ensure tracking accuracy while simultaneously considering control smoothness, stability, and energy efficiency.

[0083] Finally, an incremental control strategy is used to avoid the impact of large-amplitude control outputs on system stability by transferring the tracking error and control input increment to the composite utility function construction process. This step emphasizes the key advantages and practical application value of incremental control. A key feature of incremental control is its ability to naturally limit the rate of change of control output, preventing system oscillations or equipment damage caused by large adjustments. In wastewater treatment systems, aeration equipment typically has significant inertia and start-stop limitations; excessively rapid or large adjustments can not only cause system oscillations but also affect equipment lifespan and operating efficiency. The incremental control strategy effectively avoids these problems by directly controlling the magnitude of each adjustment. In practical applications, a maximum allowable increment limit can be set to ensure that each adjustment is within a safe range; alternatively, the allowable increment range can be dynamically adjusted based on the current system state, allowing larger adjustments when the system is stable and limiting the adjustment magnitude when the system fluctuates significantly, achieving more intelligent control behavior. Furthermore, transferring the tracking error and control input increment to the composite utility function construction process is a crucial link connecting the current control steps with the overall system performance evaluation. As the core of the system evaluation mechanism, the composite utility function needs to consider both control accuracy (tracking error) and control stability (control input increment) simultaneously, finding the optimal balance between the two. In this way, the system can maintain operational smoothness and system stability while pursuing high-precision control, achieving optimal overall control performance. In constructing the composite utility function, the weights of different objectives can be adjusted to reflect the priorities and focus of specific application scenarios. For example, when water quality fluctuates significantly, the weight of control precision can be increased; during equipment aging or high-energy-consumption phases, the weight of control stability can be increased.

[0084] Through the above steps, this embodiment accurately calculates the control input increment based on an incremental control strategy, improving the system's control accuracy and stability. Compared with traditional absolute control, the incremental control strategy is better suited to the high inertia and strong nonlinearity of wastewater treatment systems, reducing system oscillations, extending equipment lifespan, and improving energy efficiency. Simultaneously, by incorporating control accuracy and stability into a composite utility function, the system can automatically seek the optimal control equilibrium point, achieving a comprehensively optimized control effect. In practical applications, the specific parameters and implementation details of this method can be customized according to the characteristics of a particular wastewater treatment system to obtain the best control effect. With the accumulation of system operating experience and further development of control theory, the incremental control strategy can be continuously optimized and improved, providing more solid technical support for the intelligent and efficient operation of wastewater treatment systems.

[0085] To overcome large-amplitude interference and nonlinear characteristics, and to improve the stability and efficiency of the system's self-learning, this embodiment adopts an incremental control strategy. ,Right now: At this point, the goal of optimal control is to find a feedback incremental control strategy. This ensures that the dissolved oxygen concentration tracks the target setpoint while minimizing the cost function.

[0086] In one implementation method, an approximately optimal incremental control strategy is calculated based on the weights of the trained execution network to obtain an approximately optimal tracking control strategy, thereby achieving optimal tracking control of dissolved oxygen concentration. This includes: calculating the approximately optimal incremental control strategy at the current moment based on the weight vector of the trained execution network; obtaining the approximately optimal tracking control strategy at the current moment based on the approximately optimal incremental control strategy at the current moment and the control input at the previous moment; using the approximately optimal tracking control strategy as the setpoint for the oxygen transfer coefficient to adjust the aeration intensity of unit 5 in the wastewater treatment system; and achieving real-time tracking control of dissolved oxygen concentration through closed-loop feedback control to maintain the dissolved oxygen concentration within the preset range of the target setpoint.

[0087] In this embodiment, firstly, the approximate optimal incremental control strategy for the current moment is calculated based on the weight vector of the trained execution network. This step is crucial for transforming the theoretical results of reinforcement learning into practical control decisions. The execution network, a neural network structure used in this embodiment, has the core function of mapping the difference between the system state (dissolved oxygen concentration) and the target state (tracking error) into a control increment. After training, the execution network has "learned" the dynamic characteristics and control laws of the system, and its weight parameters contain rich knowledge and decision-making logic. Then, the current tracking error is used as the network input, and the network output, i.e., the approximate optimal incremental control strategy, is calculated through forward propagation. The calculated approximate optimal incremental control strategy directly reflects the control adjustment range that should be taken under the current system state, considering both control accuracy and control stability, demonstrating the comprehensive optimization capability of reinforcement learning. It is worth noting that this neural network-based control strategy has strong adaptability and generalization ability, capable of handling new states not seen in the training data, providing robust control decisions for the system.

[0088] Next, based on the approximate optimal incremental control strategy at the current moment and the control input at the previous moment, the approximate optimal tracking control strategy at the current moment is obtained. In wastewater treatment systems, the control input refers to the oxygen transfer coefficient, which directly affects the operating intensity and energy consumption level of the aeration equipment. Through incremental control, the system avoids abrupt changes in the control input, ensuring the stable operation of the aeration equipment. In practical implementation, it is necessary to maintain the historical record of the control input to ensure that the accurate control input value from the previous moment can be obtained in each calculation.

[0089] Then, the near-optimal tracking control strategy is used as the setpoint for the oxygen transfer coefficient to adjust the aeration intensity of unit 5 in the wastewater treatment system. This step transforms the calculated control decision into actual physical control operations, which is an important link in combining theory and practice. In modern wastewater treatment systems, aeration equipment typically employs variable frequency speed control technology, which can precisely adjust the aeration intensity according to the control signal.

[0090] Finally, real-time tracking and control of dissolved oxygen concentration are achieved through closed-loop feedback control, maintaining the dissolved oxygen concentration within the preset range of the target setpoint. This is the ultimate goal of the entire control system and an important criterion for evaluating the control effect. Closed-loop feedback control is a classic and effective control method. It continuously measures the system output and compares it with the target value, adjusting the control input according to the error to form a self-correcting control loop.

[0091] In this embodiment, dissolved oxygen concentration is the controlled variable, monitored in real time by an online dissolved oxygen sensor. This concentration is compared with the dynamically changing target setpoint, the tracking error is calculated, and then the optimal control strategy is calculated via network computation to adjust the aeration intensity, achieving precise control of the dissolved oxygen concentration. This process is continuously repeated, forming a complete control closed loop. In actual operation, the system needs to handle various disturbances, such as fluctuations in influent water quality, temperature changes, and changes in microbial activity, all of which affect the dynamic characteristics of dissolved oxygen. Reinforcement learning-based control methods have strong adaptability and anti-interference capabilities, enabling automatic adjustment of control strategies to cope with various changes. Simultaneously, it is also necessary to pay attention to comprehensive indicators such as control stability, energy consumption, and adjustment frequency to comprehensively evaluate the performance of the control system. Through the collection and analysis of long-term operating data, the system can continuously optimize control parameters and strategies, achieving self-improvement and evolution, further enhancing control performance and energy efficiency.

[0092] Through the above steps, this embodiment achieves precise dissolved oxygen concentration control based on reinforcement learning, effectively overcoming the limitations of traditional PID control in nonlinear dynamic systems. Compared with traditional methods, the control method in this embodiment has stronger adaptability, higher control accuracy, and better energy efficiency, minimizing energy consumption while ensuring water quality meets standards, thus creating economic and environmental benefits. Its advantages are particularly pronounced when facing complex operating conditions with large fluctuations in influent water quality and significant seasonal variations. In practical applications, this method can be integrated with existing automatic control systems, enabling rapid upgrades and transformations through simple parameter configuration and system docking, demonstrating its practicality and widespread applicability. With the continuous advancement of artificial intelligence technology and the accumulation of industrial application experience, this reinforcement learning-based control method has broad development and optimization potential, and is expected to play an important role in more industrial process control fields.

[0093] Through continuous online training and updates of the evaluation network and the execution network, the corresponding incremental control strategy can be calculated based on the weights of the trained execution network, thus obtaining an approximately optimal incremental control strategy through training. Thus, the approximate optimal tracking control strategy is obtained as follows: Set the oxygen transfer coefficient according to the obtained control strategy. It acts on the wastewater treatment system to achieve optimal tracking and control of dissolved oxygen concentration.

[0094] To verify the effectiveness of the method in this embodiment, an optimal tracking and control experiment of dissolved oxygen in the wastewater treatment system was conducted based on the BSM1 (Benchmark Simulation Model No. 1) simulation platform under three different weather conditions (sunny, rainy, and heavy rain). BSM1 is a standardized simulation platform launched by the International Water Association and has become an internationally recognized benchmark for research and performance evaluation of wastewater treatment process control strategies. The BSM1 model, based on the activated sludge process, can comprehensively simulate the dynamic operation of wastewater treatment plants under different conditions. This model not only considers major biological processes such as carbon oxidation, nitrification, and denitrification, but also includes engineering factors such as dissolved oxygen transfer and sludge return, providing a reliable simulation environment for the development and evaluation of control strategies. In this study, particular attention was paid to the dissolved oxygen control problem in the fifth biochemical reaction unit (aerobic zone) of the BSM1 model, because the dissolved oxygen level in this unit directly affects the effluent quality and system energy consumption.

[0095] like Figure 3 The figure shows the corresponding flow rates of the wastewater treatment system over 14 days under three weather conditions. It can be seen from the figure that under sunny conditions, the wastewater flow rate exhibits a stable periodic pattern. However, during rainy and heavy rainy weather, the inflow of wastewater increases significantly due to the continuous rainfall. Particularly in the last week, strong disturbances occur, which disrupt the stability of the wastewater treatment process and increase the difficulty of controller design.

[0096] Figure 4 , Figure 5 and Figure 6The tracking performance of dissolved oxygen concentration under sunny, rainy, and heavy rain conditions was demonstrated. Under sunny conditions, the load was relatively stable, external disturbances were minimal, and control was relatively easy. Under rainy conditions, the increased influent volume and water quality fluctuations caused by rainfall made the system's operating environment more complex. Under heavy rain conditions, the wastewater treatment system faced the most severe challenge. The large amount of rainwater brought by heavy rainfall led to a surge in influent flow, a shortened hydraulic retention time, and the dilution of pollutant concentrations, severely disrupting the balance of the biological system. Under these extreme conditions, the method of this invention demonstrated strong anti-interference ability and adaptability. The dissolved oxygen concentration tracking control effect of the method of this invention was compared with that of the traditional PID control method. The results showed that the dissolved oxygen concentration under the proposed method could accurately track the desired target setpoint and maintain high-precision tracking performance throughout the experimental period. In contrast, the traditional PID method exhibited significant tracking errors and fluctuations, especially leading to regulation lag or overshoot when the process dynamically changed.

[0097] To quantitatively evaluate control performance, two evaluation criteria were introduced: integral absolute error (IAE) and integral squared error (ISE). Experimental results show that under three weather conditions, the IAE and ISE values ​​of the method in this embodiment are significantly lower than those of the PID control method, indicating that the method in this embodiment has higher control accuracy and better stability, as shown in Table 1 below.

[0098] Table 1: Comparison of Tracking Control Accuracy under Three Weather Conditions The superior performance of the method in this embodiment is mainly attributed to its data-driven and adaptive learning mechanism. Unlike traditional methods that rely on precise mathematical models, this embodiment continuously optimizes the control strategy through online learning, adapting to the nonlinear characteristics and parameter changes of the system. During the experiment, it was observed that the control performance gradually improved with the extension of system operating time, especially after the first three days, when the control accuracy significantly improved, indicating that the adaptive learning mechanism was functioning effectively. Furthermore, the use of incremental control strategy effectively avoided drastic changes in the control signal, ensuring the stable operation of the system. Overall, the experimental results fully demonstrate the effectiveness and superiority of the data-driven incremental adaptive evaluation control method proposed in this embodiment in dissolved oxygen control of wastewater treatment systems, providing a powerful tool for optimizing the operation of wastewater treatment processes.

[0099] In summary, the data-driven incremental adaptive evaluation control method for wastewater treatment processes provided in this embodiment exhibits excellent performance in dissolved oxygen concentration tracking control. It can effectively overcome the nonlinear characteristics and external disturbances of wastewater treatment systems, achieve stable and accurate tracking control of dissolved oxygen concentration, and provide a high-performance intelligent control solution for wastewater treatment processes.

[0100] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data-driven incremental adaptive evaluation and control method for wastewater treatment processes, characterized in that, include: The dissolved oxygen concentration in the wastewater treatment system is used as the system state, and the oxygen transfer coefficient is used as the control input. The tracking error is calculated based on the system state and the target setpoint, and the incremental control input increment is calculated using an incremental control strategy. A composite utility function is constructed based on the tracking error and the control input increment. The composite utility function includes a tracking error term and a control increment term. An adaptive evaluation framework based on an evaluation network and an execution network is established, where the evaluation network is used to approximate the cost function and the execution network is used to approximate the incremental control strategy. The approximate cost function value is calculated using the evaluation network based on the tracking error and control input increment. The weights of the evaluation network are updated by minimizing the prediction error to complete the policy evaluation. An approximate incremental control strategy is calculated based on the tracking error using an execution network. The weights of the execution network are then updated by minimizing the approximate cost function value, thus achieving policy improvement. Based on the weights of the trained execution network, an approximately optimal incremental control strategy is calculated to obtain an approximately optimal tracking control strategy, thereby achieving optimal tracking control of dissolved oxygen concentration.

2. The method according to claim 1, characterized in that, Obtaining the dissolved oxygen concentration in the wastewater treatment system as the system state and the oxygen transfer coefficient as the control input includes: Based on the activated sludge process, the dissolved oxygen concentration data of the aerobic zone unit 5 in the biochemical reactor is collected as the system status, and the corresponding oxygen transfer coefficient data is collected as the control input. The data sampling interval is set to 45 seconds, and wastewater treatment process data is continuously collected over several days to obtain a system operation dataset containing system status and control inputs. The system operation dataset serves as the data basis for subsequent adaptive evaluation control, realizing data-driven model-free control.

3. The method according to claim 1, characterized in that, A composite utility function is constructed based on the tracking error and the control input increment, including: The tracking error and control input increment at the current moment are obtained as input parameters of the utility function. A quadratic term is constructed based on the tracking error as the tracking error term, and a quadratic term is constructed based on the control input increment as the control increment term. By setting adaptive adjustment parameters and calculating the absolute value of the tracking error, the weight of the control increment term is designed as a function related to the absolute value of the tracking error, thereby reshaping the control effectiveness and obtaining the adaptively adjusted control increment term. The tracking error term and the adaptive adjustment control increment term are combined to form a composite utility function.

4. The method according to claim 1, characterized in that, An adaptive evaluation framework based on an evaluation network and an execution network is established, including: A three-layer evaluation network and execution network are constructed. The fixed weight vectors from the input layer to the hidden layer are set to two-dimensional to fifteen-dimensional and one-dimensional to fifteen-dimensional vectors, respectively. The adjustable weight vectors from the hidden layer to the output layer are all fifteen-dimensional to one-dimensional vectors. The number of neurons in the hidden layer is set to fifteen. The hyperbolic tangent function is selected as the activation function, and the network weights are initialized to establish the forward propagation computation framework of the neural network.

5. The method according to claim 1, characterized in that, The evaluation network calculates an approximate cost function value based on the tracking error and control input increment. The weights of the evaluation network are updated by minimizing the prediction error to complete the policy evaluation, including: The tracking error and control input increment are used as inputs to the evaluation network, and the approximate cost function value is calculated through forward propagation of the evaluation network. Based on the tracking error and control input increment, the composite utility function value at the current moment is calculated using the composite utility function. The prediction error of the evaluation network is calculated based on the approximate cost function value and the composite utility function value. The performance index function of the evaluation network is set to half of the square of its prediction error, and the weight update amount of the evaluation network is calculated by gradient descent method. Based on the preset learning rate of the evaluation network, the adjustable weight vector from the hidden layer to the output layer of the evaluation network is updated online, and the training is repeated until the weights converge, thus completing the policy evaluation.

6. The method according to claim 1, characterized in that, An approximate incremental control policy is calculated using an execution network based on the tracking error. The policy is improved by updating the weights of the execution network by minimizing the approximate cost function value. This includes: The tracking error is used as the input to the execution network, and an approximate incremental control strategy is calculated through the forward propagation of the execution network. The prediction error of the execution network is calculated based on the objective of minimizing the approximate cost function value. Set the performance index function of the execution network to half of the square of its prediction error, and use the gradient descent method to calculate the weight update of the execution network. Based on the preset learning rate of the execution network, the adjustable weight vectors from the hidden layer to the output layer of the execution network are updated online, and the training is repeated until the weights converge, thus completing the policy improvement.

7. The method according to claim 1, characterized in that, Based on the trained execution network weights, an approximately optimal incremental control strategy is calculated to obtain an approximately optimal tracking control strategy, achieving optimal tracking control of dissolved oxygen concentration, including: Based on the weight vector of the trained execution network, calculate the approximate optimal incremental control strategy at the current time. Based on the approximate optimal incremental control strategy at the current moment and the control input at the previous moment, the approximate optimal tracking control strategy at the current moment is obtained. The near-optimal tracking control strategy was used as the set value for the oxygen transfer coefficient to adjust the aeration intensity of unit 5 in the wastewater treatment system. Real-time tracking and control of dissolved oxygen concentration is achieved through closed-loop feedback control, keeping the dissolved oxygen concentration within the preset range of the target value.

Images