A sewage pH value regulation system based on reinforcement learning

Abstract

The present application relates to sewage treatment technical field, specifically to a kind of sewage pH value control system based on reinforcement learning, comprising: multi-dimensional redundant perception module gathers sewage pH value, temperature, turbidity, conductivity and flow data;Time series prediction feedforward module calculates the basic dosing flow of acid-base liquid with trend compensation;Double Q learning hierarchical control module iteratively optimizes by constructing double Q-learning model containing three-dimensional extended state space, hierarchical action space and adaptive reward function, generates hierarchical optimal control strategy;Closed-loop execution feedback module accurately dosing acid-base liquid through variable frequency stepless adjustment unit, collects actual dosing amount back and completes control strategy closed-loop calibration correction;Fault self-diagnosis fault-tolerant module and the rest of the module two-way communication, monitor core component operating state, according to preset fault threshold positioning determination fault, automatically switch standby control strategy.

Description

Technical Field

[0001] This invention relates to the field of wastewater treatment technology, specifically to a wastewater pH control system based on reinforcement learning. Background Technology

[0002] In the entire wastewater treatment process, pH value is a key water quality indicator that determines treatment efficiency and system stability. Its precise control directly affects the reaction efficiency of the primary sedimentation tank, the microbial activity of the biological treatment tank, and the normal operation of subsequent advanced treatment units. Therefore, the pH value of wastewater must be strictly maintained within the set target range. Currently, traditional methods for adjusting wastewater pH value mainly rely on fixed reagent dosage, simple PID control, or manual on-site adjustment. These methods lack the ability to adapt to dynamic changes in wastewater quality and flow rate in real time. When the wastewater influent load fluctuates significantly, problems such as delayed adjustment response and excessive or insufficient reagent dosage are prone to occur. This not only results in a large waste of acid and alkali chemical reagents and increases wastewater treatment operating costs, but also causes the pH value to frequently deviate from the target range, seriously affecting the connection effect of each stage of wastewater treatment and reducing the overall treatment efficiency of the system.

[0003] In recent years, reinforcement learning technology has made significant progress in the application of industrial automatic control. Among them, Q-learning algorithms, which do not require the construction of accurate environmental models and can continuously interact with the environment to iteratively optimize control strategies, have been gradually applied to wastewater pH regulation scenarios. However, existing Q-learning-based wastewater pH regulation technologies are still limited to single algorithm design, suffering from problems such as a single state space dimension, failure to consider the prediction of water quality change trends, and a lack of refined design of control strategies. Furthermore, they have not formed a complete closed-loop system from data acquisition to execution feedback, nor do they have corresponding fault tolerance mechanisms. When facing the complex operating conditions of industrial wastewater treatment sites, the control accuracy and system robustness are insufficient to meet actual needs, and efficient, stable, and automated pH control cannot be achieved. Summary of the Invention

[0004] This application provides a wastewater pH control system based on reinforcement learning to solve the problems of slow response and insufficient control accuracy in the prior art.

[0005] The first aspect of this application provides a wastewater pH control system based on reinforcement learning, comprising: a multi-dimensional redundant sensing module, a time-series prediction feedforward module, a double-Q learning hierarchical control module, a closed-loop execution feedback module, and a fault self-diagnosis and fault tolerance module; wherein, the multi-dimensional redundant sensing module is used to collect wastewater pH, temperature, turbidity, conductivity, and wastewater flow data, and outputs effective monitoring data after processing; the time-series prediction feedforward module, based on the effective monitoring data, predicts the pH change trend through a pre-trained LSTM model, and calculates the basic acid / alkali solution addition flow rate with trend compensation by combining a preset pH target value, real-time wastewater flow rate, and acid / alkali solution concentration parameters; the double-Q learning hierarchical control module is used to adjust the basic acid / alkali solution addition flow rate... As initial input, a dual-Q-learning model containing a three-dimensional extended state space, a hierarchical action space, and an adaptive reward function is iteratively trained and optimized to generate a hierarchical optimal control strategy that matches the current actual wastewater conditions. The closed-loop execution feedback module is used to accurately add acid and alkali solutions through a variable frequency stepless adjustment unit according to the hierarchical optimal control strategy. At the same time, it collects the actual amount of acid and alkali solutions added and transmits it back to the dual-Q-learning hierarchical control module in real time to complete the closed-loop calibration and correction of the control strategy. The fault self-diagnosis and fault tolerance module is used to communicate bidirectionally with the other modules, monitor the operating status of the core components of each module in real time, complete fault location and type determination through preset fault judgment thresholds, and automatically switch to the corresponding backup control strategy according to the judgment results.

[0006] Preferably, the multi-dimensional redundant sensing module includes a data acquisition unit and a data processing unit. The data acquisition unit is equipped with sensors for detecting pH value, temperature, turbidity, conductivity, and wastewater flow rate to synchronously acquire multi-dimensional water quality parameters of wastewater. The data processing unit is used to perform basic processing on the acquired raw multi-dimensional water quality parameters, such as noise reduction and outlier removal, to obtain standardized and effective monitoring data.

[0007] Preferably, the time-series prediction feedforward module includes a pH trend prediction unit and an acid / alkali solution feedforward calculation unit. The pH trend prediction unit is used to input the effective monitoring data into a pre-trained LSTM model to predict the short-term change trend of wastewater pH and output pH trend data. The acid / alkali solution feedforward calculation unit is used to calculate the amount and flow rate of H+ substances flowing into the treatment tank based on the wastewater flow rate and pH value in the effective monitoring data, and calculate the basic addition flow rate of acid / alkali solution with trend compensation by combining the preset pH target value, acid / alkali solution concentration parameters and the pH trend data.

[0008] Preferably, the dual-Q learning hierarchical control module includes a model building unit and a strategy training and optimization unit. The model building unit is used to construct a dual-Q-learning control model, building a three-dimensional extended state space containing the current pH value, the pH change rate obtained based on pH trend data, and wastewater temperature. It divides the action space into coarse, fine, and micro-adjustment levels corresponding to different adjustment amplitudes, and designs an adaptive reward function that integrates pH deviation penalties, action smoothness rewards, and chemical reagent cost penalties. The strategy training and optimization unit uses the basic acid / alkali solution dosing flow rate as the core basic input of the dual-Q-learning control model, employs an ε-greedy strategy to dynamically balance exploration and utilization during training, and iteratively optimizes the control strategy through alternating updates of the dual-Q table to generate a hierarchical optimal control strategy matching the current wastewater conditions.

[0009] Preferably, the closed-loop execution feedback module includes a pump body execution adjustment unit and a dosing feedback calibration unit. The pump body execution adjustment unit is equipped with a variable frequency stepless adjustable acid pump and an alkali pump, which are used to dynamically adjust the start time and flow rate adjustment value of the acid pump and alkali pump according to the graded optimal control strategy, or control the pump body to enter the off state to complete the precise dosing of acid and alkali solutions. The dosing feedback calibration unit is used to collect the actual dosing amount of acid and alkali solutions in real time and send the data back to the dual-Q learning graded control module to complete the closed-loop calibration and correction of the control strategy.

[0010] Preferably, the fault self-diagnosis and fault-tolerant module includes an operation status monitoring unit and a fault-tolerant execution unit. The operation status monitoring unit is used to communicate bidirectionally with the other modules, collect the operation status parameters of sensors, acid and alkali pumps, and core components of the communication link in real time, and complete fault location and type determination through preset fault judgment thresholds. The fault-tolerant execution unit pre-stores backup control strategies that match various types of faults, and is used to automatically switch the corresponding backup control strategy according to the fault judgment result to ensure the continuity and stability of wastewater pH value control.

[0011] The second aspect of this application provides a method for regulating wastewater pH based on reinforcement learning, comprising: acquiring wastewater pH, temperature, turbidity, conductivity, and flow rate data; processing the wastewater pH, temperature, turbidity, conductivity, and flow rate data, inputting them into a pre-trained LSTM model to predict the short-term trend of wastewater pH; calculating the H+ mass flow rate flowing into the treatment tank by combining the wastewater flow rate and pH; matching preset pH target values ​​and acid / alkali concentration parameters to calculate the basic acid / alkali dosage flow rate with trend compensation; using the basic acid / alkali dosage flow rate as the initial input, performing iterative training and optimization of the strategy through a dual Q-learning regulation model to generate a graded optimal regulation strategy matching the current actual wastewater conditions; accurately adding acid / alkali according to the graded optimal regulation strategy, while simultaneously collecting and transmitting the actual acid / alkali dosage in real time for regulation strategy correction; and simultaneously monitoring the operating status of each core component of the equipment in real time, performing fault location and type determination, and automatically switching to the corresponding backup regulation strategy based on the determination result.

[0012] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement a reinforcement learning-based wastewater pH control method as described in the above embodiments.

[0013] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a reinforcement learning-based wastewater pH control method as described in the above embodiments.

[0014] A fifth aspect of this application provides a computer program product, including a computer program or instructions, for implementing a reinforcement learning-based wastewater pH control method as described in the above embodiments.

[0015] Therefore, this application has the following beneficial effects: This application embodiment utilizes a multi-dimensional redundant sensing module, configuring multiple sensors such as pH value and temperature to simultaneously collect multi-dimensional water quality data of wastewater. After noise reduction and outlier removal, it outputs effective monitoring data, improving the comprehensiveness and effectiveness of water quality parameter sensing. A time-series prediction feedforward module uses a pre-trained LSTM model to predict short-term pH changes in wastewater, and combines this with multi-parameter calculations to determine the basic acid / alkali solution dosage with trend compensation, thus compensating for water quality fluctuations in advance and eliminating the inherent lag in acid-alkali neutralization reactions. A dual-Q learning hierarchical control module builds a dual-Q-learning control model and constructs a three-dimensional extended state space, hierarchical action space, and adaptive reward function. Using the basic acid / alkali solution dosage as the core input, iteratively optimizes the control strategy, significantly improving the accuracy of the control strategy with actual wastewater conditions. The adaptability to various water conditions effectively improves the accuracy of pH control. Through a closed-loop execution feedback module, relying on a variable frequency stepless adjustment unit, acid and alkali solutions are precisely added. Real-time data collection and transmission of actual dosage complete the closed-loop calibration of the control strategy, achieving dynamic correction of the dosage and ensuring the stability of the pH control process. Through a fault self-diagnosis and fault-tolerant module, bidirectional communication with other modules and real-time monitoring of the core component's operating status automatically completes fault location and type determination and switches to the corresponding backup control strategy, avoiding control failure caused by component malfunctions and improving the overall robustness of the system. Simultaneously, each module achieves bidirectional data interaction through an industrial communication protocol, forming a complete closed-loop control system, significantly improving the automation level of pH control in wastewater treatment. Thus, it solves the problems of delayed control response and insufficient control accuracy in existing technologies.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram of a reinforcement learning-based wastewater pH control system provided according to an embodiment of this application. Figure 2 A flowchart illustrating a reinforcement learning-based method for controlling wastewater pH according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The following describes a reinforcement learning-based wastewater pH control system according to an embodiment of this application, with reference to the accompanying drawings. Addressing the issue of delayed control response mentioned in the background art, this application provides a reinforcement learning-based wastewater pH control system. In this system, a multi-dimensional redundant sensing module is used to simultaneously collect multi-dimensional water quality data of wastewater through various sensors such as pH and temperature. After noise reduction and outlier removal, effective monitoring data is output, improving the comprehensiveness and effectiveness of water quality parameter sensing. A time-series prediction feedforward module uses a pre-trained LSTM model to predict short-term pH trends in wastewater and calculates the basic acid / alkali solution dosage with trend compensation using multiple parameters, compensating for water quality fluctuations in advance and eliminating the inherent lag in acid-alkali neutralization reactions. A double-Q learning hierarchical control module is used to build a double-Q-learning control model and construct a three-dimensional extended state space, hierarchical action space, and adaptive reward function, with the basic acid / alkali solution dosage as the core. The system employs an iterative optimization approach to optimize the control strategy, significantly improving its adaptability to actual wastewater conditions and effectively enhancing pH control accuracy. A closed-loop execution feedback module, utilizing a variable frequency stepless adjustment unit, precisely adds acid and alkali solutions, collecting and transmitting actual dosage data in real time to complete closed-loop calibration of the control strategy, achieving dynamic correction of the dosage and ensuring the stability of the pH control process. A fault self-diagnosis and fault-tolerant module communicates bidirectionally with other modules and monitors the operating status of core components in real time, automatically locating and determining fault types and switching to corresponding backup control strategies. This prevents control failures caused by component malfunctions, improving the overall robustness of the system. Simultaneously, each module achieves bidirectional data interaction through an industrial communication protocol, forming a complete closed-loop control system and significantly improving the automation level of pH control in wastewater treatment. This solves the problems of delayed control response and insufficient control accuracy in existing technologies.

[0020] Figure 1 This is a schematic diagram of a wastewater pH control system based on reinforcement learning, provided as an embodiment of this application.

[0021] This application provides a wastewater pH control system based on reinforcement learning. The system 10 includes: a multi-dimensional redundant sensing module 100, a time-series prediction feedforward module 200, a dual-Q learning hierarchical control module 300, a closed-loop execution feedback module 400, and a fault self-diagnosis and fault tolerance module 500.

[0022] The multi-dimensional redundant sensing module 100 is used to collect data on the pH value, temperature, turbidity, conductivity, and flow rate of wastewater, and outputs effective monitoring data after processing. The time-series prediction feedforward module 200 predicts the pH value change trend based on the effective monitoring data through a pre-trained LSTM model, and calculates the basic acid and alkali dosage flow rate with trend compensation by combining the preset pH target value, real-time wastewater flow rate, and acid and alkali concentration parameters. The dual-Q learning hierarchical control module 300 uses the basic acid and alkali dosage flow rate as the initial input, and constructs a dual-Q-le system containing a three-dimensional extended state space, a hierarchical action space, and an adaptive reward function. The arning model undergoes iterative training and optimization to generate a graded optimal control strategy that matches the actual operating conditions of the current wastewater. The closed-loop execution feedback module 400 is used to accurately add acid and alkali solutions through a variable frequency stepless adjustment unit according to the graded optimal control strategy. At the same time, it collects the actual amount of acid and alkali solutions added and transmits it back to the dual-Q learning graded control module in real time to complete the closed-loop calibration and correction of the control strategy. The fault self-diagnosis and fault tolerance module 500 is used to communicate bidirectionally with the other modules, monitor the operating status of the core components of each module in real time, complete the fault location and type determination through preset fault judgment thresholds, and automatically switch to the corresponding backup control strategy according to the judgment results.

[0023] It is understood that the embodiments of this application utilize a multi-dimensional redundant sensing module, configuring multiple sensors such as pH value and temperature to simultaneously collect multi-dimensional water quality data of wastewater. After noise reduction and outlier removal, effective monitoring data is output, improving the comprehensiveness and effectiveness of water quality parameter sensing. A time-series prediction feedforward module uses a pre-trained LSTM model to predict short-term pH changes in wastewater, and combines this with multi-parameter calculation of the basic acid / alkali solution dosing flow rate with trend compensation to compensate for water quality fluctuations in advance and eliminate the inherent lag in acid-alkali neutralization reactions. A dual-Q learning hierarchical control module builds a dual-Q-learning control model and constructs a three-dimensional extended state space, hierarchical action space, and adaptive reward function. Using the basic acid / alkali solution dosing flow rate as the core input, the control strategy is iteratively optimized, significantly improving the control strategy. The system's adaptability to actual wastewater conditions effectively improves pH control accuracy. Through a closed-loop execution feedback module, it precisely adds acid and alkali solutions using a variable frequency stepless adjustment unit, collecting and transmitting actual dosage data in real time to complete closed-loop calibration of the control strategy, achieving dynamic correction of the dosage and ensuring the stability of the pH control process. A fault self-diagnosis and fault-tolerant module communicates bidirectionally with other modules and monitors the operating status of core components in real time, automatically locating and determining fault types and switching to corresponding backup control strategies to avoid control failures caused by component malfunctions, thus improving the overall robustness of the system. Simultaneously, each module achieves bidirectional data interaction through an industrial communication protocol, forming a complete closed-loop control system, significantly improving the automation level of pH control in wastewater treatment. This solves the problems of delayed control response and insufficient control accuracy in existing technologies.

[0024] In this embodiment, the multi-dimensional redundancy sensing module 100 includes a data acquisition unit and a data processing unit.

[0025] The data acquisition unit is equipped with sensors for pH, temperature, turbidity, conductivity and wastewater flow to simultaneously collect multi-dimensional water quality parameters of wastewater; the data processing unit is used to perform basic processing on the collected raw multi-dimensional water quality parameters, such as noise reduction and outlier removal, to obtain standardized and effective monitoring data.

[0026] It is understood that this application embodiment, through a data acquisition unit, is equipped with sensors for detecting pH value, temperature, turbidity, conductivity, and wastewater flow rate to simultaneously collect multi-dimensional water quality parameters of wastewater. This enables the comprehensive acquisition of core water quality and operating condition data affecting the efficiency of acid-base neutralization reactions, providing a complete multi-dimensional basic data source for subsequent pH value trend prediction, feedforward calculation, and control model training, thus overcoming the information limitations of traditional single pH value monitoring. Simultaneously, the data processing unit performs noise reduction and outlier removal on the collected raw multi-dimensional water quality parameters, obtaining standardized and effective monitoring data. This effectively filters environmental interference data from the wastewater treatment site, eliminates the adverse effects of outliers on subsequent calculations and model control, and ensures the validity and consistency of data input to subsequent modules, laying a reliable data foundation for the precise control of wastewater pH values.

[0027] In this embodiment of the application, the time series prediction feedforward module 200 includes: a pH trend prediction unit and an acid-base solution feedforward calculation unit.

[0028] The pH trend prediction unit is used to input effective monitoring data into a pre-trained LSTM model to predict the short-term trend of pH change in wastewater and output pH trend data. The acid and alkali feedforward calculation unit is used to calculate the amount of H+ flowing into the treatment tank based on the wastewater flow rate and pH value in the effective monitoring data, and calculate the basic addition flow rate of acid and alkali solution with trend compensation by combining the preset pH target value, acid and alkali solution concentration parameters and pH trend data.

[0029] It is understood that, through the pH trend prediction unit, the effective monitoring data is input into the pre-trained LSTM model to predict the short-term trend of wastewater pH and output pH trend data. This allows for early prediction of the dynamic changes in wastewater quality, compensating for the inherent lag in acid-base neutralization reactions from a data perspective, and providing a trend correction basis for feedforward dosage calculation. Simultaneously, through the acid-base feedforward calculation unit, the amount of H+ flowing into the treatment tank is calculated based on the wastewater flow rate and pH value in the effective monitoring data. Combined with the preset pH target value, acid-base concentration parameters, and pH trend data, the basic acid-base dosage with trend compensation is calculated. This allows for accurate calculation of the theoretical reagent dosage suitable for current and future water quality conditions, providing a reliable benchmark for the subsequent reinforcement learning control model, significantly reducing the model's ineffective exploration space, and accelerating strategy convergence.

[0030] It should be noted that the pre-trained LSTM model adopts a lightweight three-layer structure with single input and single output to balance prediction accuracy and inference speed of industrial edge devices. The input layer sets five dimensions to correspond to five core time-series features of wastewater in real time, namely, real-time pH value, temperature, turbidity, conductivity, and wastewater flow rate. The input sequence length is set to 20, which means that the prediction is completed using time-series data from the past 20 sampling periods. The hidden layer sets two cascaded LSTM core layers, with 64 LSTM neurons in each layer to extract short-term fluctuation features and long-term change patterns in water quality time-series data. After each LSTM layer, a Dropout layer with a 0.2 inactivation rate is connected to avoid model overfitting and improve the generalization ability to abnormal water quality fluctuations. The output layer sets three dimensions to correspond to the predicted pH value of wastewater for the next three sampling periods, i.e., pH trend data. The time-series features extracted by the hidden layer are mapped to the final pH prediction result through a fully connected layer.

[0031] The formula for the LSTM model is: ; ; ; ; ; ; ; in, The output of the forget gate at time t; It is the sigmoid activation function; Here is the weight matrix for the forget gate; This represents the output of the LSTM hidden layer at time t-1. Let be the input vector at time t; For the bias term of the forget gate; The output of the input gate at time t; This is the weight matrix of the input gate; This is the bias term for the input gate; Let be the candidate cell state at time t; This is the weight matrix for the candidate cell states; This is a bias term for the candidate cell state; It is the hyperbolic tangent activation function; The cell state after time t is updated. The cell state at time t-1; The output of the output gate at time t; This is the weight matrix of the output gate; This is the bias term for the output gate; This represents the final output of the LSTM hidden layer at time t. This represents the mean squared error loss value during model pre-training; This represents the total number of training samples; This represents the true pH value of the nth training sample. Predict the pH value for the model for the nth training sample.

[0032] The formulas for calculating the basic dosing flow rate of acid and alkali solutions include: Formula for calculating the flow rate of hydrogen ions flowing into the treatment tank: ; Basic formula for adding alkaline solution to acidic wastewater: ; Basic formula for adding acid to alkaline wastewater: ; Formula for calculating the basic dosing flow rate of acid and alkali solutions with trend compensation: ; in, The flow rate of hydrogen ions flowing into the treatment tank; The real-time volumetric flow rate of wastewater flowing into the treatment tank; This refers to the real-time pH value of the wastewater influent; The theoretical baseline flow rate for adding alkaline solution in acidic wastewater treatment; The preset target value for wastewater pH control; This refers to the molar concentration of the alkali solution to be added; This is the effective coefficient for the neutralization reaction of the alkali solution; The theoretical baseline flow rate for acidic solutions in alkaline wastewater treatment; This refers to the molar concentration of the acid solution used for addition; This is the effective coefficient for the neutralization reaction of the acid solution; The basic dosing flow rate for acid and alkali solutions with trend compensation; The theoretical basic flow rate for acid and alkali solutions is determined, and the flow rate is taken under acidic conditions. Take under alkaline conditions ; This is the pH trend compensation correction factor; The pH trend data output by the pre-trained LSTM model represents the predicted rate of change of wastewater pH within future sampling periods.

[0033] In this embodiment, the dual-Q learning hierarchical control module 300 includes: a model building unit and a policy training optimization unit.

[0034] The model building unit is used to construct a dual-Q-learning control model. It builds a three-dimensional extended state space that includes the current pH value, the pH change rate obtained based on pH trend data, and the wastewater temperature. It divides the action space into coarse, fine, and micro-adjustment levels corresponding to different adjustment amplitudes and designs an adaptive reward function that integrates pH deviation penalty, action smoothness reward, and chemical reagent cost penalty. The strategy training and optimization unit uses the basic acid and alkali dosing flow rate as the core basic input of the dual-Q-learning control model. It adopts an ε-greedy strategy to dynamically balance exploration and utilization during the training process. It iteratively optimizes the control strategy by alternating updates of the dual-Q table to generate a graded optimal control strategy that matches the current wastewater conditions.

[0035] It is understood that the embodiments of this application, through the model building unit, construct a dual Q-learning control model, which enriches the model's environmental perception dimensions of wastewater conditions, enabling the control strategy to adapt to the differentiated needs of different pH deviation scenarios. Simultaneously, a multi-dimensional reward function guides the model to balance control accuracy, operational stability, and reagent conservation goals. Through the strategy training and optimization unit, a graded optimal control strategy matching the current wastewater conditions is generated, effectively solving the problems of overestimation of the value function and easy getting trapped in local optima in traditional single Q-learning algorithms. This significantly improves the condition adaptability and control accuracy of the control strategy, achieving rapid convergence and stable control of wastewater pH.

[0036] It should be noted that the dual-Q-learning control model adopts a training mechanism of alternating updates and mutual verification of dual-Q tables to construct a three-dimensional extended state space, hierarchical action space, and adaptive reward function adapted to pH control scenarios. With the basic acid and alkali solution addition flow rate with trend compensation as the core basic input, the model uses an ε-greedy strategy to dynamically balance exploration and utilization during training and online inference. Through real-time interaction and iterative optimization of the control strategy with the wastewater treatment environment, the model ultimately generates a hierarchical optimal control strategy that matches the actual working conditions of the current wastewater. This can effectively improve the convergence speed, control accuracy, and operational robustness to water quality fluctuations in pH control, and adapt to the continuous and automated operation requirements of industrial wastewater treatment sites.

[0037] The formula for the dual-Q-learning regulation model includes: The formula for defining a three-dimensional extended state space is as follows: ; The formula for defining hierarchical motion space is as follows: ; Adaptive reward function calculation formula: ; 1. ; ε-greedy strategy selection formula: ; Double-Q learning: Alternating update of the double-Q table. Core formula: ; ; in, Let be the three-dimensional extended state vector of the model at time t; Let t be the real-time pH value of the wastewater at time t; Let be the rate of change of pH value of the wastewater at time t; Let t be the real-time temperature of the wastewater at time t; The control action selected for the model at time t; This is a set of large-amplitude adjustment actions corresponding to when the pH value deviation exceeds a preset first threshold. This is the set of medium-amplitude adjustment actions corresponding to when the pH deviation is between a preset first threshold and a second threshold. This is a set of small adjustment actions corresponding to when the pH value deviation is lower than a preset second threshold. Let be the total reward value obtained by the model after performing the action at time t; For the pH deviation penalty term, the adaptive weighting coefficient is used. This is a penalty for pH deviation. The preset target value for wastewater pH control; For the motion smoothness penalty term, there are adaptive weighting coefficients. This is a penalty for smoothness of motion. The control actions selected for the model at time t-1; For the chemical reagent cost penalty term, an adaptive weighting coefficient is used. This is a penalty item for the cost of chemical reagents; The actual flow rate of acid added at time t; The actual flow rate of the alkali solution added at time t; Choose a strategy for the action in state s; The optimal action that maximizes the action value function in state s; Let a be the action value function value corresponding to taking action a in state s; The exploration rate of the ε-greedy strategy; The learning rate of the model; This is the discount factor for the model; This refers to the environment state that the model transitions to at time t+1 after executing the current action; For the next state Optional control actions below; The A-table action value function of the double Q-learning model; For the B-table action value function of the dual-Q-learning model; Based on table A Select the optimal action corresponding to the maximum action value under the given state; Based on table B Select the optimal action corresponding to the maximum action value under the given state.

[0038] In this embodiment, the closed-loop execution feedback module 400 includes a pump body execution adjustment unit and an addition feedback calibration unit.

[0039] The pump body control unit is equipped with variable frequency stepless adjustment acid and alkali pumps. It is used to dynamically adjust the start time and flow rate of the acid and alkali pumps according to the graded optimal control strategy, or to control the pump body to enter the closed state to complete the precise addition of acid and alkali solutions. The addition feedback calibration unit is used to collect the actual addition amount of acid and alkali solutions in real time and send the data back to the dual-Q learning graded control module to complete the closed-loop calibration and correction of the control strategy.

[0040] It is understood that the embodiments of this application, through the pump body actuation adjustment unit, can achieve stepless and precise adjustment of the dosage, accurately matching the output requirements of the control strategy and avoiding the dosage deviation problem caused by fixed step size adjustment. At the same time, through the dosage feedback calibration unit, the deviation between the theoretical dosage and the actual dosage can be compared in real time, dynamically correcting the control strategy, forming a complete control closed loop, completely solving the dosage deviation problem caused by open loop execution, and further ensuring the accuracy and stability of pH value control.

[0041] In this embodiment, the fault self-diagnosis and fault tolerance module 500 includes an operating status monitoring unit and a fault tolerance execution unit.

[0042] The operation status monitoring unit is used to communicate bidirectionally with the other modules, collect the operation status parameters of sensors, acid and alkali pumps, and core components of the communication link in real time, and complete the fault location and type determination by preset fault judgment thresholds; the fault-tolerant execution unit pre-stores backup control strategies that match various faults, and is used to automatically switch the corresponding backup control strategy according to the fault judgment results to ensure the continuity and stability of wastewater pH control.

[0043] It is understood that, through the operational status monitoring unit, this application embodiment communicates bidirectionally with the other modules to collect real-time operational status parameters of sensors, acid and alkali pumps, and core components of the communication link. By using preset fault judgment thresholds, fault location and type determination are completed, enabling real-time monitoring of the entire system's operational status, rapid identification of component anomalies, and precise fault location, providing a reliable basis for emergency fault-tolerant handling. Simultaneously, through the fault-tolerant execution unit, backup control strategies matching various fault types are pre-stored. Based on the fault judgment results, the corresponding backup control strategy is automatically switched to ensure the continuity and stability of wastewater pH control, avoiding system control failures caused by single component failures, and improving the system's operational robustness and risk resistance in complex industrial wastewater treatment environments.

[0044] It should be noted that the preset fault judgment thresholds are multi-dimensional fault judgment boundary values ​​preset based on the operating characteristics of each core component of the system and the working conditions of the industrial site. Specifically, these include sensor fault thresholds where the sensor data exceeds the range by ±5% for three consecutive sampling cycles or the absolute value of the data deviation of the same type of main and backup sensors exceeds 10%; actuator fault thresholds where the absolute value of the deviation between the actual output flow and the commanded flow of the acid and alkali pump exceeds 15% for two consecutive control cycles; and communication fault thresholds where industrial communication between modules fails for two consecutive handshakes or the data packet loss rate in a single cycle exceeds 20%. These are used as the core judgment criteria to distinguish between normal and abnormal operating states of components.

[0045] The pre-stored backup control strategies, which are matched with various faults, are emergency control schemes that are preset to ensure the continuity and stability of system control based on the degree of impact of different fault types on pH value control. Specifically, they include a perception fault-tolerant strategy that automatically switches to the primary and backup redundant valid data and uses historical time-series data to fit and fill in missing values ​​when a sensor fails; an execution fault-tolerant strategy that automatically switches to the backup pump and synchronously adjusts the control step size and cycle when a single acid or alkali pump fails; and a degraded operation strategy that automatically switches to the local offline pre-trained control model and maintains basic stable dosing when a communication link fails. After the system completes fault location and type determination, it automatically matches and calls the corresponding scheme.

[0046] This application proposes a wastewater pH control system based on reinforcement learning. Through a multi-dimensional redundant sensing module, multiple sensors for pH, temperature, and other parameters are configured to simultaneously collect multi-dimensional water quality data from wastewater. After noise reduction and outlier removal, valid monitoring data is output, improving the comprehensiveness and effectiveness of water quality parameter sensing. A time-series prediction feedforward module uses a pre-trained LSTM model to predict short-term pH trends in wastewater and calculates trend-compensated basic acid / alkali solution dosages using multiple parameters to compensate for water quality fluctuations and eliminate the inherent lag in acid-base neutralization reactions. A dual-Q learning hierarchical control module builds a dual-Q-learning control model and constructs a three-dimensional extended state space, hierarchical action space, and adaptive reward function. Using the basic acid / alkali solution dosage as the core input, the control strategy is iteratively optimized. This system significantly improves the adaptability of the control strategy to actual wastewater conditions, effectively enhancing pH control accuracy. Through a closed-loop execution feedback module, relying on a variable frequency stepless adjustment unit, acid and alkali solutions are precisely added. Real-time data collection and transmission of actual dosage data complete the closed-loop calibration of the control strategy, enabling dynamic correction of the dosage and ensuring the stability of the pH control process. A fault self-diagnosis and fault-tolerant module communicates bidirectionally with other modules and monitors the operating status of core components in real time. It automatically locates and determines the type of fault and switches to the corresponding backup control strategy, preventing control failure due to component malfunctions and improving the overall robustness of the system. Simultaneously, each module achieves bidirectional data interaction through an industrial communication protocol, forming a complete closed-loop control system, significantly improving the automation level of pH control in wastewater treatment. This solves the problems of delayed control response and insufficient control accuracy in existing technologies.

[0047] The following will illustrate a reinforcement learning-based wastewater pH control system through a specific embodiment, including: In a domestic wastewater treatment plant's primary sedimentation tank influent pH control scenario, addressing the operational requirement of maintaining a stable influent pH within the 6.5-8.5 range for subsequent biological treatment units, the entire process was deployed with the core control objective of precisely stabilizing the wastewater pH within the range of 7.0±0.2. This approach is suitable for the complex operating conditions of the wastewater treatment plant, where the influent source is primarily municipal sewage, with intermittent industrial wastewater mixing during the rainy season and at night. It resolves the issues of response lag, excessive reagent dosage, and frequent pH overshoot inherent in the plant's original traditional PID control method. The problem is that the core hardware of the system is deployed in the inlet channel of the primary sedimentation tank and the reagent dosing room of the sewage treatment plant. The control core adopts an industrial PLC controller with an edge computing gateway. The system sampling cycle and control cycle are both set to 1 second. All algorithm models are deployed in the edge computing gateway to realize local real-time inference and strategy output. The acid solution used for dosing is 10% industrial hydrochloric acid, and the alkali solution used for dosing is 30% industrial liquid alkali. Two variable frequency stepless adjustable acid metering pumps and two alkali metering pumps are set up, one for use and one for standby. The rated flow adjustment range of the pump body is 0-50L / min, and the adjustment accuracy is ±0.5%.

[0048] The multi-dimensional redundant sensing module has two sets of multi-parameter water quality sensor groups, one main and one backup, installed in the inlet channel of the primary sedimentation tank of the wastewater treatment plant. Each sensor group is equipped with a pH sensor, temperature sensor, turbidity sensor, conductivity sensor, and a wastewater flow electromagnetic flow meter. All sensors have been installed and calibrated in accordance with wastewater treatment industry standards. The pH sensor has a measurement range of 0-14 and a measurement accuracy of ±0.02°C; the temperature sensor has a measurement range of 0-60°C and a measurement accuracy of ±0.2°C; the turbidity sensor has a measurement range of 0-1000 NTU and a measurement accuracy of ±5%; the conductivity sensor has a measurement range of 0-20 mS / cm and a measurement accuracy of ±0.5%; and the electromagnetic flow meter has a measurement range of 0-500 m³ / h and a measurement accuracy of ±0.3%. The two sets of sensor groups synchronously collect multi-dimensional water quality parameters of wastewater at a fixed sampling period of 1 second. The collected raw multi-dimensional water quality data is simultaneously processed for noise reduction and outlier removal, and the output is effective monitoring data that can be directly used for subsequent module calculation and analysis. The collected sensor operating status data is also transmitted in real time to the fault self-diagnosis and fault tolerance module.

[0049] The time-series prediction feedforward module receives valid monitoring data output from the multi-dimensional redundant sensing module and inputs the valid time-series monitoring data from 20 consecutive sampling periods into the pre-trained LSTM model. This pre-trained LSTM model uses the historical operating data of the wastewater treatment plant over the past 6 months for iterative training. After 200 rounds of iterative training, it reaches the convergence standard, and the average absolute error of the test set is controlled within 0.04. The model predicts the short-term trend of wastewater pH and outputs the corresponding pH trend data. Combining the real-time wastewater flow rate and pH parameters in the valid monitoring data with preset pH target values ​​and acid / alkali concentration parameters, it calculates and outputs the basic dosing flow rate of acid / alkali solution with trend compensation. The output basic dosing flow rate data is directly transmitted to the dual-Q learning hierarchical control module.

[0050] The dual-Q learning hierarchical control module uses the basic acid and alkali dosage flow rate output by the time-series prediction feedforward module as the core basic input to build a dual-Q-learning control model adapted to the on-site operating conditions of the wastewater treatment plant. It constructs a three-dimensional extended state space including the current pH value, pH change rate, and wastewater temperature, and divides the space into coarse-tuning, fine-tuning, and micro-tuning action spaces corresponding to different adjustment ranges. The coarse-tuning interval corresponds to the operating condition where the absolute value deviation of the pH value from the target value is ≥1.0; the fine-tuning interval corresponds to the operating condition where the absolute value deviation of the pH value from the target value is 0.2-1.0; and the micro-tuning interval corresponds to the operating condition where the absolute value deviation of the pH value from the target value is <0.2. An adaptive reward function is designed, integrating pH deviation penalty, action smoothness reward, and chemical reagent cost penalty. The weighting coefficients of the reward function are automatically adjusted under different operating conditions. An ε-greedy strategy is used to dynamically balance exploration and utilization during the iteration process. The initial exploration rate is set to 0.9, which linearly decays to 0.01 during training. The control strategy is iteratively optimized by alternating updates of the dual-Q table, generating the optimal hierarchical control strategy matching the actual operating conditions of the current wastewater and outputting it to the closed-loop execution feedback module.

[0051] The closed-loop execution feedback module receives the optimal control strategy output by the dual-Q learning hierarchical control module. Based on the control strategy instructions, it dynamically adjusts the start-up time and flow rate of the corresponding acid and alkali pumps, controlling the corresponding pumps to enter a stable operating state or shut down, thus achieving precise addition of acid and alkali solutions. It collects the actual output flow rate data of the acid and alkali pumps in real time, obtains the actual dosage information of acid and alkali solutions, and transmits the actual dosage data back to the dual-Q learning hierarchical control module in real time to complete the closed-loop calibration and correction of the control strategy. Under normal operating conditions with stable influent water quality, it continuously maintains the wastewater pH value within the range of 7.0±0.1. Under sudden operating conditions with large fluctuations in influent water quality, it quickly completes the iterative optimization of the control strategy, achieving rapid convergence of wastewater pH value without significant overshoot, and simultaneously reducing the actual consumption of acid and alkali chemical reagents throughout the process.

[0052] The fault self-diagnosis and fault-tolerant module establishes a stable bidirectional communication link with the other four modules, collects the operating status parameters of the core components of the sensor acid and alkali pump communication link in real time, and presets multi-dimensional fault judgment thresholds, including sensor fault thresholds where the sensor data exceeds the range ±5% for three consecutive sampling cycles, actuator fault thresholds where the absolute value of the deviation between the actual output flow and the command flow of the acid and alkali pump exceeds 15% for two consecutive control cycles, and communication fault thresholds where the industrial communication between modules fails for two consecutive handshakes or the single-cycle data packet loss rate exceeds 20%. By using preset fault judgment thresholds, the module completes the accurate location and type determination of faults, pre-stores backup control strategies that match various faults, and automatically switches to the corresponding backup control strategy according to the fault judgment results. When a sensor fails, the module automatically switches to the main and backup redundant effective data and uses historical time series data to fit and fill in missing values. When a single acid and alkali pump fails, the module automatically switches to the backup pump and synchronously adjusts the control step size and cycle. When a communication link fails, the module automatically switches to the local offline pre-trained control model and maintains basic stable dosing, ensuring the continuity and operational stability of the wastewater pH control process throughout.

[0053] In summary, this application's embodiment, through the coordinated operation of five modules—multi-dimensional redundant sensing, time-series prediction feedforward, dual-Q learning hierarchical control, closed-loop execution feedback, and fault self-diagnosis and fault tolerance—fully adapts to the complex influent conditions of municipal wastewater treatment plants, achieving precise and stable pH control of wastewater under various scenarios including normal operation, sudden water quality changes, and component failures. This solution effectively addresses the industry pain points of traditional control methods, such as lag response, insufficient control precision, and poor operational fault tolerance. It significantly reduces the consumption of acid and alkali reagents, greatly improves the system's automation level and operational robustness, and provides a practical and highly reliable complete solution for pH control in industrial wastewater treatment.

[0054] Next, referring to the accompanying drawings, a method for regulating wastewater pH based on reinforcement learning is described according to an embodiment of this application.

[0055] like Figure 2 As shown, the method includes the following steps: In step S101, the pH value, temperature, turbidity, conductivity and flow rate of the wastewater are acquired.

[0056] It is understood that the embodiments of this application comprehensively cover the key influencing factors affecting the efficiency of acid-base neutralization reaction and the dynamic change law of pH value by simultaneously acquiring multi-dimensional core water quality and operating condition data of wastewater such as pH value, temperature, turbidity, conductivity and wastewater flow rate. This provides a complete multi-dimensional basic data source for subsequent pH value change trend prediction and basic acid and alkali solution addition flow rate calculation. At the same time, the multi-dimensional data can form a cross-validation system, effectively identify and avoid the control deviation caused by abnormal single-dimensional data acquisition, ensure the validity and consistency of input data in subsequent control steps, improve the response speed and control accuracy of the entire pH value control method from the data source, and lay a reliable data foundation for the stable execution and precise optimization of the whole process control strategy.

[0057] In step S102, the data on the pH value, temperature, turbidity, conductivity and flow rate of the wastewater are processed and input into a pre-trained LSTM model to predict the short-term trend of pH value change in the wastewater. The amount of H+ substance flowing into the treatment tank is calculated by combining the wastewater flow rate and pH value. The preset pH target value and acid / alkali concentration parameters are matched to calculate the basic addition flow rate of acid / alkali solution with trend compensation.

[0058] Among them, the basic acid and alkali solution dosing flow rate with trend compensation refers to the dosing benchmark value obtained by trend correction of the theoretical acid and alkali solution dosing flow rate calculated based on real-time water quality and operating parameters, combined with the prediction results of short-term change trends in wastewater pH value. It is a preliminary dosing reference flow rate that adapts to the dynamic changes in wastewater quality.

[0059] It is understood that the calculated basic acid and alkali dosage flow rate in this embodiment provides a core initial input and dosage benchmark anchor point that fits the actual working conditions for the subsequent dual-Q-learning control model, which greatly narrows the scope of the model's control action exploration and accelerates the iterative convergence speed of the model's control strategy. At the same time, this flow rate is calculated based on multi-dimensional water quality parameters and pH trend prediction, which makes up for the inherent lag of acid-base neutralization reaction in advance and effectively avoids the problem of excessive or insufficient dosing of reagents when water quality fluctuates. This lays a reliable flow calculation foundation for the accurate and rapid control of wastewater pH.

[0060] In step S103, the basic dosing flow rate of acid and alkali solutions is used as the initial input, and the strategy is iteratively trained and optimized through a dual Q-learning control model to generate a graded optimal control strategy that matches the actual working conditions of the current wastewater.

[0061] Among them, the graded optimal control strategy is generated by iterative training and optimization of the dual-Q-learning control model with the basic addition flow rate of acid and alkali solution as the initial input. It matches the actual water quality conditions of the current wastewater and is a targeted pH control strategy that divides the adjustment range into coarse adjustment, fine adjustment and micro adjustment according to the degree of pH deviation.

[0062] It is understood that the embodiments of this application provide a core execution basis for the precise addition of acid and alkali solutions by generating a hierarchical optimal control strategy. The hierarchical adjustment method makes the control action highly adaptable to different pH value deviation conditions, effectively improving the accuracy and adaptability of pH value control. It solves the problem that traditional single control amplitude is prone to overshoot or control lag. At the same time, the strategy based on model iterative optimization can achieve rapid convergence of pH value while taking into account the conservation of chemical reagents and the stability of system operation.

[0063] In step S104, the acid and alkali solutions are precisely added according to the graded optimal control strategy. At the same time, the actual amount of acid and alkali solutions added is collected and transmitted back in real time to correct the control strategy. Meanwhile, the operating status of each core component of the equipment is monitored in real time, and fault location and type are determined. Based on the determination result, the corresponding backup control strategy is automatically switched.

[0064] It is understood that this application embodiment achieves precise addition of acid and alkali solutions by following a graded optimal control strategy, realizing accurate matching between reagent dosage and actual wastewater conditions. This ensures the accuracy of pH control from the execution level, avoiding cost waste and control failure caused by excessive or insufficient chemical reagent addition. This application embodiment collects and transmits the actual acid and alkali solution dosage in real time and corrects the control strategy, constructing a complete control closed loop from strategy output to execution feedback and strategy optimization. This continuously calibrates dosing deviations, ensuring the dynamic adaptability and stability of the pH control process. This application embodiment monitors the operating status of core equipment components in real time, completes fault location and type determination, and automatically switches to the corresponding backup control strategy. This effectively avoids control failure caused by a single component failure, ensuring the continuity of pH control and significantly improving the robustness and automation level of the entire control system under complex industrial wastewater treatment conditions.

[0065] According to the embodiments of this application, a wastewater pH control method based on reinforcement learning is proposed. This method utilizes a multi-dimensional redundant sensing module, configuring multiple sensors such as pH and temperature to simultaneously collect multi-dimensional water quality data from wastewater. After noise reduction and outlier removal, valid monitoring data is output, improving the comprehensiveness and effectiveness of water quality parameter sensing. A time-series prediction feedforward module uses a pre-trained LSTM model to predict short-term pH trends in wastewater. Combined with multi-parameter calculation of trend-compensated basic acid / alkali solution dosing rates, it compensates for water quality fluctuations in advance, eliminating the inherent lag in acid-base neutralization reactions. A double-Q learning hierarchical control module constructs a double-Q-learning control model and builds a three-dimensional extended state space, hierarchical action space, and adaptive reward function. The control strategy is iteratively optimized using the basic acid / alkali solution dosing rate as the core input. This system significantly improves the adaptability of control strategies to actual wastewater conditions, effectively enhancing pH control accuracy. Through a closed-loop execution feedback module, it precisely adds acid and alkali solutions using a variable frequency stepless adjustment unit, collecting and transmitting actual dosage data in real time to complete closed-loop calibration of the control strategy, achieving dynamic correction of the dosage and ensuring the stability of the pH control process. A fault self-diagnosis and fault-tolerant module communicates bidirectionally with other modules and monitors the operating status of core components in real time, automatically locating and determining fault types and switching to corresponding backup control strategies to avoid control failures caused by component malfunctions, thus improving the overall robustness of the system. Simultaneously, each module achieves bidirectional data interaction through an industrial communication protocol, forming a complete closed-loop control system, significantly improving the automation level of pH control in wastewater treatment. This solves the problems of delayed control response and insufficient control accuracy in existing technologies.

[0066] The following specific example illustrates a reinforcement learning-based method for controlling wastewater pH, including: In a domestic industrial park wastewater treatment plant's biological treatment tank pH control scenario, this method, designed for the plant's daily treatment capacity of 20,000 m³, focuses on maintaining the influent pH value within the 6.8-8.0 range required for maintaining microbial activity in the subsequent biological treatment tank. The core control objective is to precisely stabilize the wastewater pH value within the range of 7.2±0.2. The plant's influent is a mixture of industrial wastewater and domestic sewage from the park. Significant pH fluctuations, ranging from 4.0 to 10.0, are common during the morning's concentrated discharge of industrial wastewater. Existing manual control methods using fixed-dosage reagents suffer from issues such as delayed dosing timing, inaccurate dosage control, and untimely responses to sudden water quality fluctuations. This control method achieves fully automated operation, with both sampling and control cycles set to 2 seconds. The acid solution used is 8% industrial sulfuric acid, and the alkali solution is 25% industrial liquid alkali. A variable-frequency continuously variable metering pump is used for reagent dosing. Related reinforcement learning algorithms are deployed on an industrial intelligent control terminal, enabling fully autonomous control from data acquisition to strategy execution.

[0067] Multi-parameter water quality monitoring equipment was deployed at the core monitoring point of the biochemical pool inlet channel to continuously collect data on wastewater pH, temperature, turbidity, conductivity, and flow rate. All monitoring equipment underwent pre-calibration and periodic verification in accordance with the wastewater treatment testing standards of the industrial park to ensure the accuracy and continuity of the raw data collection. The collected raw water quality and operating condition data were standardized and processed using professional algorithms to reduce noise and remove outliers, filtering out invalid data caused by on-site environmental interference and instantaneous equipment malfunctions, resulting in effective data that can be directly used for subsequent trend prediction and flow calculation. This ensures the accuracy of calculations in subsequent control steps from the data collection stage, laying a reliable data foundation for pH control throughout the entire process.

[0068] Standardized continuous time-series data are input into a pre-trained LSTM model. This model is trained using historical operating data from the wastewater treatment plant in the industrial park over the past three months. After multiple iterations, it reaches the convergence standard and meets the prediction accuracy requirements for actual on-site control. The model accurately predicts the short-term trend of wastewater pH. Combining the real-time flow rate and pH value of wastewater in the effective data, the model calculates the amount and flow rate of hydrogen ions flowing into the biological treatment tank. Then, it matches the preset pH control target value and the actual concentration parameters of the acid and alkali solutions added on-site. Combined with the predicted short-term pH trend, the model performs trend compensation correction. Finally, it calculates the basic addition flow rate of acid and alkali solutions with trend compensation, ensuring that the preliminary calculation of the reagent dosage fully adapts to the dynamic changes in wastewater quality and compensates for the inherent lag of acid-base neutralization reactions from a computational perspective.

[0069] Using the basic acid and alkali solution dosing flow rate with trend compensation as the initial input, a dual-Q-learning control model is imported to conduct iterative training and optimization of the control strategy. A three-dimensional state space is constructed, including the current pH value, pH change rate, and wastewater temperature. The control is divided into coarse, fine, and micro-adjustment levels according to the degree of deviation between the pH value and the target value. Coarse adjustment corresponds to large fluctuations with an absolute deviation of ≥1.0, fine adjustment corresponds to moderate fluctuations with an absolute deviation of 0.2-1.0, and micro-adjustment corresponds to stable conditions with an absolute deviation of <0.2. A comprehensive reward function is designed that integrates pH control accuracy, control action smoothness, and chemical reagent usage cost. An ε-greedy strategy is used to dynamically balance and explore during model training. The control strategy is continuously iterated and optimized through alternating updates of the dual-Q table. Finally, a graded optimal control strategy that perfectly matches the actual wastewater quality conditions is generated, ensuring that the control action is highly adapted to the real-time on-site conditions.

[0070] Based on the generated optimal hierarchical control strategy, the precise addition of acid and alkali solutions is achieved through a variable frequency stepless metering pump. The start-up time and actual output flow of the metering pump are dynamically adjusted according to the on-site working conditions, and the pump is controlled to enter a stable operation or shut down as needed to ensure that the dosage of the reagents is accurately matched with the actual working conditions of the wastewater. The actual dosage data of acid and alkali solutions from the metering pump is collected simultaneously and transmitted back to the dual-Q-learning control model in real time. The model dynamically calibrates and corrects the subsequent control strategy based on the deviation between the actual dosage and the theoretical calculation value, thus constructing a complete control closed loop from strategy output to execution feedback and strategy optimization. At the same time, the core operating status of water quality testing equipment and reagent dosing equipment is continuously monitored, and clear equipment fault judgment criteria are preset. The fault is accurately located and the type is determined by the real-time collected operating status data. Based on the judgment result, the pre-set backup control strategy is automatically switched. Under the condition of large fluctuations in water quality or occasional equipment failure, the pH value control process is always guaranteed to be uninterrupted, and the pH value of the wastewater is continuously stabilized within the preset target range, effectively avoiding various problems of traditional control methods.

[0071] In summary, this application's embodiment utilizes a fully automated reinforcement learning-based wastewater pH control method, adapted to the fluctuating water quality conditions of mixed influent in industrial park wastewater treatment plants. It forms a complete control chain from data acquisition, trend prediction, flow calculation to strategy optimization, precise dosing, and fault tolerance. This effectively solves the problems of dosing lag, inaccurate dosage control, and insufficient response to sudden conditions inherent in traditional manual methods combined with fixed dosing. It achieves precise and stable pH control while conserving chemical reagents, ensuring the continuity and stability of the control process, and significantly improving the automation level and adaptability of pH control in industrial park wastewater treatment.

[0072] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.

[0073] When the processor 302 executes the program, it implements a wastewater pH control method based on reinforcement learning provided in the above embodiments.

[0074] Furthermore, electronic devices also include: Communication interface 303 is used for communication between memory 301 and processor 302.

[0075] The memory 301 is used to store computer programs that can run on the processor 302.

[0076] The memory 301 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0077] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0078] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.

[0079] Processor 302 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0080] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described reinforcement learning-based wastewater pH control method.

[0081] Furthermore, this application also provides a computer program product, including a computer program or instructions, which, when executed, implement the aforementioned reinforcement learning-based wastewater pH control method.

[0082] In the description of this specification, the references to the terms "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0084] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0085] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0086] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0087] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A wastewater pH control system based on reinforcement learning, characterized in that, include: The system includes a multi-dimensional redundant sensing module, a time-series prediction feedforward module, a double-Q learning hierarchical control module, a closed-loop execution feedback module, and a fault self-diagnosis and fault tolerance module; among which... The multi-dimensional redundant sensing module is used to collect data on the pH value, temperature, turbidity, conductivity and flow rate of wastewater, and outputs effective monitoring data after processing. The time-series prediction feedforward module, based on the effective monitoring data, predicts the pH value change trend through a pre-trained LSTM model, and calculates the basic acid and alkali solution addition flow rate with trend compensation by combining the preset pH target value, real-time sewage flow rate, and acid and alkali solution concentration parameters. The dual-Q learning hierarchical control module is used to take the basic acid and alkali solution addition flow rate as the initial input, and iteratively train and optimize the constructed dual-Q-learning model containing a three-dimensional extended state space, hierarchical action space and adaptive reward function to generate the hierarchical optimal control strategy that matches the current actual working conditions of the wastewater. The closed-loop execution feedback module is used to accurately add acid and alkali solutions through a variable frequency stepless adjustment unit according to the hierarchical optimal control strategy. At the same time, it collects the actual amount of acid and alkali solutions added and transmits it back to the dual-Q learning hierarchical control module in real time to complete the closed-loop calibration and correction of the control strategy. The fault self-diagnosis and fault tolerance module is used to communicate bidirectionally with the other modules, monitor the operating status of the core components of each module in real time, complete the fault location and type determination through preset fault judgment thresholds, and automatically switch the corresponding backup control strategy according to the judgment result.

2. The wastewater pH control system based on reinforcement learning according to claim 1, characterized in that, The multi-dimensional redundant sensing module includes a data acquisition unit and a data processing unit. The data acquisition unit is equipped with sensors for detecting pH value, temperature, turbidity, conductivity, and sewage flow rate to synchronously acquire multi-dimensional water quality parameters of sewage. The data processing unit is used to perform basic processing on the acquired raw multi-dimensional water quality parameters, such as noise reduction and outlier removal, to obtain standardized and effective monitoring data.

3. The wastewater pH control system based on reinforcement learning according to claim 1, characterized in that, The time-series prediction feedforward module includes a pH trend prediction unit and an acid / alkali solution feedforward calculation unit. The pH trend prediction unit is used to input the effective monitoring data into a pre-trained LSTM model to predict the short-term change trend of wastewater pH and output pH trend data. The acid / alkali solution feedforward calculation unit is used to calculate the amount of H+ flowing into the treatment tank based on the wastewater flow rate and pH value in the effective monitoring data, and calculate the basic addition flow rate of acid / alkali solution with trend compensation by combining the preset pH target value, acid / alkali solution concentration parameters and the pH trend data.

4. The wastewater pH control system based on reinforcement learning according to claim 1, characterized in that, The dual-Q learning hierarchical control module includes a model building unit and a strategy training and optimization unit. The model building unit constructs a dual-Q-learning control model, building a three-dimensional extended state space containing the current pH value, the pH change rate based on pH trend data, and wastewater temperature. It then divides the space into coarse, fine, and micro-adjustment levels corresponding to different adjustment amplitudes, and designs an adaptive reward function that integrates pH deviation penalties, action smoothness rewards, and chemical reagent cost penalties. The strategy training and optimization unit uses the basic acid / alkali dosage flow rate as the core input to the dual-Q-learning control model. It employs an ε-greedy strategy to dynamically balance exploration and utilization during training, iteratively optimizing the control strategy through alternating updates of the dual-Q table, and generating a hierarchical optimal control strategy matching the current wastewater conditions.

5. The wastewater pH control system based on reinforcement learning according to claim 1, characterized in that, The closed-loop execution feedback module includes a pump body execution adjustment unit and a dosing feedback calibration unit. The pump body execution adjustment unit is equipped with a variable frequency stepless adjustment acid pump and an alkali pump. It is used to dynamically adjust the start time and flow rate adjustment value of the acid pump and alkali pump according to the graded optimal control strategy, or control the pump body to enter the off state to complete the precise dosing of acid and alkali solutions. The dosing feedback calibration unit is used to collect the actual dosing data of acid and alkali solutions in real time and send the data back to the dual-Q learning graded control module to complete the closed-loop calibration and correction of the control strategy.

6. The wastewater pH control system based on reinforcement learning according to claim 1, characterized in that, The fault self-diagnosis and fault-tolerant module includes an operation status monitoring unit and a fault-tolerant execution unit. The operation status monitoring unit is used to communicate bidirectionally with the other modules, collect the operation status parameters of sensors, acid and alkali pumps, and core components of the communication link in real time, and complete fault location and type determination through preset fault judgment thresholds. The fault-tolerant execution unit pre-stores backup control strategies that match various types of faults, and is used to automatically switch the corresponding backup control strategy according to the fault judgment result to ensure the continuity and stability of wastewater pH control.

7. A method for regulating wastewater pH based on reinforcement learning, characterized in that, include: Obtain data on the pH value, temperature, turbidity, conductivity, and flow rate of wastewater; After processing the data on pH, temperature, turbidity, conductivity and flow rate of the wastewater, the data are input into a pre-trained LSTM model to predict the short-term trend of pH change in the wastewater. The amount of H+ substance flowing into the treatment tank is calculated by combining the wastewater flow rate and pH value. The basic addition flow rate of acid and alkali solution with trend compensation is calculated by matching the preset pH target value and acid and alkali solution concentration parameters. Using the basic dosing flow rate of the acid and alkali solutions as the initial input, the strategy is iteratively trained and optimized through a dual-Q-learning control model to generate a graded optimal control strategy that matches the actual working conditions of the current wastewater. The system accurately adds acid and alkali solutions based on a graded optimal control strategy. It also collects and transmits the actual amount of acid and alkali solutions added in real time to correct the control strategy. In addition, it monitors the operating status of each core component of the equipment in real time, locates and determines the type of fault, and automatically switches to the corresponding backup control strategy based on the determination results.

8. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the reinforcement learning-based wastewater pH control method of claim 7.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the reinforcement learning-based wastewater pH control method of claim 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the reinforcement learning-based wastewater pH control method of claim 7.

Images