Dynamic optimization method, device and equipment for whole process control quantity of sewage treatment plant

Abstract

The present disclosure provides a method, device and equipment for dynamically optimizing the control quantity of the whole process of a sewage treatment plant. The method includes: first obtaining the inlet data and control quantity of the sewage treatment plant under the current working condition, then predicting the along-process data and effluent data under the current working condition based on the inlet data and control quantity under the current working condition, and finally optimizing the control quantity under the current working condition using a pre-constructed control quantity optimization model based on the inlet data, along-process data and effluent data under the current working condition. Since the control quantity optimization model is obtained based on a reinforcement learning algorithm and the reward value involved in the reinforcement learning algorithm is the largest when the effluent data meets the standard and the comprehensive operating cost is the lowest, after optimizing the control quantity of the sewage treatment plant using the control quantity optimization model in the present disclosure, the sewage treatment plant can achieve the lowest comprehensive operating cost while achieving the effluent standard.

Description

Technical Field

[0001] The present disclosure relates to the technical field of sewage treatment, and in particular to a method, device and equipment for dynamically optimizing the control quantity of the entire process of a sewage treatment plant. Background Art

[0002] The biological system of a sewage treatment plant is very complex. The various sewage treatment units are interconnected to form a stable, comprehensive, large, slow-changing biochemical system. It is difficult to achieve the lowest comprehensive cost of aeration, dosing, backflow, and sludge discharge while meeting the effluent standards by adjusting the control volume of the sewage treatment plant based on manual experience. For example, in order to achieve effluent standards, existing sewage treatment plants will increase the amount of aeration and the amount of chemical agents added based on manual experience, resulting in an increase in comprehensive costs. Therefore, how to optimize the control volume of the sewage treatment plant so that the effluent meets the standards while achieving the lowest comprehensive cost is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0003] In view of this, the present disclosure proposes a dynamic optimization method, device and equipment for the whole process control quantity of a sewage treatment plant, which can achieve the lowest comprehensive operating cost while achieving effluent compliance.

[0004] According to a first aspect of the present disclosure, a method for dynamically optimizing the control quantity of the entire process of a sewage treatment plant is provided, comprising:

[0005] Obtain the influent data and control quantity under the current working conditions of the sewage treatment plant;

[0006] According to the water inlet data and the control amount, predicting the along-line data and the water outlet data under the current working condition;

[0007] According to the water inlet data, the data along the way and the water outlet data, the control amount is optimized by using a pre-constructed control amount optimization model;

[0008] Among them, the control quantity optimization model is obtained based on the reinforcement learning algorithm, and the reward value involved in the reinforcement learning algorithm is the largest when the water output data meets the standard and the comprehensive operating cost is the lowest.

[0009] In a possible implementation, when predicting the along-the-line data and the effluent data under the current operating conditions according to the influent data and the control quantity, it is implemented based on an activated sludge model.

[0010] In a possible implementation, the activated sludge model is debugged based on the historical influent data, historical process data and historical effluent data of the sewage treatment plant;

[0011] When the activated sludge model is debugged based on the historical influent data, historical process data and historical effluent data of the sewage treatment plant, it includes:

[0012] According to the historical water inflow data, an activated sludge model is used to obtain corresponding predicted historical along-the-line data and predicted historical water outflow data;

[0013] The kinetic coefficient and stoichiometric coefficient of the activated sludge model are adjusted so that the difference between the predicted historical along-the-way data and the historical along-the-way data and the difference between the predicted historical effluent data and the historical effluent data both meet the preset difference threshold value, and the activated sludge model is used as the activated sludge model.

[0014] In a possible implementation, the network framework of the reinforcement learning algorithm is Actor-Critic.

[0015] In a possible implementation, the step of constructing an optimization model for the control amount is also included;

[0016] When constructing the control quantity optimization model, it includes:

[0017] Based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value, constructing a four-tuple training set for training the Actor-Critic network framework;

[0018] Based on the four-tuple training set, the Actor-Critic network framework is trained, and the Actor network framework is extracted from the trained Actor-Critic network framework as the control quantity optimization model.

[0019] In a possible implementation, when constructing a first quadruple for training the Actor-Critic network framework based on the historical influent data of the sewage treatment plant, the activated sludge model, and the calculation formula of the reward value, it includes:

[0020] According to the first historical water inflow data and the corresponding control amount, the activated sludge model is used to obtain the first historical water outflow data and the first historical along-line data, and the first historical operating state data is determined according to the first historical water inflow data, the first historical water outflow data and the first historical along-line data;

[0021] According to the first historical operating status data, using the initialized Actor network framework, obtaining the control amount corresponding to the second historical water inflow data;

[0022] According to the second historical water inflow data and the control amount corresponding to the second historical water inflow data, the activated sludge model is used to obtain the second historical water outflow data and the second historical along-line data, and according to the second historical water inflow data, the second historical water outflow data and the second historical along-line data, the second historical operating state data is determined;

[0023] According to the second historical water output data, the reward value corresponding to the second historical water output data is obtained by using the calculation formula of the reward value;

[0024] The first quaternary group is determined based on the first historical operating status data, the control amount corresponding to the second historical water inlet data, the reward value corresponding to the second historical water outlet data, and the second historical operating status data.

[0025] In a possible implementation, the calculation formula of the reward value is determined based on the effluent water quality reward value, the biological phosphorus removal reward value and the operating cost reward value.

[0026] In a possible implementation, the calculation formula of the reward value is as follows:

[0027] r=w1min(r Tp ,r out )+w2r cost

[0028] In the formula, r is the reward value, r out is the effluent water quality reward value, r TP is the reward value for biological phosphorus removal, r cost is the operating cost bonus value, w1 and w2 are proportional coefficients.

[0029] According to a second aspect of the present disclosure, a device for dynamically optimizing the control quantity of the entire process of a sewage treatment plant is provided, comprising:

[0030] The inlet data acquisition module is used to obtain the inlet data and control quantity under the current working conditions of the sewage treatment plant;

[0031] A water discharge data prediction module, used to predict the along-the-line data and water discharge data under the current working condition according to the water inlet data and the control amount;

[0032] A control quantity optimization module, used to optimize the control quantity using a pre-built control quantity optimization model according to the water inlet data, the data along the process and the water outlet data;

[0033] Among them, the control quantity optimization model is obtained based on the reinforcement learning algorithm, and the reward value involved in the reinforcement learning algorithm is the largest when the water output data meets the standard and the comprehensive operating cost is the lowest.

[0034] According to a third aspect of the present disclosure, a dynamic optimization device for the whole process control quantity of a sewage treatment plant is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute the method described in the first aspect of the present disclosure.

[0035] In the present disclosure, the inlet data and control quantity of the sewage treatment plant under the current working condition are first obtained, and then the data along the process and the effluent data under the current working condition are predicted based on the inlet data and control quantity under the current working condition, and finally the control quantity under the current working condition is optimized using the pre-constructed control quantity optimization model based on the inlet data, along the process data and effluent data. Since the control quantity optimization model is obtained based on the reinforcement learning algorithm and the reward value involved in the reinforcement learning algorithm is the largest when the effluent data meets the standard and the comprehensive operating cost is the lowest, after optimizing the control quantity of the sewage treatment plant using the control quantity optimization model in the present disclosure, the sewage treatment plant can achieve the lowest comprehensive operating cost while achieving the effluent standard.

[0036] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0038] Figure 1 A schematic block diagram showing a method for dynamically optimizing the control quantity of the entire process of a sewage treatment plant according to an embodiment of the present disclosure;

[0039] Figure 2 A process flow chart of a sewage treatment plant according to an embodiment of the present disclosure is shown;

[0040] Figure 3 A comparison result diagram of effluent COD under dynamic conditions according to an embodiment of the present disclosure is shown;

[0041] Figure 4 A comparison result diagram of the outlet TN under dynamic conditions according to an embodiment of the present disclosure is shown;

[0042] Figure 5 The NH4 output water under dynamic conditions according to an embodiment of the present disclosure is shown. + -N comparison result graph;

[0043] Figure 6 A graph showing the comparison results of the outlet and outlet TP under dynamic conditions according to an embodiment of the present disclosure is shown;

[0044] Figure 7A diagram showing energy consumption comparison results under dynamic conditions according to an embodiment of the present disclosure;

[0045] Figure 8 A schematic block diagram showing a device for dynamically optimizing the control amount of the entire process of a sewage treatment plant according to an embodiment of the present disclosure;

[0046] Fig. 9 A schematic block diagram of a device for dynamically optimizing the control quantity of the entire process of a sewage treatment plant according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0048] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0049] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0050] <Method Example>

[0051] Figure 1 A schematic block diagram of a method for dynamically optimizing the control quantity of the entire process of a sewage treatment plant according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S1100-S1300.

[0052] S1100, obtaining water inlet data and control quantity under the current operating conditions of the sewage treatment plant.

[0053] The current operating condition is the actual operating condition before the control quantity is optimized.

[0054] The control amount may include at least one of dissolved oxygen (or aeration amount), chemical agent dosage, internal recirculation, external recirculation, influent distribution ratio, and sludge discharge. When optimizing the control amount of a sewage treatment plant, all the control amounts of the sewage treatment plant may be selected as optimization objects, or part of the control amounts may be selected as optimization objects according to specific needs, which is not specifically limited here.

[0055] The influent data includes influent volume and influent index values. The influent index values ​​may include influent chemical oxygen demand (i.e. influent COD), influent total ammonia content (i.e. influent TN), influent ammonia nitrogen content (i.e. influent NH4 + -N), at least one of the influent total phosphorus concentration (i.e., influent TP) and influent suspended solids (i.e., influent SS).

[0056] S1200: predicting the along-the-line data and the outflow data under the current working condition according to the inflow data and the control quantity under the current working condition.

[0057] The process data is the process data measured by the measuring device configured in the sewage treatment process. Figure 2 As shown, the data along the way may include the sludge concentration MLSS, nitrate nitrogen content NO3 measured in the anoxic tank 4 - -N, dissolved oxygen DO measured in the three aerobic tanks, ammonia nitrogen NH4 measured in the aerobic tank 3 + -N and phosphate PO4 measured in anaerobic tank 2 3- At least one of .

[0058] The effluent data includes effluent index values, which may include effluent chemical oxygen demand (ie, effluent COD), effluent total ammonia content (ie, effluent TN), effluent ammonia nitrogen content (ie, effluent NH4 + -N), at least one of the effluent total phosphorus concentration (i.e., effluent TP) and effluent suspended solids (i.e., effluent SS).

[0059] In a possible implementation, when predicting the process data and effluent data under the current working conditions based on the influent data and the control quantity, it can be implemented based on the activated sludge model. Specifically, the activated sludge model is a simulation model of the sewage treatment system based on the activated sludge method proposed by the International Water Association, which can simulate the operation process of the sewage treatment system to a certain extent. Therefore, the influent data and control quantity under the current working conditions of the sewage treatment plant are input into the activated sludge model, and the activated sludge model can simulate the process data and effluent data under the current working conditions.

[0060] It should be noted that in order for the activated sludge model to accurately simulate the actual operation process of the sewage treatment plant, after obtaining the activated sludge model, it is necessary to debug the basic activated sludge model according to the historical operation data of the sewage treatment plant, and then use the activated sludge model after debugging.

[0061] Specifically, under historical operating conditions, multiple historical operation data of the sewage treatment plant will be recorded, and each piece of historical operation data includes historical water inflow data and historical along-the-way data and historical effluent data corresponding to the historical water inflow data. When debugging the basic activated sludge model according to the historical operation data, it includes: first, according to the historical water inflow data, using the activated sludge model to obtain the corresponding predicted historical along-the-way data and predicted historical effluent data. Then, the kinetic coefficient and stoichiometric coefficient of the activated sludge model are adjusted, and the difference between the predicted historical along-the-way data and the historical along-the-way data and the difference between the predicted historical effluent data and the historical effluent data meet the preset difference threshold value. The activated sludge model after debugging is used.

[0062] For example, for any historical operation data: extract the historical influent data and input it into the activated sludge model to obtain the corresponding predicted historical along-process data and predicted historical effluent data; extract the historical along-process data and calculate the along-process difference between the predicted historical along-process data and the historical along-process data; extract the historical effluent data and calculate the effluent difference between the predicted historical effluent data and the historical effluent data; adjust the kinetic coefficient and stoichiometric coefficient of the activated sludge model according to the along-process difference and the effluent difference, so that the predicted historical effluent data and predicted historical along-process data predicted by the activated sludge model are close to the actual historical effluent data and historical along-process data. Use different historical operation data and repeat the above debugging process until the along-process difference and the effluent difference both meet the preset difference threshold, and use the activated sludge model at this time as the final activated sludge model.

[0063] S1300 optimizes the control quantity using a pre-built control quantity optimization model based on the water inlet data, along-the-way data, and water outlet data under the current working conditions.

[0064] Before executing step S1300, a control quantity optimization model needs to be constructed. In the present disclosure, the control quantity optimization model is obtained based on a reinforcement learning algorithm, and the reward value involved in the reinforcement learning algorithm is the largest when the water output data meets the standard and the comprehensive operating cost is the lowest.

[0065] In a possible implementation, the control amount optimization model is obtained based on a reinforcement learning algorithm whose network framework is Actor-Critic. When the control amount optimization model is constructed based on the Actor-Critic network framework, steps S1310-S1320 are included.

[0066] S1310, based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value, construct a four-tuple training set for training the Actor-Critic network framework.

[0067] The four-tuple training set includes multiple four-tuples. The following takes one four-tuple as an example to illustrate the steps of constructing the four-tuple. Specifically, the steps of constructing the four-tuple include S1311-S1315.

[0068] S1311, according to the first historical water inflow data and the corresponding control amount, using the activated sludge model, obtain the first historical water outflow data and the first historical data along the process, and determine the first historical operation state data according to the first historical water inflow data, the first historical water outflow data and the first historical data along the process. It should be noted that the historical water inflow data are marked as the first historical water inflow data, the second historical water inflow data, ..., the Zth historical water inflow data in the order of the collection time, so the first historical water inflow data here is the earliest historical water inflow data collected.

[0069] For example, the first historical water inflow data x in (Specifically including water volume, water COD, water TN, water NH4 + -N, influent TP and influent SS) and the corresponding control amount are input into the activated sludge model to simulate the first historical influent data corresponding to the first historical effluent data x out (Specifically including effluent COD, effluent TN, effluent NH4 + -N, effluent TP and effluent SS) and the first historical data along the way (specifically including the sludge concentration MLSS of anoxic tank 4, dissolved oxygen DO of 3 aerobic tanks and ammonia nitrogen content NH4 + -N). The first historical water inflow data x in , the first historical water output data x out The vector composed of the first historical data along the way y is used as the first historical operation status data s, that is, the first historical operation status data s includes the influent volume, influent COD, influent TN, influent NH4 + -N, influent TP, influent SS, sludge concentration MLSS of aerobic pool 4, dissolved oxygen DO of 3 aerobic pools, ammonia nitrogen content NH4 in aerobic pool 3 + -N, effluent COD, effluent TN, effluent NH4 + -N, outlet water TP and outlet water SS.

[0070] S1312: According to the first historical operating status data, the initialized Actor network framework is used to obtain a control amount corresponding to the second historical water inflow data.

[0071] It should be noted that before executing step S1312, the Actor-Critic network framework needs to be initialized and the initialized Actor network framework needs to be extracted. The Actor-Critic network framework includes the Critic network and the Actor network. When initializing the Actor-Critic network framework, it includes: randomly initializing the Critic network according to the standard normal distribution. and The network parameters θ1 and θ2, and the actor network π Φ At the same time, initialize the target network and π Φ‘ , specifically, θ'1←θ1, θ'2←θ2, Φ'←Φ, even if the target network has the same corresponding network parameters as the Actor-Critic network.

[0072] After initializing the Actor-Critic network, extract the initialized Actor network framework and input the first historical running state data s into the initialized Actor network π Φ , to predict the control amount a corresponding to the second historical water inflow data. Specifically, the calculation process of the control amount a corresponding to the second historical water inflow data is as follows:

[0073]

[0074]

[0075] ε~clip(N(0,0.1),-0.5,0.5) (3)

[0076]

[0077] In the formula, a upper is the upper limit of the control amount, a l o wer is the lower limit of the control quantity, x a is the intermediate value corresponding to the control amount.

[0078] It should be noted that there are multiple control quantities in the sewage treatment plant, and each control quantity is preset with an upper limit value and a lower limit value (as shown in Table 1). Therefore, when calculating the control quantity corresponding to the second historical water inlet data, the first historical operating status data s is first substituted into formula (1) to calculate the intermediate value corresponding to each control quantity, and then the intermediate value, upper limit value and lower limit value corresponding to each control quantity are substituted into formula (2) in turn to calculate each control quantity value corresponding to the second historical water inlet data.

[0079] Table 1

[0080]

[0081] S1313, according to the second historical water inflow data and the control amount corresponding to the second historical water inflow data, the activated sludge model is used to obtain the second historical water outflow data and the second historical data along the process, and the second historical operation status data is determined according to the second historical water inflow data, the second historical water outflow data and the second historical data along the process. The steps for constructing the second historical operation status data s' refer to the steps for constructing the first historical operation status data s, which will not be repeated here.

[0082] S1314, according to the second historical water output data, using a calculation formula for the reward value, to obtain a reward value corresponding to the second historical water output data.

[0083] In a possible implementation, the calculation formula of the reward value is determined based on the effluent water quality reward value, the biological phosphorus removal reward value and the operating cost reward value.

[0084] In a possible implementation, the calculation formula of the reward value is shown in formula (5):

[0085] r=w1min(r TP , r out )+w2r cost (5)

[0086] In the formula, r is the reward value, r out is the effluent water quality reward value, r TP is the reward value for biological phosphorus removal, r cost is the operating cost reward value, w1 and w2 are proportional coefficients, where w1, w2∈(0,1) and w1+w2=1, for example, w1=w2=0.5.

[0087] In order to make the water output index value close to the optimal water output index value x best When the reward value increases, it exceeds the optimal water discharge standard x best When the reward value is significantly reduced, and when the water output index value exceeds the standard water output index value x std When the corresponding penalty is given (the reward value is a negative value), the reward value calculation of the water output index value corresponding to the control amount can be shown as follows:

[0088]

[0089] In the formula, r out-tmp is the reward value of the water output index value, x is the water output index value, x std is the standard water index value, x best It is the best water output index value.

[0090] It should be noted that for each water output index value, a corresponding standard water output index value and an optimal water output index value are preset (as shown in Table 2). Therefore, when calculating the reward value of each water output index value, it is necessary to calculate based on the corresponding standard water output index value and the optimal water output index value.

[0091] Table 2

[0092]

[0093] After calculating the reward value of each water quality index value, the water quality reward value is calculated according to formula (7).

[0094]

[0095] In the formula, r out is the effluent water quality reward value, r out-tmp It is the reward value of the water output index value.

[0096] Specifically, when each water outlet index value is greater than the standard water outlet index value, the reward values ​​of all water outlet index values ​​are summed up as the water outlet water quality reward value. When any water outlet index value is less than or equal to the standard water outlet index value, the minimum value is selected from the reward values ​​of all water outlet index values ​​as the water outlet water quality reward value.

[0097] In order to make the operating cost corresponding to the control amount higher, the corresponding operating cost reward value is lower, and the operating cost corresponding to the lower operating cost is higher, even if the operating cost reward value is negatively correlated with the operating cost, the calculation formula of the operating cost reward value can be shown as formula (8):

[0098]

[0099] In the formula, r cost is the operating cost reward value, cost max is the maximum operating cost corresponding to the control amount. The upper limit of the control amount is input into the activated sludge model, and the resulting operating cost is the maximum operating cost. a is the actual operating cost corresponding to the control quantity.

[0100] It should be noted that after the influent data and the corresponding control quantity are input into the activated sludge model, the activated sludge model can not only predict the corresponding effluent data and along-the-process data, but also predict the corresponding power consumption. In this way, the electricity consumption cost can be calculated based on the power consumption and electricity price, and the good point cost can be used as the actual operating cost corresponding to the control quantity.

[0101] During the application of the above method, it was found that the calculation formula for the addition of chemical phosphorus removal agents used in the activated sludge simulation model sometimes deviates. In order to overcome this defect, the chemical phosphorus removal part can be removed, that is, only the biological phosphorus removal process of the activated sludge model is considered in this disclosure.

[0102] In order to make the biological phosphorus removal reward value higher when the biological phosphorus removal effect corresponding to the control amount is better, and the biological phosphorus removal reward value lower when the biological phosphorus removal effect is worse, even if the biological phosphorus removal reward value is positively correlated with the biological phosphorus removal effect, the calculation formula of the biological phosphorus removal reward value can be shown as formula (9):

[0103]

[0104] In the formula, r TP is the biological phosphorus removal bonus value, ΔTP is the biological phosphorus removal effect, TP in is the total phosphorus concentration in the influent. The calculation formula for the biological phosphorus removal effect is shown in formula (10):

[0105]

[0106] Where ΔTP is the biological phosphorus removal effect, TP in is the influent total phosphorus concentration, TP out is the effluent total phosphorus concentration.

[0107] When the reward value corresponding to the second historical water output data is calculated using the above reward value calculation formula according to the second historical water output data, the following steps are included: first, according to each water output index value in the second historical water output data and the standard water output index value and the optimal water output index value corresponding to each water output index value, the reward value r of each water output index value is calculated using formula (6): out-tmp , and according to the reward value r of each water output index value out-tmp , use formula (7) to calculate the water quality reward value r out Second, according to the total phosphorus concentration TP in the second historical influent data in And the total phosphorus concentration TP in the second historical effluent data out , the biological phosphorus removal effect ΔTP is calculated using formula (10), and the influent total phosphorus concentration TP in the second historical influent data is calculated based on the biological phosphorus removal effect ΔTP. in , use formula (9) to calculate the biological phosphorus removal reward value r TP Third, the power consumption corresponding to the control amount of the second historical water inflow data is obtained from the activated sludge model, and the actual operating cost corresponding to the control amount of the second historical water inflow data is calculated based on the power consumption and the unit price of electricity. a , and calculate the operating cost reward value r according to formula (8) cost Fourth, the calculated water quality reward value rout , Biological phosphorus removal reward value r TP And the operating cost reward value r cost Substitute into formula (5) to calculate the reward value r corresponding to the control amount of the second historical water inflow data.

[0108] S1315: Determine a first quadruple based on the first historical operating state data, the control amount corresponding to the second historical water inflow data, the reward value corresponding to the second historical water outflow data, and the second historical operating state data. Specifically, the intermediate value x corresponding to the first historical operating state data s and the control amount a corresponding to the second historical water inflow data is converted into a , the reward value r corresponding to the second historical water output data and the vector (s, x a , r, s') as the first quadruple.

[0109] The construction of each quadruple in the quadruple training set is constructed by referring to the construction process of the first quadruple, which will not be repeated here. The constructed quadruple will be stored in the replay buffer set B. When the number of quadruple in B meets the preset number threshold, the construction of the quadruple training set is completed. Among them, the preset number threshold can be 10000.

[0110] S1320, based on the four-tuple training set, the Actor-Critic network framework is trained, and the Actor network framework is extracted from the trained Actor-Critic network framework as a control quantity optimization model.

[0111] In a possible implementation, when the Actor-Critic network framework is trained based on the four-tuple training set, steps S1321-S1325 are included.

[0112] S1321, initialize the parameters of the Actor-Critic network framework (see step S1312), initialize the replay buffer set to B, and set |B|=10 6 , initial training times T = 10 10 .

[0113] S1322, let t = 1, when t ≤ T, randomly select N = 512 quadruple (s, x a , r, s'), and use the target network to complete the calculation of (11) and (12):

[0114]

[0115]

[0116] Among them, ε~clip(N(0, 0.1), -1, 1), γ is a discount factor, and the value range of γ is 0-1, for example, γ=0.98.

[0117] S1323, update the two parameters θ1 and θ2 of the critic network according to formula (13).

[0118]

[0119] S1324, when t%1000=0, update the actor network parameter Φ and the target network parameters θ'1, θ'2, Φ' according to formulas (14)-(16).

[0120]

[0121] θ′ i ←τθ i +(1-τ)θ′ i (15)

[0122] φ′←τφ′+(1-τ)φ (16)

[0123] Wherein, τ=0.05.

[0124] S1325, t←t+1, start looping from step S1322 until the number of training times T is reached, completing the training of the Actor-Critic network frame.

[0125] After completing the training of the Actor-Critic network framework, the Actor network framework is extracted from the trained Actor-Critic network framework as a control quantity optimization model.

[0126] When optimizing the control quantity based on the water inlet data, along-the-way data and water outlet data under the current working condition, using the pre-constructed control quantity optimization model, the process includes: using the vector composed of the water inlet data, along-the-way data and water outlet data under the current working condition as the operating status data under the current working condition, inputting the operating status data under the current working condition into the constructed control quantity optimization model, obtaining the intermediate value of the current control quantity optimization value, and calculating the current control quantity optimization value according to formula (2).

[0127] In order to verify the effectiveness of the dynamic optimization method of the control quantity in the present disclosure, a verification experiment was carried out.

[0128] Under steady-state conditions, the data of the original working condition and the optimized working condition are shown in Table 3, the energy consumption data under each working condition are shown in Table 4, and the effluent data under each working condition are shown in Table 5. Among them, under the #1 optimized working condition, the effluent water quality reward value is the largest when the effluent water quality meets the standard effluent water index value, and under the #2 optimized working condition, the effluent water quality reward value is the largest when the effluent water quality meets the internal control value.

[0129] Table 3

[0130]

[0131] Table 4

[0132]

[0133] Note: The unit in the table is kwh

[0134] Table 5

[0135]

[0136] Note: The unit in the table is mg / L

[0137] Under dynamic conditions, the dynamic model of the reclaimed water plant is retrieved in the same way, and the dynamic inlet data is continuously input into the control quantity optimization model. The change curve of the effluent water quality and energy consumption corresponding to the dynamic inlet are compared. Figure 3-Figure 7 shown.

[0138] It can be seen from the above experimental data that no matter which optimized operating condition is used, lower energy consumption can be obtained than the original operating condition while the effluent meets the standards, which fully demonstrates the effectiveness of the dynamic optimization of the whole process control quantity of the sewage treatment plant provided by the present invention.

[0139] In the present disclosure, the inlet data and control quantity of the sewage treatment plant under the current working condition are first obtained, and then the data along the process and the effluent data under the current working condition are predicted based on the inlet data and control quantity under the current working condition, and finally the control quantity under the current working condition is optimized using the pre-constructed control quantity optimization model based on the inlet data, along the process data and effluent data. Since the control quantity optimization model is obtained based on the reinforcement learning algorithm and the reward value involved in the reinforcement learning algorithm is the largest when the effluent data meets the standard and the comprehensive operating cost is the lowest, after optimizing the control quantity of the sewage treatment plant using the control quantity optimization model in the present disclosure, the sewage treatment plant can achieve the lowest comprehensive operating cost while achieving the effluent standard.

[0140] <Device Example>

[0141] Figure 8 A schematic block diagram of a device for dynamically optimizing the control quantity of the entire process of a sewage treatment plant according to an embodiment of the present disclosure is shown. Figure 8As shown, the dynamic optimization device 100 for the whole process control quantity of the sewage treatment plant includes:

[0142] The influent data acquisition module 110 is used to acquire the influent data and control quantity under the current working conditions of the sewage treatment plant.

[0143] The outflow data prediction module 120 is used to predict the data along the way and outflow data under the current working conditions according to the inflow data and the control quantity.

[0144] The control quantity optimization module 130 is used to optimize the control quantity according to the water inlet data, the data along the way and the water outlet data by using a pre-built control quantity optimization model.

[0145] Among them, the control quantity optimization model is obtained based on the reinforcement learning algorithm, and the reward value involved in the reinforcement learning algorithm is the largest when the water output data meets the standards and the comprehensive operating cost is the lowest.

[0146] In a possible implementation, the effluent data prediction module 120 is implemented based on an activated sludge model when predicting the along-the-line data and effluent data under the current working conditions according to the influent data and the control quantity.

[0147] In a possible implementation, the dynamic optimization device 100 for the whole process control quantity of the sewage treatment plant further includes: an activated sludge model debugging module (not shown in the figure). The activated sludge model debugging module is used to debug and obtain the activated sludge model corresponding to the sewage treatment plant based on the historical influent data, historical process data and historical effluent data of the sewage treatment plant.

[0148] The activated sludge model debugging module includes:

[0149] A debugging data generation unit is used to obtain corresponding predicted historical along-the-line data and predicted historical outflow data according to historical inflow data by using an activated sludge model;

[0150] The debugging unit is used to adjust the kinetic coefficient and stoichiometric coefficient of the activated sludge model so that the difference between the predicted historical along-the-process data and the historical along-the-process data and the difference between the predicted historical effluent data and the historical effluent data both meet the preset difference thresholds as the activated sludge model.

[0151] In a possible implementation, the control amount optimization model adopted by the control amount optimization module 130 is obtained based on a reinforcement learning algorithm, and the network framework of the reinforcement learning algorithm is Actor-Critic.

[0152] In a possible implementation, the dynamic optimization device 100 for the whole-process control quantity of a sewage treatment plant further includes: a control quantity optimization model building module.

[0153] The control quantity optimization model building module includes:

[0154] A four-tuple training set construction unit, which is used to construct a four-tuple training set for training the Actor-Critic network framework based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value;

[0155] The training unit is used to train the Actor-Critic network framework based on the four-tuple training set, and extract the Actor network framework from the trained Actor-Critic network framework as a control quantity optimization model.

[0156] In a possible implementation, the four-tuple training set construction unit includes:

[0157] A first operation state construction subunit is used to obtain first historical effluent data and first historical along-stream data according to the first historical influent data and the corresponding control amount by using an activated sludge model, and determine first historical operation state data according to the first historical influent data, the first historical effluent data and the first historical along-stream data;

[0158] A control quantity construction subunit is used to obtain a control quantity corresponding to the second historical water inflow data by using an initialized Actor network framework according to the first historical operation status data;

[0159] A second operation state constructing subunit is used to obtain second historical effluent data and second historical along-stream data according to the second historical influent data and the control amount corresponding to the second historical influent data by adopting an activated sludge model, and determine second historical operation state data according to the second historical influent data, the second historical effluent data and the second historical along-stream data;

[0160] A reward value constructing subunit, used to obtain a reward value corresponding to the second historical water output data by using a reward value calculation formula according to the second historical water output data;

[0161] The quadruple construction subunit is used to determine the first quadruple based on the first historical operating status data, the control amount corresponding to the second historical water inlet data, the reward value corresponding to the second historical water outlet data, and the second historical operating status data.

[0162] In a possible implementation, the reward value calculation formula adopted by the reward value construction subunit is determined based on the effluent water quality reward value, the biological phosphorus removal reward value and the operating cost reward value.

[0163] In a possible implementation, the reward value calculation formula used by the reward value construction subunit is as follows:

[0164] r=w1min(r Tp,r out )+w2r cost

[0165] In the formula, r is the reward value, r out is the effluent water quality reward value, r TP is the reward value for biological phosphorus removal, r cost is the operating cost bonus value, w1 and w2 are proportional coefficients.

[0166] <Equipment Embodiment>

[0167] Fig. 9 A schematic block diagram of a device for dynamically optimizing the control quantity of the entire process of a sewage treatment plant according to an embodiment of the present disclosure is shown. Fig. 9 As shown, the dynamic optimization device 200 for the whole process control quantity of the sewage treatment plant includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the above-mentioned dynamic optimization methods for the whole process control quantity of the sewage treatment plant when executing the executable instructions.

[0168] Here, it should be noted that the number of processors 210 may be one or more. At the same time, in the dynamic optimization device 200 of the whole process control amount of the sewage treatment plant in the embodiment of the present disclosure, an input device 230 and an output device 240 may also be included. Among them, the processor 210, the memory 220, the input device 230 and the output device 240 may be connected through a bus or in other ways, which are not specifically limited here.

[0169] The memory 220 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the dynamic optimization method for the whole process control quantity of the sewage treatment plant in the embodiment of the present disclosure. The processor 210 executes various functional applications and data processing of the dynamic optimization device 200 for the whole process control quantity of the sewage treatment plant by running the software program or module stored in the memory 220.

[0170] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include display devices such as display screens.

[0171] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A dynamic optimization method for the whole process control quantity of a sewage treatment plant, characterized in that: include: Obtain the influent data and control quantity under the current working conditions of the sewage treatment plant; According to the water inlet data and the control amount, predicting the along-line data and the water outlet data under the current working condition; According to the water inlet data, the data along the way and the water outlet data, the control amount is optimized by using a pre-constructed control amount optimization model; The control quantity optimization model is obtained based on a reinforcement learning algorithm and the reward value involved in the reinforcement learning algorithm is the largest when the water output data meets the standard and the comprehensive operating cost is the lowest; When the control quantity is optimized according to the water inlet data, the data along the process and the water outlet data, a pre-built control quantity optimization model is used to optimize the control quantity, including: The water inlet data, the along-the-way data and the water outlet data are combined into a vector as the operating status data under the current working condition; Inputting the operating state data under the current working condition into the control quantity optimization model to obtain the intermediate value corresponding to the control quantity; Calculating an optimized value of the control amount according to the corresponding intermediate value of the control amount and the upper limit value and the lower limit value configured for the control amount; The network framework of the reinforcement learning algorithm is Actor-Critic; When constructing the control quantity optimization model, it includes: Based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value, constructing a four-tuple training set for training the Actor-Critic network framework; Based on the four-tuple training set, the Actor-Critic network framework is trained, and an Actor network framework is extracted from the trained Actor-Critic network framework as the control quantity optimization model; When constructing a first quadruple for training the Actor-Critic network framework based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value, it includes: According to the first historical water inflow data and the corresponding control amount, the activated sludge model is used to obtain the first historical water outflow data and the first historical along-line data, and the first historical operating state data is determined according to the first historical water inflow data, the first historical water outflow data and the first historical along-line data; According to the first historical operating status data, using the initialized Actor network framework, obtaining the control amount corresponding to the second historical water inflow data; According to the second historical water inflow data and the control amount corresponding to the second historical water inflow data, the activated sludge model is used to obtain the second historical water outflow data and the second historical along-line data, and according to the second historical water inflow data, the second historical water outflow data and the second historical along-line data, the second historical operating state data is determined; According to the second historical water output data, the reward value corresponding to the second historical water output data is obtained by using the calculation formula of the reward value; Determining the first quaternary group based on the first historical operating state data, the control amount corresponding to the second historical water inflow data, the reward value corresponding to the second historical water outflow data, and the second historical operating state data; When the reward value corresponding to the second historical water output data is obtained by using the calculation formula of the reward value according to the second historical water output data, the method includes: First, according to each water output index value in the second historical water output data and the standard water output index value and the optimal water output index value corresponding to each water output index value, the reward value r of each water output index value is calculated using formula (6): out-tmp , and according to the reward value r of each water output index value out-tmp , use formula (7) to calculate the water quality reward value r out ; In the formula, r out-tmp is the reward value of the water output index value, x is the water output index value, x std is the standard water index value, x best is the best water output index value; In the formula, r out is the effluent water quality reward value, r out-tmp It is the reward value of the water output index value; Second, according to the total phosphorus concentration TP in the second historical influent data in And the total phosphorus concentration TP in the second historical effluent data out , the biological phosphorus removal effect ΔTP is calculated using formula (10), and the influent total phosphorus concentration TP in the second historical influent data is calculated based on the biological phosphorus removal effect ΔTP. in , use formula (9) to calculate the biological phosphorus removal reward value r TP ; ΔTp=Tp in -Tp ou (10) Where ΔTP is the biological phosphorus removal effect, TP in is the influent total phosphorus concentration, TP out is the effluent total phosphorus concentration; In the formula, r TP is the biological phosphorus removal bonus value, ΔTP is the biological phosphorus removal effect, TP in is the total phosphorus concentration in the influent; Third, the power consumption corresponding to the control amount of the second historical water inflow data is obtained from the activated sludge model, and the actual operating cost corresponding to the control amount of the second historical water inflow data is calculated based on the power consumption and the unit price of electricity. a , and calculate the operating cost reward value r according to formula (8) cost ; In the formula, r cost is the operating cost reward value, cost max is the maximum operating cost corresponding to the control amount. The upper limit of the control amount is input into the activated sludge model, and the resulting operating cost is the maximum operating cost. a is the actual operating cost corresponding to the control quantity; Fourth, the calculated water quality reward value r out , Biological phosphorus removal reward value r TP And the operating cost reward value r cost Substitute into formula (5) to calculate the reward value r corresponding to the control amount of the second historical water inflow data; r=w i min(r Tp ,r out )+w2r cost (5) In the formula, r is the reward value, r out is the effluent water quality reward value, r TP is the reward value for biological phosphorus removal, r cost is the running cost reward value, w1 and w2 are proportional coefficients, where w1, w2∈(0, 1) and w1+w2=1, for example, w1=w2=0.

5.

2. The method according to claim 1, characterized in that When predicting the along-the-way data and the effluent data under the current working condition according to the influent data and the control amount, it is achieved based on the activated sludge model.

3. The method according to claim 2, characterized in that The activated sludge model is debugged based on the historical influent data, historical process data and historical effluent data of the sewage treatment plant; When the activated sludge model is debugged based on the historical influent data, historical process data and historical effluent data of the sewage treatment plant, it includes: According to the historical water inflow data, an activated sludge model is used to obtain corresponding predicted historical along-the-line data and predicted historical water outflow data; The kinetic coefficient and stoichiometric coefficient of the activated sludge model are adjusted so that the difference between the predicted historical along-the-way data and the historical along-the-way data and the difference between the predicted historical effluent data and the historical effluent data both meet the preset difference threshold value, and the activated sludge model is used as the activated sludge model.

4. A dynamic optimization device for the whole process control quantity of a sewage treatment plant, characterized in that: include: The inlet data acquisition module is used to obtain the inlet data and control quantity under the current working conditions of the sewage treatment plant; A water discharge data prediction module, used to predict the along-the-line data and water discharge data under the current working condition according to the water inlet data and the control amount; A control quantity optimization module, used to optimize the control quantity using a pre-built control quantity optimization model according to the water inlet data, the data along the process and the water outlet data; The control quantity optimization model is obtained based on a reinforcement learning algorithm and the reward value involved in the reinforcement learning algorithm is the largest when the water output data meets the standard and the comprehensive operating cost is the lowest; When the control quantity is optimized according to the water inlet data, the data along the process and the water outlet data, a pre-built control quantity optimization model is used to optimize the control quantity, including: The water inlet data, the along-the-way data and the water outlet data are combined into a vector as the operating status data under the current working condition; Inputting the operating state data under the current working condition into the control quantity optimization model to obtain the intermediate value corresponding to the control quantity; Calculating an optimized value of the control amount according to the corresponding intermediate value of the control amount and the upper limit value and the lower limit value configured for the control amount; The network framework of the reinforcement learning algorithm is Actor-Critic; When constructing the control quantity optimization model, it includes: Based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value, constructing a four-tuple training set for training the Actor-Critic network framework; Based on the four-tuple training set, the Actor-Critic network framework is trained, and an Actor network framework is extracted from the trained Actor-Critic network framework as the control quantity optimization model; When constructing a first quadruple for training the Actor-Critic network framework based on the historical influent data of the sewage treatment plant, the activated sludge model and the calculation formula of the reward value, it includes: According to the first historical water inflow data and the corresponding control amount, the activated sludge model is used to obtain the first historical water outflow data and the first historical along-line data, and the first historical operating state data is determined according to the first historical water inflow data, the first historical water outflow data and the first historical along-line data; According to the first historical operating status data, using the initialized Actor network framework, obtaining the control amount corresponding to the second historical water inflow data; According to the second historical water inflow data and the control amount corresponding to the second historical water inflow data, the activated sludge model is used to obtain the second historical water outflow data and the second historical along-line data, and according to the second historical water inflow data, the second historical water outflow data and the second historical along-line data, the second historical operating state data is determined; According to the second historical water output data, the reward value corresponding to the second historical water output data is obtained by using the calculation formula of the reward value; Determining the first quaternary group based on the first historical operating state data, the control amount corresponding to the second historical water inflow data, the reward value corresponding to the second historical water outflow data, and the second historical operating state data; When the reward value corresponding to the second historical water output data is obtained by using the calculation formula of the reward value according to the second historical water output data, the method includes: First, according to each water output index value in the second historical water output data and the standard water output index value and the optimal water output index value corresponding to each water output index value, the reward value r of each water output index value is calculated using formula (6): out-tmp , and according to the reward value r of each water output index value out-tmp , use formula (7) to calculate the water quality reward value r out ; In the formula, r out-tmp is the reward value of the water output index value, x is the water output index value, x std is the standard water index value, x best is the best water output index value; In the formula, r out is the effluent water quality reward value, r out-tmp It is the reward value of the water output index value; Second, according to the total phosphorus concentration TP in the second historical influent data in And the total phosphorus concentration TP in the second historical effluent data out , the biological phosphorus removal effect ΔTP is calculated using formula (10), and the influent total phosphorus concentration TP in the second historical influent data is calculated based on the biological phosphorus removal effect ΔTP. in , use formula (9) to calculate the biological phosphorus removal reward value r TP ; ΔTp=Tp in -Tp ou (10) Where ΔTP is the biological phosphorus removal effect, TP in is the influent total phosphorus concentration, TP out is the effluent total phosphorus concentration; In the formula, r TP is the biological phosphorus removal bonus value, ΔTP is the biological phosphorus removal effect, TP in is the total phosphorus concentration in the influent; Third, the power consumption corresponding to the control amount of the second historical water inflow data is obtained from the activated sludge model, and the actual operating cost corresponding to the control amount of the second historical water inflow data is calculated based on the power consumption and the unit price of electricity. a , and calculate the operating cost reward value r according to formula (8) cost ; In the formula, r cost is the operating cost reward value, cost max is the maximum operating cost corresponding to the control amount. The upper limit of the control amount is input into the activated sludge model, and the resulting operating cost is the maximum operating cost. a is the actual operating cost corresponding to the control quantity; Fourth, the calculated water quality reward value r out , Biological phosphorus removal reward value r TP And the operating cost reward value r cost Substitute into formula (5) to calculate the reward value r corresponding to the control amount of the second historical water inflow data; r=w1min(r Tp ,r out ) + w2r cost (5) In the formula, r is the reward value, r out is the effluent water quality reward value, r TP is the reward value for biological phosphorus removal, r cost is the running cost reward value, w1 and w2 are proportional coefficients, where w1, w2∈(0, 1) and w1+w2=1, for example, w1=w2=0.

5.

5. A dynamic optimization device for the whole process control quantity of a sewage treatment plant, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 3 when executing the executable instructions.

Images