Pruned double q-learning echo state network-based prediction method for effluent ammonia concentration

Abstract

The application discloses a kind of based on pruned double Q learning echo state network effluent ammonia nitrogen concentration prediction method, the real-time monitoring data of sewage treatment is collected, the real-time monitoring data is preprocessed, and monitoring data set is obtained;Preliminary screening is obtained by Pearson correlation coefficient, and auxiliary variable is selected, and the final auxiliary variable set is determined based on maximum correlation-minimum redundancy algorithm further screening auxiliary variable;Echo state network including input layer, reserve pool layer and output layer is constructed, network output weight is trained, and optimal reserve pool structure is obtained;The auxiliary variable in test sample is input into the echo state network, and the effluent ammonia nitrogen concentration prediction result is output.The application reduces the overestimation bias of traditional network, avoids premature convergence or falls into suboptimal pruning configuration;Selection matrix keeps reserve pool weight norm monotone not increasing, overcome the problem that single network is easily trapped in local optimum, further improve the convergence of algorithm and the robustness of model.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of concentration prediction, in particular to an effluent ammonia nitrogen concentration prediction method based on a pruned double Q learning echo state network. BACKGROUND

[0002] With the rapid development of economic globalization and urban globalization, the earth's water resources are facing serious pollution problems, which not only cause excessive harm to the water ecosystem, but also bring unprecedented challenges to the water safety of human society. As the world's largest developing country, China has increasingly attached importance to environmental protection and water pollution prevention and control, and actively promoted the construction of sewage treatment infrastructure. Sewage treatment can effectively remove harmful substances remaining in sewage through physical, chemical and biological methods. At present, the most widely used biological treatment technology in urban sewage treatment plants in various countries is the activated sludge method, which degrades pollutants in sewage through microorganisms in sludge to achieve sewage purification (such as Figure 1 ). However, the existing sewage treatment technology and process are relatively limited and the quality control level is not high, and there is a lack of real-time monitoring of water quality and process state during operation, which not only increases the operation cost and energy consumption of sewage treatment facilities, but also affects the overall effect of sewage treatment. Therefore, studying a sewage water quality soft measurement model with multi-disciplinary cross-fusion can further improve the prediction performance of sewage water quality, realize accurate control and rapid response of process parameters and effluent quality, so the application has a wide application prospect.

[0003] In the urban sewage treatment process, ammonia nitrogen is not only an important monitoring index for measuring the nitrogen pollution level in water bodies, but also an important index for measuring water eutrophication. Ammonia nitrogen mainly exists in the form of ammonia ions and free ammonia in sewage. High concentration of ammonia nitrogen in sewage not only causes harm to aquatic organisms and reduces the self-purification ability of water bodies, but also easily causes water hypoxia and destroys the ecological balance. Therefore, real-time detection of effluent ammonia nitrogen concentration in the urban sewage treatment process is of great importance to perfecting the urban sewage treatment monitoring mechanism and urban environmental protection, and lays a solid foundation for the further promotion of the intelligent water system.

[0004] The traditional ammonia nitrogen concentration measurement of sewage treatment process mainly reacts ammonia nitrogen in a certain form existing in sewage with Nessler's reagent to generate a red colloidal complex, carries out colorimetry at a certain wavelength, carries out calculation through a corresponding specific relationship, and then obtains the concentration. However, the prediction process of the traditional method is tedious and has low precision, and it is difficult to meet the needs of fine control of urban sewage treatment. At the same time, due to the complexity of chemical reaction and the uncertainty of the environment, the sewage treatment process has the characteristics of diversity, randomness and high nonlinearity, and the prediction results obtained by the mechanism model cannot meet the industrial requirements. In recent years, as a new type of intelligent data-driven model, artificial neural network can achieve higher accuracy with less prior assumption, and can solve the complex nonlinear and uncertain problems in sewage. Therefore, the present application designs an effluent ammonia nitrogen concentration prediction method based on artificial neural network, which not only reduces the model complexity but also improves the model prediction accuracy, effectively solves the prediction problem of high-dimensional nonlinear dynamic system, and provides a feasible solution for real-time prediction of effluent ammonia nitrogen of urban sewage treatment. SUMMARY

[0005] In view of the above existing problems, the present application provides an effluent ammonia nitrogen concentration prediction method based on a pruned double Q learning echo state network, which solves the problems of poor network performance and low prediction accuracy of ammonia nitrogen concentration data processing model in the prior art.

[0006] To solve the above technical problems, the present application provides the following technical scheme:

[0007] In a first aspect, the present application provides an effluent ammonia nitrogen concentration prediction method based on a pruned double Q learning echo state network, comprising the following steps,

[0008] Collecting real-time monitoring data of sewage treatment, preprocessing the real-time monitoring data to obtain a monitoring data set;

[0009] The monitoring data set is divided into training samples and test samples; the selected auxiliary variables are obtained by preliminary screening based on the Pearson correlation coefficient, and the final auxiliary variable set is determined by further screening the auxiliary variables based on the maximum correlation-minimum redundancy algorithm;

[0010] An echo state network containing an input layer, a reservoir pool layer and an output layer is constructed, the reservoir pool neurons are pre-pruned by principal component analysis, the neuron contribution degree is calculated and the low contribution neurons are removed; based on the pre-pruned neurons, secondary pruning is carried out by a pruned double deep Q network algorithm, a main network and a target network are constructed, an action is selected by an epsilon greedy strategy, the network is updated in combination with a reward function and a TD target, an optimal reservoir pool structure is obtained, the network output weight is trained, and an optimal echo state network is obtained;

[0011] The auxiliary variables in the test sample are input into the echo state network, and a predicted result of the water ammonia nitrogen concentration is output.

[0012] Further, the preprocessing step includes noise reduction, outlier removal, missing value filling, and normalization. The normalization processing adopts a Min-Max method to unify the variable scale to the range of [0, 1], and the calculation formula is:

[0013]

[0014] where z is the original value of the variable at a certain moment, z min represents the minimum value of the variable in the entire sample, z max represents the maximum value of the variable in the entire sample.

[0015] Further, the Pearson correlation coefficient is used to preliminarily analyze the correlation between the auxiliary variables X i and the predicted variable Y, and the formula is as follows:

[0016]

[0017] where represents the correlation coefficient between the auxiliary variable X i and the predicted variable Y; represents the lth observation value of the ith auxiliary variable; y l represents the lth observation value of the predicted variable; represents the mean value of the ith auxiliary variable; represents the mean value of the predicted variable. By calculating the correlation coefficient , it is determined which auxiliary variables X i have a greater impact on the predicted variable Y, and the auxiliary variables are preliminarily screened out and stored in the auxiliary variable set to be selected; when the Pearson correlation coefficient is used to screen the auxiliary variables, the correlation of the predicted variable is used as the screening principle, and the auxiliary variables with an absolute value of the correlation coefficient greater than or equal to 0.4 are retained;

[0018] In the maximum correlation-minimum redundancy algorithm, the mutual information value between the auxiliary variables in the auxiliary variable set to be selected and the predicted variable is calculated to determine the linear relationship:

[0019]

[0020] where p(x i ) and p(y) are the marginal probability densities of the auxiliary variable X i and the predicted variable Y, respectively. p(x i , y) is the auxiliary variable X iThe joint probability density of the prediction variable Y and the auxiliary variable X is calculated, the correlation between the auxiliary variable and the prediction variable and the redundancy between the auxiliary variables are calculated by the mutual information value, and finally 5-12 auxiliary variables are selected.

[0021] Further, the reserve pool internal state and output state update equation of the echo state network is:

[0022] x(t+1)=f(W in u(t+1)+W res x(t)) (5)

[0023] y(t+1)=W out x(t+1) (6)

[0024] Wherein, x(t+1) is the reserve pool internal state at t+1, y(t+1) is the output state, W in is the input weight matrix, W res is the reserve pool internal connection weight matrix, W out is the output weight matrix, and f is the activation function.

[0025] Further, the principal component analysis pre-pruning includes constructing a covariance matrix using the reserve pool internal state matrix, obtaining eigenvalues and eigenvectors by eigenvalue decomposition, calculating the normalized load value of neurons in the first several principal components as the contribution degree, and removing neurons with low contribution degree in proportion.

[0026] Further, in the pruning double deep Q network algorithm, the reward function calculation formula is:

[0027]

[0028] Wherein, ζ old is the prediction error after pre-pruning, ζ new is the prediction error after re-pruning, α(s t ,a t ) is an adaptive coefficient, a t is a pruning action, and N prune is the number of remaining neurons after pre-pruning.

[0029] Further, the TD target calculation formula of the pruning double deep Q network algorithm is:

[0030]

[0031] Wherein, r t is the reward value; γ∈[0,1] is the discount factor; a ＊ is the optimal action; the min function selects the smaller Q value after the action a ＊ is taken in the target network,

[0032] Further, in the division of the training sample and the test sample, the training sample accounts for 65%-70%, and the test sample accounts for 30%-35%; the initial state influence is eliminated by idling operation during training, and the pseudo-inverse method is used to optimize the output weight.

[0033] In the second aspect, the present application provides a computer device comprising a memory and a processor, and the memory stores a computer program, wherein: the computer program is executed by the processor to realize any step of the effluent ammonia nitrogen concentration prediction method based on the pruned double Q learning echo state network of the first aspect of the present application.

[0034] In the third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein: the computer program is executed by the processor to realize any step of the effluent ammonia nitrogen concentration prediction method based on the pruned double Q learning echo state network of the first aspect of the present application.

[0035] The present application has the following advantages:

[0036] (1) The present application proposes an intelligent prediction method for effluent ammonia nitrogen concentration in urban sewage treatment process based on principal component analysis combined with pruned double deep Q network algorithm of echo state network, aiming at the problem that the diversity, randomness and high nonlinearity of sewage treatment process lead to low measurement accuracy of effluent ammonia nitrogen concentration. Pearson correlation coefficient analysis and maximum correlation-minimum redundancy algorithm are used to obtain research variables, PCA is used to pre-prune the neurons with low contribution in the reserve pool, and ClippedDoubleDQN algorithm is used to re-prune the remaining neurons, finally an echo state network with simplified structure and excellent prediction effect is obtained, which can realize real-time and accurate prediction of effluent ammonia nitrogen in urban sewage treatment process.

[0037] (2) The present application uses principal component analysis method to obtain the contribution of each neuron in the reserve pool, which provides a theoretical basis for the experience and trial of the subsequent ClippedDoubleDQN algorithm. The two main networks of ClippedDoubleDQN algorithm are used to update the action state alternately, which reduces the overestimation bias of traditional network; the two target networks are used to constrain the pruning action to dynamically balance between exploration and utilization, avoiding premature convergence or falling into suboptimal pruning configuration; the selection matrix keeps the weight norm of the reserve pool monotone and does not increase, which ensures that the pruning process always meets the echo state characteristics. This design not only overcomes the problem that single network is easy to fall into local optimum, but also further improves the convergence of the algorithm and the robustness of the model through slow updating of the target network and noise clipping. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The present application is a biological activated sludge process for sewage treatment;

[0039] Figure 2 The original echo state network structure diagram of the application;

[0040] Figure 3 The principal component analysis combined with the pruning double Q learning echo state network process schematic diagram of the application;

[0041] Figure 4 The prediction result graph of effluent ammonia nitrogen in the municipal sewage treatment process of the application;

[0042] Figure 5 The prediction error optimization graph of effluent ammonia nitrogen in the municipal sewage treatment process of the application. DETAILED DESCRIPTION

[0043] In order to make the above objectives, characteristics and advantages of the application more apparent, the specific embodiments of the application are described in detail below with reference to the accompanying drawings.

[0044] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the application, therefore the application is not limited by the specific embodiments disclosed below.

[0045] Secondly, the "one embodiment" or "embodiment" referred to herein means that a specific feature, structure or characteristic can be included in at least one implementation of the application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0046] The application establishes an effluent ammonia nitrogen real-time monitoring model based on the echo state network of the principal component analysis combined with the pruning double deep Q network algorithm, can realize accurate prediction of the effluent ammonia nitrogen concentration, uses the principal component analysis method to evaluate the importance of the initialized randomly generated reservoir neurons, and in view of the problem that the traditional importance evaluation method has poor extraction ability in the face of high-dimensional nonlinear data, introduces the pruning double deep Q network algorithm. The fusion of the model and the data is continuously learned by using the Markov decision mechanism, the reservoir structure of the echo state network is further improved through prior knowledge and experience trial and error, the nonlinear relationship in the time series is effectively grasped, and the model prediction ability is improved.

[0047] The embodiment of the application provides an effluent ammonia nitrogen concentration prediction method based on a pruning double Q learning echo state network, which comprises the following steps:

[0048] The experimental data selects the test data of sewage treatment of a sewage treatment plant in Beijing from August 26, 2014 to September 16, 2014, and there are a total of 12 variables left after data preprocessing and variable determination process.

[0049]

[0050]

[0051] The present application comprises the following specific embodiments and steps:

[0052] 1. Data acquisition and pretreatment module

[0053] 1.1 Data acquisition

[0054] According to the actual operation condition of the sewage treatment plant, a portable multi-parameter water quality analyzer is used to collect, detect and record real-time monitoring data.

[0055] 1.2 Data pretreatment

[0056] After the collected data are sequentially subjected to noise reduction processing, removal of abnormal values, filling of missing values and normalization processing, 21 auxiliary variables and a predicted variable effluent ammonia nitrogen are obtained, and a total of 456 groups of data samples of 21 variables are obtained. The 456 groups of data samples need to be divided into two parts, the first 300 groups of data are used as training samples, and the first 50 groups of data are used to eliminate the initial transient influence, and the remaining 156 groups of data are used as test samples.

[0057] 1.3 Determine the predicted variable and auxiliary variable

[0058] Initialize the candidate auxiliary variable set A = {X1, X2,..., X i ,...,X K} and the predicted variable effluent ammonia nitrogen Y, and the observation vector corresponding to the auxiliary variable is where i ∈ [1, K] is the auxiliary variable arrangement number and the observation vector corresponding to the predicted variable is

[0059] 1.3.1) Min-Max normalization processing is performed on the variables X i in the candidate auxiliary variable set A and the predicted variable Y, so that the scales of all variables are unified to the range of [0, 1], so that the model training will not be biased towards the features with larger numerical range, and the convergence speed is accelerated, and the training stability is improved;

[0060]

[0061] where z is the original value of the variable at a certain time, z min represents the minimum value of the variable in the entire sample, and z max represents the maximum value of the variable in the entire sample.

[0062] 1.3.2) The auxiliary variables X iand the predictor Y using the Pearson correlation coefficient The correlation between variables is analyzed initially.

[0063]

[0064] where denotes the correlation coefficient between the auxiliary variable X i and the predictor Y; denotes the 1th observation of the ith auxiliary variable; y l denotes the 1th observation of the predictor; denotes the mean of the ith auxiliary variable; denotes the mean of the predictor. By calculating the correlation coefficient it is determined which auxiliary variables X i have a large impact on the predictor Y, and the auxiliary variables are initially selected and stored in the set of auxiliary variables to be selected B;

[0065] 1.3.3) If the correlation between two auxiliary variables is very high (i.e. , it can lead to a multicollinearity problem, so the maximum correlation-minimum redundancy algorithm is used to extract the main features from high-dimensional data and reduce the dimension of the data. The mutual information value between the auxiliary variables in the set of auxiliary variables to be selected and the predictor is calculated to determine the linear relationship.

[0066]

[0067] where p(x i ) and p(y) are the marginal probability densities of the auxiliary variable X i and the predictor Y, respectively. p(x i ,y) is the joint probability density of the auxiliary variable X i and the predictor Y. The set of determined auxiliary variables containing 8 variables is initialized Based on the calculation of the mutual information value, the auxiliary variables with strong representation ability are selected iteratively.

[0068]

[0069] where I(X j ,Y) is the mutual information value between the auxiliary variable X i and the predictor Y, the larger the value, the better the auxiliary variable X i explains the target Y; I(X j ,X i ) is the mutual information value between two auxiliary variables X j and X i ; is the measure of the auxiliary variable X jThe greater the value of the average mutual information of each feature in the current selected feature set C, the more information is repeated with the selected features. When φ(X j ) is greater, it indicates that the auxiliary variable X j is highly correlated with the predicted variable Y and less redundant with the selected features. The algorithm selects the X j that maximizes φ(X j ) to join the set C at each iteration update, and finally obtains an auxiliary variable set C containing 8 variables by maximizing the correlation and minimizing the redundancy.

[0070] According to the auxiliary variable set C, eight auxiliary variables including temperature T, anaerobic end ORP, effluent ORP, aerobic front DO, aerobic end DO, aerobic end TSS, effluent pH and effluent NO3-N are finally selected to predict the effluent ammonia nitrogen.

[0071] 2. Echo state network design module combining principal component analysis and pruned double-depth Q network

[0072] The echo state network mainly consists of an input layer of K nodes, a reservoir layer of N nodes and an output layer of L nodes (as shown in Figure 2 ). The update equations of the internal state x(t+1) and the output state y(t+1) of the reservoir at t+1 are represented by formulas (5) and (6). The input weight matrix W in of the echo state network and the internal connection weight matrix W res of the reservoir are randomly generated and fixed during model initialization, and the output weight matrix W out is the only weight matrix that needs to be trained, and the pseudo-inverse method is usually used to optimize the output weight of the network. The reservoir node activation function f of the model is selected as the tangent function tanh.

[0073] x(t+1)=f(W in u(t+1)+W res x(t))(5)

[0074] y(t+1)=W out x(t+1) (6)

[0075] Although Echo-State Networks (ESNs) have evolved from traditional recurrent neural networks (RNNs), there is still room for improvement in areas such as model parameter selection and structural optimization. The core of an ESN is a dynamic reservoir, which transforms the input signal from a low-dimensional input space to a high-dimensional state space, enabling feature extraction from complex nonlinear sequences. Linear regression is then used to map the high-dimensional state space back to a low-dimensional output space, and key information is extracted from the features through weighted summation. However, the high-dimensional state space of the reservoir may suffer from collinearity, leading to ill-conditioned matrices in the output weight calculation process and reducing the model's generalization performance. Therefore, optimization algorithms are needed to dynamically adjust the reservoir structure, obtaining a set of neurons in the reservoir to achieve similar or better performance than all neurons in the original reservoir.

[0076] Principal component analysis (PCA) transforms high-dimensional raw data into a low-dimensional data space in an unsupervised manner through linear transformation. The data are then arranged in descending order of variance to construct a few uncorrelated principal components. This approach maximizes the preservation of the main features and information of the original data, thereby simplifying the data structure and reducing computational complexity. PCA can be used to reduce the number of neurons in the reserve pool, starting from the perspective of contribution. This can be achieved by using the "cleaned" internal state matrix X = [X(m), X(m+1), ..., X(N)]. s )] T Construct the covariance matrix G.

[0077]

[0078] Through eigenvalue decomposition Gv i =λ i v i i = 1, 2, ..., n yields a set of eigenvalues ​​λ1 ≥ λ2 ≥ ... ≥ λ n and eigenvector v i Take the eigenvectors W = [v1, v2, ..., v1] corresponding to the w largest eigenvalues. w ]∈ N×w W T W = I w The contribution of the j-th neuron can be calculated using its normalized loading value from the first w principal components.

[0079]

[0080] Where v j,i It is the feature vector of the j-th neuron on the i-th component; It is the sum of the loads of all neurons. The load value after normalization. The range is within [0,1]. Sort in ascending order, remove neurons with low contribution, and reconstruct the model.

[0081] However, this method performs poorly in feature extraction of nonlinear or non-Gaussian dynamic processes. Therefore, the Clipped Double DQN algorithm is introduced to dynamically evaluate and decide whether to retain or prune the remaining neurons, thereby improving the model's generalization performance and prediction accuracy while retaining nonlinear features.

[0082] After pre-pruning the reserve pool neurons through principal component analysis, the remaining neurons are sorted in ascending order according to the normalized load The state vector s j is calculated. j The action space A is formed. The main network Q A (s,a; θ A ), Q B (s,a; θ B ), and the target network

[0083] A main network (such as Q A ) is used to select action a t according to current state s t using an ε-greedy policy. After performing the action, the corresponding submatrix is extracted from the weight matrix, the subnetwork is reconstructed and retrained, and the new prediction error ζ new is calculated. According to the PCA pre-pruned prediction error ζ old , the reward r t is calculated and the model structure is determined whether to update.

[0084]

[0085] where α(s t ,a t ) is a coefficient that can be self-adaptively adjusted according to the current state and action, usually α>0;

[0086] is the number of pruned neurons; N prune is the number of remaining neurons after pre-pruning. If the pruning is excessive, i.e. is large, the penalty term will increase, and the value of α(s t ,a t ) will increase, avoiding the removal of too many neurons at once; if the pruning is appropriate, the penalty term will decrease, encouraging further exploration of accuracy improvement in fine-tuning.

[0087] get next state s t+1 , store (s t , a t , r t , s t+1 ) into experience replay and sample a batch of samples from experience replay, use the network to select the optimal action a t+1 in the next state s * .

[0088]

[0089] where Q A (s t+1 , a';θ A ) is the Q value of the main network under parameters θ A for state s t+1 adopting action a'; the argmax function is to find the action with the maximum Q value in all possible actions a' in the action space A. The a ＊ obtained is the optimal action that is most likely to bring the highest cumulative return in the next state s t+1 .

[0090] According to the optimal action a ＊ , use the target network to calculate the TD target y t of the algorithm.

[0091]

[0092] where r t is the reward value; γ∈[0,1] is the discount factor; a ＊ is the optimal action; the min function selects the smaller Q value after adopting action a ＊ in the target network, so as to reduce the overestimation error existing in the network, so that the network update is in structural stability and model convergence.

[0093] Two main networks respectively calculate the error L i (θ i )=Ε (s,a)～D [Q i (s,a;θ i )-y] 2 i=A,B, and use the gradient descent method to update the weights θ A and θ B of the main network in the ClippedDoubleDQN algorithm, and every update steps, the weights θ A and θ B of the main network are updated to the target network and

[0094] When the iterative update reaches a certain number of episodes or after a certain number of patience rounds of error values do not change, the main network Q A Given the final state s fin Select the optimal action

[0095]

[0096] Extract the corresponding submatrix from the weight matrix to complete the final pruning processing, and obtain the most concise and optimal prediction performance echo state network reservoir structure. Use principal component analysis method to pre-prune the reservoir neurons and obtain the importance value, which provides information reference for the subsequent ClippedDoubleDQN algorithm. Use ClippedDoubleDQN algorithm to further prune the reservoir structure, capture nonlinear features, reduce low-contribution neurons, and improve the prediction performance of the model structure while simplifying the model structure. Specifically, the ClippedDoubleQ-Learning algorithm can improve the prediction accuracy of the pruned model, and the DeepQ-Network is introduced to model the high-dimensional and continuous neuron contribution vector to further improve the learning efficiency of the pruning strategy. The algorithm combines experience-based trial and error and dynamic adjustment to optimize the model structure until the model structure corresponding to a certain number of iterations is the best structure.

[0097] According to the number of prediction variables and auxiliary variables, the number of input nodes and output nodes of the model is determined, eight auxiliary variables are finally selected for the prediction of effluent ammonia nitrogen, so the number of input nodes is set to 8; the prediction variable is effluent ammonia nitrogen, so the number of output nodes is set to 1. Then we need to determine the specific structure of the reservoir.

[0098] 2.1 Initialization

[0099] 2.1.1) The principal component analysis method needs to determine the cleaned internal state matrix X = [X(m), X(m+1), …, X(N s ) T and the pre-pruning ratio prune_ratio;

[0100] 2.1.2) The ClippedDoubleDQN algorithm needs to determine the initial exploration probability ε0, the exploration decay degree ε t , the minimum exploration probability ε min, learning rate learning_rate, discount factor γ, minimum batch size batch_size, memory capacity memory_size, target network update round target_update, maximum number of iterations episodes, pruning lower limit ratio target_lower_ratio and pruning upper limit ratio target_upper_ratio, maximum tolerance of loss value loss_tolerance and patience value patience;

[0101] 2.1.3) Echo State Network needs to determine the reservoir size N DR , sparsity SD, spectral radius p, input shrinkage factor IS, and the specific settings of the parameters can be seen in the table.

[0102] Parameter Value Parameter Value prune_ratio 0.95 episodes 100 [CDATA[ε0]]> 0.9 target_lower_ratio 0.85 e t ]] 0.995 target_upper_ratio 0.95 e min ]] 0.1 loss_tolerance 1 x 10 -20 ]]> learning_rate 0.001 patience 40 γ 0.9

[0010] N DR ]]> 300 batch_size 5 SD 1.0 memory_size 100 ρ 0.9 target_update 10 IS 1.0

[0103] 2.2) Pre-pruning of reservoir neurons using principal component analysis

[0104] 2.2.1) Use the "cleaned" internal state matrix X = [X(m), X(m+1),..., X(N s )] T According to formula (7), the covariance matrix G is constructed;

[0105] 2.2.2) Use the eigenvalue decomposition Gv i = λ i v i i = 1, 2,..., n to obtain a set of eigenvalues λ1≥ λ2≥... ≥ λ n and eigenvectors v i ;

[0106] 2.2.3) For the n sets of eigenvalues and corresponding eigenvectors obtained, respectively, take the first w largest eigenvalues corresponding to the eigenvectors W = [v1, v2,..., v w ] ∈ N×w , W T W = I w . According to formula (8), the contribution degree of each neuron is calculated Sort in ascending order, record the current neuron contribution value;

[0107] 2.2.4) Remove the first prune% of neurons with low contribution to form a new set of neuron indexes The number of elements in the set, i.e. the number of neurons remaining after pre-pruning N prune ;

[0108] 2.2.5) Extract the corresponding rows and columns from the original weight matrix of the echo state network to reconstruct a new input weight matrix W'in The internal connection weight matrix W' of the reserve pool res and output weight matrix W' out Its parameters were also adjusted accordingly;

[0109] 2.2.6) Calculate the prediction error ζ after pre-pruning based on the pre-pruning model using the wastewater dataset as input. old .

[0110] 2.3) Use the ClippedDoubleDQN algorithm to reprune neurons in the reserve pool.

[0111] 2.3.1) Contribution values ​​obtained using principal component analysis Obtain the state vector State space Constructing Action Space Main network Q A (s,a;θ A ), Q B (s,a;θ B ) and target network

[0112] 2.3.2) In the main network Q A According to the current state s t Using an ε-greedy strategy to select action a t Extract the corresponding rows and columns from the pre-pruned weight matrix to reconstruct the weight matrix W”. in (t), W” res (t) and W” out (t) Obtain the new pruning model and train it to obtain the corresponding new prediction error ζ. new ;

[0113] 2.3.3) Calculate the reward r according to formula (9). t And discuss it. If r t If the value is greater than 0, accept the current pruning and record the current pruned subnetwork as the best subnetwork, using ζ. new Update ζ old If the pruning effect is no better than the previous effect, then revert to the subnetwork before pruning.

[0114] 2.3.4) Obtain the next state s t+1 , will (s t ,a t ,r t ,s t+1 Save the experience replay;

[0115] 2.3.5) Randomly select a batch of samples from the playback, and use the network to select the next state s according to formula (10).t+1 optimal action a ＊ TD error is calculated according to equation (11);

[0116] 2.3.6) Two master networks respectively calculate error L i (θ i ) = E (s,a)～D [Q i (s, a; θ i ) - y] 2 i = A, B, update the weights θ A and θ B of the master networks;

[0117] 2.3.7) Every target_update steps, the weights θ A and θ B of the master networks are synchronously updated to the target network and

[0118] 2.3.8) When the iterative update reaches a certain number of episodes or after patience rounds of error values do not change, the master network Q A is used to select the optimal action a fin according to equation (12) given the final state s The neural pruning is performed from the final action a to reconstruct the weight matrix W" in , W" res and W" out . The optimal policy π ＊ under the echo state network structure is finally obtained;

[0119] 2.3.9) The trained data set is used to drive the network. After the "idling" operation and the elimination of the initial state influence, the pseudo-inverse method is used to train the output weights of equation (6).

[0120] The echo state network of the trained principal component analysis combined with pruning double deep Q network algorithm is obtained through 2.1) to 2.3). The self-organizing process of the echo state network structure can be seen in the table.

[0121]

[0122]

[0123] 3. Online prediction and performance monitoring module

[0124] The trained principal component analysis combined with pruning double deep Q network (PCA-CDDQN) algorithm is used to obtain the optimal structure of the echo state network for predicting the test data, the observed value of the auxiliary variable in the test sample is taken as the input of the trained echo state network, and the output of the model is the prediction result of the effluent ammonia nitrogen concentration. 2 ) is used to reflect the explanation ability of the model to the variance of the observation data. Figure 4 The optimal test result graph of the original principal component analysis method, the original pruning double deep Q network algorithm and the proposed principal component analysis combined with pruning double deep Q network algorithm is shown in the figure, the X axis is time, the unit is sample number / day, the Y axis is the effluent ammonia nitrogen NH4-N concentration, the unit is mg / L, the red line is the actual data, the blue line is the output effluent ammonia nitrogen prediction concentration of the original echo state network, the yellow line is the output effluent ammonia nitrogen prediction concentration of the original principal component analysis method optimized echo state network structure, the green line is the output effluent ammonia nitrogen prediction concentration of the original pruning double deep Q network algorithm optimized echo state network structure, and the pink line is the output effluent ammonia nitrogen prediction concentration of the principal component analysis combined with pruning double deep Q network algorithm optimized echo state network. The closer the image is to the red line, the more ideal the prediction effect of the model is. Compared with the actual result and the actual ammonia nitrogen output, the prediction result of the proposed method is closer to the actual value.

[0125] As shown in Figure 5 , the iteration update error of the principal component analysis combined with pruning double deep Q network algorithm is shown, the horizontal axis represents the iteration number of the model, and the vertical axis represents the error of the model, that is, the mean square error. With the increase of the iteration number, the error of the model gradually decreases, the prediction result deviates relatively small from the true value, the explanation ability to the variance of the observation data is further improved, which indicates that the echo state network of the principal component analysis combined with pruning double deep Q network algorithm is effective for designing the effluent ammonia nitrogen concentration prediction method of the sewage treatment process.

[0126] The present application proposes a kind of effluent ammonia nitrogen prediction model construction method based on pruning double Q learning echo state network, and the prediction model of effluent ammonia nitrogen in urban sewage treatment process is established based on the echo state network of principal component analysis and pruning double deep Q network.The method is pre-reduction to reserve pool neuron by principal component analysis, then the contribution degree obtained by principal component analysis is re-reduction to reserve pool neuron by pruning double deep Q network algorithm, the adjustment of echo state network reserve pool structure not only guarantees the simplicity of model parameter and network structure, but also can more accurately capture the time series nonlinear relationship in data sequence, significantly improve the prediction accuracy of network.

[0127] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A method for predicting effluent ammonia nitrogen concentration based on a pruned double Q-learning echo state network, characterized in that: The method comprises the following steps, Real-time monitoring data of sewage treatment is collected, and the real-time monitoring data is preprocessed to obtain a monitoring data dataset; The monitoring data dataset is divided into training samples and test samples; a preliminary screening is performed on the auxiliary variables to be selected based on a Pearson correlation coefficient, and a final auxiliary variable set is determined based on a maximum correlation-minimum redundancy algorithm by further screening the auxiliary variables; An echo state network including an input layer, a reservoir layer, and an output layer is constructed, principal component analysis is used to pre-prune the reservoir neurons, the contribution degree of the neurons is calculated and low-contribution neurons are removed; based on the pre-pruned neurons, a double pruning deep Q network algorithm is used for secondary pruning to construct a main network and a target network, an action is selected by using an epsilon-greedy strategy, a reward function and a TD target are used to update the network, an optimal reservoir structure is obtained, the output weight of the network is trained, and an optimal echo state network is obtained; The auxiliary variables in the test samples are input into the echo state network, and an effluent ammonia nitrogen concentration prediction result is output; In the double pruning deep Q network algorithm, the reward function calculation formula is: (9) wherein, is the prediction error after pre-pruning, is the prediction error after re-pruning, is the adaptive coefficient, is the pruning action, is the number of remaining neurons after pre-pruning; In the double pruning deep Q network algorithm, the TD target calculation formula is: ； wherein, is a reward value; is a discount factor; is an optimal action; the function selects the action with the smaller Q-value. 2.The pruned double Q-learning echo state network-based effluent ammonia concentration prediction method according to claim 1, wherein: The preprocessing step includes noise reduction, removal of outliers, filling of missing values, and normalization. The normalization processing uses a Min-Max method to unify the variable scale to the range of [0, 1], and the calculation formula is: (1) wherein is the original value of the variable at a certain time, denotes the minimum value of the variable over the entire sample, denotes the maximum value of the variable over the entire sample. 3.The pruned double Q-learning echo state network-based effluent ammonia concentration prediction method according to claim 1, wherein: Using Pearson's correlation coefficient The auxiliary variable and the predicted variable The correlation between the variables is analyzed initially, according to the formula: (2) in Representing auxiliary variables With predictor variables The correlation coefficient between them; Indicates the first The first auxiliary variable One observation value; Represents the first predictor variable One observation value; Indicates the first The mean of the auxiliary variables; The mean of the predictor variable is represented by the correlation coefficient. Determine which auxiliary variables For predictor variables The impact is significant; preliminary screening has identified... The auxiliary variables are stored in the candidate auxiliary variable set; when screening auxiliary variables using the Pearson correlation coefficient, the correlation of the predictor variables is used as the screening principle, and auxiliary variables with an absolute value of correlation coefficient greater than 0.4 are retained; In the maximum correlation-minimum redundancy algorithm, the mutual information value between the auxiliary variables in the auxiliary variable set to be selected and the predicted variables is calculated to determine the linear relationship therebetween: (3) where and are the marginal probability densities of the auxiliary variables and the prediction variable , is the joint probability density of the auxiliary variables and the prediction variable , the correlation between the auxiliary variables and the prediction variable and the redundancy among the auxiliary variables are calculated by the mutual information value, and finally 5-12 auxiliary variables are selected. 4.The pruned double Q-learning echo state network-based effluent ammonia concentration prediction method according to claim 1, wherein: The update equations of the internal state and the output state of the reservoir of the echo state network are: (5) (6) wherein, is the internal state of the reservoir at time t, is the output state, is the input weight matrix, is the internal connection weight matrix of the reservoir, is the output weight matrix, and f is an activation function.

5. The pruned double Q-learning echo state network-based effluent ammonia concentration prediction method of claim 4, wherein: The principal component analysis pre-pruning includes constructing a covariance matrix using the internal state matrix of the reservoir, obtaining eigenvalues and eigenvectors through eigenvalue decomposition, calculating the normalized load value of the neurons in the first several principal components as the contribution degree, and removing the neurons with low contribution degree in proportion. 6.The pruned double Q-learning echo state network based effluent ammonia concentration prediction method according to claim 1, wherein: When dividing the training samples and the test samples, the proportion of the training samples is 65%-70%, and the proportion of the test samples is 30%-35%; during training, the initial state influence is eliminated by idling operation, and the pseudo-inverse method is used to optimize the output weight.

7. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: The processor executes the computer program to implement the steps of the effluent ammonia nitrogen concentration prediction method based on the double Q learning echo state network pruning of any one of claims 1-6.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the effluent ammonia nitrogen concentration prediction method based on the double Q learning echo state network pruning of any one of claims 1-6.

Images