A method for optimizing parameters of a dry quenching cascade control loop based on reinforcement learning

Abstract

The present application relates to a kind of based on reinforcement learning's dry quenching cascade control loop parameter optimization method, the waste heat boiler feed water flow cascade control loop data of dry quenching system actual operation in site is collected, the nonlinear model between the PID parameter of dry quenching cascade control loop controller and the integral ITAE of performance index error absolute value multiplied by time item to time is established, the PID parameter boundary of dry quenching system cascade loop is selected using multi-objective genetic algorithm NSGA-Ⅱ, the PID parameter optimization problem of dry quenching cascade control loop is changed into combination optimization problem, using reinforcement learning mechanism, realize the online optimization method of reinforcement learning cascade control loop parameter based on dry quenching waste heat boiler drum level cascade three impulse systems, significantly reduce the maximum overshoot under the condition that the rise time increment is little, it has wide application value in the field of intelligent control of industrial equipment, to effectively guarantee that field personnel timely and reliably maintain and control dry quenching cascade control loop.

Description

Technical Field

[0001] This invention relates to the field of dry quenching control technology, and in particular to a method for optimizing the parameters of a dry quenching cascade control loop based on reinforcement learning. Background Technology

[0002] The emergence of intelligent manufacturing has made the optimization and control of industrial processes increasingly important. Cascade control is one of the important controls in dry quenching systems. Control technology represented by PID is widely used in the field of cascade control loops for dry quenching waste heat boilers. Its safe and efficient operation is of great significance to enterprises. (DING, J., et al., Research Progress and Prospects of Intelligent Optimization Decision Making in Complex Industrial Process. ACTA AUTOMATICA SINICA, 2018. 44(11): p. 1931-1943.) Design a proportional-integral (PI) control system. Integral (PI) cascade controllers are used to solve problems such as inertia, time delay, and time-varying parameters in the cascade control of main steam temperature in power plant boilers. (Zhang Kun, Wang Hua, Wang Jianjun, et al. Simulation study of fuzzy neural control for waste heat boiler drum water level [J]. Control Engineering, 2010, 17(S3): 73-77.) Fuzzy neural control algorithm is introduced, and the composite control strategy based on fuzzy neural network and self-tuning PID is successfully applied to the control system of waste heat boiler drum water level. (Elhosseini, MA., et al., Heat recovery steam generator (HRSG) three-element drum level control utilizing Fractional order PID and fuzzy controllers. 2022, ISATRANSACTIONS.p.281-293.) A fractional order PID controller is proposed to solve the temperature regulation problem of cascade steam system. The performance index is based on overshoot, rise time, and the integral of the absolute value of error multiplied by the time term over time (ITAE). The fractional order PID is superior.

[0003] The cascade control loop for dry quenching coke is still mainly based on PID control. Its control difficulties are: 1) The cascade control loop for dry quenching coke has problems such as multiple operating conditions and non-stationarity; 2) Cascade control involves global non-convex and non-stationary optimization problems; 3) Cascade control loop is difficult to achieve real-time online automatic adjustment and optimization. At present, there is no effective method to systematically solve all the above problems at the same time. Summary of the Invention

[0004] This invention provides a reinforcement learning-based method for optimizing parameters of a dry quenching coke cascade control loop. Addressing the uncertainty of the state combination between the PID control parameters of the dry quenching coke cascade control loop and the steam drum liquid level, a BP neural network is used to nonlinearly model the integral of the absolute value of the error between the PID parameters and the optimization index multiplied by the time term over time (ITAE). The NSGA-II algorithm is used to select the PID parameter boundary of the dry quenching coke cascade control loop system, transforming the PID parameter optimization problem into a combinatorial optimization problem. Finally, a reinforcement learning mechanism is proposed to provide the optimal PID control strategy for the dry quenching coke system cascade control loop under multiple operating conditions. This invention not only has the advantages of fast solution speed and offline training with real-time online tuning, but also significantly reduces the maximum overshoot of the dry quenching coke cascade control loop under conditions of small rise time increments, thereby achieving significant energy-saving and emission-reduction effects in on-site production. This is of great significance for improving enterprise safety and reducing enterprise energy consumption.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A method for optimizing the parameters of a dry quenching coke cascade control loop based on reinforcement learning includes the following steps:

[0007] S1. Data Acquisition and Storage: Data of the waste heat boiler feedwater cascade control loop in actual operation of the dry quenching system is acquired through switches and wireless networks, and then stored and retrieved after data preprocessing.

[0008] S2. Control loop tuning model construction: Based on the BP neural network, a nonlinear model is established between the PID parameters of the dry quenching coke cascade control loop controller and the integral of the absolute value of the error multiplied by the time term over time, ITAE.

[0009] S3. Multi-objective optimization solution and population selection: The nonlinear model for tuning the dry quenching coke cascade control loop is constructed based on the BP neural network. The optimization indexes are the absolute value of the error of each working condition of the dry quenching coke cascade control loop multiplied by the integral of the time term over time (ITAE) and the root mean square error (RMSE). The PID parameters of the dry quenching coke cascade control loop are used as boundary constraints. The multi-objective genetic algorithm NSGA-II is used for optimal iteration to select the optimal PID control parameter population.

[0010] S4. Optimization and solution of PID parameters for dry quenching coke cascade control loop based on reinforcement learning: When the value neural network evaluates the change of PID parameters of the dry quenching coke cascade control loop at the current moment, the policy neural network gives the next PID control parameter action based on the absolute value of the current performance index error multiplied by the integral of the time term over time, ITAE. Finally, the optimal PID parameters of the dry quenching coke cascade control loop in this paper are obtained through optimization and iteration by the deep reinforcement learning value neural network and the policy neural network.

[0011] S5. Optimal Parameter Distribution: The adjusted PID controller parameters of the dry quenching cascade control loop are distributed to optimize the actual dry quenching cascade control loop in real time.

[0012] Furthermore, in step S2, the input vector of the BP neural network is the PID parameter U of the dry quenching cascade control loop, the input layer has N_bp neurons, the output vector is D, which is the absolute value of the error multiplied by the integral of the time term over time, ITAE, the output layer has M_bp neurons, and the length of the input / output sample pair is L. Specifically, the steps include:

[0013] S2.1 Set the initial weights W(0) of the BP neural network. They are random non-zero values ​​that conform to a Gaussian distribution with a standard deviation of 0.01.

[0014] S2.2. Given a sample pair of PID parameters / absolute error multiplied by the integral of the time term over time in the dry quenching cascade control loop (ITAE), calculate the output of the BP neural network:

[0015] Let the PID parameter sample input of the dry quenching cascade control loop of group P be: U p =(U 1P U 2P ,...,U nP Let the absolute value of the output error of the Pth sample be the integral of the time term over time, ITAE: D p =(D 1P D 2P ,...,D nP ), P = 1, 2, ..., L, node m outputs Y when the Pth sample is input. mP :

[0016]

[0017] In the formula, I nP When the PID parameter sample input of the dry quenching cascade control loop in group P is given, the nth input of node m is given, and f(·) is the S-shaped action function of the energy take-off, i.e. Y mP (t) represents the output of node m when the PID parameter sample input of the dry quenching coke cascade control loop in group P is received, x mP(t) is the product of the PID parameter sample input and weights of node m in the P-th group of dry quenching cascade control loop, W mn (t) represents the weight of node m, I nP For node n, the PID parameter sample input of the dry quenching cascade control loop in group P is used.

[0018] S2.3 Calculate the objective function J_bp of the BP neural network:

[0019] Let E P Let (t) be the objective function of the network when the PID parameter samples of the dry quenching cascade control loop in group P are input. Then:

[0020]

[0021] In the formula, Y lP (t) represents the network output after t weight adjustments when the PID parameter samples of the dry quenching coke cascade control loop of group P are input; l is the l-th node of the output layer; D lP The corresponding absolute value of the error is multiplied by the integral of the time term over time, ITAE.

[0022] The overall objective function serves as an evaluation of the BP neural network's learning performance. The network's overall objective function is:

[0023]

[0024] S2.4 Algorithm Stop Judgment:

[0025] If J_bp(t)≤ε, the algorithm ends, where ε is a predetermined convergence error and ε>0; otherwise, proceed to step S2.5.

[0026] S2.5, Backpropagation Calculation:

[0027] The ITAE index, calculated by multiplying the absolute value of the output layer error by the integral of the time term over time, is used in reverse gradient descent based on J_bp. Weights are adjusted layer by layer with a constant step size to obtain the formula for adjusting the connection weights from neuron n to neuron m for the (t+1)th iteration:

[0028]

[0029] In the formula, η is the step size, referred to here as the learning operator, and ΔW mn (t) represents the connection weights from neuron n to neuron m adjusted by t+1 times.

[0030] Furthermore, step S3, which uses the multi-objective genetic algorithm NSGA-II to perform optimal iteration of PID parameters in the dry quenching cascade control loop to select the optimal PID parameter population, includes the following steps:

[0031] S3.1, Fast Non-Dominated Sort Strategy:

[0032] The initial dry quenching coke cascade control loop PID parameter population size of the multi-objective genetic algorithm NSGA-II is defined as N, and two parameters θ are set for each PID parameter individual i in the population. i and χ i θ i and χ i Let θ represent the number of individual PID parameters dominating i and the set of individual PID parameters dominated by i, respectively. The first step is to find the set of PID parameters in this dry quenching cascade control loop that satisfy θ. i Individuals with a value of 0 are added to set F. i In the middle; the second step is for F i For all individual PID parameters j, examine the set of individuals χ governed by them. i And reduce the number of solutions that govern the individual PID parameter k by 1, if θ k -1 = 0, so store k in set H; third step, put F i The set of first-level non-dominated PID parameters is considered, and all PID parameters in the set are assigned the same non-dominated order. Then, H is graded and assigned a non-dominated order, until all PID parameters are graded. The computational complexity of the algorithm is O(MN). 2 ), where N is the population size and M is the number of individuals in the objective function;

[0033] S3.2, Diversity Preservation Strategy:

[0034] The multi-objective genetic algorithm NSGA-II maintains population diversity by comparing the crowding distances of individual PID parameters in the dry quenching cascade control loop. Before calculating the crowding distance, the PID parameter population is quickly non-dominated and sorted according to the ITAE value (the integral of the time term over time) of the absolute value of the objective function error of each objective function. All PID parameter populations are divided into V layers. Then, an infinite crowding distance d = ∞ is assigned to the two solutions at the outermost boundary of the ITAE multiplied by the absolute value of the objective function error. The smaller the crowding distance of a PID parameter individual, the denser its distribution with surrounding PID parameter individuals. NSGA-II uses the crowding distance comparison strategy to guide the solution PID parameter individuals to be evenly distributed on an ideal Pareto graph. Each PID parameter individual i is assigned two basic attributes: non-dominated sorting i. rank and crowded distance i distance We give a partial order Definition: If the non-dominated ranking of PID parameter individual i is less than the non-dominated ranking of PID parameter individual j, then... rank <j rankOr, i and j have the same non-dominated order. rank =j rank However, the crowding distance of i is greater than the crowding distance of j. distance >j distance , just say

[0035] S3.3, Select the crossover / mutation operation:

[0036] The multi-objective genetic algorithm NSGA-II employs a tournament selection strategy. Two parent PID parameter individuals are crossbred using simulated binary crossover to produce two offspring PID parameter individuals. The offspring PID parameter individuals retain the pattern information from their parents. g represents the current generation number. For two PID parameter individuals in the dry quenching coke cascade control loop, arbitrarily selected from the PID parameter population, These are the j-th gene loci of the two individuals with PID parameters. yes Two offspring PID parameter individuals are generated by simulating binary crossover. The generation process is shown in the following formula:

[0037]

[0038] In the formula, β q (j) is the uniform distribution factor, given by formula (6), where μ(j) is a random number belonging to [0,1], and η c ∈[0,1] represents the crossover index:

[0039]

[0040] β q The probability density function of (j) is:

[0041]

[0042] The polynomial mutation operator prevents the population of PID parameters in the dry quenching coke cascade control loop from getting trapped in local optima by mutating genes. For the j-th gene of the i-th individual with PID parameters in the g-th generation, PID parameters Upper and lower bounds, individual PID parameters Mutation generates offspring PID parameter individuals The mutation process is shown in the following formula:

[0043]

[0044] Pick It is a uniformly distributed random number in the range [0,1]. It is a variable-length variable, the magnitude of which determines the degree of variation, and is calculated by the following formula:

[0045]

[0046] In the formula, η m It is the distribution index of variation, and its magnitude affects the distribution of variation.

[0047] Furthermore, the optimization solution of PID parameters for the dry quenching cascade control loop based on reinforcement learning in step S4 includes the following:

[0048] Based on the Deep Deterministic Policy Ladder (DDPG), a deep neural network is used as a nonlinear function approximator to construct the policy μ(s|θ). μ ) and value Q(s,a|θ Q The network model is set, and the policy target network μ′(s|θ) is also set. μ′ ) and value target network Q′(s,a|θ Q′ An exploratory policy network μ′(s) is constructed by adding exploratory noise to the independent policy network. t ),

[0049] μ′(s t )=μ(s t |θ μ )+Noise (10)

[0050] In the formula, μ′(s t (This refers to a policy network for exploring noise, which is an independent policy network.)

[0051] During network model learning, the value network minimizes the loss function:

[0052]

[0053] In the formula, Let Q(s) be the sign of minimizing the expected value of the loss function. t ,a t |θ Q ) represents the value network of the dry quenching cascade control loop at the current moment, s t a is the current state value of the dry quenching coke cascade control loop. t The current action value of the PID parameters of the dry quenching coke cascade control loop;

[0054] The cumulative reward for the dry quenching coke cascade control loop is as follows:

[0055] y t =r(s t ,a t)+γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ (12)

[0056] In the formula, r(s) t ,a t ) represents the current reward value of the dry quenching coke cascade control loop, γ represents the reward discount factor of the dry quenching coke cascade control loop, and Q′(s) represents the reward value of the dry quenching coke cascade control loop. t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ The cumulative network reward for the target value of the dry quenching coke cascade control loop;

[0057] The dry quenching coke cascade control loop strategy network uses the Monte Carlo method for sampling to approximate the desired value, and the update is shown in Equation (13):

[0058]

[0059] In the formula, For the gradient of the dry quenching coke cascade control loop strategy network, J(θ) μ ) is the network reward for the dry quenching coke cascade control loop strategy, s i Let μ(s|θ) be the state of the dry quenching cascade control loop at time i. μ ) represents the cascade control loop strategy network for dry quenching coke, μ(s) i The state of the dry quenching coke cascade control loop is s. i At that time, the PID control parameters generated by the strategy network take action, s is the state of the dry quenching coke cascade control loop, and a is the PID parameter action of the dry quenching coke cascade control loop.

[0060] The target network uses a soft update method to track and update the parameters of the learned dry quenching cascade control loop network.

[0061] θ′←τθ+(1-τ)θ′ (14)

[0062] In the formula, θ′ represents the updated target network parameters, τ represents the update coefficient, which ranges from 0 to 1, and θ represents the target network parameters.

[0063] The Deep Deterministic Policy Gradient (DDPG) also employs an experience retrieval mechanism, which utilizes the experience samples e = (s) generated by the interaction between the dry quenching coke cascade control loop policy network and the dry quenching coke cascade control loop environment. t ,a t ,r t ,s t+1 The data are stored in the experience pool, and during network training, experience samples are randomly selected from the experience pool to update the network parameters.

[0064] Compared with the prior art, the beneficial effects of the present invention are:

[0065] To address the uncertainty in the state combination between the control parameters of the dry quenching coke cascade loop and the steam drum liquid level, this invention utilizes existing historical data from the industrial field of the dry quenching coke cascade control loop. First, the real-time data is pre-processed. Then, a BP neural network is used to model the integral of the absolute value of the error between the PID controller parameters and the optimization index multiplied by the time term over time in the dry quenching coke waste heat boiler steam drum liquid level cascade three-impulse system. The multi-objective genetic algorithm NSGA-II is used to select the PID parameter boundary of the dry quenching coke cascade control loop system, transforming the PID parameter optimization problem into a combinatorial optimization problem. Finally, a reinforcement learning mechanism is used to realize an online optimization method for the cascade control loop parameters based on the dry quenching coke waste heat boiler steam drum liquid level cascade three-impulse system. The proposed dry quenching coke cascade control loop control strategy can significantly reduce the maximum overshoot with a small rise time increment. This method has wide application value in the field of intelligent control of industrial equipment, effectively ensuring timely and reliable maintenance and control of the dry quenching coke cascade control loop by on-site personnel.

[0066] This invention proposes a reinforcement learning-based parameter optimization method for a given dry quenching cascade control loop, which can optimize and adjust the controller loop in a timely and effective manner, thereby providing effective support for on-site optimization and control. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0068] Figure 2 This is a schematic diagram of the BP neural network topology described in this invention.

[0069] Figure 3 This is a schematic diagram of the BP neural network learning process described in this invention.

[0070] Figure 4 This is a schematic diagram of the multi-objective genetic algorithm NSGA-II described in this invention.

[0071] Figure 5 This is a diagram showing the overall structure of the reinforcement learning optimization method of this invention.

[0072] Figure 6 This is a flowchart of the Deep Deterministic Policy Network (DDPG) algorithm described in this invention.

[0073] Figure 7 This is a control effect diagram of the dry quenching coke cascade control loop described in this invention. Detailed Implementation

[0074] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0075] See Figure 1 This is a schematic diagram of the structure of the present invention. The present invention provides a method for optimizing the parameters of a dry quenching coke cascade control loop based on reinforcement learning, comprising the following steps:

[0076] S1. Data Acquisition and Storage: Data on the waste heat boiler feedwater cascade control loop of the dry quenching system in actual operation is acquired through switches and wireless networks. This includes the controlled variable CV, manipulated variable MV, process measurement value PV, loop setpoint SP, and PID parameter big data of the dry quenching cascade control loop. After data preprocessing, the data is stored and retrieved.

[0077] S2. Control loop tuning model construction: Based on the BP neural network, a nonlinear model is established between the PID controller parameters and the performance index error absolute value multiplied by the time term and the integral of the time term ITAE in the dry quenching coke cascade control loop.

[0078] Using a BP neural network, the structure of the three-layer neural network is as follows: Figure 2 As shown, each neuron is represented by a node. The network consists of input layer, hidden layer, and output layer nodes. For the PID parameters of the quenching cascade control loop, the input signal must first propagate forward to the hidden layer nodes. After passing through the activation function, the output signal of the hidden layer nodes is then propagated to the output layer. The learning process of the BP neural network is as follows: Figure 3 As shown, the learning steps of the algorithm are as follows:

[0079] Let the input vector of the BP neural network be the PID parameters U of the dry quenching coke cascade control loop, the input layer have N_bp neurons, the output vector be D, which is the absolute value of the error multiplied by the integral of the time term over time ITAE, the output layer have M_bp neurons, and the length of the input / output sample pair be L. The specific steps are as follows:

[0080] S2.1 Set the initial weights W(0) of the BP neural network. They are random non-zero values ​​that conform to a Gaussian distribution with a standard deviation of 0.01.

[0081] S2.2. Given a sample pair of PID parameters / absolute error multiplied by the integral of the time term over time in the dry quenching cascade control loop (ITAE), calculate the output of the BP neural network:

[0082] Let the PID parameter sample input of the dry quenching cascade control loop of group P be: U p =(U 1P U 2P ,...,U nP Let the absolute value of the output error of the Pth sample be the integral of the time term over time, ITAE: D p=(D 1P D 2P ,...,D nP ), P = 1, 2, ..., L, node m outputs Y when the Pth sample is input. mP :

[0083]

[0084] In the formula, I nP When the PID parameter sample input of the dry quenching cascade control loop in group P is given, the nth input of node m is given, and f(·) is the S-shaped action function of the energy take-off, i.e. Y mP (t) represents the output of node m when the PID parameter sample input of the dry quenching coke cascade control loop in group P is received, x mP (t) is the product of the PID parameter sample input and weights of node m in the P-th group of dry quenching cascade control loop, W mn (t) represents the weight of node m, I nP For node n, the PID parameter sample input of the dry quenching cascade control loop in group P is used.

[0085] S2.3 Calculate the objective function J_bp of the BP neural network:

[0086] Let E P Let (t) be the objective function of the network when the PID parameter samples of the dry quenching cascade control loop in group P are input. Then:

[0087]

[0088] In the formula, Y lP (t) represents the network output after t weight adjustments when the PID parameter samples of the dry quenching coke cascade control loop of group P are input; l is the l-th node of the output layer; D lP The corresponding absolute value of the error is multiplied by the integral of the time term over time, ITAE.

[0089] The overall objective function serves as an evaluation of the BP neural network's learning performance. The network's overall objective function is:

[0090]

[0091] S2.4 Algorithm Stop Judgment:

[0092] If J_bp(t)≤ε, the algorithm ends, where ε is a predetermined convergence error and ε>0; otherwise, proceed to step S2.5.

[0093] S2.5, Backpropagation Calculation:

[0094] The ITAE index, calculated by multiplying the absolute value of the output layer error by the integral of the time term over time, is used in reverse gradient descent based on J_bp. Weights are adjusted layer by layer with a constant step size to obtain the formula for adjusting the connection weights from neuron n to neuron m for the (t+1)th iteration:

[0095]

[0096] In the formula, η is the step size, referred to here as the learning operator, and ΔW mn (t) represents the connection weights from neuron n to neuron m adjusted by t+1 times.

[0097] S3. Multi-objective optimization solution and population selection: Based on the constructed control loop tuning model, the absolute value of the error of each working condition multiplied by the integral of the time term over time (ITAE) and the root mean square error (RMSE) are used as optimization indicators. The PID parameters of the dry quenching coke cascade control loop are used as boundary constraints. The multi-objective genetic algorithm NSGA-II is used for optimal iteration to select the optimal population of PID parameters for the dry quenching coke cascade control loop.

[0098] S3.1, Fast Non-Dominated Sort Strategy:

[0099] The initial dry quenching coke cascade control loop PID parameter population size of the multi-objective genetic algorithm NSGA-II is defined as N, and two parameters θ are set for each PID parameter individual i in the population. i and χ i θ i and χ i Let θ represent the number of individual PID parameters dominating i and the set of individual PID parameters dominated by i, respectively. The first step is to find the set of PID parameters in this dry quenching cascade control loop that satisfy θ. i Individuals with a value of 0 are added to set F. i In the middle; the second step is for F i For all individual PID parameters j, examine the set of individuals χ governed by them. i And reduce the number of solutions that govern the individual PID parameter k by 1, if θ k -1 = 0, so store k in set H; third step, put F i The set of first-level non-dominated PID parameters is considered, and all PID parameters in the set are assigned the same non-dominated order. Then, H is graded and assigned a non-dominated order, until all PID parameters are graded. The computational complexity of the algorithm is O(MN). 2 ), where N is the population size and M is the number of individuals in the objective function;

[0100] S3.2, Diversity Preservation Strategy:

[0101] The multi-objective genetic algorithm NSGA-II maintains population diversity by comparing the crowding distances of individual PID parameters in the dry quenching cascade control loop. Before calculating the crowding distance, the PID parameter population is quickly non-dominated and sorted according to the ITAE value (the integral of the time term over time) of the absolute value of the objective function error of each objective function. All PID parameter populations are divided into V layers. Then, an infinite crowding distance d = ∞ is assigned to the two solutions at the outermost boundary of the ITAE multiplied by the absolute value of the objective function error. The smaller the crowding distance of a PID parameter individual, the denser its distribution with surrounding PID parameter individuals. NSGA-II uses the crowding distance comparison strategy to guide the solution PID parameter individuals to be evenly distributed on an ideal Pareto graph. Each PID parameter individual i is assigned two basic attributes: non-dominated sorting i. rank and crowded distance i distance We give a partial order Definition: If the non-dominated ranking of PID parameter individual i is less than the non-dominated ranking of PID parameter individual j, then... rank <j rank Or, i and j have the same non-dominated order. rank =j rank However, the crowding distance of i is greater than the crowding distance of j. distance >j distance , just say

[0102] S3.3, Select the crossover / mutation operation:

[0103] The multi-objective genetic algorithm NSGA-II employs a tournament selection strategy. The algorithm's flowchart is as follows: Figure 4 As shown, the multi-objective genetic algorithm NSGA-II uses simulated binary crossover to obtain two offspring PID parameter individuals from two parent PID parameter individuals. The offspring PID parameter individuals retain the pattern information from the parent PID parameter individuals, where g is the current generation number. For two PID parameter individuals in the dry quenching coke cascade control loop, arbitrarily selected from the PID parameter population, These are the j-th gene loci of the two individuals with PID parameters. yes Two offspring PID parameter individuals are generated by simulating binary crossover. The generation process is shown in the following formula:

[0104]

[0105] In the formula, β q (j) is the uniform distribution factor, given by formula (20), where μ(j) is a random number belonging to [0,1], and η c∈[0,1] represents the crossover index:

[0106]

[0107] β q The probability density function of (j) is:

[0108]

[0109] The polynomial mutation operator prevents the population of PID parameters in the dry quenching coke cascade control loop from getting trapped in local optima by mutating genes. For the j-th gene of the i-th individual with PID parameters in the g-th generation, PID parameters Upper and lower bounds, individual PID parameters Mutation generates offspring PID parameter individuals The mutation process is shown in the following formula:

[0110]

[0111] Pick It is a uniformly distributed random number in the range [0,1]. It is a variable-length variable, and its magnitude determines the degree of variation, which is calculated by the following formula:

[0112]

[0113] In the formula, η m It is the distribution index of variation, and its magnitude affects the distribution of variation.

[0114] S4. Optimization and solution of PID parameters for dry quenching coke cascade control loop based on reinforcement learning: When the value neural network of the dry quenching coke cascade control loop evaluates the change of PID parameters of the dry quenching coke cascade control loop at the current moment, the policy neural network of the dry quenching coke cascade control loop gives the next action of PID control parameters of the dry quenching coke cascade control loop based on the absolute value of the current performance index error multiplied by the integral of the time term over time (ITAE). Finally, the optimal PID parameters of the dry quenching coke cascade control loop in this paper are obtained through optimization and iteration by the deep reinforcement learning value neural network and the policy neural network.

[0115] The flowchart of the Deep Deterministic Policy Gradient (DDPG) algorithm is as follows: Figure 6 As shown, a deep neural network is used as a nonlinear function approximator to construct the strategy μ(s|θ). μ ) and value Q(s,a|θ Q The network model is set, and the policy target network μ′(s|θ) is also set. μ′ ) and value target network Q′(s,a|θ Q′An exploratory policy network μ′(s) is constructed by adding exploratory noise to the independent policy network. t ),

[0116] μ′(s t )=μ(s t |θ μ )+Noise (24)

[0117] In the formula, μ′(s t (This refers to a policy network for exploring noise, which is an independent policy network.)

[0118] During network model learning, the value network minimizes the loss function:

[0119]

[0120] In the formula, Let Q(s) be the sign of minimizing the expected value of the loss function. t ,a t |θ Q ) represents the value network of the dry quenching cascade control loop at the current moment, s t a is the current state value of the dry quenching coke cascade control loop. t This represents the current action value of the dry quenching cascade control loop.

[0121] The cumulative reward for the dry quenching coke cascade control loop is as follows:

[0122] y t =r(s t ,a t )+γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ (26)

[0123] In the formula, r(s) t ,a t ) represents the current reward value of the dry quenching coke cascade control loop, γ represents the reward discount factor of the dry quenching coke cascade control loop, and Q′(s) represents the reward value of the dry quenching coke cascade control loop. t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ The cumulative network reward for the target value of the dry quenching coke cascade control loop;

[0124] The dry quenching coke cascade control loop strategy network uses the Monte Carlo method for sampling to approximate the desired value, and the update is shown in Equation (27):

[0125]

[0126] In the formula, For the gradient of the dry quenching coke cascade control loop strategy network, J(θ) μ ) is the network reward for the dry quenching coke cascade control loop strategy, s i Let μ(s|θ) be the state of the dry quenching cascade control loop at time i. μ ) represents the cascade control loop strategy network for dry quenching coke, μ(s) i The state of the dry quenching coke cascade control loop is s. i At that time, the PID control parameters generated by the strategy network are activated, s is the state of the dry quenching coke cascade control loop, and a is the action of the dry quenching coke cascade control loop.

[0127] The target network uses a soft update method to track and update the parameters of the learned dry quenching cascade control loop network.

[0128] θ′←τθ+(1-τ)θ′ (28) In the formula, θ′ is the updated target network parameter, τ is the update coefficient, which takes a value of 0 to 1, and θ is the target network parameter;

[0129] The Deep Deterministic Policy Gradient (DDPG) also employs an experience retrieval mechanism, which utilizes the experience samples e = (s) generated by the interaction between the dry quenching coke cascade control loop policy network and the dry quenching coke cascade control loop environment. t ,a t ,r t ,s t+1 The data are stored in the experience pool, and during network training, experience samples are randomly selected from the experience pool to update the network parameters.

[0130] S5. Optimal Parameter Distribution: The adjusted PID controller parameters of the dry quenching cascade control loop are distributed to optimize the actual dry quenching cascade control loop in real time.

[0131] Figure 5 This is a diagram illustrating the overall structure of the reinforcement learning optimization method of this invention. The dry quenching cascade control loop mainly consists of a PI controller, a P controller, feedwater valves, a waste heat boiler drum, steam flow rate, and steam flow transmitters, feedwater flow transmitters, and drum level transmitters. The process of this invention is as follows: First, the PID parameters and error data of the waste heat boiler drum level control loop in the dry quenching cascade control loop are collected; second, a nonlinear model is established using a BP neural network between the PID parameters of the cascade control loop and the ITAE index (the integral of the absolute value of the error multiplied by the time term over time); then, the multi-objective genetic algorithm NSGA-II is used to optimize and search for the PID parameters of the cascade control loop, obtaining the optimal PID parameter population for the cascade loop; finally, the optimal PID control parameters of the dry quenching cascade control loop are obtained according to the deep deterministic policy gradient (DDPG), and the control effect is as follows. Figure 7As shown, the long dashed line and the dotted dashed line represent the effects of the present invention and the grid optimization control, respectively. In comparison, the overshoot of the present invention can be reduced by 1.33%.

[0132] Table 1 shows the control and adjustment results of the present invention, where the mean square error (MSE) and integral absolute error (IAE) are the control performance evaluation indicators.

[0133] Table 1. Evaluation of Control Performance of Two Control Methods

[0134]

[0135] It can be seen that the PID controller parameters obtained by the grid optimization method can achieve stable operation of the waste heat boiler drum liquid level in the dry quenching coke cascade control loop. However, the grid optimization method has a long solution time and high computer resource requirements, making it difficult to directly apply to the drum liquid level control loop with rapidly changing dynamic performance. The method in this paper can be trained offline, can perform real-time online optimization of the drum liquid level, can achieve stable operation of the control loop, and can significantly reduce the maximum overshoot of the waste heat boiler drum liquid level control loop under the condition that the rise time increment is not large.

[0136] The above embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the above embodiments. Unless otherwise specified, the methods used in the above embodiments are conventional methods.

Claims

1. A method for optimizing parameters of a dry quenching cascade control loop based on reinforcement learning, characterized in that, Includes the following steps: S1. Data Acquisition and Storage: Data of the waste heat boiler feedwater cascade control loop in actual operation of the dry quenching system is acquired through switches and wireless networks, and then stored and retrieved after data preprocessing. S2. Control loop tuning model construction: Based on the BP neural network, a nonlinear model is established between the PID parameters of the dry quenching coke cascade control loop controller and the integral of the absolute value of the error multiplied by the time term over time, ITAE. Let the input vector of the BP neural network be the PID parameters U of the dry quenching coke cascade control loop, the input layer have N_bp neurons, the output vector be D, which is the absolute value of the error multiplied by the integral of the time term over time ITAE, the output layer have M_bp neurons, and the length of the input / output sample pair be L. The specific steps are as follows: S2.1 Set the initial weights W(0) of the BP neural network. They are random non-zero values ​​that conform to a Gaussian distribution with a standard deviation of 0.

01. S2.

2. Given a sample pair of PID parameters / absolute error multiplied by the integral of the time term over time in the dry quenching cascade control loop (ITAE), calculate the output of the BP neural network: Suppose the PID parameter sample input for the Pth group of dry quenching cascade control loop is: Let the absolute value of the output error of the Pth sample group be multiplied by the integral of the time term over time, ITAE: , P=1,2,...,L, and the output of node m when the Pth sample group is input is: : （1） In the formula, It is the nth input of node m when the PID parameter sample input of the dry quenching cascade control loop of group P is received. To obtain the S-shaped action function with different energy, i.e. , The output of node m when the PID parameter sample input of the dry quenching cascade control loop in group P is given. Let m be the product of the PID parameter sample input and weights of node m in the P-th group of dry quenching cascade control loop. Let be the weight of node m. For node n, the PID parameter sample input of the dry quenching cascade control loop in group P is used. S2.3 Calculate the objective function J_bp of the BP neural network: set up Let the objective function of the network be the PID parameter sample input of the P-th group of dry quenching cascade control loop, then: （2） In the formula, It is the output of the network after t weight adjustments when the PID parameter samples of the dry quenching cascade control loop of the Pth group are input; l is the lth node of the output layer; The corresponding absolute value of the error is multiplied by the integral of the time term over time, ITAE. The overall objective function serves as an evaluation of the BP neural network's learning performance. The network's overall objective function is: （3） S2.4 Algorithm Stop Judgment: like The algorithm then ends. It is a predetermined convergence error and Otherwise, proceed to step S2.

5. S2.5, Backpropagation Calculation: The ITAE index, calculated by multiplying the absolute value of the output layer error by the integral of the time term over time, is used in reverse gradient descent based on J_bp. Weights are adjusted layer by layer, with a constant step size, to obtain the formula for adjusting the connection weights from neuron n to neuron m for the (t+1)th iteration: （4） In the formula, Let the step size be denoted as the learning operator. The connection weights from neuron n to neuron m are adjusted by increment t+1 times; S3. Multi-objective optimization solution and population selection: The nonlinear model for tuning the dry quenching coke cascade control loop is constructed based on the BP neural network. The optimization indexes are the absolute value of the error of each working condition of the dry quenching coke cascade control loop multiplied by the integral of the time term over time (ITAE) and the root mean square error (RMSE). The PID parameters of the dry quenching coke cascade control loop are used as boundary constraints. The multi-objective genetic algorithm NSGA-II is used for optimal iteration to select the optimal PID control parameter population. The following steps are involved in using the multi-objective genetic algorithm NSGA-II to iterate and select the optimal PID parameter population for the dry quenching cascade control loop: S3.1, Fast Non-Dominated Sort Strategy: The initial PID parameters and population size of the dry quenching cascade control loop of the multi-objective genetic algorithm NSGA-II are defined as follows: For each individual PID parameter in the population Set two parameters for each and , and Representing domination The number of individual PID parameters and the The first step is to find the set of individual PID parameters that dominate this dry quenching cascade control loop. Individuals are stored in a set In the middle; the second step is... All individual PID parameters Examine the set of individuals it governs and control the individual PID parameters. If the number of solutions is reduced by 1, then... Then Store in collection In the middle; the third step, will Treat it as a set of first-order undominated PID parameters, assign the same undominated order to all PID parameters in the set, and then... The algorithm performs a classification process and assigns non-dominated orders until all individual PID parameters are classified. The computational complexity of this algorithm is O(n log n). ,in It is the population size. It is the number of individuals in the objective function; S3.2, Diversity Preservation Strategy: The multi-objective genetic algorithm NSGA-II maintains population diversity by comparing the crowding distances of individual PID parameters in the dry quenching cascade control loop. Before calculating the crowding distance, the PID parameter population is quickly non-dominated and sorted according to the ITAE value (the integral of the time term over time) of the absolute value of the error of each objective function. This divides the entire PID parameter population into V layers. Then, an infinitely large crowding distance is assigned to each of the two solutions at the outermost boundary of the ITAE multiplied by the absolute value of the error of each objective function. The smaller the crowding distance of a PID parameter individual, the denser its distribution with surrounding PID parameter individuals. The multi-objective genetic algorithm NSGA-II uses a crowding distance comparison strategy to guide the solution to uniformly distribute PID parameter individuals onto an ideal Pareto graph. They are all assigned two basic properties: non-dominated ordering and crowded distance We give a partial order Definition: If the PID parameters are individual The non-dominated ranking is less than the PID parameter individual. Non-dominated sorting ,or and Non-dominated ordering is equal ,but The crowding distance is greater than crowded distance , just say ; S3.3, Select the crossover / mutation operation: The multi-objective genetic algorithm NSGA-II employs a tournament selection strategy. Two parent PID parameter individuals are used in NSGA-II to generate two offspring PID parameter individuals through simulated binary crossover. The offspring PID parameter individuals retain the pattern information from their parents. For the current generation, For two PID parameter individuals in the dry quenching coke cascade control loop, arbitrarily selected from the PID parameter population, These are the first two individual PID parameters. One gene locus, yes Two offspring PID parameter individuals are generated by simulating binary crossover. The generation process is shown in the following formula: （5） In the formula, The uniform distribution factor is given by formula (6), where It belongs to random numbers, Cross-distribution index: （6） The probability density function is: （7） The polynomial mutation operator prevents the population of PID parameters in the dry quenching coke cascade control loop from getting trapped in local optima by mutating genes. For the first The generation The first PID parameter individual One gene, , PID parameters Upper and lower bounds, individual PID parameters Mutation generates offspring PID parameter individuals The mutation operation process is shown in the following formula: （8） Pick , is a uniformly distributed random number in the range [0,1]. It is a variable-length variable, the magnitude of which determines the degree of variation, and is calculated by the following formula: （9） In the formula, This is the distribution index of variation, and its magnitude affects the distribution of variation. S4. Optimization and solution of PID parameters for dry quenching coke cascade control loop based on reinforcement learning: When the value neural network evaluates the change of PID parameters of the dry quenching coke cascade control loop at the current moment, the policy neural network gives the next PID control parameter action based on the absolute value of the current performance index error multiplied by the integral of the time term over time, ITAE. Finally, the optimal PID parameters of the dry quenching coke cascade control loop in this paper are obtained through optimization and iteration by the deep reinforcement learning value neural network and the policy neural network. The optimization solution of PID parameters for the dry quenching coke cascade control loop based on reinforcement learning includes the following: Based on the Deep Deterministic Policy Ladder (DDPG), a deep neural network is used as a nonlinear function approximator to construct policies. and value The network model, and also set the policy target network. and value target network By adding exploration noise to the independent policy network Constructing exploratory policy networks , （10） In the formula, To add exploration noise to the independent policy network The strategy network; During network model learning, the value network minimizes the loss function: （11） In the formula, The sign represents the minimization of the expected value of the loss function. The value network of the dry quenching coke cascade control loop at the current moment. This represents the current state value of the dry quenching coke cascade control loop. The current action value of the PID parameters of the dry quenching coke cascade control loop; The cumulative reward for the dry quenching coke cascade control loop is as follows: （12） In the formula, This is the reward value for the current dry quenching coke cascade control loop. The reward discount factor for the dry quenching coke cascade control loop. Accumulated network reward for the target value of the dry quenching coke cascade control loop; The dry quenching coke cascade control loop strategy network uses the Monte Carlo method for sampling to approximate the desired value, and the update is shown in Equation (13): （13） In the formula, The gradient of the dry quenching coke cascade control loop strategy network. The network reward is used for the dry quenching coke cascade control loop strategy. The state of the dry quenching cascade control loop at time i. For the dry quenching coke cascade control loop strategy network, The state of the dry quenching coke cascade control loop is s i At that time, the PID control parameters generated by the strategy network take action, s is the state of the dry quenching coke cascade control loop, and a is the PID parameter action of the dry quenching coke cascade control loop. The target network uses a soft update method to track and update the parameters of the learned dry quenching cascade control loop network. （14） In the formula, To update the target network parameters, The updated coefficients range from 0 to 1. For target network parameters; The Deep Deterministic Policy Gradient (DDPG) also employs an experience retrieval mechanism, which utilizes experience samples generated from the interaction between the dry quenching coke cascade control loop policy network and the dry quenching coke cascade control loop environment. The samples are stored in the experience pool, and during network training, experience samples are randomly selected from the experience pool to update the network parameters. S5. Optimal Parameter Distribution: The adjusted PID controller parameters of the dry quenching cascade control loop are distributed to optimize the actual dry quenching cascade control loop in real time.

Images