AI model adaptive tuning method based on deep reinforcement learning

Abstract

The invention discloses an AI model adaptive tuning method based on deep reinforcement learning, and belongs to the technical field of industrial process control, the method comprises the following steps: firstly, constructing a deep reinforcement learning environment comprising a state space, an action space and a reward function, the state space comprising a process variable measurement value, a controller parameter and a performance index; the action space is a parameter adjustment quantity vector, and the reward function comprehensively considers the stability, rapidity and accuracy of the system; then, based on a deep Q learning network of a dual-network structure, training data are stored and sampled by adopting an experience playback mechanism, and network parameters are iteratively optimized through a deep Q learning algorithm; and finally, realizing online adaptive adjustment of controller parameters by using the optimized deep Q network. The method has the advantages of being high in adaptability, good in optimization performance, high in practicability and the like, intelligent optimization control over an industrial process control system can be achieved, and the overall performance of the system is improved.

Description

Technical Field

[0001] The present invention belongs to the field of industrial process control technology, and specifically relates to an AI model adaptive tuning method based on deep reinforcement learning. Background Art

[0002] At present, the PID parameter tuning methods commonly used in industrial sites mainly include: empirical tuning method, ZN tuning method, genetic algorithm and fuzzy control methods.

[0003] These traditional methods have the following problems: the empirical tuning method overly relies on the professional knowledge and field experience of operators, the tuning process is time-consuming, and it is difficult to ensure the optimality of parameters; the ZN tuning method requires the system to be adjusted to a critical state, which may bring safety hazards in actual industrial processes, and the obtained parameters often need further fine-tuning; although the genetic algorithm can achieve global optimization, it has a slow convergence speed and is prone to falling into local optimality, making it difficult to meet the real-time requirements of online adjustment; the control rule formulation of the fuzzy control method relies on expert experience, the process of establishing the rule base is complicated, and it is difficult to accurately describe complex nonlinear industrial processes;

[0004] Existing methods generally lack adaptive capabilities and find it difficult to cope with the impact of dynamic factors such as changes in process conditions and equipment characteristic drift, resulting in a decline in control performance. They often only focus on the optimization of a single performance indicator and find it difficult to achieve a comprehensive balance of multiple objectives, such as the coordinated optimization of stability, speed and accuracy. In addition, due to the lack of an intelligent online optimization mechanism, control parameters cannot be automatically adjusted according to real-time operating data, making it difficult for system performance to remain in the optimal state.

[0005] Therefore, there is an urgent need for an intelligent method that can adaptively adjust PID parameters and achieve multi-objective optimization to improve the overall performance of industrial process control systems. The AI ​​model adaptive tuning method based on deep reinforcement learning proposed in the present invention can solve the above problems and realize online adaptive optimization of controller parameters. Summary of the invention

[0006] The present invention provides an AI model adaptive tuning method based on deep reinforcement learning, which is used for online adaptive adjustment of controller parameters of industrial process control systems, and automatically adjusts control parameters according to real-time operating data to achieve the purpose of continuously maintaining system performance in an optimal state.

[0007] The scheme of the present invention is as follows:

[0008] An AI model adaptive tuning method based on deep reinforcement learning is used for online adaptive adjustment of controller parameters of an industrial process control system, the method comprising the following steps:

[0009] Step S1, for the industrial process control system, construct a deep reinforcement learning environment including a state space, an action space and a reward function.

[0010] Furthermore, the state space s t The expression is: t =[x t ,θ t ,p t ]; where x t is the process variable measurement vector, x t =[x 1,t , x 2,t , ..., x n,t ], x n,t is the measured value of the nth process variable at time t, n is the number of process variables; θ t is the controller parameter vector, θ t =[θ 1，t ,θ 2,t , ..., θ m,t ],θ m,t is the parameter of the mth controller at time t, m is the number of controller parameters; p t is the performance index vector, p t =[p 1，t , p 2,t , ..., p k,t ], p k,t is the value of the kth performance indicator at time t, where k is the number of performance indicators.

[0011] Furthermore, the action space a t is the parameter adjustment vector, expressed as: t =[Δθ 1,t ,Δθ 2,t , ..., Δθ m,t ]; where Δθ m,t Represents the adjustment amount of the mth controller parameter at time t.

[0012] Furthermore, the reward function r t The expression is: t =w1r stab +w2r speed +w3r acc , where r stab is the stability bonus, r stab =-α|MP t |, r speed is the speed bonus item, r speed =-β|t r |, r acc is the accuracy bonus, r acc =-γ|ess |; w1, w2, w3 are weight coefficients, satisfying w1+w2+w3=1, MP t is the overshoot, unit %, t r is the rise time, e ss is the steady-state error; α, β, γ are the influencing factors for balancing stability, rapidity and accuracy, α: [0.1, 1.0], β: [0.1, 1.0], γ: [0.1, 1.0].

[0013] Step S2, based on the deep Q learning network with a dual network structure, an experience replay mechanism is used to store and sample training data, and the network parameters are iteratively optimized through a deep Q learning algorithm.

[0014] Furthermore, the deep Q learning network includes an evaluation network and a target network. The evaluation network adopts a four-layer feedforward neural network, and the mathematical model is:

[0015] h 1 =ReLU(W 1 s t +b 1 );h 2 =ReLU(W 2 h 1 +b 2 );h 3 =ReLU(W 3 h 2 +b 3 );

[0016] Q=W 4 h 3 +b 4 ; Among them, W 1 , W 2 , W 3 , W 4 is the weight matrix, b 1 , b 2 , b 3 , b 4 is the bias vector;

[0017] The structure of the target network is the same as that of the evaluation network, and the parameters of the target network are copied from the evaluation network by soft updating; the input layer dimension is: n+m+k, (corresponding to the state space dimension); the output layer dimension is: m (corresponding to the action space dimension).

[0018] The use of the experience replay mechanism to store and sample training data and the iterative optimization of network parameters through the deep Q learning algorithm include:

[0019] Furthermore, an experience replay pool D is set up to store the transfer samples (s t , at , r t ,s t+1 ); randomly sample training data from the experience replay pool at a fixed batch size and use a greedy strategy for action selection:

[0020]

[0021] Calculate the temporal difference loss function: Among them, the target value is: is the expected value of the sample in the experience replay pool D.

[0022] Furthermore, the stochastic gradient descent method is used to update the evaluation network parameters: It is a gradient operator, and the network parameters are updated along the negative gradient direction of the loss function.

[0023] Furthermore, the target network parameters are soft-updated: θ′ Q ←τθ Q +(1-τ)θ′ Q ; Among them, η is the learning rate, ranging from [0.0001, 0.01], τ is the soft update coefficient, ranging from [0.001, 0.1], ε is the exploration rate, the initial value is in the range of [0.1, 0.9], and decreases with training; γ is the discount factor, ranging from [0.9, 0.99].

[0024] Step S3, based on the optimized deep Q network, realize online adaptive adjustment of controller parameters.

[0025] Specifically include:

[0026] Real-time collection of current status: s t =[x t ,θ t ,p t ];

[0027] Compute the optimal action by evaluating the network:

[0028] Update controller parameters:

[0029] Execute control and get the new state: s t+1 =[x t+1 ,θ t+1 ,p t+1 ].

[0030] Furthermore, the experience replay mechanism also includes: setting a fixed-capacity experience replay pool with a capacity range of [1000, 100000], using batch random sampling with a batch size range of [32, 256], and when the experience replay pool is full, using a first-in-first-out strategy to update the sample.

[0031] Furthermore, the number of hidden layer neurons of the evaluation network and the target network is: the first hidden layer: [64, 256] neurons, the second hidden layer: [32, 128] neurons, the third hidden layer: [16, 64] neurons; the number of neurons in the output layer is the same as the dimension of the action space.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention realizes intelligent optimization of controller parameters by constructing a deep Q learning network, which has the following significant advantages: it realizes online adaptive adjustment of control parameters without human intervention; the framework based on deep reinforcement learning can optimize multiple performance indicators at the same time, thereby improving the comprehensive performance of the system; the experience replay mechanism is adopted to improve the training efficiency and model generalization ability; the introduction of a safety protection mechanism ensures the reliability of parameter adjustment; the system has good scalability and can be extended to other industrial process control scenarios; the actual application effect is remarkable, effectively improving the control accuracy and reducing energy consumption and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of the AI ​​model adaptive tuning method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] like Figure 1 As shown, it is a flow chart of the AI ​​model adaptive tuning method based on deep reinforcement learning of the present invention, which is used for online adaptive adjustment of controller parameters of industrial process control systems. The method comprises the following steps:

[0037] Step S1, for the industrial process control system, construct a deep reinforcement learning environment including a state space, an action space and a reward function.

[0038] State space s t The expression is: t =[xt ,θ t , p t ]; where x t is the process variable measurement vector, x t =[x 1,t , x 2,t , ..., x n,t ], x n,t is the measured value of the nth process variable at time t, n is the number of process variables; θ t is the controller parameter vector, θ t =[θ 1，t ,θ 2,t , ..., θ m,t ],θ m,t is the parameter of the mth controller at time t, m is the number of controller parameters; p t is the performance index vector, p t =[p 1，t , p 2,t , ..., p k,t ], p k,t is the value of the kth performance indicator at time t, where k is the number of performance indicators.

[0039] Action Space a t is the parameter adjustment vector, expressed as: t =[Δθ 1,t ,Δθ 2,t , ..., Δθ m,t ]; where Δθ m,t Represents the adjustment amount of the mth controller parameter at time t.

[0040] Reward function r t The expression is: t =w1r stab +w2r speed +w3r acc , where r stab is the stability bonus, r stab =-α|MP t |, r speed is the speed bonus item, r speed =-β|t r |, r acc is the accuracy bonus, r acc =-γ|e ss |; w1, w2, w3 are weight coefficients, satisfying w1+w2+w3=1, MP t is the overshoot, unit %, t r is the rise time, e ssis the steady-state error; α, β, γ are the influencing factors for balancing stability, rapidity and accuracy, α: [0.1, 1.0], β: [0.1, 1.0], γ: [0.1, 1.0].

[0041] Take the temperature control system of a chemical plant as an example. The system includes a heating furnace and multiple temperature measurement points. In this embodiment, the process variable x t Contains real-time data of 5 measurement points (n=5), including furnace temperature, material temperature, etc.; controller parameter θ t It contains three parameters of PID controller: proportional coefficient, integral time and differential time (m=3); performance index p t It includes three indicators (k=3): temperature overshoot, rise time, and steady-state error. The system samples data every 500ms through the distributed control system (DCS).

[0042] In this temperature control system, the specific settings of the weight coefficients are: w_1 = 0.4, w_2 = 0.3, w_3 = 0.3, to balance the stability, rapidity and accuracy of the system. The influence factors are set as: α = 0.5, β = 0.3, γ = 0.2. This setting is suitable for fine chemical production processes that require high temperature control stability. When the temperature fluctuation exceeds ±2% of the set value, the stability reward item will give a large negative reward, prompting the system to adjust the control parameters to reduce the fluctuation.

[0043] Step S2, based on the deep Q learning network with a dual network structure, an experience replay mechanism is used to store and sample training data, and the network parameters are iteratively optimized through a deep Q learning algorithm.

[0044] The deep Q learning network includes an evaluation network and a target network. The evaluation network adopts a four-layer feedforward neural network, and the mathematical model is:

[0045] h 1 =ReLU(W 1 s t +b 1 );h 2 =ReLU(W 2 h 1 +b 2 );h 3 =ReLU(W 3 h 2 +b 3 );

[0046] Q=W 4 h 3 +b 4 ; Among them, W 1 ,W 2 ,W 3, W 4 is the weight matrix, b 1 ,b 2 ,b 3 ,b 4 is the bias vector;

[0047] The structure of the target network is the same as that of the evaluation network, and the parameters of the target network are copied from the evaluation network by soft updating; the input layer dimension is: n+m+k, (corresponding to the state space dimension); the output layer dimension is: m (corresponding to the action space dimension).

[0048] The use of the experience replay mechanism to store and sample training data and the iterative optimization of network parameters through the deep Q learning algorithm include:

[0049] Set up the experience replay pool D to store the transfer samples (s t ,a t ,r t ,s t+1 ); randomly sample training data from the experience replay pool at a fixed batch size and use a greedy strategy for action selection:

[0050]

[0051] Calculate the temporal difference loss function: Among them, the target value is: is the expected value of the sample in the experience replay pool D.

[0052] In order to improve training efficiency and model performance, this embodiment adopts the following training strategy:

[0053] 1) Using the pre-training method, first generate basic training data through the simulation model;

[0054] 2) Introduce a priority experience replay mechanism to give higher sampling probability to important samples;

[0055] 3) Use the target network double buffering technology to update the target network every 100 steps to improve training stability;

[0056] 4) Adopt a decaying exploration strategy, maintain a high exploration rate in the early stage of training to obtain more samples, and gradually reduce the exploration rate as training progresses to improve the certainty of the strategy.

[0057] Use stochastic gradient descent to update the evaluation network parameters: It is a gradient operator, and the network parameters are updated along the negative gradient direction of the loss function.

[0058] Soft update target network parameters: θ′ Q←τθ Q +(1-τ)θ′ Q ; Among them, η is the learning rate, ranging from [0.0001, 0.01], τ is the soft update coefficient, ranging from [0.001, 0.1], ε is the exploration rate, the initial value is in the range of [0.1, 0.9], and decreases with training; γ is the discount factor, ranging from [0.9, 0.99].

[0059] For this temperature control system, the specific structure of the evaluation network is: 11 neurons in the input layer (5 measurements + 3 controller parameters + 3 performance indicators), 128 neurons in the first hidden layer, 64 neurons in the second hidden layer, 32 neurons in the third hidden layer, and 3 neurons in the output layer (corresponding to the adjustment of 3 PID parameters). The use of the ReLU activation function can effectively avoid the gradient vanishing problem and improve the training effect of the network.

[0060] Step S3, based on the optimized deep Q network, realize online adaptive adjustment of controller parameters.

[0061] Specifically include:

[0062] Real-time collection of current status: s t =[x t ,θ t ,p t ];

[0063] Compute the optimal action by evaluating the network:

[0064] Update controller parameters:

[0065] Execute control and get the new state: s t+1 =[x t+1 ,θ t+1 ,p t+1 ].

[0066] The experience replay mechanism also includes: setting a fixed-capacity experience replay pool with a capacity range of [1000, 100000], using batch random sampling with a batch size range of [32, 256], and when the experience replay pool is full, using a first-in-first-out strategy to update the sample.

[0067] The number of hidden layer neurons of the evaluation network and the target network is: the first hidden layer: [64, 256] neurons, the second hidden layer: [32, 128] neurons, the third hidden layer: [16, 64] neurons; the number of neurons in the output layer is the same as the dimension of the action space.

[0068] In this temperature control system, the online adjustment of controller parameters takes 30 seconds as a cycle. At the beginning of each cycle, the system collects data from the most recent 60 sampling points to calculate performance indicators, and then calculates the parameter adjustment amount through the trained deep Q network. In order to ensure the safety of the control system, limits are set for the parameter adjustment amount: the adjustment range of the proportional coefficient is ±10%, the adjustment range of the integral time is ±15%, and the adjustment range of the differential time is ±20%. The actual application results show that this method can reduce the overshoot of temperature control by 40% and the steady-state error by 50%, which has obvious advantages over the traditional PID parameter tuning method.

[0069] The method of the present invention has been applied in the temperature control system of a heating furnace of a chemical enterprise for 6 months, and has achieved remarkable results. The temperature fluctuation has been reduced by 40%, and the steady-state error has been reduced by 50%. Energy consumption has been reduced by about 15% through optimized control. Due to the improved temperature control accuracy, the product qualification rate has been increased by 8%. The number of manual debugging times has been reduced, and the operation and maintenance costs have been reduced by about 20%.

[0070] The method of the present invention has good scalability and adaptability, and can be extended to the control of other process parameters such as flow, pressure, liquid level, etc.; it supports parameter optimization of multivariable coupling control systems; it can be integrated with the industrial Internet platform to achieve remote monitoring and optimization; it has transfer learning capabilities and can quickly adapt to similar process steps.

[0071] It should be noted that those skilled in the art should understand that various changes and equivalent substitutions may be made to the present invention without departing from the scope of the present invention. In addition, various modifications may be made to the present invention for specific situations or materials without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed, but should include all embodiments falling within the scope of the claims of the present invention.

Claims

1. An AI model adaptive tuning method based on deep reinforcement learning is used for online adaptive adjustment of controller parameters for industrial process control systems, characterized in that: The method comprises the following steps: Step S1, constructing a deep reinforcement learning environment including a state space, an action space and a reward function for an industrial process control system; Step S2, according to the deep Q learning network with a dual network structure, an experience replay mechanism is used to store and sample training data, and the network parameters are iteratively optimized through a deep Q learning algorithm; Step S3, based on the optimized deep Q network, realize online adaptive adjustment of controller parameters.

2. The AI ​​model adaptive tuning method based on deep reinforcement learning according to claim 1 is characterized in that: State space s t The expression is: t =[x t ,θ t ,p t ]; where x t is the process variable measurement vector, x t =[x 1,t ,x 2,t ,…,x n,t ], x n,t is the measured value of the nth process variable at time t, n is the number of process variables; θ t is the controller parameter vector, θ t =[θ 1,t ,θ 2,t ,…,θ m,t ],θ m,t is the parameter of the mth controller at time t, m is the number of controller parameters; p t is the performance index vector, p t =[p 1,t ,p 2,t ,…,p k,t ], p k,t is the value of the kth performance indicator at time t, where k is the number of performance indicators; Action Space a t is the parameter adjustment vector, expressed as: t =[Δθ 1,t ,Δθ 2,t ,…,Δθ m,t ]; where Δθ m,t represents the adjustment amount of the mth controller parameter at time t; Reward function r t The expression is: t =w1r stab +w2r speed +w3r acc , where r stab is the stability bonus, r stab =-α|MP t |, r speed is the speed bonus item, r speed =-β|t r |, r acc is the accuracy bonus, r acc =-γ|e ss |; w1, w2, w3 are weight coefficients, satisfying w1+w2+w3=1, MP t is the overshoot, unit %, t r is the rise time, e ss is the steady-state error.

3. The AI ​​model adaptive tuning method based on deep reinforcement learning according to claim 2 is characterized in that: The deep Q learning network includes an evaluation network and a target network. The evaluation network adopts a four-layer feedforward neural network, and the mathematical model is: h 1 =ReLU(W 1 s t +b 1 );h 2 =ReLU(W 2 h 1 +b 2 );h 3 =ReLU(W 3 h 2 +b 3 ); Q=W 4 h 3 +b 4 ; Among them, W 1 ,W 2 ,W 3 , W 4 is the weight matrix, b 1 ,b 2 ,b 3 ,b 4 is the bias vector; The structure of the target network is the same as that of the evaluation network, and the parameters of the target network are copied from the evaluation network by soft updating; The use of the experience replay mechanism to store and sample training data and the iterative optimization of network parameters through the deep Q learning algorithm include: Set up the experience replay pool D to store the transfer samples (s t ,a t ,r t ,s t+1 ); randomly sample training data from the experience replay pool at a fixed batch size and use a greedy strategy for action selection: Calculate the temporal difference loss: Among them, the target value is: Use stochastic gradient descent to update the evaluation network parameters: Soft update target network parameters: θ′ Q ←τθ Q +(1-τ)θ′ Q ; Among them, η is the learning rate, ranging from [0.0001, 0.01], τ is the soft update coefficient, ranging from [0.001, 0.1], ε is the exploration rate, the initial value is in the range of [0.1, 0.9], and decreases with training; γ is the discount factor, ranging from [0.9, 0.99].

4. The AI ​​model adaptive tuning method based on deep reinforcement learning according to claim 3 is characterized in that: The online adaptive adjustment of controller parameters based on the optimized deep Q network specifically includes: Real-time collection of current status: s t =[x t ,θ t ,p t ]; Compute the optimal action by evaluating the network: Update controller parameters: Execute control and get the new state: s t+1 =[x t+1 ,θ t+1 ,p t+1 ].

5. The AI ​​model adaptive tuning method based on deep reinforcement learning according to claim 4 is characterized in that: The experience replay mechanism also includes: setting a fixed-capacity experience replay pool with a capacity range of [1000, 100000], using batch random sampling with a batch size range of [32, 256], and when the experience replay pool is full, using a first-in-first-out strategy to update the sample.

6. The AI ​​model adaptive tuning method based on deep reinforcement learning according to claim 5 is characterized in that: The number of hidden layer neurons of the evaluation network and the target network is: the first hidden layer: [64, 256] neurons, the second hidden layer: [32, 128] neurons, the third hidden layer: [16, 64] neurons; the number of neurons in the output layer is the same as the dimension of the action space.

Images