Adaptive PID temperature control algorithm based on reinforcement learning

Abstract

The invention discloses a self-adaptive PID temperature control algorithm based on reinforcement learning, and belongs to the technical field of self-adaptive temperature control. A temperature control strategy based on fuzzy PID is abstracted into the form of a neural network, weight parameters in the neural network are trained online in real time by means of a reinforcement learning strategy, and a training data set in the temperature control field, an execution strategy containing a temperature control algorithm module and an interaction step of the execution strategy and a temperature control element are obtained. Compared with traditional PID temperature control, after the neural network is introduced, control parameters can be adjusted through training and learning so as to deal with attenuation of temperature control stability caused by environment change and equipment aging.

Description

Technical Field

[0001] The invention belongs to the technical field of adaptive temperature control and relates to an adaptive PID temperature control algorithm based on reinforcement learning. Background Art

[0002] In the technical field of precision instruments and equipment, many system components are extremely sensitive to temperature, so stable control of system temperature is crucial to ensure the accuracy of the equipment. Traditional PID-based temperature control strategies have fixed control parameters and cannot be adjusted with changes in the environment or system characteristics. In harsh working conditions, the stability of temperature control will be greatly weakened. Secondly, the control parameters of the PID control strategy are mainly adjusted manually, which reduces development efficiency and makes it difficult to obtain the optimal parameter combination to achieve the best control performance. Summary of the invention

[0003] The present invention aims at solving the problems in the prior art and provides an adaptive PID temperature control algorithm based on reinforcement learning.

[0004] An adaptive PID temperature control algorithm based on reinforcement learning includes the following steps: abstracting the temperature control strategy based on fuzzy PID into the form of a neural network, using the strategy of reinforcement learning to train the weight parameters in the neural network in real time online, obtaining a training data set in the field of temperature control, and including the execution strategy of the temperature control algorithm module and its interaction steps with the temperature control element.

[0005] The advantages of the present invention are:

[0006] Compared with traditional PID temperature control, after the introduction of neural networks, the control parameters can be adjusted through training and learning to cope with the attenuation of temperature control stability caused by environmental changes and aging of the equipment itself.

[0007] In the temperature control scenario, it is difficult or even impossible to collect labeled samples for training neural networks as training sets. Therefore, using reinforcement learning strategies instead of traditional supervised learning strategies can reduce the dependence on pre-collected training sets. In addition, reinforcement learning strategies are online learning methods that can enhance the adaptive ability of temperature control systems and enable control parameters to better adapt to changes in external conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. As shown in the figure:

[0009] Figure 1 It is the overall framework diagram of the temperature control system of the present invention.

[0010] Figure 2 This is a diagram of the temperature control strategy based on reinforcement learning and its interaction process with the temperature control element of the present invention.

[0011] Figure 3 It is a structural diagram of the action strategy network based on fuzzy PID neural network of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] Example 1: Figure 1 , Figure 2 and Figure 3 As shown, an adaptive PID temperature control algorithm based on reinforcement learning enables the control system to adaptively adjust the control parameters according to the continuously updated system operation data until the requirements are met, so as to solve the problems of low efficiency of manual adjustment of control parameters and the inability of control parameters to adapt to changes.

[0014] The present invention is mainly applied to a PID-based temperature control system to solve the problem that the control stability of fixed PID parameters decreases after the system characteristics of the controlled system change due to environmental changes or aging of the system itself.

[0015] An adaptive PID temperature control algorithm based on reinforcement learning includes the following components or steps:

[0016] Step S1, step S1, as Figure 1 As shown in the figure, the preset temperature t_target is used as the control target of the output temperature, and the actual temperature is continuously collected and calculated. t with de t As the input quantity of temperature control calculation, the action execution quantity is output to the temperature control element through the calculation of the temperature control algorithm (including customized neural network and historical data) with specific control parameters. Under the influence of the action execution quantity, the output temperature of the temperature control element changes accordingly, constantly approaching the control target and finally remaining stable near the control target.

[0017] t_target is the target temperature that the system wants to control.

[0018] h tIt is the output temperature actually controlled by the temperature control element.

[0019] e t is the error between the target temperature and the actual temperature.

[0020] de t It is the difference between the current actual temperature and the actual temperature measured last time, which is equivalent to the differential of the actual temperature change.

[0021] a t It is the action value output by the temperature control algorithm, which is used to indicate the temperature control behavior of the temperature control element. Taking the thermoelectric cooler (TEC) as the temperature control element as an example, a t Should be the duty cycle of the pulse width modulation (PWM) driving the TEC.

[0022] Step S2: Figure 2 As shown, the execution strategy of the temperature control algorithm module and its interaction steps with the temperature control element.

[0023] Step (1), composition:

[0024] The continuous time will be discretized into multiple time steps in the algorithm calculation, each time step is counted as i, and the temperature h output by the temperature control element is t The calculated e t with de t The state quantity s i ;

[0025] The action strategy network π is a customized neural network for PID temperature control scenarios. It is used to make decisions based on the current state of the temperature control element and output the subsequent actions to be performed to the control element in the form of state quantity s i As input, the action execution amount a i is the output.

[0026] The value evaluation network Q is used to evaluate the action execution amount a i The supervision network of the execution effect is implemented by an ordinary fully connected multi-layer perceptron, which is based on s i and a i As input, the evaluation value is output. However, because this strategy uses the target network as a buffer transition network, the evaluation value of the network Q is not directly output to other components.

[0027] The historical data buffer stores the results of all the interactions between the temperature control algorithm and the temperature control element (s i ,a i ,r i+1 ,s i+1) as historical data samples. When the buffer is full, the new samples will overwrite the old samples according to the first-in-first-out (FIFO) principle. i+1 is a i The expected reward value under the action of execution volume represents the temperature control element at a i The expected effect of making the output temperature approach the target temperature under the action of , is obtained through a specific reward value algorithm, s i+1 Yes i The state quantity of the next time step of the current time step.

[0028] The action strategy target network π' and the value assessment target network Q' are the shadow networks of network π and network Q respectively. Their network structures are the same as those of network π and network Q respectively, and the weight parameters will approach the weight parameters of network π and network Q according to the soft update rules. The purpose of setting the target network and adopting the soft update rules is to slow down the weight update speed of network π and network Q to avoid unstable temperature control due to rapid mutation of weights.

[0029] Step (2): Basic interaction process between temperature control algorithm and temperature control element

[0030] When e t The value does not exceed a certain threshold e th , reinforcement learning is not performed, and the network weight parameters are not updated. Figure 2 The dotted data flow in the figure does not occur. At this time, only network π is in operation, and the calculation is performed according to the following formula:

[0031] a i =π(s i ) (Formula 1)

[0032] As the core of the temperature control algorithm, the network π interacts with the temperature control elements in real time, making the output temperature of the system approach the target temperature and keep it stable, while storing the data samples of each time step in the historical data buffer.

[0033] The soft update of network π' and network Q' is run normally until their weight parameters θ' and μ' are exactly the same as the weight parameters θ and μ of network π and network Q.

[0034] Step (3): Reinforcement Learning Process

[0035] When e t The value exceeds the threshold e thAfter that, reinforcement learning is started and the weight parameters of the network are updated and iterated. While the basic interaction process described in step (2) continues to run, other networks and the interaction process shown by the dotted line are carried out simultaneously. The temperature control algorithm module randomly takes out N samples from the historical data buffer according to specific rules for network training and learning.

[0036] Step (a): Weight update process of the value assessment target network Q

[0037] For each historical data sample, the action strategy target network π' directly receives s in the sample i+1 , and according to the calculation method of formula 1, calculate π'(s i+1 ) and outputs it to the value evaluation target network Q', which receives r from each buffer sample. i+1 With s i+1 , calculate the target evaluation value y according to formula 2 i :

[0038] y i =r i+1 +γQ'(s i+1 ,π'(s i+1 )) (Formula 2)

[0039] Among them, γ is the influence coefficient, which represents the influence of network Q' on the target evaluation value. The smaller the coefficient, the closer the target evaluation value is to the reward value r i+1 Subsequently, the value evaluation target network Q' converts the target evaluation value y i Output to the value evaluation network Q, the network Q will compare the target evaluation value yi with the real-time a i The mean square error of the evaluation value calculated by and si is used as the loss function to update its own weight parameter μ. The calculation formula of the mean square error is:

[0040]

[0041] The update of weight μ is based on the gradient of loss function to weight parameter implement.

[0042] Step (b), weight update process of action strategy network π

[0043] The network π updates θ by combining the gradient of the evaluation value calculated by the comprehensive network Q with the action execution amount and the gradient of the action execution amount with the network π's own weight parameter θ. The formula for the double gradient compound is:

[0044]

[0045] The update of weight θ is based on implement.

[0046] Step (4): Soft Update Process

[0047] The weight parameters of the action strategy target network π' and the value assessment target network Q' are not updated directly based on the gradient, but gradually approach the weights of the network π and the network Q based on the soft update rule after the weights of the network π and the network Q are updated. The soft update rule is:

[0048] μ'=τμ+(1-τ)μ'(Formula 5)

[0049] θ'=τθ+(1-τ)θ'(Formula 6)

[0050] τ is the soft update coefficient, which takes a value between 0 and 1. The closer it is to 0, the slower the update speed, and vice versa. When τ is 1, it is a direct update without soft update.

[0051] Step S3: Customized action strategy network for PID temperature control scenario

[0052] The action strategy network π is a neural network customized for PID temperature control scenarios, combining the advantages of fuzzy logic and neural networks. The composition of the neural network is as follows: Figure 3 shown.

[0053] Step (1), fuzzification and calculation process of state value L1

[0054] The error e and the differential of the error de are the state values ​​received by the action strategy network. At each time step, the L1 calculation first fuzzifies the two received state values ​​into seven fuzzy sets, namely, negative large (NB), negative medium (NM), negative small (NS), zero (ZO), positive small (PS), positive medium (PM), and positive large (PB). The fuzzification method is to calculate the membership of the state value to each fuzzy set based on the membership function, and obtain the vectors respectively. and

[0055] Step (2), calculation process L2

[0056] Since the membership function and its coefficients are fixed, in order to increase the adaptability of the membership calculation process, let the vector and Do element-wise product with adjustment factor vectors k1 and k2 respectively to get vector and The L2 calculation process can be described by the following formula:

[0057]

[0058] and are the membership value vectors of e and de, respectively, containing 7 elements. k1 and k2 are adjustment factor vectors, adding variable coefficients to the membership values, containing 7 elements. and It is the result vector of the L2 calculation process and the input vector of the L3 calculation process, each containing 7 elements.

[0059] because k1k2

[0060] ,, and are all vectors, so Formula 7 and Formula 8 add subscripts when describing these vectors to indicate that they are vectors with multiple elements.

[0061] As adjustment factor vectors, k1 and k2 are regarded as one of the weight parameters of the action policy network π, which are trained and updated during the reinforcement learning process.

[0062] Step (3), calculation process L3

[0063] The calculation process L3 describes two membership vectors and Fuzzy reasoning calculation between two 7-dimensional vectors: The elements of two 7-dimensional vectors are multiplied by two-by-two permutations and combinations to obtain a 49-dimensional vector.

[0064] The calculation process is shown in the following formula:

[0065]

[0066] m in f erence It is the result vector of the L3 calculation process and also the result vector of the fuzzy reasoning part. It contains 49 elements. The L1, L2, and L3 calculation processes are the fuzzy reasoning part of the action strategy network.

[0067] Step (4), calculation process L4

[0068] The result vector m of the fuzzy reasoning part inference After full connection multiplication and addition operation, the integrated result o is obtained total , the calculation process L4 is shown in the following formula:

[0069]

[0070] o total It is the result of the L4 process and the result of the integration part (the L4 process is the integration part of the action strategy network). total is the weight vector of the L4 process, containing 49 elements. totalis regarded as one of the weight parameters of the action policy network π, which is trained and updated during reinforcement learning.

[0071] Step (5), calculation process L5

[0072] (1) Obtain the base value K of the PID control parameter

[0073] Since the fuzzy part also performs normalized calculations, the calculation process L5 first amplifies and restores the result of fuzzy reasoning to obtain the base value K of the PID control parameter. The process is calculated according to the following formula:

[0074] K=o total *w r +K0(Formula 11)

[0075] K is the base value of the PID control parameter and is the intermediate result of the calculation process L5. K0 is the offset of the base value calculation. r is the magnification factor of the base value calculation. K and w r is regarded as one of the weight parameters of the action policy network π, which is trained and updated during reinforcement learning.

[0076] (2) Obtaining PID control parameters and action amount a

[0077] On the basis of the base value K, the adjustment factors w of the three control parameters are respectively p 、w i and w d Multiply to get the three basic control parameters K of PID p , K i and K d Finally, the action amount a output by the temperature control element is calculated based on the basic algorithm of PID control.

[0078] The basic algorithm of PID control:

[0079] a=K p *e+K i *Σ t e+K d *de(Formula 12)

[0080] K p , K i and K d These are the three basic control parameters of the PID process. p 、w i and w d K p , K i and K dThe adjustment factor is regarded as one of the weight parameters of the action strategy network π, which is trained and updated during reinforcement learning. In Formula 12, t is the discrete time step, and a is the final action output of the action strategy network π.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive PID temperature control algorithm based on reinforcement learning, characterized in that: The method includes the following steps: abstracting the temperature control strategy based on fuzzy PID into the form of a neural network, training the weight parameters in the neural network in real time online with the help of reinforcement learning strategy, obtaining a training data set in the field of temperature control, and including the execution strategy of the temperature control algorithm module and its interaction steps with the temperature control elements.

2. The adaptive PID temperature control algorithm based on reinforcement learning according to claim 1, characterized in that: The following steps are included: Taking the preset temperature t_target as the control target of the output temperature, the actual temperature is continuously collected and calculated e t with de t As the input of temperature control calculation, the action execution amount is output to the temperature control element through the temperature control algorithm with specific control parameters, including customized neural network and historical data calculation. Under the influence of the action execution amount, the output temperature of the temperature control element changes accordingly, constantly approaching the control target and finally remaining stable near the control target.

3. The adaptive PID temperature control algorithm based on reinforcement learning according to claim 1, characterized in that: The execution strategy of the temperature control algorithm module and its interaction steps with the temperature control element include the following steps: Step (1), composition: The continuous time will be discretized into multiple time steps in the algorithm calculation, each time step is counted as i, and the temperature h output by the temperature control element is t The calculated e t with de t The state quantity s i ; The action strategy network π is a customized neural network for PID temperature control scenarios. It is used to make decisions based on the current state of the temperature control element and output the subsequent actions to be performed to the control element in the form of state quantity s i As input, the action execution amount a i is the output; The value evaluation network Q is used to evaluate the action execution amount a i The supervision network of the execution effect is implemented by an ordinary fully connected multi-layer perceptron, which is based on s i and a i As input, the evaluation value is output. However, because this strategy uses the target network as a buffer transition network, the evaluation value of the network Q is not directly output to other components. The historical data buffer stores the results of all the interactions between the temperature control algorithm and the temperature control element (s i ,a i ,r i+1 ,s i+1 ) as historical data samples. When the buffer is full, the new samples will overwrite the old samples according to the first-in-first-out (FIFO) principle. i+1 is a i The expected reward value under the action of execution volume represents the temperature control element at a i The expected effect of making the output temperature approach the target temperature under the action of is obtained through a specific reward value algorithm; The action strategy target network π' and the value assessment target network Q' are the shadow networks of network π and network Q respectively. Their network structures are the same as those of network π and network Q respectively. The weight parameters will approach the weight parameters of network π and network Q according to the soft update rules. The purpose of setting the target network and adopting the soft update rules is to slow down the weight update speed of network π and network Q to avoid unstable temperature control due to rapid mutation of weights. Step (2): Basic interaction process between temperature control algorithm and temperature control element When e t The value does not exceed a certain threshold e th When , reinforcement learning is not performed, the network weight parameters are not updated, and the dotted data flow in Figure 2 does not occur. At this time, only the network π is in operation and is calculated according to the following formula: a i = π(s i ) (Formula 1) As the core of the temperature control algorithm, network π interacts with the temperature control element in real time to make the output temperature of the system approach the target temperature and keep it stable. At the same time, the data samples of each time step are stored in the historical data buffer. The soft update of network π' and network Q' runs normally until their weight parameters θ' and μ' are exactly the same as the weight parameters θ and μ of network π and network Q. Step (3): Reinforcement Learning Process When e t The value exceeds the threshold e th After that, reinforcement learning is started, and the weight parameters of the network are updated and iterated. While the basic interaction process described in step (2) continues to run, the interaction process shown by the other networks and the dotted line is carried out simultaneously. The temperature control algorithm module randomly takes out N samples from the historical data buffer according to specific rules for network training and learning. Step (4): Soft Update Process The weight parameters of the action strategy target network π' and the value assessment target network Q' are not updated directly based on the gradient, but gradually approach the weights of the network π and the network Q based on the soft update rule after the weights of the network π and the network Q are updated. The soft update rule is: μ'=τμ+(1-τ)μ'(Formula 5) θ'=τθ+(1-τ)θ'(Formula 6) τ is the soft update coefficient, which takes a value between 0 and 1. The closer it is to 0, the slower the update speed, and vice versa. When τ is 1, it is a direct update without soft update.

4. The adaptive PID temperature control algorithm based on reinforcement learning according to claim 3 is characterized in that: Step (3), the reinforcement learning process also includes the following steps: Step (a): Weight update process of the value assessment target network Q For each historical data sample, the action strategy target network π' directly receives s in the sample i+1 , and according to the calculation method of formula 1, calculate π'(s i+1 ) and outputs it to the value evaluation target network Q', which receives r from each buffer sample. i+1 With s i+1 , calculate the target evaluation value y according to formula 2 i : y i = r i+1 + γQ'(s i+1 , π'(s i+1 )) (Formula 2) Among them, γ is the influence coefficient, which represents the influence of network Q' on the target evaluation value. The smaller the coefficient, the closer the target evaluation value is to the reward value r i+1 , then, the value evaluation target network Q' converts the target evaluation value y i Output to the value evaluation network Q, the network Q will compare the target evaluation value yi with the real-time a i The mean square error of the evaluation value calculated by and si is used as the loss function to update its own weight parameter μ. The calculation formula of the mean square error is: The update of weight μ is based on the gradient of loss function to weight parameter implement, Step (b), weight update process of action strategy network π The network π updates θ by combining the gradient of the evaluation value calculated by the comprehensive network Q with the action execution amount and the gradient of the action execution amount with the network π's own weight parameter θ. The formula for the double gradient compound is: The update of weight θ is based on implement.

5. The adaptive PID temperature control algorithm based on reinforcement learning according to claim 2 is characterized in that: It also includes a customized action strategy network for PID temperature control scenarios. The action strategy network π is a neural network customized for PID temperature control scenarios, combining the advantages of fuzzy logic and neural networks. Step (1), fuzzification and calculation process of state value L1 The error e and the differential of the error de are the state values ​​received by the action strategy network. At each time step, the L1 calculation first fuzzifies the two received state values ​​into seven fuzzy sets, namely, negative large (NB), negative medium (NM), negative small (NS), zero (ZO), positive small (PS), positive medium (PM), and positive large (PB). The fuzzification method is to calculate the membership of the state value to each fuzzy set based on the membership function, and obtain the vectors respectively. and Step (2), calculation process L2 Since the membership function and its coefficients are fixed, in order to increase the adaptability of the membership calculation process, let the vector and Do element-wise product with adjustment factor vectors k1 and k2 respectively to get vector and The L2 calculation process can be described by the following formula: and are the membership value vectors of e and de, each containing 7 elements. k1 and k2 are adjustment factor vectors, which add variable coefficients to the membership values ​​and contain 7 elements. and is the result vector of the L2 calculation process and the input vector of the L3 calculation process, each containing 7 elements, because k1, k2, and are all vectors, so Formula 7 and Formula 8 add subscripts when describing these vectors to indicate that they are vectors with multiple elements. k1 and k2 as adjustment factor vectors are regarded as one of the weight parameters of the action strategy network π, which are trained and updated in the process of reinforcement learning. Step (3), calculation process L3 The calculation process L3 describes two membership vectors and Fuzzy reasoning calculation between: The elements of two 7-dimensional vectors are multiplied by two-by-two permutations and combinations to obtain a 49-dimensional vector. The calculation process is shown in the following formula: m inference It is the result vector of the L3 calculation process and the result vector of the fuzzy reasoning part. It contains 49 elements. The L1, L2, and L3 calculation processes are the fuzzy reasoning part of the action strategy network. Step (4), calculation process L4 The result vector m of the fuzzy reasoning part inference After full connection multiplication and addition operation, the integrated result o is obtained total , the calculation process L4 is shown in the following formula: o total is the result of the L4 process and the result of the integration part (the L4 process is the integration part of the action strategy network), w total is the weight vector of the L4 process, containing 49 elements, w total is regarded as one of the weight parameters of the action policy network π, which is trained and updated during reinforcement learning. Step (5), calculation process L5 (1) Obtain the base value K of the PID control parameter Since the fuzzy part also performs normalized calculations, the calculation process L5 first amplifies and restores the result of fuzzy reasoning to obtain the base value K of the PID control parameter. The process is calculated according to the following formula: K=o total *w r +K0 (Formula 11) K is the base value of the PID control parameter, which is the intermediate result of the calculation process L5. K0 is the offset of the base value calculation. r is the amplification factor of the base value calculation, K and w r is regarded as one of the weight parameters of the action policy network π, which is trained and updated during reinforcement learning. (2) Obtaining PID control parameters and action amount a On the basis of the base value K, the adjustment factors w of the three control parameters are respectively p 、w i and w d Multiply to get the three basic control parameters K of PID p , K i and K d Finally, according to the basic algorithm of PID control, the action amount a output by the temperature control element is calculated. The basic algorithm of PID control: a = K p *e + K i *Σ t e + K d *de (Equation 12) K p , K i and K d are the three basic control parameters of the PID process, w p 、w i and w d K p , K i and K d The adjustment factor is regarded as one of the weight parameters of the action strategy network π, which is trained and updated during reinforcement learning. In Formula 12, t is the discrete time step, and a is the final action output of the action strategy network π.

Images