Internal mixing temperature iterative learning prediction control method based on reinforcement learning

Abstract

The invention discloses an internal mixing temperature iterative learning prediction control method based on reinforcement learning. The method comprises the following steps: S1, establishing an internal mixing process system temperature control model based on a reaction energy conservation principle; s2, establishing a system temperature control model in a state space expression form; s3, designing an iterative learning control law according to batch characteristics of the process, and establishing an iterative axis error model; s4, designing a two-dimensional system iterative learning model prediction control method, and determining a cost performance index function; s5, solving a standard quadratic programming problem about optimization variables; s6, setting a hyper-parameter of a Soft Actor-Critic algorithm, combining the trained strategy network with the output of the ILMPC controller, and acting on the system; and S7, updating network parameters according to the sample data in the cache region. According to the method, the control robustness of the control system under non-repeated disturbance and unknown dynamic change is optimized, and the stability of the rubber discharging temperature of the internal mixing process is effectively improved.

Description

Technical Field

[0001] The present invention relates to a rubber tire processing mixing process, and specifically but not limited to a mixing temperature iterative learning predictive control method based on reinforcement learning. Background Art

[0002] Internal mixing is an essential step in rubber processing, and its quality directly determines subsequent processing and the physical properties of the final product. Poor mixing not only impacts the proper functioning of processes like vulcanization but also reduces the quality, performance, and service life of rubber products. Rubber mixing is a typical rapid, intermittent production process, and traditional control methods struggle to achieve the desired results under varying material and process conditions. New. Due to the diverse parameters influencing the mixing process, process control and adjustment of these parameters to optimize mixing performance have become key areas for intelligent internal mixer control.

[0003] In actual production, the process parameters that primarily influence mixing quality include main engine power, ejector pressure, and rotor speed. While all parameters for a single batch are configured by the PLC system based on the selected compound formula, process parameters must be adjusted promptly for different formulas and batches. Controlling the discharge time solely by observing the discharge temperature still relies on manual adjustments made by the operator's experience, rendering traditional control methods ineffective. The mixing process is highly complex and nonlinear. Changes in the viscoelastic properties of the compound are primarily due to the intense mechanical friction within the mixing chamber, and the compound's plasticity increases as it heats up. Therefore, temperature is the most critical factor affecting discharge quality. Precise control schemes ensure smoother operation and more precise control of key process parameters, ensuring they remain within reasonable ranges. Integrating intelligent control algorithms with the actual mixing process and production requirements has become an effective technical approach for lean product quality control. Summary of the Invention

[0004] In order to address the problems of fast mixing process, lag, batch difference, etc., the actual values ​​of process parameters cannot accurately track the given targets. Based on the two-dimensional iterative learning model predictive control and reinforcement learning, dynamic performance indicators containing two-dimensional error information are introduced to solve the open-loop disadvantages of iterative learning in the time domain. The reinforcement learning algorithm effectively suppresses the parameter mismatch and non-repetitive external disturbances of the model, and uses data-driven capabilities to compensate for the adverse effects of errors, thereby improving the stability and robustness of the system. The discharge temperature of each rubber mixing batch can track the preset trajectory.

[0005] In order to achieve the technical purpose of the present invention, the technical solution adopted is:

[0006] A reinforcement learning-based iterative learning predictive control method for mixing temperature, the method comprising the following steps:

[0007] S1. Establish a temperature control model for the mixing process system based on the principle of reaction energy conservation;

[0008] S2. Establishing a system temperature control model in the form of state space expression;

[0009] S3. Design an iterative learning control law based on the batch characteristics of the process and establish an iterative axis error model;

[0010] S4. Design a two-dimensional system iterative learning model predictive control method and determine the cost-performance indicator function;

[0011] S5. Solve the standard quadratic programming problem with respect to the optimization variables;

[0012] S6. Set the hyperparameters of the Soft Actor-Critic algorithm, combine the trained policy network with the ILMPC controller output, and apply it to the system.

[0013] S7. Update network parameters according to the sample data in the buffer area.

[0014] To optimize the above technical solutions, specific measures taken also include:

[0015] The S1 establishes a temperature control model with the debinding temperature as the controlled object based on the thermodynamic principle of the reaction occurring in the mixing chamber:

[0016]

[0017] Where n is the rotor speed; K1 is the equivalent heat transfer coefficient; Q c is the cooling water flow rate; K2 is the thermal resistance coefficient of the flow rate; T1 is the temperature of the rubber compound in the mixing chamber, the equivalent heat capacity C3, T2 represents the temperature of the mixing chamber wall, the equivalent heat capacity C4; T3 represents the temperature of the cooling water in contact with the mixing chamber wall, the equivalent heat capacity C5; R6 represents the degree of resistance to heat transfer between the medium in the mixing chamber and the mixing chamber wall, R7 represents the degree of resistance to heat transfer between the mixing chamber wall and the contact part of the cooling water, R8 represents the degree of resistance to heat transfer between the mixing chamber wall and the air, R9 represents the degree of resistance to heat transfer between the cooling water in contact with the furnace wall and the cooling water in non-contact part. The values ​​of the above parameters are determined according to the material characteristics and equipment data of the actual production process.

[0018] Derive the state space model described in S2, taking the state variable x = [T1 T2 T3] T , control variable u=[n Q c ] T , at sampling time T s ,(x d ,u d ) The linearized and discretized state space expression at the point is:

[0019]

[0020] Where k is the number of iterations; are the state, input and output of the system respectively; f(·) is the batch feature v k The uncertainty non-repeating characteristics of (t). The coefficient matrix is ​​expressed as:

[0021]

[0022] A two-dimensional incremental generalized error model including the dynamic characteristics of the iteration domain and time domain is further derived:

[0023]

[0024] in

[0025]

[0026] δu k (t) = u k (t)-u k (t-1),w k (t) = f(x k (t),u k (t),ν k (t))-f(x k-1 (t),u k-1 (t),ν k-1 ).

[0027] Perturbation w k (t) is simulated as the difference between two consecutive evaluations of the function f(·), representing the state x k , control input u k and external parameter ν k This formulation captures the inherent non-repetitive, time-varying nature of the batch process. The above S3 establishes an error model on the batch axis to update the iterative learning control law:

[0028]

[0029] in:

[0030] Combining the error information of adjacent batch axes with model predictive control, we get the output prediction:

[0031]

[0032] in:

[0033]

[0034] δu k (t) = u k (t)-u k (t-1),w k (t) = f(x k (t),u k (t),ν k (t))-f(x k-1 (t),u k-1 (t),ν k-1 ),

[0035]

[0036] The error model between the predicted output and the actual output of the system is:

[0037] For repetitive disturbances:

[0038] Then the predicted value of the kth iteration error sequence at time t is:

[0039]

[0040] Determine the S4 cost performance indicator function:

[0041]

[0042] Among them, t represents the current time, R x and R u is a weight matrix of appropriate dimensions that describes the effect of state and input action on the cost function, using the optimality condition Solve for the control increment:

[0043]

[0044] In each time step, only the first control action in the calculated optimal sequence is used, so the control input signal at time t of the kth batch is:

[0045]

[0046] The parameters of the value network and policy network in the reinforcement learning Soft Actor-Critic (SAC) algorithm in S6 are further designed, and the temperature control model in S2 is used as the virtual interactive environment.

[0047] The core of reinforcement learning is to find the optimal strategy through the interaction between the agent and the environment. Assume that in the discrete time process, the environment state s t is input to the agent, and the resulting policy π(a t |st ) will select action a that acts on the environment state again t , transfer the current state to the next new state s t+1 , At the same time, the agent will receive a reward value r(t) for learning new strategies.

[0048] The state-action value function is used to measure the transition from state s t Start and take action t The expected cumulative reward is expressed as:

[0049]

[0050] Among them, γ represents the discount factor, α is the entropy factor, Indicates that in the strategy π(a t |s t ) under the expected value of the state-action, ρ π Represents the trajectory state s under the current strategy t and action a t probability.

[0051] Then define the objective function of the optimal strategy:

[0052]

[0053] in, is the strategy π(a t |s t By adjusting the entropy parameter, the randomness of the strategy can be changed, thereby changing the size of the strategy's exploration ability.

[0054] Introduce a parameterized policy network and use the mean square error minimization method to train the target network:

[0055]

[0056] in, is the distribution of previously sampled states and actions, or the replay buffer; θ' represents the target network parameters, Represents the state value function:

[0057]

[0058] The Q function is updated by minimizing the Bellman residual, while adding an entropy term to ensure exploration:

[0059]

[0060] Among them, Q soft (s t ,a t) represents the Q value to be updated, Indicates s t+1 and a t+1 The training process uses the target network θ i ' and the original network θ i By minimizing the policy network J π (φ) Training policy network:

[0061]

[0062] is the objective function J π (φ) Estimated gradient about parameter φ. According to the environment state space information in SAC algorithm The output of the policy network is used as the action of the agent, with a k (t)=π(s k (t)|φ) means that a k (t) and the iterative learning model predictive controller Δδu k (t) is combined and applied to the controlled process, and the new feedback state and immediate reward of the process are obtained:

[0063] In iterative predictive control, the goal of the reward function is to minimize the tracking error, thereby ensuring that the system output closely follows the desired reference trajectory. The reward at each time step can be defined as

[0064] r(s k (t),a k (t))=-l*‖e k (t)‖ 2

[0065] Among them, e k =y k (t)-y k-1 (t) is the tracking error of the kth batch at time t, representing the difference between the system output and the reference output, and l is a constant greater than zero. This reward function penalizes deviations from the reference trajectory, encouraging the strategy to minimize the tracking error. The parameters of the target network described in S5 are updated and optimized using the exponential moving average method:

[0066]

[0067] Where τ is the soft update rate, which is used to stabilize the training process of the target network.

[0068] The effect of the present invention is mainly manifested in the simultaneous consideration of the two-dimensional structure of time and batch. By combining the iterative learning control method and the generalized predictive control method, the rapidity and robustness of the reaction process of the batch bioreactor are comprehensively optimized. By adjusting the weight coefficient, the control performance of the batch bioreactor can be improved, and the rapid tracking control target of multiple batches under multiple disturbance conditions can be achieved.

[0069] The effect of the present invention is mainly reflected in the combination of iterative learning predictive control and reinforcement learning methods, which optimizes the stability and robustness of mixing temperature control, significantly improves the system's adaptability under non-repetitive disturbances and unknown dynamic changes, and effectively improves the tracking control performance of the mixing process debinding temperature. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide a further understanding of the present invention and, together with the description, to explain the embodiments of the present invention, but do not constitute a limitation of the present invention. In the accompanying drawings:

[0071] Figure 1 It is the basic framework diagram of temperature control in the internal mixing process of the present invention.

[0072] Figure 2 This is a schematic diagram of the principle of combining iterative learning model predictive control with reinforcement learning in the present invention.

[0073] Figure 3 This is a comparison chart of the control effects of the present invention and traditional iterative learning predictive control under repeated disturbances.

[0074] Figure 4 This is a comparison chart of the control effects of the present invention and traditional iterative learning predictive control under non-repetitive disturbance.

[0075] Figure 5 This is the learning curve of the agent under non-repeated perturbations.

[0076] Figure 6 This is a comparison chart of the tracking performance of the present invention and traditional iterative learning under non-repeated disturbance.

[0077] Figure 7 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0078] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0079] Combine Figure 7 , a reinforcement learning-based iterative learning predictive control method for mixing temperature, specifically including:

[0080] S1, such as Figure 1 The basic framework of the internal mixing process temperature control of the present invention is shown. The main component of the upper push bolt, the pressing block 1, is located above the feed hopper 2. The cooling water pipe 3 is installed inside the internal mixer wall. A motor 5 drives a pair of rotors 4 with different speed ratios to complete the main mixing process. After completion, the rubber material is flipped out of the internal mixing chamber through the lower push bolt 6 and discharged through the discharge door 7. The rotor speed is selected as the manipulated variable, and the temperature of the rubber material at the discharge port is the controlled variable. To implement the subsequent iterative learning predictive control algorithm, the differential equation needs to be converted into a state space expression, taking the state variable x = [T1 T2 T3] T , control variable u=[n Q c ] T , at sampling time T s ,(x d ,u d ) The linearized and discretized state space expression at the point is:

[0081]

[0082] Where k is the number of iterations; are the state, input and output of the system respectively; f(·) is the batch feature v k The uncertainty non-repeating characteristics of (t). The coefficient matrix is ​​expressed as:

[0083]

[0084] According to the actual reaction process principle and rubber characteristics, Table 1 gives the parameter values ​​in the above coefficient matrix:

[0085] Table 1 Process mechanism model parameters

[0086]

[0087] S2. Using the above kinetic model and its parameter values, a prediction model is constructed and the two-dimensional performance index function is solved:

[0088]

[0089] Prediction time domain n x =10, control time domain n u =10, weight coefficient R x =I,R u =0.1I. The control sequence is calculated by the optimization principle:

[0090]

[0091] S3, such as Figure 2As shown, the output of the iterative learning predictive controller is combined with the action of the reinforcement learning agent, and the training hyperparameters of the agent are set as shown in Table 2:

[0092] Table 2 Soft Actor-critic algorithm hyperparameters

[0093]

[0094] Based on the two-dimensional iterative learning model predictive control design, two different types of disturbances are added to the system: 1) Starting from the fourth batch, the cooling water temperature is increased by 1°C; 2) Starting from the fourth batch, random load noise with an amplitude in [-1, 1] is added. The dynamic response performance indicators of the introduced chemical learning system are analyzed and the effectiveness of the design scheme is evaluated. Figure 3 and Figure 4 The results show that the iterative learning model predictive control method based on reinforcement learning introduced in this paper has faster and more stable tracking performance as the batch size increases, and has superior robustness under non-repetitive, unknown noise disturbances compared to traditional iterative learning model predictive control. Figure 5 As shown in the figure, the tracking error decreases with the increase of batches, and the root mean square error and absolute error can converge to 0 after 20 batches, highlighting the superior tracking performance. Figure 6 The learning curve of this method under non-repeated interference is shown. The strategy proposed in this paper continuously explores the virtual environment, optimizes control performance, and achieves the maximum cumulative reward within about 20 training rounds, so that the reward value quickly approaches zero with high speed and accuracy.

Claims

1. A method for iterative learning and predictive control of mixing temperature based on reinforcement learning, characterized in that: The method acts on a rubber mixing system and specifically comprises the following steps: S1. Establish a temperature control model for the mixing process system based on the principle of reaction energy conservation; S2. Establishing a system temperature control model in the form of state space expression; S3. Design iterative learning control law according to the batch characteristics of the process and establish an iterative axis error model; S4. Design a two-dimensional system iterative learning model predictive control method and determine the cost-performance indicator function; S5. Solve the standard quadratic programming problem on the optimization variables; S6. Set the hyperparameters of the Soft Actor-Critic algorithm, combine the trained policy network with the ILMPC controller output, and apply it to the rubber mixing system. S7. Update network parameters according to the sample data in the buffer area.

2. The method according to claim 1, characterized in that: According to the energy conservation principle of the reaction in the mixing chamber, S1 establishes a system temperature control model with the debinding temperature as the controlled object: Where n is the rotor speed; K1 is the equivalent heat transfer coefficient; Q c is the cooling water flow rate; K2 is the thermal resistance coefficient of the flow rate; T1 is the temperature of the rubber compound in the mixing chamber, the equivalent heat capacity C3, T2 represents the temperature of the mixing chamber wall, the equivalent heat capacity C4; T3 represents the temperature of the cooling water in contact with the mixing chamber wall, the equivalent heat capacity C5; R6 represents the degree of resistance to heat transfer between the medium in the mixing chamber and the mixing chamber wall, R7 represents the degree of resistance to heat transfer between the mixing chamber wall and the contact part of the cooling water, R8 represents the degree of resistance to heat transfer between the mixing chamber wall and the air, R9 represents the degree of resistance to heat transfer between the cooling water in contact with the furnace wall and the cooling water in non-contact part. The values ​​of the above parameters are determined according to the material characteristics and equipment data of the actual production process.

3. The method according to claim 2, characterized in that The S2 is based on the linearization and discretization of differential equations to obtain the following state space model, where the rubber temperature, chamber wall temperature, and cooling water temperature are selected as state variables x = [T1T2 T3] T , select rotor speed and cooling water flow as control variables u=[n Q c ] T , get the status expression: Where k is the number of iterations; are the state, input and output of the system respectively; f(·) is the batch feature v k (t) is an uncertain non-repeating feature; T s is the sampling time; A, B, C are matrices of corresponding dimensions, and the expressions are as follows: Where T s is the sampling time, V2 is the inlet temperature of cooling water, x d 、u d are reaction equilibrium points, which are collected based on the equipment operation data of the internal mixing production process.

4. The method according to claim 3, characterized in that The S3 takes into account the non-repetitive characteristics of the intermittent process for different formulation specifications and different mixing stages of the internal mixing process, and establishes an iterative axis error model based on the axis differences of adjacent batches: in: k=1,2,3... represents the number of iteration batches, t=0,1,2,3... represents the sampling time of each batch; x k (t) represents the n of the kth batch at time t x dimensional state vector, u k (t) represents the n of the kth batch at time t u dimensional control vector, y k (t) represents the n of the kth batch at time t y dimensional output vector, A, B, C are coefficient matrices of corresponding dimensions; Δ represents the adjacent iteration axis error operator, and δ represents the adjacent time axis error operator.

5. The method according to claim 4, characterized in that The S4 is based on the two-dimensional system information of the time domain and the iteration domain to achieve the complementarity of the iteration domain performance and the time domain performance, and ensure the two-dimensional stability of the batch process control; First, define the temperature prediction output at N sampling points Iterative predictive controller input Δδu and external disturbance w k The sequence is as follows: w k (t)=f(x k (t),u k (t),ν k (t))-f(x k-1 (t),u k-1 (t),ν k-1 ), Combining the iterative axis error model and the system temperature control model, the prediction model is derived: in Represents the incremental form of each variable, is the initial state change along the batch axis, G and F are the input coefficient matrix and initial state coefficient matrix of the prediction model respectively, and the expression is: Based on the prediction model, a quadratic cost performance index is designed. The index consists of the error between the output reference trajectory and the predicted output, and the control input values ​​on the batch and time axes. It is used to measure the amplitude of the control input, the convergence robustness in the batch domain, and the dynamic stability in the time domain. A weight matrix is ​​added to each item, and the control performance function that is most suitable for the refining process is determined by adjusting the size of the weight coefficient: in, Represents the system state x k (t) Regarding the control input change Δδu k (t), t represents the current time, is the error sequence at the prediction time t+1, R x and R u is a weight matrix of appropriate dimension, used to quantify the relative influence of state deviation and control increment on the cost function; n x is the maximum time domain step size for the predicted state, n u is the maximum time domain step size of the predictive control input.

6. The method according to claim 5, characterized in that The optimality condition θJ / θΔδu is used in S5 k (t) = 0 to solve the control increment: In each time step, only the first control action in the calculated optimal sequence is used, so the control input signal at time t of the kth batch is:

7. The method according to claim 6, characterized in that The S6 designs the parameters of the value network and the policy network in the Soft Actor-Critic algorithm, interacts with the system temperature control model established by S1, and iteratively updates the network parameters; In the complex system of the mixing process, reinforcement learning is used as a key technical means to solve the problem of system temperature control. The key lies in determining the optimal temperature control strategy through the close interaction between the intelligent agent and the mixing environment. Given that the temperature control model of the mixing process system has been constructed in the form of a discrete state space, the actual output value of the debinding temperature, the set value, the control signal input, etc. can be selected to form the environmental state s t ; After these environmental states closely related to the refinement conditions are input into the agent, the agent generates the corresponding strategy π(a t |s t ), in order to clarify the optimal temperature control strategy, the objective function is defined: in, is the strategy π(a t |s t ) information entropy function, r(s t ,a t ) is the reward value of the environment for the current action, In the strategy π(a t |s t ) is the expected value of the state-action under the condition of α is the entropy factor; by adjusting the entropy parameter, the size of the strategy exploration ability can be changed, thereby transferring the current debonding temperature state to the next new state s t+1 ; The training process is done by minimizing the policy network J π (φ) Training policy network: π φ (a t ∣s t ) represents the strategy when the network parameter is φ, Denote the network parameter as θ i , i=1,2; In this process, the agent will also obtain a reward value r(t) closely related to the current working environment, based on which it continuously learns and updates network parameters and continuously optimizes strategies; To evaluate the current environment s t For action a t The reward value r(t) is designed, and the state-action value function is: Among them, γ represents the discount factor, α is the entropy factor, In the strategy π(a t |s t ) under the expected value of state-action, ρ π Represents the trajectory state s under the current strategy t and action a t The probability of ; the Q function is updated by minimizing the Bellman residual, while adding an entropy term to ensure exploration: Among them, Q soft (s t ,a t ) represents the Q value to be updated, In the strategy π(a t |s t ) Next action a t+1 The expected value of Indicates t+1 and a t+1 The target Q network is trained using the mean square error minimization method: in, is the distribution of previously sampled states and actions, or the replay buffer; θ' represents the target network parameters, represents the state value function; According to the environmental state space information in the SAC algorithm The output of the policy network is used as the action of the agent, with a k (t)=π(s k (t)|φ) means that, then, a k (t) and the highly optimized model predictive controller Δδu after multiple iterative learning k (t) are organically combined and precisely act on the controlled object of the mixing process; at the same time, the system will quickly feedback new status information and immediate rewards, thus forming a closed-loop, continuously optimized temperature control loop; in each control cycle, the intelligent agent can continuously adjust its own strategy network and value network parameters according to the latest feedback information to adapt to various working conditions that may occur in the mixing process, such as material batch differences, equipment performance fluctuations, and ambient temperature changes, etc., to ensure that the system temperature of the mixing process is always within the ideal control range, providing a solid and reliable technical guarantee for the high-quality production of mixing products.

8. The method according to claim 6, characterized in that In order to maintain the stability of the network during the training process, S7 adopts the exponential moving average EMA update strategy of the target network parameters for optimization; in this method, the parameters of the target network are not directly replaced by the parameters of the current network, but gradually approach the current network parameters to achieve a smoother and more stable update; this optimization process controls the update amplitude by introducing a decay factor τ, thereby reducing the volatility during the training process: Among them, θ Q represents the parameters of the Q network, θ′ Q is the parameter of the target Q network; φ π represents the parameters of the policy network, φ π ′ is the parameter of the target policy network; the update coefficient ranges from 0 to 1 to ensure that the target network parameters gradually approach the current network parameters without causing mutations.

Images