Non-strategy output feedback optimal tracking control method for time-delay industrial process with unknown interference

Abstract

A non-strategy output feedback optimal tracking control method for a time-delay industrial process with unknown interference belongs to the field of advanced control of industrial processes, can overcome the problem that a conventional time-delay process control method excessively depends on a system model, and specifically comprises the following steps: step 1, describing a time-delay space expression form with interference, introducing a state increment and an output error to establish a new augmentation model; 2, introducing a time delay performance index, and designing a control strategy; 3, proposing a Q learning algorithm with fault-tolerant performance under a time delay condition, and solving an optimal control gain through continuous learning; according to the method, the optimal controller gain matrix can be found through continuous learning under the condition that a system accurate model is unknown, and a system output track can well track an expected value and achieve a good control effect under the condition that the system has state time delay and unknown interference.

Description

Technical Field

[0001] The invention belongs to the field of advanced industrial process control and relates to a non-strategy output feedback optimal tracking control method for a time-delay industrial process with unknown interference. Background Art

[0002] In a globalized and highly competitive market environment, all industries need to continuously improve production efficiency and product quality and reduce production costs to maintain competitiveness. To this end, industrial production methods that adopt advanced technologies such as digitalization, intelligence, and automation have become an indispensable trend. This production method can not only meet diverse production needs, but also achieve efficient use of resources and sustainable development of the environment.

[0003] In the actual industrial production process, time lag is common and inevitable. The existence of time lag in the control system will have a great impact on the industrial system, causing corresponding lags and fluctuations, and even causing system instability. The time lag phenomenon brings great difficulties to the optimal control and problems of modern industrial production.

[0004] In the injection molding production process, accurate model data is often difficult to obtain, and a large amount of historical data will be generated and stored. In the absence of an accurate process model, it is very important to effectively use historical data to directly design the controller of the injection molding production process. Therefore, data-dependent methods are more applicable in problems with time-delay interference. Since reinforcement learning has the advantage of optimizing the control of complex systems by relying solely on data when the precise model of the system is unknown, it has also been studied in the optimal tracking control of systems with state delays. For this reason, it is very important to study a control method based on reinforcement learning for industrial production processes with faults and delays, which can enable the system to not rely on the model and only rely on data to continuously learn to obtain the optimal control law. Summary of the invention

[0005] The present invention is a non-strategy output feedback optimal tracking control method for a time-delay industrial process with unknown interference, which belongs to the field of advanced control of industrial processes. The method overcomes the limitation of time-varying system dynamic parameters in traditional control methods. The specific steps are as follows: Step 1: Describe the spatial expression of time-delay with interference, introduce state increment and output error to establish a new augmented model; Step 2: Introduce time-delay performance indicators and design control strategies; Step 3: Propose a Q learning algorithm with fault-tolerant performance under time-delay conditions, and solve the optimal control gain through continuous learning. After multiple learning with the data generated by the injection molding process, the optimal controller gain of the injection molding process can be obtained, and the optimal control law under the performance indicator can be obtained through the controller gain, which can then act on the actuator control system to gradually track the output of the system to the set value.

[0006] The present invention is achieved through the following technical solutions:

[0007] Step 1: Describe the time-delay spatial expression with interference, introduce state increment and output error to establish a new augmented model;

[0008] First, consider a state-delay process with interference:

[0009]

[0010] Where k represents time, x k ,y k ,u k and ω k denote the system state, system output, control input and disturbance at time k, respectively. A, A d , B, C, and E represent system matrices of appropriate dimensions, x k , x k-1 and x k-d They represent the state variables at time k, time k-1 and time kd respectively, and d represents the delay at time k.

[0011] The new augmented model can be derived as follows:

[0012]

[0013] in, in and is a correlation matrix of appropriate dimension, x Δk =x k -x k-1 , x Δk+1 =x k+1 -x k ,y Δk =y k -y k-1 , x Δk-d =x k-d -x k-d-1 ,y Δk-d =y k-d -y k-d-1 , where x k-d-1 , x k+1 They represent the state variables at time kd-1 and time k+1 respectively, y k-d-1 and k-1 They represent the system output at time kd-1 and time k-1 respectively, and X k , X k-d ,u Δk and ω ΔkThey are the state variables of the new system model at time k, the state variables at time kd, the difference in control input and disturbance between time k and time k-1, and R n+1 is represented as a real number matrix, and n+1 is represented as a real number matrix R n+1 The appropriate dimension of α is the actuator fault gain. By changing the value of α, faults in different situations can be represented.

[0014] Step 2: Introduce time-delay performance indicators and design control strategies;

[0015]

[0016] Among them, Q1 and Q2 are state X i and X i-d The weight matrix, R is a positive definite matrix, which represents the weight of the control variable, and γ is the weight matrix of the external disturbance;

[0017] In order to ensure that the system output can track the required reference trajectory by finding the optimal control strategy, the control strategy can be expressed as:

[0018]

[0019] When the control strategy u Δk Optimal and interference ω Δk In the worst case, by comparing the value function and the Q function, it can be inferred that the value function and the Q function are equal.

[0020]

[0021] Define the following equation:

[0022]

[0023] Among them, λ r =X k-d+i -X k-d+(i-1) , by comparing the relationship between the time-delay value function and the time-delay Q function, the H matrix can be obtained as follows:

[0024]

[0025] in, H d1d1 =d, H 1d1 =0, H d1dd =0,H d1u =0,H d1ω =0, definition

[0026] f=G1+d 2 G3, b = -d 2 G3,

[0027] c=G 2 +(2d 2 -d)G;

[0028] Substituting formula (7) into formula (3), we can obtain:

[0029]

[0030] Step 3: A Q-learning algorithm with fault-tolerant performance under time-delay conditions is proposed to solve the optimal control gain through continuous learning;

[0031] The simplified optimal Bellman equation is as follows:

[0032]

[0033] According to the expression of Kronecker product, it can be rewritten as:

[0034]

[0035] in,

[0036]

[0037] The proposed algorithm has good convergence, and the delayed Q function with interference can be expressed in the following form:

[0038]

[0039] in, is the column vector form of the Kronecker product, and q represents the sum of the dimensions of the state input and the control input. h j+1 =v(H j+1 ) is the upper triangular element vector in the matrix H. Since and There is a linear correlation between them, and it is necessary to introduce noise e that is linearly independent of the three k1 and e k2 ensure as well as Linearly independent, the form is as follows:

[0040]

[0041] τ k Can be written as:

[0042]

[0043] h j+1 It can be derived from the following formula:

[0044]

[0045] in, The following formula can be derived:

[0046]

[0047] After continuous iteration, the following formula can be obtained:

[0048]

[0049] in, From this we can deduce that h j+1 With H j+1 The iterations of are equivalent;

[0050] H j+1 It can be written as follows:

[0051]

[0052] in, Therefore, H j+1 Use G j Iterative solution;

[0053] H j+1 Iteration process and G j The iterative process is similar. Substituting formula (18) into formula (19), we can get G j The expression is as follows:

[0054]

[0055]

[0056] Under the condition of initial iteration, as the number of iterations approaches infinity and satisfies the solution of the Riccati equation, the controller gain matrix solved by the proposed algorithm gradually converges to the optimal controller gain matrix. Therefore, as the number of iterations increases, the learned controller gain matrix and It will also gradually approach the optimal control gain matrix.

[0057] The advantages and effects of the present invention are:

[0058] This invention is mainly aimed at the time-delay process with actuator failure in this chapter, and proposes a new Q learning algorithm. This algorithm does not rely on the system model parameters, but only relies on the real-time data of the system to continuously learn to obtain the optimal controller gain matrix, and then obtain the optimal control law. The advantages of this algorithm are: on the one hand, it overcomes the problem of excessive dependence on the system model in previous time-delay process control methods; on the other hand, it is different from other data-driven control methods. This method can find the optimal controller gain matrix through continuous learning when the precise model of the system is unknown. And when there is state lag and unknown interference in the system, the system output trajectory can track the expected value well and have a good control effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is the convergence process of H matrix;

[0060] Figure 2 It is the convergence process of controller gains K1, K2, K3, K4;

[0061] Figure 3 Controls the tracking curve of the output y. DETAILED DESCRIPTION

[0062] In order to further illustrate the present invention, the present invention is described in detail below with reference to the accompanying drawings and examples, but they should not be construed as limiting the protection scope of the present invention.

[0063] Embodiment 1:

[0064] Injection molding is a typical industrial production process, which plays an important role in the plastic products industry. It is mainly composed of three stages: filling, packaging and cooling. The packaging stage is an important stage that determines the quality of the product, and the injection speed is one of the key variables, which affects the flow behavior of the melt in the cavity. In order to ensure product quality, the injection speed should be controlled within the given profile range. In this section, in order to verify that the non-strategy Q learning algorithm proposed in this paper has a good control effect in the time-delay process of actuator failure, in this section, taking injection molding as an example, the proposed control method is used to control the injection speed parameters in the injection molding production process.

[0065] Based on a large number of experiments, the response of the injection rate to the proportional valve was identified as an autoregressive model, which was converted into a state space model as shown in equation (21):

[0066]

[0067] Here, after a large amount of test data, it can be determined that the parameters of the injection stage performance index Q1 = Q2 = diag [2,2,2], R = 1, α = 0.8, the output setting value is y d=40. Given the initial H matrix, Figure 1 and 2 We can find that in either case, after multiple iterations of learning, the matrix H j , and Gradually converge to the optimal matrix H, and Figure 3 is the system output curve. It can be seen that the expected output trajectory can be tracked as time increases. This proves the effectiveness and feasibility of the proposed method.

[0068] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the control method can be simply improved and modified. These improvements and improvements should also be regarded as the scope of protection of the present invention.

Claims

1. A non-strategy output feedback optimal tracking control method for a time-delayed industrial process with unknown disturbances, characterized in that: The specific steps are as follows: Step 1: Describe the time-delay spatial expression with interference, introduce state increment and output error to establish a new augmented model; First, consider a state-delay process with interference: Where k represents time, x k ,y k ,u k and ω k denote the system state, system output, control input and disturbance at time k, respectively. A, A d , B, C, and E represent system matrices of appropriate dimensions, x k , x k-1 and x k-d They represent the state variables at time k, time k-1 and time kd respectively, and d represents the delay at time k; The new augmented model can be derived as follows: in, in and is a correlation matrix of appropriate dimension, x Δk =x k -x k-1 , x Δk+1 =x k+1 -x k ,y Δk =y k -y k-1 , x Δk-d =x k-d -x k-d-1 ,y Δk-d =y k-d -y k-d-1 , where x k-d-1 , x k+1 They represent the state variables at time kd-1 and time k+1 respectively, y k-d-1 and k-1 They represent the system output at time kd-1 and time k-1 respectively, and X k , X k-d ,u Δk and ω Δk They are the state variables of the new system model at time k, the state variables at time kd, the difference in control input and disturbance between time k and time k-1, and R n+1 is represented as a real number matrix, and n+1 is represented as a real number matrix R n +1 The appropriate dimension of α is the actuator fault gain. By changing the value of α, faults in different situations can be represented. Step 2: Introduce time-delay performance indicators and design control strategies; Among them, Q1 and Q2 are state X i and X i-d The weight matrix, R is a positive definite matrix, which represents the weight of the control variable, and γ is the weight matrix of the external disturbance; In order to ensure that the system output can track the required reference trajectory by finding the optimal control strategy, the control strategy can be expressed as: When the control strategy u Δk Optimal and interference ω Δk In the most serious case, the strategy obtained at this time is the optimal strategy. By comparing the value function and the Q function, it can be inferred that the values ​​of the value function and the Q function are equal. Define the following equation: Among them, λ r =X k-d+i -X k-d+(i-1) , by comparing the relationship between the time-delay value function and the time-delay Q function, the H matrix can be obtained as follows: in, H d1d1 =d, H 1d1 =0, H d1dd =0,H d1u =0,H d1ω =0, definition f=G1+d 2 G3, b=-d 2 G3, c=G 2 +(2d 2 -d)G; Substituting formula (7) into formula (3), we can obtain: Step 3: A Q-learning algorithm with fault-tolerant performance under time-delay conditions is proposed to solve the optimal control gain through continuous learning; The simplified optimal Bellman equation is as follows: According to the expression of Kronecker product, it can be rewritten as: in, The proposed algorithm has good convergence, and the delayed Q function with interference can be expressed in the following form: in, is the column vector form of the Kronecker product, q represents the sum of the dimensions of the state input and the control input, and h j+1 =v(H j+1 ) is the upper triangular element vector in the matrix H, because and There is a linear correlation between them. At this time, it is necessary to introduce noise e which is linearly independent of the three. k1 and e k2 ensure as well as Linearly independent, the form is as follows: τ k Can be written as: h j+1 It can be derived from the following formula: in, The following formula can be derived: After continuous iterations, the following formula can be obtained: in, From this we can deduce that h j+1 With H j+1 The iterations of are equivalent; H j+1 It can be written as follows: in, Therefore, H j+1 Use G j Iterative solution; H j+1 Iteration process and G j The iterative process is similar. Substituting formula (18) into formula (19), we can get G j The expression is as follows: Under the condition of initial iteration, as the number of iterations approaches infinity and satisfies the solution of the Riccati equation, the controller gain matrix solved by the proposed algorithm gradually converges to the optimal controller gain matrix. Therefore, as the number of iterations increases, the learned controller gain matrix and It will also gradually approach the optimal control gain matrix.

Images