Feed-forward feedback synchronous reinforcement learning control method under feed-forward bidirectional association

Abstract

The invention relates to the technical field of automatic control, and discloses a feed-forward feedback synchronous reinforcement learning control method under feed-forward bidirectional association, comprising the following steps: establishing an inverse disturbance model and a composite disturbance model of control output to disturbance, and obtaining a system dynamics model under feed-forward bidirectional association; converting a set tracking problem into a system error framework, and decomposing a component caused by disturbance from an error generated by state self-transfer as an intermediate variable; system errors and intermediate variables are broadened to new states, a feedback controller is designed based on the broadened state to ensure limited stability, and the residual stability problem is converted into an augmented system state convergence problem only related to feedforward control output; under the environment that feed-forward control serves as an action and an augmented system state serves as feedback, the optimal augmented state under the optimal action is obtained through the DQN reinforcement learning algorithm, then the optimal augmented state is connected to the optimal feedback controller for output, the control effect is better, and the solving speed is higher.

Description

Technical Field

[0001] The present application relates to the field of automatic control technology, and specifically discloses a feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association. Background Art

[0002] Feedforward feedback control is a widely used control system structure, which has been widely used in the fields of industrial control, mechanical control, and intelligent driving. The feedback control part is the basis of control system design. It can collect adjustment errors and perform corresponding actions according to the feedback control rate to continuously reduce tracking errors. Feedforward control is used to overcome the negative effects of external disturbances. By monitoring the changes in external disturbances, the output of the controller is compensated in advance to achieve the effect of quickly offsetting disturbances. In actual applications, disturbances are universal and complex, and the characteristics of different disturbances require the design of specific feedforward control methods.

[0003] Normally, the disturbance comes entirely from the outside of the system, that is, it follows the direction of the external influence on the system. Feedforward control only needs to be designed according to the characteristics of the external disturbance to achieve the goal. However, when the adjustment of the actuator inside the system can also affect the change of the external disturbance, the feedforward variable has both the direction of the external influence on the system and the direction of the system's internal influence on the external disturbance, that is, the feedforward is bidirectionally associated. This phenomenon is widely present in many actual control processes. For example, the abnormal steam pressure fluctuation provided by the boiler is regarded as an external disturbance of the internal pressure of the rear-end heat exchanger. By adjusting the valve to offset the disturbance, it will also cause changes in the external steam pressure; the abnormal pressure provided by the fan in the sewage treatment process is regarded as an external disturbance of the pressure of the multi-channel aeration pipeline. By adjusting the valve to offset the disturbance, it will also affect the pressure change of the fan. In this case, the existing feedforward control design method is no longer applicable, and the controller design problem under the feedforward bidirectional association faces challenges. The main challenges are: Challenge 1. Feedforward control is used to improve the system's ability to quickly adjust when a disturbance occurs, and feedback control is used to solve the problem of stable tracking. The incompatible characteristics between rapidity and stability make the two controllers have a trade-off between performance. Challenge 2: Solving the coordination problem between the performance of the two controllers requires the use of optimization algorithms or reinforcement learning algorithms. Since the decision variables are two-dimensional and have unquantifiable internal correlations, the burden of solving the problem is increased.

[0004] Therefore, in view of this, the inventor provides a feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association to solve the above problems. Summary of the invention

[0005] The object of the present invention is to provide a feedforward feedback synchronous reinforcement learning control method with better control effect and faster solution speed under feedforward bidirectional association.

[0006] In order to achieve the above object, the basic scheme of the present invention provides a feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association, comprising the following steps:

[0007] Step 1, establish an inverse disturbance model and a composite disturbance model of the control output to the disturbance, and obtain a system dynamics model under feedforward bidirectional correlation;

[0008] Step 2: Transform the setting tracking problem into the system error framework, and decompose the disturbance-caused component from the error caused by the state self-transition as an intermediate variable;

[0009] Step 3: Augment the system error and intermediate variables into new states, design a feedback controller based on the augmented states to ensure limited stability, and transform the remaining stability problem into an augmented system state convergence problem related only to the feedforward control output;

[0010] Step 4: In an environment where feedforward control is used as the action and the augmented system state is used as the feedback, the DQN reinforcement learning algorithm is used to obtain the optimal augmented state under the optimal action, and then the optimal augmented state is connected to the optimal feedback controller output.

[0011] Furthermore, in step 1, the process in which the controller output affects the feedforward is an inverse disturbance model, and the influence of the composite disturbance composed of the inverse disturbance and the external disturbance on the system output is a composite disturbance model. The controlled model, the inverse disturbance model and the composite disturbance model are used as sub-models to construct a system dynamics model under the feedforward bidirectional association.

[0012] Furthermore, the state space model of the system dynamics model under the feedforward bidirectional association is expressed as follows:

[0013]

[0014] In the formula, represents the state in the state space model, x1(t) represents the controlled model state, x2(t) represents the inverse disturbance model state, x3(t) represents the composite disturbance model state, u(t) = u g (t)+u e (t), represents the controller output, and the feedforward controller output u g (t) and the sum of the feedback controller output u e (t) composition, Represents a composite disturbance, and is composed of an external disturbance g(t) and the inverse disturbance model output C2x2(t), C2 represents the output matrix of the inverse disturbance model from state to output, y(t) is the output of the system, x(t+1) is the new state after a state transition occurs once, and A, B, C, and E are the state space model parameters respectively.

[0015] Further, the step 2 includes the following sub-steps:

[0016] Step 2.1, using the state space model to transform the system tracking set value and system output into the error system framework to obtain the error system model;

[0017] Step 2.2, in the error system model, obtain the integral form of the influence of the state self-transition process on the error system by recursion;

[0018] Step 2.3, from the integral form, decompose the component caused by the disturbance as an intermediate variable, and further push it to the differential form.

[0019] Further, the error system model expression in step 2.1 is as follows:

[0020] e(t+1)=r(t+1)-y(t+1)

[0021] =r(t+1)-Cx(t+1)

[0022] =r(t+1)-[CAx(t)+CB(u e (t)+u g (t))+CEg(t)]

[0023] In the formula, e(t+1) represents the system error at time t+1, CBu e (t), CBu g (t) and CEg(t) are the effects of the feedback controller, feedforward controller and external disturbance on the error system at time t, respectively, and CAx(t) is the effect of the state self-transition process on the error system;

[0024] The expression of step 2.2 is as follows:

[0025]

[0026] In the formula, i represents the summation parameter;

[0027] In step 2.3, the component caused by the disturbance is decomposed as the expression of the intermediate variable as follows:

[0028]

[0029] The expression reduced to differential form is as follows:

[0030]

[0031] In the formula, Stands for pseudo-inverse operation.

[0032] Further, the step 3 includes the following sub-steps:

[0033] Step 3.1, expand the system error and intermediate variables into the new state expression as follows:

[0034]

[0035] Where e(t) is the system error, is an intermediate variable;

[0036] Step 3.2, the expression of the feedback controller designed based on the augmented state is as follows:

[0037]

[0038] In the formula, κ1 and κ2 are controller parameters respectively;

[0039] Step 3.3, based on the Lyapnuov error energy function Analyze the stability of the error system, and the differential form of the energy function after substituting it into the feedback controller is:

[0040]

[0041] Where ΔV e (e(t)) = e T (t+1)e(t+1)-e T (t)e(t) represents the difference form;

[0042] Based on the energy function of Lyapnuov intermediate variable system Analyze the stability of the intermediate variables, and the differential form of the energy function after substituting the intermediate variables is:

[0043]

[0044] In the formula, represents the differential form;

[0045] Based on Lyapnuov overall system energy function Analyze the stability of the augmented state, the differential form of the overall energy function after substituting into the first two equations is:

[0046]

[0047] In the formula,

[0048] represents the differential form;

[0049] Step 3.4, transform the differential form of the overall system energy function into an augmented system, and from the controller perspective, the input of the augmented system is only related to the feedforward control u g(t) is related to the convergence problem of the system, which is transformed into the convergence problem of the augmented system state:

[0050]

[0051] In the formula, ζ 1,1 (t) = e(t), which represents the first row and first column element of the augmented state. A nonlinear form representing the first row and first column elements of the augmented state.

[0052] Further, the step 4 includes the following sub-steps:

[0053] Step 4.1: Use feedforward control as the action of reinforcement learning, use the augmented system state as the feedback of the DQN reinforcement learning algorithm, and calculate the state value function The expression is as follows:

[0054]

[0055] In the formula, u g (t) is the feedforward control, is the augmented system state, π k represents the reinforcement learning strategy executed into the environment, represents the comprehensive reward, R t represents the state reward obtained by executing the strategy at time t, It represents the predicted state reward from time t+1 to the end of strategy execution. In the strategy π k Get the state reward G at the next t moment t The conditional expectation of

[0056] Step 4.2: Calculate the action-state value function based on the state value function The expression is as follows:

[0057]

[0058] In the formula, γ represents the discount factor, R (t) is the reward function, Represents a discount reward, In the strategy π k Get the discount status reward at the next t moment The conditional expectation of

[0059] Step 4.3, use the online deep neural network and the target deep neural network to replace the action-state value function, perform a synchronization operation on the two network parameters at a fixed period, and construct the objective function for updating the network parameters as follows:

[0060]

[0061] In the formula, θ i are the online deep neural network parameters, is the target deep neural network parameter, i represents the number of parameter updates using the objective function, L(θ i ) represents the parameter θ i The objective function is Ak Represents a set of actions, represents the action-state value function calculated by the target deep neural network, represents the action-state value function calculated by the online deep neural network, Indicates action u g (t) maximize the action-state value function calculated by the target deep neural network;

[0062] Step 4.4, use the trained online deep neural network for the decision-making process of the feedforward controller output:

[0063]

[0064] In the formula, represents the output of the feedforward controller that maximizes the value of the action state calculated by the online deep neural network;

[0065] Step 4.5, obtain the optimal feedforward control output u g (t), corresponding to the optimal augmented system state Then connected to the optimal feedback control output u e (t), thereby synchronously obtaining the output of the feedforward feedback controller.

[0066] Furthermore, the reward function R in step 4.2 (t) for:

[0067] R(t)=ω a A a (t)+τ s V s (t)

[0068] In the formula, ω a and τ s are constant coefficients, represents the reward obtained by the error tracking effect, where e mr and e max Indicates the upper and lower bounds of the error allowed, D a is a constant, represents the reward for the convergence speed of the augmented state, where D v is a constant.

[0069] Based on the same inventive concept, the present invention provides a feedforward feedback synchronous reinforcement learning control system under feedforward bidirectional association, including a storage module, wherein the storage module includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes each step of the feedforward feedback synchronous reinforcement learning control under the above-mentioned feedforward bidirectional association.

[0070] Based on the same inventive concept, the present invention provides an electronic device, including the above-mentioned feedforward feedback synchronous reinforcement learning control system under a feedforward bidirectional association.

[0071] Compared with the prior art, the present invention has the following effects:

[0072] 1. In the present invention, in step 1, a system dynamics model under feedforward bidirectional association is established, in step 2, the influence form of the disturbance is separated from the self-transfer part of the error system state as an intermediate variable, in step 3, the intermediate variable and the system error are augmented into a new state, a feedback controller is designed based on the augmented state to ensure limited stability, an augmented system is formed by the augmented state and the input containing only the feedforward controller, and the convergence of the augmented system is equivalent to the convergence of the system, in step 4, the feedforward controller is learned by using the DQN reinforcement learning algorithm, and is synchronously connected to the feedback controller through the augmented state, so that the optimal trade-off problem between the feedforward control and feedback control performance can be solved with a lower solution burden, and the control effect can be improved.

[0073] 2. The present invention realizes the connection path from the feedforward controller learning to the augmented state determination and then to the feedback controller determination by constructing an augmented system, reducing the decision-making problem of the two controllers to the dimension of one controller learning, reducing the number of variables to be learned, and reducing the burden of solving. In addition, in the process of optimizing the decision of the feedforward control through reinforcement learning, the feedback controller can ensure limited stability, further reducing the action space that needs to be explored by reinforcement learning, further accelerating the learning process, and being able to solve the optimal trade-off problem between the feedforward control and feedback control performance with a lower burden of solving, and improving the control effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 A schematic diagram showing the association between the boiler and the heat exchanger;

[0076] Figure 2A schematic flow chart of a feedforward feedback synchronous reinforcement learning control method under a feedforward bidirectional association proposed in an embodiment of the present application is shown;

[0077] Figure 3 The effect comparison diagram of the invention and the traditional control method in the design of the heat exchanger pressure control system is shown, wherein (a) is a comparison diagram of the invention and the feedforward based on the predictive control design, (b) is a comparison diagram of the invention and the feedforward based on the optimization control design, (c) is a comparison diagram of the invention and the proportional control feedforward (K=0.4), and (d) is a comparison diagram of the invention and the proportional control feedforward (K=0.8);

[0078] Figure 4 A comparison diagram of the effects of the invention and the reinforcement learning control method of independent learning of the controller in the design of a heat exchanger pressure control system is shown, wherein (a) is a comparison diagram of the average rewards of the invention and independent learning of the controller, and (b) is a training comparison diagram of independent learning of the controller. DETAILED DESCRIPTION

[0079] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0080] A feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association is used to improve the control effect of feedforward feedback control under feedforward bidirectional association and the speed of solving the controller. The design of the heat exchanger pressure control system with the steam pressure provided by the boiler as the disturbance is taken as an example. The equipment schematic diagram is shown in Figure 1 As shown in the figure, the hot steam generated by the boiler is transported to the inlet of the heat exchanger through the steam pipe network. The heat exchanger uses the hot steam to physically exchange heat with the material. The internal pressure of the heat exchanger is P I It is the controlled parameter index, the steam network pressure P provided by the boiler E It is a parameter index of the previous stage. When the boiler production conditions or operation mode change, the pipe network pressure P E Fluctuations occur, with the pipe network pressure P E As an example, based on the principle of feedforward control, the heat exchanger inlet valve quickly takes measures to reduce the opening to suppress the disturbance on the internal pressure P I However, at the same time, reducing the opening of the heat exchanger inlet valve will reversely increase the pipe network pressure P E , which in turn causes the heat exchanger inlet valve to decrease, and the network pressure P E The descent will have a similar phenomenon.

[0081] In this process, traditional feedforward control will cause one-way excessive adjustment of the heat exchanger inlet valve, which will suppress the disturbance while affecting the disturbance change in reverse. The excessive adjustment of the feedforward control will also affect the output of the feedback controller. The feedforward controller and the feedback controller will continue to work independently, not only failing to achieve mutual coordination, but will even worsen the control effect of the system. This phenomenon of bidirectional feedforward correlation brings difficulties to the design of the control system.

[0082] In view of the control system design problem under feedforward bidirectional association, this embodiment adopts the following steps for design:

[0083] Step S1, define the influence of heat exchanger inlet valve adjustment on pipe network pressure P E The process is the inverse disturbance model, and the inverse disturbance and pipe network pressure P are defined. E Composite disturbance The impact on the system output is a composite disturbance model. The heat exchanger internal pressure controlled model, the inverse disturbance model and the composite disturbance model are used as sub-models to jointly constitute the system dynamics model under feedforward bidirectional association. Its state space model form is defined as:

[0084]

[0085] In the formula, represents the state in the state space model, x1(t) represents the controlled model state, x2(t) represents the inverse disturbance model state, x3(t) represents the composite disturbance model state, u(t) = u g (t)+u e (t), represents the controller output, and the feedforward controller output u g (t) and the sum of the feedback controller output u e (t) composition, Represents a composite disturbance, and is composed of an external disturbance g(t) and the inverse disturbance model output C2x2(t), C2 represents the output matrix of the inverse disturbance model from state to output, y(t) is the output of the system, x(t+1) is the new state after a state transition occurs once, and A, B, C, and E are the state space model parameters respectively.

[0086] In such Figure 1 In the embodiment shown, the sub-models are all set as first-order models, the output coefficient of the inverse disturbance model from state to output is set to C2=1, and the system matrix, control matrix, output matrix and disturbance matrix in the state space model are respectively:

[0087]

[0088] The experimental time is set to 150 minutes, and the internal pressure of the heat exchanger is P I The target setting is 0.43Mpa, and the pipe network pressure PE A step disturbance of 0.5 MPa was generated at the 75th minute.

[0089] Step 2: Transform the setting tracking problem into the system error e(t) framework, and decompose the disturbance-caused component from the error generated by the state self-transition as an intermediate variable. Specifically include:

[0090] Step 2.1, Heat exchanger internal pressure P I The target is set to r(t) is 0.43Mpa for any time t, e(t+1) represents the system error at time t+1, and it is transformed into the error system framework using the state space model, then the error system model is:

[0091] e(t+1)=r(t+1)-y(t+1)

[0092] =r(t+1)-Cx(t+1)

[0093] =r(t+1)-[CAx(t)+CB(u e (t)+u g (t))+CEg(t)]

[0094] Step 2.2, in the error system model, CBu e (t), CBu g (t) and CEg(t) are the influence of the feedback controller, feedforward controller and external disturbance on the error system at time t, respectively. CAx(t) is the influence of the state self-transfer process on the error system. The integral form of the influence of the state self-transfer process on the error system is obtained by recursion:

[0095]

[0096] Where i represents the summation parameter.

[0097] Step 2.3: From the integral form, decompose the component caused by the disturbance as an intermediate variable Right now

[0098]

[0099] Further derivation to the differential form is:

[0100]

[0101] In the formula, Stands for pseudo-inverse operation.

[0102] Step 3: The system error e(t) and the intermediate variable Augmented to a new state, the feedback controller is designed based on the augmented state to ensure limited stability, and the remaining stability problem is transformed into the augmented system state convergence problem related only to the feedforward control output, including:

[0103] Step 3.1: The system error e(t) and the intermediate variable Expanded to new state:

[0104]

[0105] Step 3.2, design the feedback controller based on the augmented state:

[0106]

[0107] Where κ1 and κ2 are controller parameters respectively.

[0108] Step 3.3, based on the Lyapnuov error energy function Analyze the stability of the error system, and the differential form of the energy function after substituting it into the feedback controller is:

[0109]

[0110] Where ΔV e (e(t)) = e T (t+1)e(t+1)-e T (t)e(t) represents the difference form.

[0111] Based on the energy function of Lyapnuov intermediate variable system Analyze the stability of the intermediate variables, and the differential form of the energy function after substituting the intermediate variables is:

[0112]

[0113] In the formula, Indicates the differential form.

[0114] Based on Lyapnuov overall system energy function Analyze the stability of the augmented state, the differential form of the overall energy function after substituting into the first two equations is:

[0115]

[0116] In the formula,

[0117] Indicates the differential form.

[0118] Step 3.4, transform the differential form of the overall system energy function into an augmented system, and from the controller perspective, the input of the augmented system is only related to the feedforward control u g(t) is related to the convergence problem of the system, which is transformed into the convergence problem of the augmented system state:

[0119]

[0120] In the formula, ζ 1,1 (t) = e(t), which represents the first row and first column element of the augmented state. A nonlinear form representing the first row and first column elements of the augmented state.

[0121] Step 4: In an environment where feedforward control is used as the action and the augmented system state is used as the feedback, the DQN reinforcement learning algorithm is used to obtain the optimal augmented state under the optimal action, and then the optimal augmented state is connected to the optimal feedback controller output, which specifically includes:

[0122] Step 4.1, feedforward control u g (t) As a reinforcement learning action to augment the system state As feedback to the DQN reinforcement learning algorithm, calculate the state value function

[0123]

[0124] In the formula, π k represents the reinforcement learning strategy executed into the environment, represents the comprehensive reward, R (t) represents the state reward obtained by executing the strategy at time t, It represents the predicted state reward from time t+1 to the end of strategy execution. In the strategy π k Get the state reward G at the next t moment t conditional expectation.

[0125] Step 4.2: Calculate the action-state value function based on the state value function

[0126]

[0127] In the formula, γ represents the discount factor, R (t) is the reward function, Represents a discount reward, In the strategy π k Get the discount status reward at the next t moment conditional expectation.

[0128] Reward function R (t) for:

[0129] R(t)=ω a Aa (t)+τ s V s (t)

[0130] In the formula, ω a and τ s are constant coefficients, represents the reward obtained by the error tracking effect, where e mr and e max Indicates the upper and lower bounds of the error allowed, D a is a large constant. represents the reward for the convergence speed of the augmented state, where D v is a large constant, D a and D v It is a constant that is several times larger than the reward already obtained, and specifies the penalty measures when the error exceeds the limit;

[0131] Step 4.3, use the online deep neural network and the target deep neural network to replace the action-state value function, and the online deep neural network parameters are θ i , the target deep neural network parameters are The two network parameters are synchronized once at a fixed period, and the objective function for updating the network parameters is constructed as follows:

[0132]

[0133] In the formula, i represents the number of parameter updates using the objective function, L(θ i ) represents the parameter θ i The objective function is Ak Represents a set of actions, represents the action-state value function calculated by the target deep neural network, represents the action-state value function calculated by the online deep neural network, Indicates action u g (t) maximize the action-state value function calculated by the target deep neural network;

[0134] Step 4.4, use the trained online deep neural network for the decision-making process of the feedforward controller output:

[0135]

[0136] In the formula, Represents the output of the feedforward controller that maximizes the value of the action state calculated by the online deep neural network.

[0137] Step 4.5, obtain the optimal feedforward control output u g (t), corresponding to the optimal augmented system state Then connected to the optimal feedback control output u e (t), thereby synchronously obtaining the output of the feedforward feedback controller.

[0138] In step 4, the part to be designed in the feedback controller is completely composed of intermediate variables. The intermediate variables, as the state of the augmented system, are completely determined by the feedforward control output. With the help of intermediate variables and the framework of the Lyapunov function method, the design of the feedforward control output and the feedback control output are connected, and the design problem of the two-dimensional decision variables is reduced to a one-dimensional decision space. The feedback controller is in a given form and partially stable performance is guaranteed. The reinforcement learning process is subject to partial stability constraints, avoiding unnecessary trial and error processes and improving the solution speed.

[0139] The present invention discloses a feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association, which takes the disturbance accumulation influence term separated from the error system as an intermediate variable and constitutes an augmented state together with the system error. The augmented state participates in the design of the feedback controller on the one hand, and is used to construct an augmented system related only to the feedforward control on the other hand, thereby constructing a process from reinforcement learning to solve the feedforward control to the optimal augmented state and then to synchronously obtain the feedback controller, reducing the two-dimensional decision space required to be explored by reinforcement learning to one dimension, and enabling reinforcement learning to start the learning process on the basis of limited stability, thereby improving the solution speed and control effect. It is possible to solve the optimal trade-off problem between feedforward control and feedback control performance with a lower solution burden and improve the control effect.

[0140] In this embodiment, Figure 3 The control effect comparison between the invention method and the traditional control method is shown. Different traditional control methods are used in the control experiment method, namely, a feedforward feedback control method based on predictive control design, a feedforward feedback control method based on optimized control design, and a proportional feedforward feedback control method with different proportional gains. The results show that the invention method has a good control effect on the internal pressure P of the heat exchanger. I In the control system design, it is possible to reduce the pipe network pressure P E The results of the feedforward control output in the sub-graph show that the control method proposed in the present invention only works in the initial stage with large errors and the stage when disturbances occur, which is more in line with the purpose of introducing feedforward control.

[0141] In this embodiment, Figure 4The comparison between the inventive method and the reinforcement learning control method of independent controller learning is demonstrated. The control experimental method explores the feedforward controller and the feedback controller as independent decision variables in a two-dimensional decision space. The results show that from the perspective of reward convergence, the reward convergence speed of the inventive method is nearly 200 iteration cycles earlier than that of the comparison method. From the perspective of average reward, the average reward of the inventive method is much greater than that of the comparison method. From the perspective of variance band, the reward variance obtained by the inventive method when starting learning under multiple random initial conditions is smaller. From the perspective of control effect, the inventive method tracks the set value better with smaller errors. The inventive method achieves the optimal trade-off problem between feedforward control and feedback control performance with a lower solution burden and improves the control effect.

[0142] Based on the same inventive concept, the present invention provides a feedforward feedback synchronous reinforcement learning control system under feedforward bidirectional association, including a storage module, the storage module includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes the various steps of the above-mentioned feedforward feedback synchronous reinforcement learning control under a feedforward bidirectional association.

[0143] Based on the same inventive concept, the present invention provides an electronic device, including the above-mentioned feedforward feedback synchronous reinforcement learning control system under a feedforward bidirectional association.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0145] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association, characterized in that: The following steps are involved: Step 1, establish an inverse disturbance model and a composite disturbance model of the control output to the disturbance, and obtain a system dynamics model under feedforward bidirectional correlation; Step 2: Transform the setting tracking problem into the system error framework, and decompose the disturbance-caused component from the error caused by the state self-transition as an intermediate variable; Step 3: Augment the system error and intermediate variables into new states, design a feedback controller based on the augmented states to ensure limited stability, and transform the remaining stability problem into an augmented system state convergence problem related only to the feedforward control output; Step 4: In an environment where feedforward control is used as the action and the augmented system state is used as the feedback, the DQN reinforcement learning algorithm is used to obtain the optimal augmented state under the optimal action, and then the optimal augmented state is connected to the optimal feedback controller output.

2. According to the feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 1, it is characterized in that: In step 1, the process in which the controller output affects the feedforward is an inverse disturbance model, and the influence of the composite disturbance composed of the inverse disturbance and the external disturbance on the system output is a composite disturbance model. The controlled model, the inverse disturbance model and the composite disturbance model are used as sub-models to construct the system dynamics model under the feedforward bidirectional association.

3. According to the feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 2, it is characterized in that: The state space model form of the system dynamics model under the feedforward bidirectional association is expressed as follows: In the formula, x(t)=[x1 T (t),x2 T (t),x3 T (t)] T , represents the state in the state space model, x1(t) represents the controlled model state, x2(t) represents the inverse disturbance model state, x3(t) represents the composite disturbance model state, u(t) = u g (t)+u e (t), represents the controller output, and the feedforward controller output u g (t) and the sum of the feedback controller output u e (t) composition, Represents a composite disturbance, and is composed of an external disturbance g(t) and the inverse disturbance model output C2x2(t), C2 represents the output matrix of the inverse disturbance model from state to output, y(t) is the output of the system, x(t+1) is the new state after a state transition occurs once, and A, B, C, and E are the state space model parameters respectively.

4. A feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to any one of claims 2 or 3, characterized in that: The step 2 includes the following sub-steps: Step 2.1, using the state space model to transform the system tracking set value and system output into the error system framework to obtain the error system model; Step 2.2, in the error system model, obtain the integral form of the influence of the state self-transition process on the error system by recursion; Step 2.3, from the integral form, decompose the component caused by the disturbance as an intermediate variable, and further push it to the differential form.

5. According to the feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 4, it is characterized in that: The error system model expression in step 2.1 is as follows: e(t+1)=r(t+1)-y(t+1) =r(t+1)-Cx(t+1) =r(t+1)-[CAx(t)+CB(u e (t)+u g (t))+CEg(t)] In the formula, e(t+1) represents the system error at time t+1, CBu e (t), CBu g (t) and CEg(t) are the effects of the feedback controller, feedforward controller and external disturbance on the error system at time t, respectively, and CAx(t) is the effect of the state self-transition process on the error system; The expression of step 2.2 is as follows: In the formula, i represents the summation parameter; In step 2.3, the component caused by the disturbance is decomposed as the expression of the intermediate variable as follows: The expression reduced to differential form is as follows: In the formula, Stands for pseudo-inverse operation.

6. A feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to any one of claims 2, 3 or 5, characterized in that: The step 3 includes the following sub-steps: Step 3.1, expand the system error and intermediate variables into the new state expression as follows: Where e(t) is the system error, is an intermediate variable; Step 3.2, the expression of the feedback controller designed based on the augmented state is as follows: In the formula, κ1 and κ2 are controller parameters respectively; Step 3.3, based on the Lyapnuov error energy function Analyze the stability of the error system, and the differential form of the energy function after substituting it into the feedback controller is: Where ΔV e (e(t)) = e T (t+1)e(t+1)-e T (t)e(t) represents the difference form; Based on the energy function of Lyapnuov intermediate variable system Analyze the stability of the intermediate variables, and the differential form of the energy function after substituting the intermediate variables is: In the formula, represents the differential form; Based on Lyapnuov overall system energy function Analyze the stability of the augmented state, the differential form of the overall energy function after substituting into the first two equations is: In the formula, represents the differential form; Step 3.4, transform the differential form of the overall system energy function into an augmented system, and from the controller perspective, the input of the augmented system is only related to the feedforward control u g (t) is related to the convergence problem of the system, which is transformed into the convergence problem of the augmented system state: In the formula, ζ 1,1 (t) = e(t), which represents the first row and first column element of the augmented state. A nonlinear form representing the first row and first column elements of the augmented state.

7. The feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 6 is characterized in that: The step 4 includes the following sub-steps: Step 4.1: Use feedforward control as the action of reinforcement learning, use the augmented system state as the feedback of the DQN reinforcement learning algorithm, and calculate the state value function The expression is as follows: In the formula, u g (t) is the feedforward control, is the augmented system state, π k represents the reinforcement learning strategy executed into the environment, represents the comprehensive reward, R (t) represents the state reward obtained by executing the strategy at time t, It represents the predicted state reward from time t+1 to the end of strategy execution. In the strategy π k Get the state reward G at the next t moment t The conditional expectation of Step 4.2: Calculate the action-state value function based on the state value function The expression is as follows: In the formula, γ represents the discount factor, R (t) is the reward function, Represents a discount reward, In the strategy π k Get the discount status reward at the next t moment The conditional expectation of Step 4.3, use the online deep neural network and the target deep neural network to replace the action-state value function, perform a synchronization operation on the two network parameters at a fixed period, and construct the objective function for updating the network parameters as follows: In the formula, θ i are the online deep neural network parameters, is the target deep neural network parameter, i represents the number of parameter updates using the objective function, L(θ i ) represents the parameter θ i The objective function is Ak Represents a set of actions, represents the action-state value function calculated by the target deep neural network, represents the action-state value function calculated by the online deep neural network, Indicates action u g (t) maximize the action-state value function calculated by the target deep neural network; Step 4.4, use the trained online deep neural network for the decision-making process of the feedforward controller output: In the formula, represents the output of the feedforward controller that maximizes the value of the action state calculated by the online deep neural network; Step 4.5, obtain the optimal feedforward control output u g (t), corresponding to the optimal augmented system state Then connected to the optimal feedback control output u e (t), thereby synchronously obtaining the output of the feedforward feedback controller.

8. The feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 7 is characterized in that: The reward function R in step 4.2 (t) for: R(t)=ω a A a (t)+τ s V s (t) In the formula, ω a and τ s are constant coefficients, represents the reward obtained by the error tracking effect, where e mr and e max Indicates the upper and lower bounds of the error allowed, D a is a constant, represents the reward for the convergence speed of the augmented state, where D v is a constant.

9. A feedforward feedback synchronous reinforcement learning control system under feedforward bidirectional association, characterized in that: It includes a storage module, which includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes each step of a feedforward feedback synchronous reinforcement learning control under a feedforward bidirectional association according to any one of claims 1-8.

10. An electronic device, characterized in that: It includes a feedforward feedback synchronous reinforcement learning control system under feedforward bidirectional association according to claim 9.

Images