A feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional linkage

Abstract

The present invention relates to the field of automatic control technology and discloses a feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional linkage, comprising the following steps: establishing an inverse disturbance model and a composite disturbance model of a control output to a disturbance to obtain a system dynamics model under feedforward bidirectional linkage; transforming a set tracking problem into a system error framework, and decomposing a component caused by the disturbance from the error generated by state self-transition as an intermediate variable; augmenting the system error and the intermediate variable into a new state, designing a feedback controller based on the augmented state to ensure limited stability, and transforming the remaining stability problem into an augmented system state convergence problem related only to the feedforward control output; in an environment where feedforward control is used as an action and the augmented system state is used as feedback, utilizing a DQN reinforcement learning algorithm to obtain an optimal augmented state under the optimal action, and then connecting the optimal augmented state to an optimal feedback controller output. The present invention has better control effect and faster solution speed.

Description

Technical Field

[0001] The present application relates to the field of automatic control technology, and specifically discloses a feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association. Background Art

[0002] Feedforward-feedback control is a widely used control system architecture, widely applied in industrial control, mechanical control, and intelligent driving. The feedback control component is fundamental to control system design. It collects the regulation error and executes corresponding actions according to the feedback control rate to continuously reduce the tracking error. Feedforward control is used to overcome the negative effects of external disturbances. By monitoring changes in external disturbances, it preemptively compensates the controller output to quickly offset the disturbance. In practical applications, disturbances are common and complex, and the characteristics of different disturbances require the design of specific feedforward control methods.

[0003] Typically, disturbances originate entirely from outside the system, meaning they follow the direction of external influences on the system's internal systems. Feedforward control only needs to be designed based on the characteristics of the external disturbance to achieve its objectives. However, when adjustments to actuators within the system can also influence changes in external disturbances, the feedforward variable has both the direction of external influences on the system's internal systems and the direction of internal influences on the external disturbances, meaning the feedforward is bidirectionally correlated. This phenomenon is common in many practical control processes. For example, abnormal steam pressure fluctuations provided by a boiler are considered external disturbances in the pressure inside a downstream heat exchanger. Adjusting the valve to offset the disturbance will also cause changes in the external steam pressure. Alternatively, abnormal pressure from a blower in a sewage treatment process is considered external disturbances in the pressure of multiple aeration pipes. Adjusting the valve to offset the disturbance will also affect the pressure changes in the blower. In such cases, existing feedforward control design methods are no longer applicable, and controller design challenges in bidirectional feedforward control are emerging. The main challenges are: First, feedforward control is used to improve the system's ability to quickly adjust to disturbances, while feedback control is used to address stable tracking. The incompatible characteristics of rapidity and stability lead to a performance trade-off between the two controllers. Challenge 2: Solving the coordination problem between the performance of the two controllers requires the use of optimization algorithms or reinforcement learning algorithms. Since the decision variables are two-dimensional and have non-quantifiable internal correlations, the burden of solving the problem is increased.

[0004] Therefore, in view of this, the inventors provide a feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a feedforward feedback synchronous reinforcement learning control method with better control effect and faster solution speed under feedforward bidirectional association.

[0006] In order to achieve the above objectives, the basic solution of the present invention provides a feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association, comprising the following steps:

[0007] Step 1: Establish an inverse disturbance model and a composite disturbance model of the control output to obtain the system dynamics model under feedforward bidirectional correlation;

[0008] Step 2: Transform the setting tracking problem into the system error framework and decompose the component caused by the disturbance from the error caused by the state self-transition as an intermediate variable;

[0009] Step 3: Augment the system error and intermediate variables into new states. Design a feedback controller based on the augmented states to ensure limited stability. The remaining stability problem is transformed into an augmented system state convergence problem related only to the feedforward control output.

[0010] Step 4: In an environment where feedforward control is used as the action and the augmented system state is used as the feedback, the DQN reinforcement learning algorithm is used to obtain the optimal augmented state under the optimal action, and then the optimal augmented state is connected to the optimal feedback controller output.

[0011] Furthermore, in step 1, the process in which the controller output affects the feedforward is an inverse disturbance model, and the influence of the composite disturbance composed of the inverse disturbance and the external disturbance on the system output is a composite disturbance model. The controlled model, the inverse disturbance model and the composite disturbance model are used as sub-models to construct a system dynamics model under the feedforward bidirectional association.

[0012] Furthermore, the state space model of the system dynamics model under the feedforward bidirectional association is expressed as follows:

[0013]

[0014] Where, represents the state in the state space model, x1(t) represents the controlled model state, x2(t) represents the inverse disturbance model state, x3(t) represents the composite disturbance model state, u(t) = u g (t)+u e (t), represents the controller output, and is output by the feedforward controller u g (t) and the sum of the feedback controller output u e (t) composition, Represents a composite disturbance, and is composed of the external disturbance g(t) and the inverse disturbance model output C2x2(t), C2 represents the output matrix of the inverse disturbance model from state to output, y(t) is the output of the system, x(t+1) is the new state after a state transition occurs, A, B, C, E are the state space model parameters respectively.

[0015] Furthermore, the step 2 includes the following sub-steps:

[0016] Step 2.1, using the state space model to transform the system tracking setpoint and system output into the error system framework to obtain the error system model;

[0017] Step 2.2, in the error system model, obtain the integral form of the influence of the state self-transition process on the error system through recursion;

[0018] Step 2.3, from the integral form, decompose the component caused by the disturbance as an intermediate variable, and further push it to the differential form.

[0019] Furthermore, the error system model expression in step 2.1 is as follows:

[0020] e(t+1)=r(t+1)-y(t+1)

[0021] =r(t+1)-Cx(t+1)

[0022] =r(t+1)-[CAx(t)+CB(u e (t)+u g (t))+CEg(t)]

[0023] Where, e(t+1) represents the system error at time t+1, CBu e (t), CBu g (t) and CEg(t) are the effects of the feedback controller, feedforward controller and external disturbance on the error system at time t, respectively; CAx(t) is the effect of the state self-transition process on the error system;

[0024] The expression of step 2.2 is as follows:

[0025]

[0026] Where i represents the summation parameter;

[0027] In step 2.3, the component caused by the disturbance is decomposed as the expression of the intermediate variable as follows:

[0028]

[0029] The expression reduced to differential form is as follows:

[0030]

[0031] Where, Stands for pseudo-inverse operation.

[0032] Furthermore, the step 3 includes the following sub-steps:

[0033] Step 3.1, expand the system error and intermediate variables into the new state expression as follows:

[0034]

[0035] Where e(t) is the system error, is an intermediate variable;

[0036] In step 3.2, the expression of the feedback controller designed based on the augmented state is as follows:

[0037]

[0038] Where, κ1 and κ2 are controller parameters respectively;

[0039] Step 3.3, based on Lyapnuov error energy function Analyze the stability of the error system, and the differential form of the energy function after substituting it into the feedback controller is:

[0040]

[0041] Where, ΔV e (e(t))=e T (t+1)e(t+1)-e T (t)e(t) represents the differential form;

[0042] Based on the energy function of Lyapnuov intermediate variable system Analyze the stability of the intermediate variables. The differential form of the energy function after substituting the intermediate variables is:

[0043]

[0044] Where, represents the differential form;

[0045] Based on Lyapnuov overall system energy function Analyzing the stability of the augmented state, the differential form of the overall energy function after substituting the first two equations is:

[0046]

[0047] Where,

[0048] represents the differential form;

[0049] Step 3.4, transform the differential form of the overall system energy function into an augmented system, and from the controller perspective, the input of the augmented system is only related to the feedforward control u g(t), the convergence problem of the system is transformed into the convergence problem of the augmented system state:

[0050]

[0051] Where, ζ 1,1 (t) = e(t), which represents the first row and first column element of the augmented state. A nonlinear form representing the first row and first column elements of the augmented state.

[0052] Furthermore, step 4 includes the following sub-steps:

[0053] Step 4.1: Use feedforward control as the reinforcement learning action, use the augmented system state as the feedback of the DQN reinforcement learning algorithm, and calculate the state value function The expression is as follows:

[0054]

[0055] Where u g (t) is the feedforward control, is the augmented system state, π k represents the reinforcement learning strategy executed into the environment, represents the comprehensive reward, R t represents the state reward obtained by executing the strategy at time t, Represents the state reward predicted from time t+1 to the end of strategy execution, In the strategy π k Get the state reward G at the next t moment t The conditional expectation of

[0056] Step 4.2: Calculate the action-state value function based on the state value function The expression is as follows:

[0057]

[0058] Where γ represents the discount factor, R (t) is the reward function, Represents a discount reward, In the strategy π k Get the discount status reward at the next t moment The conditional expectation of

[0059] In step 4.3, the online deep neural network and the target deep neural network are used to replace the action-state value function. The two network parameters are synchronized once at a fixed period, and the objective function for updating the network parameters is constructed as follows:

[0060]

[0061] Where θ i are the online deep neural network parameters, is the target deep neural network parameter, i represents the number of parameter updates using the objective function, L(θ i ) represents the parameter θ i The objective function, Ak Represents a set of actions, represents the action-state value function calculated by the target deep neural network, represents the action-state value function calculated by the online deep neural network, Indicates action u g (t) Maximize the action-state value function calculated by the target deep neural network;

[0062] Step 4.4, use the trained online deep neural network for the decision-making process of the feedforward controller output:

[0063]

[0064] Where, represents the output of the feedforward controller that maximizes the value of the action state calculated by the online deep neural network;

[0065] Step 4.5, obtain the optimal feedforward control output u g (t), corresponding to the optimal augmented system state Then connected to the optimal feedback control output u e (t), thereby synchronously obtaining the output of the feedforward feedback controller.

[0066] Furthermore, the reward function R in step 4.2 (t) for:

[0067] R(t)=ω a A a (t)+τ s V s (t)

[0068] Where, ω a and τ s are all constant coefficients, Represents the reward obtained from the error tracking effect, where e mr and e max Indicates the upper and lower bounds of the error allowed, D a is a constant, represents the reward for the convergence speed of the augmented state, where D v is a constant.

[0069] Based on the same inventive concept, the present invention provides a feedforward feedback synchronous reinforcement learning control system under feedforward bidirectional association, including a storage module, wherein the storage module includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes the various steps of the feedforward feedback synchronous reinforcement learning control under the above-mentioned feedforward bidirectional association.

[0070] Based on the same inventive concept, the present invention provides an electronic device, including the above-mentioned feedforward feedback synchronous reinforcement learning control system under a feedforward bidirectional association.

[0071] Compared with the prior art, the present invention has the following effects:

[0072] 1. In the present invention, in step 1, a system dynamics model under feedforward bidirectional correlation is established, in step 2, the influence form of the disturbance is separated from the self-transfer part of the error system state as an intermediate variable, in step 3, the intermediate variable and the system error are augmented into a new state, a feedback controller is designed based on the augmented state to ensure limited stability, an augmented system is constructed with the augmented state and an input containing only the feedforward controller, and the convergence of the augmented system is equivalent to replacing the convergence of the system, in step 4, the feedforward controller is learned using the DQN reinforcement learning algorithm, and is synchronously connected to the feedback controller through the augmented state, which can achieve the optimal trade-off problem between feedforward control and feedback control performance with a lower solution burden and improve the control effect.

[0073] 2. By constructing an augmented system, the present invention implements a connection path from feedforward controller learning to augmented state determination and then to feedback controller determination. This reduces the decision-making problem of the two controllers to the dimension of a single controller learning, thus reducing the number of variables to be learned and the solution burden. Furthermore, during the process of optimizing feedforward control decisions through reinforcement learning, the feedback controller can ensure limited stability, further reducing the action space that reinforcement learning needs to explore, further accelerating the learning process, and achieving the optimal trade-off between feedforward and feedback control performance with a lower solution burden, thereby improving control effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 A schematic diagram showing the relationship between the boiler and the heat exchanger;

[0076] Figure 2A schematic flow chart of a feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association proposed in an embodiment of the present application is shown;

[0077] Figure 3 Figures comparing the effects of the invention and conventional control methods in the design of a heat exchanger pressure control system are shown, wherein (a) compares the invention with a feedforward based on predictive control design, (b) compares the invention with a feedforward based on optimization control design, (c) compares the invention with a proportional control feedforward (K=0.4), and (d) compares the invention with a proportional control feedforward (K=0.8).

[0078] Figure 4 A comparison diagram of the effects of the invention and the reinforcement learning control method of independent controller learning in the design of a heat exchanger pressure control system is shown, where (a) is a comparison diagram of the average rewards of the invention and independent controller learning, and (b) is a training comparison diagram of independent controller learning. DETAILED DESCRIPTION

[0079] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0080] A feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional linkage is used to improve the control effect of feedforward feedback control under feedforward bidirectional linkage and the speed of solving the controller. The design of a heat exchanger pressure control system with the steam pressure provided by the boiler as the disturbance is taken as an example. The equipment schematic diagram is shown as follows Figure 1 As shown in the figure, the hot steam generated by the boiler is transported to the inlet of the heat exchanger through the steam pipe network. The heat exchanger uses the hot steam to physically exchange heat with the material. The internal pressure of the heat exchanger is P I It is the controlled parameter index, the steam network pressure P provided by the boiler E It is a parameter index of the previous section. When the boiler production conditions or operation mode changes, the pipe network pressure P E Fluctuations occur, with the pipe network pressure P E As an example, based on the principle of feedforward control, the heat exchanger inlet valve quickly takes measures to reduce the opening to suppress the disturbance on the internal pressure P I However, at the same time, reducing the opening of the heat exchanger inlet valve will reversely increase the pipe network pressure P E , which in turn causes the heat exchanger inlet valve to decrease, and the pipe network pressure P E The descent will have a similar phenomenon.

[0081] In this process, traditional feedforward control will cause one-way excessive adjustment of the heat exchanger inlet valve, which will suppress the disturbance while inversely affecting the disturbance change. The excessive adjustment of the feedforward control will also transmit and affect the output of the feedback controller. The feedforward controller and the feedback controller will continue to work independently, not only failing to achieve mutual coordination, but even worsening the control effect of the system. This phenomenon of bidirectional feedforward correlation brings difficulties to the design of the control system.

[0082] To solve the control system design problem under feedforward bidirectional correlation, this embodiment adopts the following steps for design:

[0083] Step S1, define the influence of heat exchanger inlet valve adjustment on pipe network pressure P E The process is the inverse disturbance model, and the inverse disturbance and the pipe network pressure P are defined. E Composite disturbance The impact on the system output is a composite disturbance model. The heat exchanger internal pressure controlled model, the inverse disturbance model, and the composite disturbance model serve as sub-models, which together constitute the system dynamics model under feedforward bidirectional correlation. Its state space model form is defined as:

[0084]

[0085] Where, represents the state in the state space model, x1(t) represents the controlled model state, x2(t) represents the inverse disturbance model state, x3(t) represents the composite disturbance model state, u(t) = u g (t)+u e (t), represents the controller output, and is output by the feedforward controller u g (t) and the sum of the feedback controller output u e (t) composition, Represents a composite disturbance, and is composed of the external disturbance g(t) and the inverse disturbance model output C2x2(t), C2 represents the output matrix of the inverse disturbance model from state to output, y(t) is the output of the system, x(t+1) is the new state after a state transition occurs, A, B, C, E are the state space model parameters respectively.

[0086] In such Figure 1 In the embodiment shown, the submodels are all set as first-order models, the output coefficient of the inverse disturbance model from state to output is set to C2=1, and the system matrix, control matrix, output matrix and disturbance matrix in the state space model are respectively:

[0087]

[0088] The experimental time is set to 150 minutes, and the internal pressure of the heat exchanger is P I The target setting is 0.43Mpa, and the pipe network pressure PE A step disturbance of 0.5 MPa was generated at the 75th minute.

[0089] Step 2: Transform the setting tracking problem into the system error e(t) framework, and decompose the component caused by the disturbance from the error caused by the state self-transition as an intermediate variable Specifically include:

[0090] Step 2.1, the internal pressure of the heat exchanger P I The target is set as r(t) is 0.43 MPa for any time t, and e(t+1) represents the system error at time t+1. Using the state space model, it is transformed into the error system framework, and the error system model is:

[0091] e(t+1)=r(t+1)-y(t+1)

[0092] =r(t+1)-Cx(t+1)

[0093] =r(t+1)-[CAx(t)+CB(u e (t)+u g (t))+CEg(t)]

[0094] Step 2.2, in the error system model, CBu e (t), CBu g (t) and CEg(t) are the effects of the feedback controller, feedforward controller and external disturbance on the error system at time t, respectively. CAx(t) is the effect of the state self-transfer process on the error system. The integral form of the effect of the state self-transfer process on the error system is obtained by recursion:

[0095]

[0096] Where i represents the summation parameter.

[0097] Step 2.3: From the integral form, decompose the component caused by the disturbance as an intermediate variable Right now

[0098]

[0099] Further derivation to the differential form is:

[0100]

[0101] Where, Stands for pseudo-inverse operation.

[0102] Step 3: The system error e(t) and the intermediate variable The system is augmented to a new state. A feedback controller is designed based on the augmented state to ensure limited stability. The remaining stability problem is transformed into the state convergence problem of the augmented system related only to the feedforward control output, including:

[0103] Step 3.1, the system error e(t), the intermediate variable Expanded to new state:

[0104]

[0105] Step 3.2, design the feedback controller based on the augmented state as:

[0106]

[0107] Where κ1 and κ2 are controller parameters respectively.

[0108] Step 3.3, based on Lyapnuov error energy function Analyze the stability of the error system, and the differential form of the energy function after substituting it into the feedback controller is:

[0109]

[0110] Where, ΔV e (e(t))=e T (t+1)e(t+1)-e T (t)e(t) represents the differential form.

[0111] Based on the energy function of Lyapnuov intermediate variable system Analyze the stability of the intermediate variables. The differential form of the energy function after substituting the intermediate variables is:

[0112]

[0113] Where, Indicates the differential form.

[0114] Based on Lyapnuov overall system energy function Analyzing the stability of the augmented state, the differential form of the overall energy function after substituting the first two equations is:

[0115]

[0116] Where,

[0117] Indicates the differential form.

[0118] Step 3.4, transform the differential form of the overall system energy function into an augmented system, and from the controller perspective, the input of the augmented system is only related to the feedforward control u g(t), the convergence problem of the system is transformed into the convergence problem of the augmented system state:

[0119]

[0120] Where, ζ 1,1 (t) = e(t), which represents the first row and first column element of the augmented state. A nonlinear form representing the first row and first column elements of the augmented state.

[0121] Step 4: In an environment where feedforward control is used as the action and the augmented system state is used as the feedback, the DQN reinforcement learning algorithm is used to obtain the optimal augmented state under the optimal action. The optimal augmented state is then connected to the output of the optimal feedback controller. Specifically,

[0122] Step 4.1, feedforward control u g (t) as reinforcement learning action to augment the system state As feedback for the DQN reinforcement learning algorithm, calculate the state value function

[0123]

[0124] Where, π k represents the reinforcement learning strategy executed into the environment, represents the comprehensive reward, R (t) represents the state reward obtained by executing the strategy at time t, Represents the state reward predicted from time t+1 to the end of strategy execution, In the strategy π k Get the state reward G at the next t moment t conditional expectation of .

[0125] Step 4.2: Calculate the action-state value function based on the state value function

[0126]

[0127] Where γ represents the discount factor, R (t) is the reward function, Represents a discount reward, In the strategy π k Get the discount status reward at the next t moment conditional expectation of .

[0128] Reward function R (t) for:

[0129] R(t)=ω a Aa (t)+τ s V s (t)

[0130] Where, ω a and τ s are all constant coefficients, Represents the reward obtained from the error tracking effect, where e mr and e max Indicates the upper and lower bounds of the error allowed, D a is a large constant. represents the reward for the convergence speed of the augmented state, where D v is a large constant, D a and D v It is a constant that is several times larger than the reward obtained, and specifies the penalty measures when the error exceeds the limit;

[0131] Step 4.3, use the online deep neural network and the target deep neural network to replace the action-state value function, and the online deep neural network parameters are θ i , the target deep neural network parameters are The two network parameters are synchronized once at a fixed period, and the objective function for updating the network parameters is constructed as follows:

[0132]

[0133] Where i represents the number of times the objective function is used to update the parameters, L(θ i ) represents the parameter θ i The objective function, Ak Represents a set of actions, represents the action-state value function calculated by the target deep neural network, represents the action-state value function calculated by the online deep neural network, Indicates action u g (t) Maximize the action-state value function calculated by the target deep neural network;

[0134] Step 4.4, use the trained online deep neural network for the decision-making process of the feedforward controller output:

[0135]

[0136] Where, Represents the output of the feedforward controller that maximizes the value of the action state calculated by the online deep neural network.

[0137] Step 4.5, obtain the optimal feedforward control output u g (t), corresponding to the optimal augmented system state Then connected to the optimal feedback control output u e (t), thereby synchronously obtaining the output of the feedforward feedback controller.

[0138] In step 4, the part to be designed in the feedback controller is composed entirely of intermediate variables. The intermediate variables, as the state of the augmented system, are completely determined by the feedforward control output. With the help of intermediate variables and the framework of the Lyapunov function method, the design of the feedforward control output and the feedback control output are connected, and the design problem of the two-dimensional decision variables is reduced to a one-dimensional decision space. The feedback controller is in a given form and partially stable performance is guaranteed. The reinforcement learning process is subject to partial stability constraints, avoiding unnecessary trial and error processes and improving the solution speed.

[0139] The present invention discloses a feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association, which uses the disturbance accumulation influence term separated from the error system as an intermediate variable and forms an augmented state together with the system error. The augmented state participates in the design of the feedback controller on the one hand, and is used to construct an augmented system related only to the feedforward control on the other hand. In this way, a process is constructed from reinforcement learning to solve the feedforward control, to the optimal augmented state, and then to the synchronous acquisition of the feedback controller. The two-dimensional decision space that reinforcement learning needs to explore is reduced to one dimension, and reinforcement learning starts the learning process on the basis of limited stability, thereby improving the solution speed and control effect. It is possible to solve the optimal trade-off problem between feedforward control and feedback control performance with a lower solution burden and improve the control effect.

[0140] In this embodiment, Figure 3 The control effect comparison between the invention method and the traditional control method is shown. The control experimental methods use different traditional control methods, namely the feedforward feedback control method based on predictive control design, the feedforward feedback control method based on optimized control design and the proportional feedforward feedback control method with different proportional gains. The results show that the invention method has a better control effect on the internal pressure P of the heat exchanger. I In the control system design, it is possible to reduce the pipe network pressure P E The results of the feedforward control output in the sub-graph show that the control method proposed in this invention only works in the initial stage with large errors and the stage when disturbances occur, which is more in line with the purpose of introducing feedforward control.

[0141] In this embodiment, Figure 4The paper shows a comparison between the inventive method and the reinforcement learning control method of independent controller learning. The control experimental method explores the feedforward controller and the feedback controller as independent decision variables in a two-dimensional decision space. The results show that from the perspective of reward convergence, the reward convergence speed of the inventive method is nearly 200 iteration cycles earlier than that of the comparison method. From the perspective of average reward, the average reward of the inventive method is much greater than that of the comparison method. From the perspective of variance band, the reward variance obtained by the inventive method when starting learning under multiple random initial conditions is smaller. From the perspective of control effect, the inventive method tracks the set value better with smaller error. The inventive method achieves the optimal trade-off problem between feedforward control and feedback control performance with a lower solution burden and improves the control effect.

[0142] Based on the same inventive concept, the present invention provides a feedforward feedback synchronous reinforcement learning control system under feedforward bidirectional association, including a storage module, the storage module including instructions loaded and executed by a processor, and when the instructions are executed, the processor performs the various steps of the above-mentioned feedforward feedback synchronous reinforcement learning control under feedforward bidirectional association.

[0143] Based on the same inventive concept, the present invention provides an electronic device, including the above-mentioned feedforward feedback synchronous reinforcement learning control system under a feedforward bidirectional association.

[0144] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0145] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association, characterized in that: The following steps are involved: Step 1: Establish an inverse disturbance model and a composite disturbance model of the control output to obtain the system dynamics model under feedforward bidirectional correlation; Step 2: Transform the setting tracking problem into the system error framework and decompose the component caused by the disturbance from the error caused by the state self-transition as an intermediate variable; Step 3: Augment the system error and intermediate variables into new states. Design a feedback controller based on the augmented states to ensure limited stability. The remaining stability problem is transformed into an augmented system state convergence problem related only to the feedforward control output. Step 4: In an environment where feedforward control is used as the action and the augmented system state is used as the feedback, the DQN reinforcement learning algorithm is used to obtain the optimal augmented state under the optimal action, and then the optimal augmented state is connected to the optimal feedback controller output.

2. The feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 1, characterized in that: In step 1, the process in which the controller output affects the feedforward is an inverse disturbance model, and the influence of the composite disturbance composed of the inverse disturbance and the external disturbance on the system output is a composite disturbance model. The controlled model, the inverse disturbance model and the composite disturbance model are used as sub-models to construct a system dynamics model under the feedforward bidirectional association.

3. The feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 2, characterized in that: The state space model of the system dynamics model under the feedforward bidirectional connection is expressed as follows: Where x(t)=[x1 T (t),x2 T (t),x3 T (t)] T , represents the state in the state space model, x1(t) represents the controlled model state, x2(t) represents the inverse disturbance model state, x3(t) represents the composite disturbance model state, u(t)=u g (t)+u e (t), represents the controller output, and is output by the feedforward controller u g (t) and the sum of the feedback controller output u e (t) composition, Represents a composite disturbance, and is composed of the external disturbance g(t) and the inverse disturbance model output C2x2(t), C2 represents the output matrix of the inverse disturbance model from state to output, y(t) is the output of the system, x(t+1) is the new state after a state transition occurs, A, B, C, E are the state space model parameters respectively.

4. The feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to any one of claims 2 or 3, characterized in that: The step 2 includes the following sub-steps: Step 2.1, using the state space model to transform the system tracking setpoint and system output into the error system framework to obtain the error system model; Step 2.2, in the error system model, obtain the integral form of the influence of the state self-transition process on the error system through recursion; Step 2.3, from the integral form, decompose the component caused by the disturbance as an intermediate variable, and further push it to the differential form.

5. The feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 4, characterized in that: The error system model expression in step 2.1 is as follows: e(t+1)=r(t+1)-y(t+1) =r(t+1)-Cx(t+1) =r(t+1)-[CAx(t)+CB(u e (t)+u g (t))+CEg(t)] Where, e(t+1) represents the system error at time t+1, CBu e (t), CBu g (t) and CEg(t) are the effects of the feedback controller, feedforward controller and external disturbance on the error system at time t, respectively; CAx(t) is the effect of the state self-transition process on the error system; The expression of step 2.2 is as follows: Where i represents the summation parameter; In step 2.3, the component caused by the disturbance is decomposed as the expression of the intermediate variable as follows: The expression reduced to differential form is as follows: Where, Stands for pseudo-inverse operation.

6. The feedforward feedback synchronous reinforcement learning control method under feedforward bidirectional association according to any one of claims 2, 3 or 5, characterized in that: The step 3 includes the following sub-steps: Step 3.1, expand the system error and intermediate variables into the new state expression as follows: Where e(t) is the system error, is an intermediate variable; In step 3.2, the expression of the feedback controller designed based on the augmented state is as follows: Where, κ1 and κ2 are controller parameters respectively; Step 3.3, based on Lyapnuov error energy function Analyze the stability of the error system, and the differential form of the energy function after substituting it into the feedback controller is: Where, ΔV e (e(t))=e T (t+1)e(t+1)-e T (t)e(t) represents the differential form; Based on the energy function of Lyapnuov intermediate variable system Analyze the stability of the intermediate variables. The differential form of the energy function after substituting the intermediate variables is: Where, represents the differential form; Based on Lyapnuov overall system energy function Analyzing the stability of the augmented state, the differential form of the overall energy function after substituting the first two equations is: Where, represents the differential form; Step 3.4, transform the differential form of the overall system energy function into an augmented system, and from the controller perspective, the input of the augmented system is only related to the feedforward control u g (t), the convergence problem of the system is transformed into the convergence problem of the augmented system state: Where, ζ 1,1 (t) = e(t), which represents the first row and first column element of the augmented state. A nonlinear form representing the first row and first column elements of the augmented state.

7. The feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 6, characterized in that: The step 4 includes the following sub-steps: Step 4.1: Use feedforward control as the reinforcement learning action, use the augmented system state as the feedback of the DQN reinforcement learning algorithm, and calculate the state value function The expression is as follows: Where u g (t) is the feedforward control, is the augmented system state, π k represents the reinforcement learning strategy executed into the environment, represents the comprehensive reward, R (t) represents the state reward obtained by executing the strategy at time t, Represents the state reward predicted from time t+1 to the end of strategy execution, In the strategy π k Get the state reward G at the next t moment t The conditional expectation of Step 4.2: Calculate the action-state value function based on the state value function The expression is as follows: Where γ represents the discount factor, R (t) is the reward function, Represents a discount reward, In the strategy π k Get the discount status reward at the next t moment The conditional expectation of In step 4.3, the online deep neural network and the target deep neural network are used to replace the action-state value function. The two network parameters are synchronized once at a fixed period, and the objective function for updating the network parameters is constructed as follows: Where θ i are the online deep neural network parameters, is the target deep neural network parameter, i represents the number of parameter updates using the objective function, L(θ i ) represents the parameter θ i The objective function, Ak Represents a set of actions, represents the action-state value function calculated by the target deep neural network, represents the action-state value function calculated by the online deep neural network, Indicates action u g (t) Maximize the action-state value function calculated by the target deep neural network; Step 4.4, use the trained online deep neural network for the decision-making process of the feedforward controller output: Where, represents the output of the feedforward controller that maximizes the value of the action state calculated by the online deep neural network; Step 4.5, obtain the optimal feedforward control output u g (t), corresponding to the optimal augmented system state Then connected to the optimal feedback control output u e (t), thereby synchronously obtaining the output of the feedforward feedback controller.

8. The feedforward-feedback synchronous reinforcement learning control method under feedforward bidirectional association according to claim 7, characterized in that: The reward function R in step 4.2 (t) for: R(t)=ω a A a (t)+τ s V s (t) Where, ω a and τ s are all constant coefficients, Represents the reward obtained from the error tracking effect, where e mr and e max Indicates the upper and lower bounds of the error allowed, D a is a constant, represents the reward for the convergence speed of the augmented state, where D v is a constant.

9. A feedforward-feedback synchronous reinforcement learning control system under feedforward bidirectional association, characterized in that: It includes a storage module, which includes instructions loaded and executed by a processor, and when the instructions are executed, the processor performs the various steps of the feedforward feedback synchronous reinforcement learning control under a feedforward bidirectional association according to any one of claims 1 to 8.

10. An electronic device, characterized in that: It includes a feedforward feedback synchronous reinforcement learning control system under a feedforward bidirectional association according to claim 9.

Images