Reinforcement learning based motor system h-infinity reduced order output tracking control method

Abstract

The application discloses a motor system H-infinity reduced-order output tracking control method based on reinforcement learning, and can solve the interference suppression tracking control problem of a motor system containing unmodeled dynamics and imperfect data. Specifically, the original motor system H-infinity output tracking control problem is decomposed into a reduced-order system problem by using singular perturbation theory; a state reconstruction mechanism of a virtual subsystem is proposed based on the output state data of the original system to solve the problem that the data of the virtual subsystem is unmeasurable; a reinforcement learning H-infinity output tracking iterative algorithm based on the reconstructed data is further derived; an execution-evaluation-disturbance neural network approximate controller, a performance index and a disturbance are introduced; the weights of the neural network are iteratively updated based on the least square method to obtain a reduced-order H-infinity output tracking controller based on reinforcement learning. The application avoids potential high-dimensional and ill-conditioned numerical problems when designing a double-time-scale motor system tracking controller in the reinforcement learning framework.

Description

Technical Field

[0001] This invention belongs to the field of motor system drive control, specifically involving a method for reducing the order of output tracking control of an H-infinity motor system based on reinforcement learning. Background Technology

[0002] Nonlinear dual-time-scale motor systems, widely found in power systems, process industries, and other fields, are a class of systems with complex characteristics such as high order and fast-slow coupling. In practice, these systems are often required to operate along a pre-set reference trajectory while also possessing a certain degree of interference resistance. Robust tracking control aims to design controllers that enable the system to meet these requirements, and has therefore received extensive research attention.

[0003] Existing tracking control methods for nonlinear dual-time-scale motor systems are mainly based on sliding mode control and active disturbance rejection control. However, these methods lack quantitative analysis of disturbance suppression, leading to the development of H-infinity control as an effective means of addressing disturbance rejection issues. However, directly applying general system tracking control methods to singularly perturbed systems can result in ill-conditioned numerical problems and the curse of dimensionality. Therefore, a feasible solution based on system decomposition has been applied to control such systems. While some results have introduced time-scale decomposition to design nonlinear dual-time-scale combined robust controllers, this requires a completely known system model and fully measurable virtual subsystem states. Currently, there is no H-infinity reduced-order output tracking control for nonlinear dual-time-scale motor systems with unknown dynamics.

[0004] In actual industrial production, establishing accurate models of systems is often difficult. Reinforcement learning, benefiting from the interactive nature of agents and their environment, possesses a unique advantage in handling model-free control problems. It utilizes system input-output data to obtain ideal control laws, thus solving optimal tracking control problems. Currently, many methods have emerged to overcome the adverse effects of disturbances within the reinforcement learning framework. As a mainstream disturbance rejection method, H-infinity control based on reinforcement learning has attracted widespread attention. Transforming the H-infinity control problem into a zero-sum game and solving it using optimal control principles has proven to be an effective approach. However, due to the high dimensionality and fast-slow dynamic coupling characteristics of dual-timescale systems, existing reinforcement learning methods are not suitable for the aforementioned motor systems, and may even lead to ill-conditioned numerical problems during iterative learning. Therefore, there is an urgent need to develop a self-learning H-infinity reduced-order output tracking control method for motor systems, capable of achieving H-infinity reduced-order output tracking control for systems other than those described above, even with unknown dynamics and imperfect data. Summary of the Invention Summary of the Invention

[0006] To address the aforementioned technical shortcomings, the purpose of this invention is to provide a reinforcement learning-based method for reducing the order of output tracking control of an H-infinity motor system. This method can solve the problem of interference suppression tracking control in motor systems containing unmodeled dynamics and imperfect data, and avoids the potential high-dimensionality and ill-conditioned numerical problems when designing a dual-timescale motor system tracking controller under the reinforcement learning framework.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A reinforcement learning-based reduced-order output tracking control method for an H-infinity motor system, applicable to servo motors, process industry systems, etc., includes the following steps:

[0009] Step 1: Using singular perturbation theory, the original motor system H infinite output tracking control problem is decomposed into a reduced-order problem;

[0010] Step 2: Based on the output state data of the original system, a state reconstruction mechanism for the virtual subsystem is proposed to solve the problem of unmeasurable data in the virtual subsystem. Furthermore, an H-infinite output tracking reinforcement learning iterative algorithm based on the reconstructed data is derived.

[0011] Step 3: Introduce the execution-evaluation-perturbation neural network approximation controller, performance metrics, and perturbations. Iteratively update the weights of the neural network using the least squares method to obtain a reduced-order tracking controller based on reinforcement learning.

[0012] Preferably, in step one, the motor system is described using the following state-space model:

[0013]

[0014]

[0015] Where x1 and x2 are the state variables of the motor system, and u = [u1, ..., u2]. m ] is the control input, w = [w1, ... w q ] is an external disturbance, f 11 f 12 f 21 f 22 This is the system dynamics, g1 and g2 are the input dynamics, k is the perturbation dynamics and 0 < ε << 1 is the singular perturbation parameter; assume f 11 f 12 f 21 f 22 g1, g2, k are completely unknown and Lipchitz continuous, f(0) = 0 and f 22 Reversible; the fast subsystem asymptotically stabilizes within a short time without the application of a fast controller.

[0016] To make the system's slow state x1 track a bounded reference trajectory r(t), assume there exists a Lipchitz continuous function such that...

[0017]

[0018] Define the tracking error as

[0019] ρ=Cx1-r(t);

[0020] Tracking error dynamics

[0021]

[0022] The original H-infinity output tracking control problem is: design a state feedback controller u=χ(ρ,r) that satisfies the L2 gain condition defined by the following formula when there is a disturbance, and the tracking error converges to 0 when there is no disturbance;

[0023]

[0024] Where ||z|| 2 =ρ T Qρ+u T Ru is the defined virtual control output, α>0 is the discount factor, γ represents the attenuation level from the disturbance input w(t) to the defined performance output variable z(t), and Q=[C1 C2] T [C1 C2]>0, R>0;

[0025] The original system can be simplified to the following reduced-order system:

[0026]

[0027] y = Cx 1s ;

[0028] Where C is the system output matrix, x 1s For the reduced-order system state and

[0029] F s (x 1s )=f 11 (x 1s )-f 12 (x 1s )f 22 -1 (x 1s )f 21 (x 1s )

[0030] G s (x 1s )=g1(x 1s )-f 12 (x1s )f 22 -1 (x 1s )g2(x 1s );

[0031] K s (x 1s )=k(x 1s )

[0032] The H-infinity reduced-order output tracking control problem simplifies to the following reduced-order output tracking problem:

[0033] Design controller u s This makes the output state trajectory Cx of the reduced-order system... 1s Track the reference trajectory r(t);

[0034] Define the tracking error of the reduced-order system as

[0035] ρ s =Cx 1s -r(t);

[0036] Tracking error dynamics

[0037]

[0038] The virtual control output is defined as follows:

[0039] ||z|| 2 =ρ s T Qρ s +u s T Ru s ;

[0040] The objective of the H-infinite reduced-order output tracking control problem is to determine the tracking error ρ. s Given a reference trajectory r, find a control strategy u with a smooth function x. s =χ(ρ s (r), such that it satisfies the following conditions:

[0041] 1) In the presence of disturbances, the system satisfies the following L2 gain condition:

[0042]

[0043] 2) In the absence of disturbances, the output tracking error approaches 0.

[0044] Preferably, in step two, the state reconstruction mechanism of the virtual subsystem is as follows: the unmeasurable virtual subsystem state is reconstructed using the original system's slow dynamic state x1, and the infinite reinforcement learning iterative algorithm for the slow subsystem H based on the reconstructed data x1 is as follows:

[0045]

[0046] in, i represents the iteration metric for the slow controller.

[0047] Preferably, in step three, the slow controller design method based on reinforcement learning specifically includes:

[0048] a: Select the linear independent activation function vectors for the evaluation neural network, the execution neural network, and the perturbation neural network as follows: Design Evaluation-Execution-Perturbation Neural Network for Approximate Performance Metrics J rec Controller u rec and disturbance w rec :

[0049]

[0050]

[0051]

[0052] in, These represent the weight vectors of the evaluation neural network, the execution neural network, and the perturbation neural network, respectively.

[0053] b: Initialize the neural network weight vector Given an initially stable execution network and perturbed network weights Different behavioral strategies u s Under the influence of w, data is collected from the original system for {X}. 1(n) ,u s(n) ,w,X′ 1(n)} and put it into the sample set In this case, the number of samples collected is N. s n = 1, ..., N s ;

[0054] c: Utilize and W (i) Further construct the database and update the weights of the evaluation-execution-perturbation neural network simultaneously using the least squares method:

[0055]

[0056] in,

[0057] Preferably, the reinforcement learning-based reduced-order output tracking controller for the motor system H is:

[0058]

[0059] The beneficial effects of this invention are:

[0060] 1) By using singular perturbation theory, the original motor system H infinite output tracking control problem is decomposed into a reduced-order slow subsystem problem, thus avoiding the occurrence of ill-conditioned numerical problems;

[0061] 2) Based on the output state data of the original system, a state reconstruction mechanism for the virtual subsystem is proposed to solve the problem of unmeasurable data in the virtual subsystem. Furthermore, an H-infinite output tracking reinforcement learning iterative algorithm based on the reconstructed data is derived.

[0062] 3) By introducing reinforcement learning algorithms into the motor control system, and using the execution-evaluation-disturbance neural network to approximate the controller, performance index and disturbance, the weights of the neural network are iteratively updated based on the least squares method to obtain a reduced-order H-infinite output tracking controller based on reinforcement learning. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a diagram of the reduced-order output tracking control framework for an H-type motor system based on reinforcement learning, provided in an embodiment of the present invention.

[0065] Figure 2 This is a schematic diagram of the convergence process of the evaluation neural network weights provided in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the first execution neural network weight convergence process provided in this embodiment of the invention;

[0067] Figure 4 This is a schematic diagram of the second execution neural network weight convergence process provided in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the convergence process of the perturbation neural network weights provided in an embodiment of the present invention;

[0069] Figure 6 This is the state trajectory tracking curve of the closed-loop motor system under the action of the optimal control law provided in the embodiment of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Example 1; see Figure 1 A reinforcement learning-based method for reducing the order of output tracking control of an infinite motor system H includes the following steps:

[0072] Step 101: Using singular perturbation theory, the original motor system H infinite output tracking control problem is decomposed into a reduced-order subproblem, thus avoiding the occurrence of ill-conditioned numerical problems;

[0073] The specific method is as follows:

[0074] (1-1) For the nonlinear dual-time-scale motor system, without loss of generality, the state-space model of the system is described as follows:

[0075]

[0076]

[0077] Where x1 and x2 are the state variables of the motor system, and u = [u1, ..., u2]. m ] is the control input, w = [w1, ... w q ] is an external disturbance, f 11 ,f 12 ,f 21 ,f 22 This represents the system dynamics, g1 and g2 are the input dynamics, and k is the perturbation dynamics, where 0 < ε << 1 and is the singular perturbation parameter. Assume f... 11 ,f 12 ,f 21 ,f 22 g1, g2, k are completely unknown and Lipchitz continuous, f(0) = 0 and f 22 It is reversible, and the fast subsystem asymptotically stabilizes in a very short time without the application of a fast controller.

[0078] To make the system's slow state x1 track a bounded reference trajectory r(t), assume there exists a Lipschitz continuous function such that...

[0079]

[0080] Define the tracking error as

[0081] ρ=Cx1-r(t)(3)

[0082] Tracking error dynamics

[0083]

[0084] (1-2) The original H-infinity output tracking control problem is: design a state feedback controller u=χ(ρ,r) that satisfies the L2 gain condition defined by the following formula when there is a disturbance, and the tracking error converges to 0 when there is no disturbance.

[0085]

[0086] Where ||z|| 2 =ρ T Qρ+u T Ru is the defined virtual control output, α>0 is the discount factor, γ represents the attenuation level from the disturbance input w(t) to the defined performance output variable z(t), and Q=[C1 C2] T [C1 C2]>0, R>0.

[0087] The decomposed slow subproblem (1-3) is: design a controller u s This causes the slow subsystem to output the state trajectory Cx. 1s Track the reference trajectory r(t).

[0088] Define the tracking error of the reduced-order system as

[0089] ρ s =Cx 1s -r(t)(6)

[0090] Tracking error dynamics

[0091]

[0092] The virtual control output is defined as follows:

[0093] ||z|| 2 =ρ s T Qρ s +u s T Ru s (8)

[0094] The objective of the H-infinite reduced-order output tracking control problem is to determine the tracking error ρ. s Given a reference trajectory r, find a control strategy u with a smooth function x. s =χ(ρ s (r), such that it satisfies the following conditions:

[0095] 1) In the presence of disturbances, the system satisfies the following L2 gain condition:

[0096]

[0097] 2) In the absence of disturbances, the output tracking error approaches 0.

[0098] Step 102: Based on the output state data of the original system, a state reconstruction mechanism for the virtual subsystem is proposed to solve the problem of unmeasurable data in the virtual subsystem. Furthermore, an H-infinite output tracking reinforcement learning iterative algorithm based on the reconstructed data is derived; including the following steps:

[0099] (2-1) The state of the unmeasurable virtual subsystem is reconstructed using the slow dynamic state x1 of the original system. The infinite reinforcement learning iterative algorithm for the slow subsystem H based on the reconstructed data x1 is as follows:

[0100]

[0101] in, i represents the iteration metric for the slow controller.

[0102] Step 103: Introduce reinforcement learning algorithms into the motor control system, and use an execution-evaluation-perturbation neural network to approximate the controller, performance indicators, and perturbations. Iteratively update the weights of the neural network based on the least squares method to obtain a reduced-order H-infinite output tracking controller based on reinforcement learning. This includes the following steps:

[0103] (3-1) Design a slow controller based on reinforcement learning, specifically as follows:

[0104] a: Select the linear independent activation function vectors of the slow evaluation neural network, the execution neural network, and the perturbation neural network as follows: Design Evaluation-Execution-Perturbation Neural Network for Approximate Chronic Energy Index J rec Slow controller u rec and disturbance w rec :

[0105]

[0106]

[0107]

[0108] in, These represent the weight vectors of the slow evaluation neural network, the slow execution neural network, and the l-th perturbation neural network, respectively.

[0109] b: Initialize the neural network weight vector Given an initially stable execution network and perturbed network weights Different behavioral strategies u sUnder the influence of w, data is collected from the original system. and put it into the sample set In this case, the number of samples collected is N. s n = 1, ..., N s .

[0110] c: Utilize and W (i) Further construct the database and update the weights of the evaluation-execution-perturbation neural network simultaneously using the least squares method:

[0111]

[0112] in,

[0113] Will Acting on the original motor system;

[0114] Design a reinforcement learning-based reduced-order output tracking controller for an H-type motor system.

[0115] u = u s (15).

[0116] Example 2

[0117] To enable those skilled in the art to better understand the present invention, a method for reducing the order of output tracking control of an H-infinity motor system based on reinforcement learning will be described in detail below with reference to specific embodiments;

[0118] Consider the following permanent magnet synchronous motor model:

[0119]

[0120]

[0121]

[0122] Among them, the extreme pair number n p =4, coefficient of viscous friction B υ =0.005 N·m·s, stator resistance R s =10.7Ω, composite rotor flux Direct-axis and quadrature-axis inductors L d =L q =0.0098mH, moment of inertia Select state variables Let u be the motor speed, direct-axis current, and quadrature-axis current, and let u = [u1 u2]. T =[u d u q ] T For direct-axis and quadrature-axis voltages, external disturbances For load torque, time-scale parameter get

[0123] x1=-0.238x1+2.0114x2-4.7619w

[0124]

[0125] The control objective of this embodiment is to design a state feedback controller that enables the motor system to operate along a given reference trajectory when w≡0, while satisfying the L2 gain. Choose Q and R as first-order and second-order identity matrices, respectively, and γ = 1.

[0126] In designing the H-infinite output tracking controller, four neural networks are introduced, including one evaluation neural network, two execution neural networks, and one perturbation neural network. The reference trajectory is chosen as r = 0.2cos(0.2t), with an initial value of 0, x1 with an initial value of 1, C = 1, Q = 1, R = 1, α = 0.2, and γ = 1. The neural network basis function for the evaluation function is σ = [ρ 2 ,ρ 3 ,ρ 4 ,r,r 3 The basis functions for the execution network and the perturbation network are: Initial weights are

[0127] Probe noise is applied, sample data is collected, and after iteration, the weights of each neural network converge. The iterative process of evaluating the neural network weights in the slow subsystem is as follows: Figure 2 As shown, the neural network weight iteration process is as follows: Figures 2-4 As shown, the iterative process of the perturbation neural network weights is as follows: Figure 5 As shown. Based on the weights of the execution neural network and combined with equation (12), the H-infinite reduced-order tracking controller (15) can be obtained.

[0128] The state trajectory curve of the closed-loop motor system under the action of the reduced-order tracking controller is as follows: Figure 6 As shown, the system operates according to the given reference trajectory without disturbance.

[0129] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for reducing the order of output tracking control of an H-type motor system based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Using singular perturbation theory, the original motor system H infinite output tracking control problem is decomposed into a reduced-order system problem; Step 2: Based on the output state data of the original system, a state reconstruction mechanism for the virtual subsystem is proposed to solve the problem of unmeasurable data in the virtual subsystem. Furthermore, an H-infinite output tracking reinforcement learning iterative algorithm based on the reconstructed data is derived. Step 3: Introduce an execution-evaluation-perturbation neural network approximation controller, performance metrics, and perturbations. Iteratively update the weights of the neural network using the least squares method to obtain a reduced-order controller based on reinforcement learning. In step one, the motor system is described using the following state-space model: ； in , For the state variables of the motor system, It is a control input. It is an external disturbance. It is a system dynamic. It is input dynamic. It is a perturbation dynamic and These are singular perturbation parameters; assuming Completely unknown and Lipchitz continuous and Reversible; the fast subsystem asymptotically stabilizes within a short time without the application of a fast controller. To make the system slow state Tracking a bounded reference trajectory Suppose there exists a Lipchitz continuous function such that ； Define the tracking error as ； Tracking error dynamics ； The original problem of infinite output tracking control is: design a state feedback controller. The L2 gain condition, as defined by the following formula, is satisfied when there is a disturbance, and the tracking error converges to 0 when there is no disturbance. ； in For the defined virtual control output, It is a discount factor. Indicates from interference input To the defined performance output variable The attenuation level, , ; The original system can be simplified to the following reduced-order system: ； ； Where C is the system output matrix. For the reduced-order system state and ； The original H-infinity output tracking control problem simplifies to the following H-infinity reduced-order output tracking problem: Design controller This makes the output state trajectory of the reduced-order system... Tracking reference trajectory ; Define the tracking error of the reduced-order system as ； Tracking error dynamics ； The virtual control output is defined as follows: ； The goal of the H-infinite reduced-order output tracking control problem is to determine the tracking error... and reference trajectory Find a smooth function control strategy Make it satisfy the following conditions: 1) In the presence of disturbances, the system satisfies the following: Gain condition: ； 2) In the absence of disturbances, the output tracking error approaches 0.

2. The reinforcement learning-based reduced-order output tracking control method for an H-type motor system according to claim 1, characterized in that, In step two, the state reconstruction mechanism of the virtual subsystem is as follows: utilizing the slow dynamic state of the original system. Reconstructing the state of unpredictable virtual subsystems based on reconstruction data. The infinite reinforcement learning iterative algorithm for the slow subsystem H is as follows: ； in, , This is the iteration metric for the slow controller.

3. The reinforcement learning-based reduced-order output tracking control method for an H-type motor system according to claim 1, characterized in that, In step three, the slow controller design method based on reinforcement learning is as follows: a: Select the linear independent activation function vectors for the evaluation neural network, the execution neural network, and the perturbation neural network as follows: Design Evaluation-Execution-Perturbation Neural Network for Approximate Performance Metrics controller and disturbances : ； ； ； in, , , These represent the weight vectors of the evaluation neural network, the execution neural network, and the perturbation neural network, respectively. b: Initialize the neural network weight vector Given the initial stable execution network and the perturbation network weights , In different behavioral strategies , Under this effect, data collected from the original system is... and put it into the sample set In this process, the number of samples collected was... , ; c: Utilize and Further construct the database and update the weights of the evaluation-execution-perturbation neural network simultaneously using the least squares method: ； in, .

4. The reinforcement learning-based reduced-order output tracking control method for an H-type motor system according to claim 1, characterized in that, The reinforcement learning-based motor system H infinite order reduced output tracking controller is: 。

Images