Electromagnetic micromirror output feedback tracking control method and device based on adaptive Q learning

Abstract

The invention discloses an electromagnetic micromirror output feedback tracking control method and device based on adaptive Q learning, and the method comprises the steps: constructing an electromagnetic micromirror model based on a physical structure of an electromagnetic micromirror; combining the electromagnetic micro-mirror model with a reference trajectory to construct an augmentation system so as to convert a trajectory tracking problem into a state adjustment problem; based on the augmented system, designing a Q function Bellman equation to establish a data-driven performance evaluation framework; based on the augmented system, the state of the augmented system is reconstructed through historical input and output data, so that angular velocity measurement is avoided; obtaining an optimal control input and a new Q function Bellman equation based on the reconstructed augmented system state and the Q function Bellman equation; and based on the optimal control input and the new Q function Bellman equation, solving a control problem by using an adaptive Q learning algorithm. The method breaks through the limitation that a traditional model depends on a control method, and achieves optimal tracking control only through historical input and output data.

Description

Technical Field

[0001] The present invention relates to the field of micro-electromechanical system intelligent control technology, and in particular to an output feedback tracking control method, device, terminal equipment and computer-readable storage medium of an electromagnetic micromirror based on adaptive Q learning. Background Art

[0002] Microelectromechanical system (MEMS) electromagnetic micromirrors, core components in modern optical sensing and communications, have been widely used in applications such as lidar scanning, fiber optic signal modulation, and endoscopic imaging, thanks to their high-precision dynamic deflection capabilities at the micro- and nanoscale. Their operating principle is based on an electromagnetic drive mechanism, which adjusts the mirror's deflection angle by controlling the coil current, enabling precise beam pointing and scanning. However, as optical systems demand ever-increasing resolution and response speed, micromirror control faces multiple technical bottlenecks.

[0003] Current control schemes, such as PID control, sliding mode control, and flat control, rely on precise dynamic modeling. In practice, frequent calibration is required due to the time-varying nature of parameters, making them difficult to meet industrial-grade reliability requirements. While robust control can mitigate parameter uncertainty, its conservative design based on worst-case scenarios sacrifices dynamic performance and energy efficiency. In recent years, model-free reinforcement learning techniques (such as Q-learning) have provided new insights into micromirror control, but their practical application remains limited by two major issues. First, policy iteration (PI) algorithms require an initial stable policy, but the nonlinear nature of micromirrors makes initial controller design difficult. Second, value iteration (VI) algorithms, while relaxing initial conditions, converge slowly, making them unsuitable for high-speed scanning scenarios. Furthermore, existing learning algorithms often rely on full-state feedback, which conflicts with the engineering reality that the micromirror's angular velocity cannot be measured. Forcibly introducing an observer increases system complexity and noise sensitivity. Summary of the Invention

[0004] The present invention addresses key control challenges faced by MEMS electromagnetic micromirrors in practical applications, such as parameter uncertainty, limited state measurement, and noise interference. It provides an output feedback tracking control method, device, terminal equipment, and computer-readable storage medium for an electromagnetic micromirror based on adaptive Q learning. This method breaks through the limitations of traditional model-dependent control methods and achieves optimal tracking control by relying solely on system input and output data.

[0005] The first object of the present invention is to provide an output feedback tracking control method of an electromagnetic micromirror based on adaptive Q learning.

[0006] The second object of the present invention is to provide an output feedback tracking control device of an electromagnetic micromirror based on adaptive Q learning.

[0007] The third object of the present invention is to provide a terminal device.

[0008] A fourth object of the present invention is to provide a computer-readable storage medium.

[0009] The first object of the present invention can be achieved by adopting the following technical solutions:

[0010] An output feedback tracking control method for an electromagnetic micromirror based on adaptive Q-learning, the method comprising:

[0011] Based on the physical structure of the electromagnetic micromirror, an electromagnetic micromirror model is constructed. The electromagnetic micromirror model is combined with the reference trajectory to construct an augmented system to transform the trajectory tracking problem into a state adjustment problem.

[0012] Based on the augmented system, the Q-function Bellman equation is designed to establish a data-driven performance evaluation framework;

[0013] Based on the augmented system, the historical input and output data are used to reconstruct the augmented system state to avoid angular velocity measurement;

[0014] Based on the reconstructed augmented system state and the Q-function Bellman equation, the optimal control input and the new Q-function Bellman equation are obtained;

[0015] Based on the optimal control input and the new Q-function Bellman equation, an adaptive Q-learning algorithm is used to solve the control problem; the adaptive Q-learning algorithm optimizes the iterative strategy through dynamic hyperparameters.

[0016] Furthermore, define x k 、 are the states of the electromagnetic micromirror model and the reference trajectory generation system at time k, respectively. The augmented system is:

[0017] X k+1 =A1X k +B1u k

[0018] e k =C1X k

[0019] in: B1=[B 0] T , C1=[CG]

[0020] Where, X k+1 、X k are the states of the augmented system at time k+1 and time k, respectively, e k is the output error at time k, u k is the control input, A, B, C, F, and G are all constant matrices.

[0021] Furthermore, the Q-function Bellman equation is designed based on the augmented system, including:

[0022] Based on the electromagnetic micromirror model and the reference trajectory generation system, the output error is:

[0023] e k =y k -r k

[0024] Where y k 、r k are the outputs of the electromagnetic micromirror model and the reference trajectory generation system at time k respectively;

[0025] The optimal discount performance indicator function is defined as:

[0026]

[0027] Where x k ∈R d is the state of the electromagnetic micromirror model, Q and R are both positive weight matrices, Υ is the discount factor and satisfies 0<Υ≤1, e i 、u i are the output error and control input at time i respectively;

[0028] The input control of the augmented system is denoted as K x 、 If both are constant matrices, the optimal discounted performance index function is transformed into a quadratic form:

[0029]

[0030] Where P = P T >0; is the reference state of the system at time k;

[0031] will u k Substituting into formula (3-1), we have:

[0032]

[0033] Then we have:

[0034]

[0035] Substituting equation (3-2) into equation (3-3), we get the Bellman equation of the Q function:

[0036]

[0037] in,

[0038] The Hamiltonian function is defined as follows:

[0039]

[0040] According to the necessary conditions for optimality, find u for the Hamiltonian function k The partial derivative of We can get:

[0041]

[0042] in, P satisfies:

[0043]

[0044] Furthermore, the optimal control input and the new Q-function Bellman equation are obtained based on the reconstructed augmented system state and the Q-function Bellman equation, including:

[0045] definition Introduce the following Q function:

[0046]

[0047] Combining the augmented system and Q function, we have:

[0048]

[0049] in,

[0050] According to the optimality principle, let Available Q(Z k )=V(X k ), so the Q function Bellman equation is converted to the following Q function Bellman equation:

[0051]

[0052] The reconstructed augmented system state X k Substituting into formula (4-1) we have:

[0053]

[0054] in:

[0055]

[0056]

[0057] U N =[B AB...A N-1 B],

[0058] W N =[(CA N-1 ) T …(CA) T C] T ,

[0059] Where, r k-N are the historical input, output data and reference trajectory data of the augmented system at the time interval [kN, k-1]; N≤n, n is the dimension of the state space;

[0060] Pick Obtain the OPFB learning control strategy under the optimal criterion for:

[0061]

[0062] Substituting equation (4-2) into the transformed Q-function Bellman equation, we get the new Q-function Bellman equation as follows:

[0063]

[0064] in, v k =[y k r k ] T ,

[0065] Furthermore, the adaptive Q-learning algorithm includes the following steps:

[0066] Step 1, initialization: j = 0, with any initial control law Start the iterative process;

[0067] Step 2, strategy evaluation: For j = 0, 1, ..., collect data samples and use the new Q function Bellman equation to calculate H j+1 ,Right now:

[0068]

[0069] Step 3: Strategy improvement: Update the control law using the optimal control input Right now:

[0070]

[0071] Step 4: When ||H is satisfied j+1 -H j ||<ε, terminate the iteration, where ε is a pre-set threshold; otherwise, j=j+1, and return to step 2 to continue executing subsequent steps;

[0072] Among them, β j is a dynamic hyperparameter.

[0073] Furthermore, the augmented system-based reconstruction of the augmented system state using historical input and output data includes:

[0074] Define X k =[M u M y M r ],in:

[0075]

[0076] U N =[B AB...A N-1 B],

[0077] W N =[(CA N-1 ) T …(CA) T C] T ,

[0078] For an augmented system with a time interval of [kN, k-1], N≤n, n is the dimension of the state space; if the matrix [A, C] is observable, then the historical input and output data And the reference trajectory data r k-N , reconstruct the state X of the augmented system k for:

[0079]

[0080] in:

[0081]

[0082] Furthermore, the electromagnetic micromirror model is:

[0083] x k+1 =Ax k +Bu k

[0084] y k =Cx k

[0085] Where x k ∈R d is the system status, y k =θ k ∈R q is the system output, and the matrices (A, B) and (A, C) are controllable and detectable respectively;

[0086] The reference trajectory generation system is:

[0087]

[0088] Where, are the reference states of the reference trajectory generation system at time k+1 and time k, r k is the reference output at time k, F and G are both constant matrices;

[0089] The output error is:

[0090] e k =y k -r k

[0091] definition Combining the electromagnetic micromirror model, the reference trajectory generation system and the output error, the expression of the augmented system is obtained.

[0092] The second object of the present invention can be achieved by adopting the following technical solutions:

[0093] An output feedback tracking control device for an electromagnetic micromirror based on adaptive Q learning, the device comprising:

[0094] A building module is used to construct an electromagnetic micromirror model based on the physical structure of the electromagnetic micromirror; the electromagnetic micromirror model is combined with a reference trajectory to construct an augmented system to transform the trajectory tracking problem into a state adjustment problem;

[0095] A design module for designing the Q-function Bellman equation based on the augmented system to establish a data-driven performance evaluation framework;

[0096] A reconstruction module is used to reconstruct the state of the augmented system based on the augmented system using historical input and output data to avoid angular velocity measurement;

[0097] A calculation module, used for obtaining an optimal control input and a new Q-function Bellman equation based on the reconstructed augmented system state and the Q-function Bellman equation;

[0098] The solution module is used to solve the control problem based on the optimal control input and the new Q-function Bellman equation using an adaptive Q-learning algorithm; the adaptive Q-learning algorithm optimizes the iterative strategy through dynamic hyperparameters.

[0099] The third object of the present invention can be achieved by adopting the following technical solutions:

[0100] A terminal device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the output feedback tracking control method of the electromagnetic micromirror based on adaptive Q learning is implemented.

[0101] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0102] A computer-readable storage medium stores a program, which, when executed by a processor, implements the output feedback tracking control method of an electromagnetic micromirror based on adaptive Q learning.

[0103] The present invention has the following beneficial effects compared to the prior art:

[0104] 1. By constructing an augmented system and utilizing state reconstruction technology, the present invention effectively solves the engineering problem of the difficulty in directly measuring the angular velocity of a micromirror system. State estimation can be achieved with only historical input and output data, greatly enhancing the engineering practicality of the method.

[0105] 2. The adaptive Q-learning algorithm designed in this paper creatively combines the dual advantages of policy iteration and value iteration. By introducing an adjustable hyperparameter mechanism, it not only breaks away from the dependence on the initial stable policy, but also maintains a relatively fast convergence speed, thus improving learning efficiency while ensuring the universality of the algorithm.

[0106] 3. The adaptive Q-learning algorithm designed in this invention exhibits strong robustness to system parameter changes and external noise interference. Through the design of the Bellman equation in the form of a Q function, the control strategy has a natural immunity to excitation noise. This feature is particularly important for MEMS devices working in complex electromagnetic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0108] Figure 1 Flowchart of the output feedback tracking control method of the electromagnetic micromirror based on adaptive Q learning according to Example 1 of the present invention;

[0109] Figure 2 Schematic diagram of the structure of the electromagnetically driven micromirror according to embodiment 1 of the present invention;

[0110] Figure 3 The different moments of inertia I of Example 1 of the present invention mThe tracking error curve below:

[0111] Figure 4 : Tracking error curves under different stiffness coefficients K of Example 1 of the present invention;

[0112] Figure 5 The tracking error curve when the first element of the output matrix C changes in embodiment 1 of the present invention;

[0113] Figure 6 The moment of inertia I under different excitation noises in Example 1 of the present invention is m Tracking error curve when changing;

[0114] Figure 7 This is a structural block diagram of an output feedback tracking control device of an electromagnetic micromirror based on adaptive Q learning according to Example 2 of the present invention;

[0115] Figure 8 This is a structural block diagram of the terminal device of embodiment 3 of the present invention. DETAILED DESCRIPTION

[0116] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. It should be understood that the specific embodiments described are only used to explain this application and are not used to limit this application.

[0117] Example 1:

[0118] like Figure 1 As shown, this embodiment provides an output feedback tracking control method of an electromagnetic micromirror based on adaptive Q learning, comprising the following steps:

[0119] S101. Based on the physical structure of the electromagnetic micromirror, construct an electromagnetic micromirror model; combine the electromagnetic micromirror model with the reference trajectory to construct an augmented system to transform the trajectory tracking problem into a state adjustment problem;

[0120] S102. Based on the augmented system, design the Q-function Bellman equation to establish a data-driven performance evaluation framework;

[0121] S103, converting the Q function Bellman equation; based on the augmented system, reconstructing the state information of the augmented system using historical input and output data and reference trajectory data to avoid angular velocity measurement;

[0122] S104, obtaining an optimal control input and a new Q-function Bellman equation based on the reconstructed state information of the augmented system and the converted Q-function Bellman equation;

[0123] S105. Based on the optimal control input and the new Q-function Bellman equation, the control problem is solved using an adaptive Q-learning algorithm.

[0124] Furthermore, step S101 specifically includes:

[0125] The physical structure of the MEMS electromagnetically driven micromirror can be referred to Figure 2 , the dynamic model of the MEMS electromagnetic micromirror is constructed as follows:

[0126]

[0127] Among them, θ, and are deflection angle, deflection velocity and deflection acceleration respectively, I m is the moment of inertia, B m is the damping coefficient, K m is the angular spring constant, T field is the electromagnetic torque.

[0128] Based on the physical structure, the driving current i generates a magnetic torque T through the planar microcoil. field , and for the electromagnetic micromirror whose mirror plate is composed of polymer composite hard magnetic material, the driving current i and the electromagnetic torque T field The relationship can be considered as a linear relationship, that is, T field (θ)=Ki (K is a constant). Therefore, the driving current i can be regarded as the control input of the dynamic model.

[0129] Define x1 = θ, u=i, then the dynamic model can be rewritten as:

[0130]

[0131] Inconsistency in industrial manufacturing can lead to parameter uncertainty. To solve this problem, the nominal value of the parameter vector is defined as Parameter change value in In the range, the actual value of the parameter vector Ω=(K m ,B m ,I m ,K) can be decomposed into

[0132] The actual control system of MEMS micromirrors is usually based on a digital processor (such as a DSP or microcontroller), so the dynamic model (2) is discretized:

[0133]

[0134] Among them, x k ∈R d is the system status, u k ∈R p is the control input, y k =θ k ∈R q is the system output, A, B, and C are given constant matrices (in the model-free tracking control framework), and the matrices (A, B) and (A, C) are controllable and detectable, respectively.

[0135] Considering the scanning application of MEMS, assume that the reference trajectory generation system is as follows:

[0136]

[0137] in, is the reference state of the system at time k, r k is the reference output at time k, and F and G are constant matrices of appropriate dimensions.

[0138] The output error can be expressed as:

[0139] e k =y k -r k (5)

[0140] The optimal discount performance indicator function is defined as follows:

[0141]

[0142] Where Q and R are positive definite weight matrices, and Υ is a discount factor that satisfies 0<Υ≤1.

[0143] Further definition Then we have the augmented system:

[0144]

[0145] in, B1=[B 0] T , C1=[C -G].

[0146] X k+1 、X k are the states of the augmented system at time k+1 and time k respectively.

[0147] This embodiment aims to propose an OPFB-based control scheme for the micromirror dynamic model (2), so that the deflection angle θ of the micromirror can achieve excellent tracking performance in a certain optimal sense given a sinusoidal trajectory. Therefore, the control problem can be formulated as the design of an optimal tracking controller for a system with unknown parameter values (i.e., the augmented system (7)).

[0148] Furthermore, step S102 specifically includes:

[0149] The input control of the augmented system is denoted as Then the optimal discounted performance index function (6) can be transformed into a quadratic form:

[0150]

[0151] Where P = P T >0.

[0152] Further u k Substituting into (6), we have:

[0153]

[0154] Then we have:

[0155]

[0156] Substituting (8) into (9), we obtain the Bellman equation of the Q function:

[0157]

[0158] in,

[0159] The Hamiltonian function is defined as follows:

[0160]

[0161] According to the necessary conditions for optimality, we can find u in the above formula. k The partial derivative of We can get:

[0162]

[0163] in, P satisfies:

[0164]

[0165] Due to the nonlinear nature, it is difficult to solve P directly from the above equation. Therefore, substituting (11) into (10), we have:

[0166] Π-P+Υ(A1-B1K) TP(A1-B1K)+K T RK=0 (12)

[0167] When all the parameter values of the augmented system (7) are available, the solution of (12) can be obtained by an iterative process. However, this approach is conservative and in practical applications, it is desirable to design the optimal controller without relying on the parameter information of the system model.

[0168] Furthermore, step S103 specifically includes:

[0169] definition Introduce the following Q function:

[0170]

[0171] Combining (7) and (13), we have:

[0172]

[0173] in,

[0174] According to the optimality principle, let Available Further note that Q(Z k )=V(X k ), so the Q-function Bellman equation (10) can be transformed into the following Q-function Bellman equation:

[0175]

[0176] In order to reconstruct the state information of the system using the historical input and output data of the system and the reference trajectory data, the observation matrix is defined as follows:

[0177] X k =[M u M y M r ], where M u 、M y and M r They are defined as:

[0178]

[0179] Among them, U N =[B AB...A N-1 B],

[0180] W N =[(CA N-1 ) T …(CA) T CT ,

[0181] For the augmented system (7) with a time interval of [kN, k-1] (N≤n, where n is the dimension of the state space), assuming that the matrix [A, C] is observable, it can be obtained by inputting and outputting the data sequence Historical information and reference trajectory data r k-N , reconstruct the state X of the augmented system k ,as follows:

[0182]

[0183] in:

[0184]

[0185] Furthermore, step S104 specifically includes:

[0186] To transform Equation (15) into a new Q-function Bellman equation represented by the historical input and output data of the augmented system (7), the reconstructed state X k Substituting into formula (14), we have:

[0187]

[0188] in,

[0189]

[0190] The element expressions in formula (17) are as follows:

[0191]

[0192] On this basis, take To give the OPFB learning control strategy under the optimal criterion

[0193]

[0194] Substituting equation (16) into equation (15), we get the new Bellman equation of the Q function as follows:

[0195]

[0196] in, v k =[y k r k ] T , Furthermore, step S105 specifically includes:

[0197] Based on Equations (19) and (20), an adaptive Q-learning algorithm is proposed. The adaptive Q-learning algorithm integrates policy iteration (PI) and value iteration (VI), adjusting the iteration strategy through a dynamic hyperparameter β and updating the kernel matrix H online. Online kernel matrix updates eliminate the prior reliance on micromirror dynamic parameters (such as moment of inertia and damping coefficient). Online kernel matrix updates enable the algorithm to automatically compensate for parameter perturbations (such as moment of inertia drift) caused by manufacturing process inconsistencies. Dynamic adjustment of the kernel matrix suppresses noise interference and enhances robustness to electromagnetic environment noise.

[0198] The adaptive Q-learning algorithm includes the following steps:

[0199] Step 1, initialization: j = 0, with any initial control law (No need to stabilize H j ) starts the iteration process;

[0200] Step 2, strategy evaluation: For j = 0, 1, ..., collect its data samples and calculate H using Bellman equation (20) j+1 :

[0201]

[0202] Step 3: Strategy improvement: Update the control law by using (19)

[0203]

[0204] Step 4: Termination condition, when ||H is satisfied j+1 -H j If ||<ε (ε is a very small pre-set threshold, algorithm accuracy), the iteration is terminated; otherwise, j=j+1 is set and the process returns to step 2 to continue the subsequent steps.

[0205] When j→∞, the control strategy in the algorithm It will converge to the optimal OPFB strategy, i.e.,

[0206] By introducing the hyperparameter β j , the algorithm has the potential advantage of fully utilizing the advantages of PI and VI learning frameworks. From the formula (21) in the algorithm, it can be seen that if β is directly taken j =1, then (21) is simplified to:

[0207]

[0208] This is exactly the policy evaluation in the PI algorithm.

[0209] On the other hand, setting βj = 0 will simplify (21) to the following formula:

[0210]

[0211] This is the strategy evaluation in the VI algorithm.

[0212] This algorithm is a compromise between PI and VI, taking advantage of the fact that VI does not require an initial stable control strategy while maintaining the fast convergence of PI. j In addition to being set as a constant, it can also be designed to change with the number of iterations. At the beginning of the iteration process, a smaller β j To avoid the initial stable control strategy, gradually increase β as the number of iterations increases j To speed up the convergence. Based on this consideration, the adaptive parameter rule can be selected as β j =tanh[aln(i+j)], where parameter a> 0. However, the selection of parameter a also needs to be carefully determined based on actual experience, and different systems require different settings.

[0213] The following simulation experiments will verify the effectiveness of the above OPFB-based Q-learning scheme in the electromagnetic torsion micromirror system.

[0214] The nominal parameter values of the micromirror are set as:

[0215] Torque constant

[0216] Damping coefficient

[0217] moment of inertia

[0218] Stiffness coefficient

[0219] Select the sampling interval T s =10 -4 s, the discretized model of the micromirror system can be obtained, and its state space expression is:

[0220] C=[1 0]

[0221] To verify the effectiveness of the algorithm, the angular displacement output of the micromirror system is set to asymptotically track a sinusoidal reference trajectory with an amplitude of 10 and a frequency of 50 Hz. The discretization form of this reference trajectory is:

[0222] G=[1 0]

[0223] It should be noted that although the nominal parameters of the micromirror system and the reference trajectory are provided in the simulation, it is not necessary to know this information in advance when applying the proposed OPFB-based Q-learning algorithm. H in Equation (21) can be calculated by simply measuring the historical input, output, and reference trajectory data. j+1 , and updates the current control strategy based on Equation (22). Therefore, the algorithm can achieve optimal OPFB tracking control without the need for system model parameters.

[0224] The moment of inertia of the micromirror system is easily affected by the MEMS manufacturing process. Therefore, the proposed algorithm needs to ensure that the micromirror system can be used in different moments of inertia. m Good reference trajectory tracking performance can be achieved under all conditions.

[0225] In the optimal discount performance index function (Equation (6)), set the parameters γ = 0.01, Q = 1, R = 0.00001,

[0226] σ k =7sin(k)+5cos(2k)+9sin(8k)+2cos(6k)

[0227] The injected excitation noise is: According to the above settings, the experimental results are as follows Figure 3-5 As shown, Figure 3 Indicates that when the moment of inertia I m At 5.80×10 -12 to 6.00×10 -12 When the angular displacement output of the micromirror system changes within the range, it can still accurately track the sinusoidal reference signal. Figure 4 It means that when the stiffness coefficient K is 5.3×10 -4 to 5.6×10 -4 The tracking error remains converged when the Figure 5 This indicates that when the first element of the output matrix C varies within the range of 0.96 to 1.04, the algorithm can still maintain stable tracking performance. The above results show that the OPFB-based Q-learning algorithm we designed can still maintain excellent tracking performance when the system parameters change.

[0228] In order to verify the robustness of the algorithm to the excitation noise, under the same parameter settings, the following excitation noises with different frequencies and amplitudes are injected: k =3sin(2k)+8cos(3k)+10sin(4k)+8cos(7k) The experimental results are as follows Figure 6As shown in the figure, despite slight variations in transient performance, the tracking error converges to zero. Compared to traditional value function approximation (VFA) methods, the Q-learning algorithm demonstrates strong robustness to excitation noise. This experimental result demonstrates that the algorithm is particularly suitable for the noise-prone operating environment of MEMS micromirrors.

[0229] The above experimental results show that the MEMS electromagnetic micromirror output feedback tracking control method based on OPFB Q learning provided in this embodiment can achieve optimal tracking control without the need for system model parameters, while maintaining stable performance under parameter perturbations and strong noise interference. It is suitable for precision control systems such as MEMS electromagnetic micromirrors and reduces dependence on manufacturing process consistency.

[0230] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0231] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0232] Example 2:

[0233] like Figure 7 As shown, this embodiment provides an output feedback tracking control device for an electromagnetic micromirror based on adaptive Q learning, the device comprising a construction module 701, a design module 702, a reconstruction module 703, a calculation module 704, and a solution module 705, wherein:

[0234] A construction module 701 is configured to construct an electromagnetic micromirror model based on the physical structure of the electromagnetic micromirror; and to combine the electromagnetic micromirror model with a reference trajectory to construct an augmented system, thereby converting the trajectory tracking problem into a state adjustment problem.

[0235] A design module 702 is used to design a Q-function Bellman equation based on the augmented system to establish a data-driven performance evaluation framework;

[0236] A reconstruction module 703 is used to reconstruct the state of the augmented system based on the augmented system using historical input and output data to avoid angular velocity measurement;

[0237] A calculation module 704 is configured to obtain an optimal control input and a new Q-function Bellman equation based on the reconstructed augmented system state and the Q-function Bellman equation;

[0238] The solution module 705 is used to solve the control problem based on the optimal control input and the new Q-function Bellman equation using an adaptive Q-learning algorithm; the adaptive Q-learning algorithm optimizes the iterative strategy through dynamic hyperparameters.

[0239] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the device provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0240] Example 3:

[0241] This embodiment provides a terminal device, which can be a computer, such as Figure 8 As shown, it includes a processor 802, a memory, an input device 803, a display 804 and a network interface 805 connected via a system bus 801. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 806 and an internal memory 807. The non-volatile storage medium 806 stores an operating system, a computer program and a database. The internal memory 807 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 802 executes the computer program stored in the memory, the output feedback tracking control method of the electromagnetic micromirror based on adaptive Q learning in the above-mentioned embodiment 1 is implemented.

[0242] Example 4:

[0243] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the output feedback tracking control method of the electromagnetic micromirror based on adaptive Q learning of the above-mentioned embodiment 1 is implemented.

[0244] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0245] In summary, the present invention provides a model-free intelligent control method (output feedback tracking control method of electromagnetic micromirrors based on adaptive Q-learning) that only requires output feedback and integrates parameter adaptability. It can meet the following multiple requirements: no prior knowledge of micromirror dynamics is required, it is compatible with parameter perturbations, it supports a pure output feedback architecture, it takes into account both tracking accuracy and optimal energy consumption, and at the same time avoids the compromise of traditional reinforcement learning on initial strategy or convergence speed.

[0246] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. An output feedback tracking control method for an electromagnetic micromirror based on adaptive Q-learning, characterized in that: The method comprises: Based on the physical structure of the electromagnetic micromirror, an electromagnetic micromirror model is constructed. The electromagnetic micromirror model is combined with the reference trajectory to construct an augmented system to transform the trajectory tracking problem into a state adjustment problem. Based on the augmented system, the Q-function Bellman equation is designed to establish a data-driven performance evaluation framework; Based on the augmented system, the historical input and output data are used to reconstruct the augmented system state to avoid angular velocity measurement; Based on the reconstructed augmented system state and the Q-function Bellman equation, the optimal control input and the new Q-function Bellman equation are obtained; Based on the optimal control input and the new Q-function Bellman equation, an adaptive Q-learning algorithm is used to solve the control problem; the adaptive Q-learning algorithm optimizes the iterative strategy through dynamic hyperparameters.

2. The output feedback tracking control method according to claim 1, characterized in that: definition x k 、 are the states of the electromagnetic micromirror model and the reference trajectory generation system at time k, respectively. The augmented system is: X k+1 =A1X k +B1u k And k =C1X k in: B1=[B 0] T , C1=[CG] Where, X k+1 、X k are the states of the augmented system at time k+1 and time k, respectively, e k is the output error at time k, u k is the control input, A, B, C, F, and G are all constant matrices.

3. The output feedback tracking control method according to claim 2, characterized in that: The Q-function Bellman equation is designed based on the augmented system, including: Based on the electromagnetic micromirror model and the reference trajectory generation system, the output error is: e k =y k -r k Where y k 、r k are the outputs of the electromagnetic micromirror model and the reference trajectory generation system at time k respectively; The optimal discount performance indicator function is defined as: Where x k ∈R d is the state of the electromagnetic micromirror model, Q and R are both positive weight matrices, Υ is the discount factor and satisfies 0<Υ≤1, e i 、u i are the output error and control input at time i respectively; The input control of the augmented system is denoted as K x 、 If both are constant matrices, the optimal discounted performance index function is transformed into a quadratic form: Where P = P T >0; is the reference state of the system at time k; will u k Substituting into formula (3-1), we have: Then we have: Substituting equation (3-2) into equation (3-3), we get the Bellman equation of the Q function: in, The Hamiltonian function is defined as follows: According to the necessary conditions for optimality, find u for the Hamiltonian function k The partial derivative of We can get: in, P satisfies:

4. The output feedback tracking control method according to claim 3, characterized in that: The optimal control input and the new Q-function Bellman equation are obtained based on the reconstructed augmented system state and the Q-function Bellman equation, including: definition Introduce the following Q function: Combining the augmented system and Q function, we have: in, According to the optimality principle, let Available Q(Z k )=V(X k ), so the Q function Bellman equation is converted to the following Q function Bellman equation: The reconstructed augmented system state X k Substituting into formula (4-1) we have: in: Where, r k-N are the historical input, output data and reference trajectory data of the augmented system at the time interval [kN, k-1]; N≤n, n is the dimension of the state space; Pick Obtain the OPFB learning control strategy under the optimal criterion for: Substituting equation (4-2) into the transformed Q-function Bellman equation, we get the new Q-function Bellman equation as follows: in, 5. The output feedback tracking control method according to claim 4, characterized in that: The adaptive Q-learning algorithm includes the following steps: Step 1, initialization: j = 0, with any initial control law Start the iterative process; Step 2, strategy evaluation: For j = 0, 1, ..., collect data samples and use the new Q function Bellman equation to calculate H j+1 ,Right now: Step 3: Strategy improvement: Update the control law using the optimal control input Right now: Step 4: When ||H is satisfied j+1 -H j ||<ε, terminate the iteration, where ε is a pre-set threshold; otherwise, j=j+1, and return to step 2 to continue executing subsequent steps; Among them, β j is a dynamic hyperparameter.

6. The output feedback tracking control method according to any one of claims 2 to 5, characterized in that: The augmented system-based method of reconstructing the augmented system state using historical input and output data includes: Define X k =[M u M y M r ],in: For an augmented system with a time interval of [kN, k-1], N≤n, n is the dimension of the state space; if the matrix [A, C] is observable, then the historical input and output data And the reference trajectory data r k-N , reconstruct the state X of the augmented system k for: in:

7. The output feedback tracking control method according to any one of claims 2 to 5, characterized in that: The electromagnetic micromirror model is: x k+1 =Ax k +Bu k y k =Cx k Where x k ∈R d is the system status, y k =θ k ∈R q is the system output, and the matrices (A, B) and (A, C) are controllable and detectable respectively; The reference trajectory generation system is: Where, are the reference states of the reference trajectory generation system at time k+1 and time k, r k is the reference output at time k, F and G are both constant matrices; The output error is: e k =y k -r k definition Combining the electromagnetic micromirror model, the reference trajectory generation system and the output error, the expression of the augmented system is obtained.

8. An output feedback tracking control device for an electromagnetic micromirror based on adaptive Q learning, characterized in that: The device comprises: A building module is used to construct an electromagnetic micromirror model based on the physical structure of the electromagnetic micromirror; the electromagnetic micromirror model is combined with a reference trajectory to construct an augmented system to transform the trajectory tracking problem into a state adjustment problem; A design module for designing the Q-function Bellman equation based on the augmented system to establish a data-driven performance evaluation framework; A reconstruction module is used to reconstruct the state of the augmented system based on the augmented system using historical input and output data to avoid angular velocity measurement; A calculation module, used for obtaining an optimal control input and a new Q-function Bellman equation based on the reconstructed augmented system state and the Q-function Bellman equation; The solution module is used to solve the control problem based on the optimal control input and the new Q-function Bellman equation using an adaptive Q-learning algorithm; the adaptive Q-learning algorithm optimizes the iterative strategy through dynamic hyperparameters.

9. A terminal device comprising a processor and a memory for storing a program executable by the processor, wherein when the processor executes the program stored in the memory, the output feedback tracking control method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the output feedback tracking control method according to any one of claims 1 to 7 is implemented.

Images