Permanent magnet synchronous motor control method based on reinforcement learning and model predictive control

Abstract

The application discloses a permanent magnet synchronous motor control method based on reinforcement learning and model predictive control and belongs to the technical field of motor control. The application adopts two state space expressions to respectively describe the motor which is a multi-time scale system, the inner loop and the outer loop can be decoupled through reasonable equivalence and assumption; the optimal control theory is combined with integral reinforcement learning, the model error has certain self-adaptability, and the optimal reference signal under certain parameters can be given; the model predictive control algorithm is used to improve the inner loop performance, and the actual current can quickly track the reference signal.

Description

Technical Field

[0001] This invention relates to the field of motor control technology, and specifically to a control method for permanent magnet synchronous motors based on reinforcement learning and model predictive control. Background Technology

[0002] Permanent magnet synchronous motors (PMSMs) are widely used in various motor drive applications due to their high power density and simple structure. However, because of the strong coupling between the variables in a PMSM, the dynamic response times of the speed loop and current loop differ significantly. Therefore, the PMSM control system is a multi-timescale system, and finding appropriate strategies to improve the control performance of the inner current loop is a key research focus.

[0003] Traditionally, PI control is used, generating d- and q-axis voltage reference signals from d- and q-axis current reference signals. These are then transformed using an inverse dq transformation to obtain reference signals in the ABC natural coordinate system. These signals are then modulated using PWM or SVPWM to generate switching signals on the inverter, thereby driving the motor. This process requires tuning the parameters of two PI controllers; inappropriate parameter selection will severely affect the motor's speed control performance. Model Predictive Control (MPC) is a relatively new control algorithm. With the development of microprocessors, the high computing power required for MPC is no longer a bottleneck limiting its application in motor drive scenarios, and it has been widely used in motor speed control and torque control in recent years.

[0004] However, the MPC algorithm requires specific parameters of the motor system to traverse the inverter's voltage vector. If the motor system parameters are incorrect, the MPC algorithm will select the wrong vector, resulting in motor torque ripple, poor speed regulation performance, and in severe cases, even motor reversal or runaway. To overcome this deficiency, it is necessary to use various parameter identification and system identification algorithms to obtain the motor system parameters for the MPC algorithm to use.

[0005] Commonly used system identification algorithms mainly include neural networks, Kalman filters, Romberg observers, and least squares methods. These algorithms either have high computational requirements, cannot track system changes in real time, or are easily affected by various disturbances, and may identify parameters incorrectly. Therefore, a permanent magnet synchronous motor control method based on reinforcement learning and model predictive control is proposed. Summary of the Invention

[0006] The technical problem to be solved by this invention is: how to obtain motor system parameters and the optimal linear state feedback control law under given parameters through integral reinforcement learning, and how to obtain inverter voltage vector drive motor using MPC algorithm in the inner loop, thus providing a permanent magnet synchronous motor control method based on reinforcement learning and model predictive control.

[0007] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0008] S1: Define the operator

[0009] Define multiple operators;

[0010] S2: Obtain the gain matrix

[0011] The optimal control law, i.e. the gain matrix, under given parameters is obtained through an integral reinforcement learning algorithm;

[0012] S3: Obtain voltage vector

[0013] The inner loop uses a model predictive control algorithm to obtain inverter voltage vector drive motor;

[0014] S4: Motor speed control

[0015] By designing a loss function and minimizing it, a voltage vector that minimizes the loss function is applied to the inverter, enabling the motor current to track the reference signal and thus achieve the preset speed.

[0016] Furthermore, for a three-phase permanent magnet synchronous motor, its stator voltage equation in the dq coordinate system is expressed as:

[0017]

[0018] Among them, u d ,u q These are the d-axis and q-axis voltages, respectively, i d i q These are the d-axis and q-axis currents, respectively, L d ,L q These are the d-axis and q-axis inductances, respectively; ω is the electric angular velocity of the motor; and ψ... f It is a permanent magnet flux linkage.

[0019] Furthermore, for surface-mounted three-phase permanent magnet synchronous motors, L d =L q Using L instead of L d L q , take i d =0 control to obtain the maximum current-to-torque ratio, the nonlinear dynamic equation of the motor is described as follows:

[0020]

[0021]

[0022] Where, n p J is the number of pole pairs of the motor, J is the sum of the moments of inertia of the motor and the load, and T is the number of pole pairs of the motor.L It is the load torque.

[0023] Furthermore, in step S1, the defined operator is as follows:

[0024] Given a complex number U, real(U) denotes its real part and imag(U) denotes its imaginary part;

[0025] Given two row vectors Y1 and Y2, col(Y1,Y2) = [Y1 T Y1 T ] T , It is the Kronecker product of Y1 and Y2;

[0026] Given a matrix It is the vectorization of M;

[0027] Given a matrix

[0028] Given vector

[0029] For any two vectors c1, c2, and multiple time intervals [t0, t1], [t1, t2], ..., [t], ... s-1 ,t s Define the following two operators:

[0030]

[0031]

[0032] Where c1(t1) represents the value of vector c1 at time t1, and c1(t2) represents the value of vector c1 at time t2.

[0033] Furthermore, step S2 specifically includes the following sub-steps:

[0034] S21: For the outer loop speed of the motor, a slow-time variable, it is assumed that the inner loop current can flawlessly track the reference signal, i.e., Motor speed is selected as the state variable, q-axis reference current is selected as the input, and tracking error is selected as the input. The motor mechanical equations and electromagnetic equations are combined to form the error tracking system, which is described as follows:

[0035]

[0036]

[0037] Where x = ω, e = ω ref -ω, ωref This is the desired motor speed;

[0038] S22: Taking into account the input and output of the error tracking system, the corresponding design performance indicators are:

[0039]

[0040] Where Q and R are weight parameters;

[0041] S23: Transform the performance metrics in step S22 into a standard LQR problem, reconstruct the auxiliary system, and select the following state variables and inputs:

[0042]

[0043] S24: The reconstructed state-space expression of the auxiliary system is obtained as follows:

[0044]

[0045] The corresponding performance metrics have been reconstructed as follows:

[0046]

[0047] in:

[0048]

[0049] S25: Let K be the gain matrix that minimizes the performance index, then the input is represented as:

[0050]

[0051] The corresponding algebraic Riccati equation is:

[0052]

[0053] Where P is the unique solution to the equation;

[0054] After obtaining the P matrix, the gain matrix K is represented as follows:

[0055]

[0056] S26: The corresponding algebraic Riccati equation is solved using the iterative method:

[0057]

[0058]

[0059] in, K kThe value of the gain matrix K is represented by the value of the gain matrix K at the kth iteration; the following integral equation is obtained:

[0060]

[0061] Where δt is the sampling interval;

[0062] S27: Reconstruct it using the operators defined in step S1 as follows:

[0063]

[0064] Further organized as follows:

[0065]

[0066] in:

[0067]

[0068]

[0069] The least squares solution to this equation is expressed as:

[0070]

[0071] Then, the optimal control law, i.e., the gain matrix, under given parameters Q and R is obtained through an integral reinforcement learning algorithm, and the q-axis current reference signal is obtained from this.

[0072] Furthermore, in step S3, the specific processing procedure is as follows:

[0073] S31: For the inner current loop, select the d-axis and q-axis currents i d i q As state variables, the d-axis and q-axis voltages u d u q As input variables, the electromagnetic equations of the motor are expressed as:

[0074]

[0075] S32: In the model predictive control algorithm, the electromagnetic equations of the motor are discretized using the first-order Euler forward prediction formula:

[0076] x e (k+1)=A e x e (k)+B e u e (k)+E e

[0077] Where x e =[i di q ] T ,u e =[u d ,u q ] T And there are:

[0078]

[0079] Among them, T s It is a control interval;

[0080] S33: For a three-phase two-level inverter, the inverter output voltage has a finite number of states, which can be represented in the αβ coordinate system as follows:

[0081] U = u dc S

[0082]

[0083]

[0084]

[0085] Among them, u dc It is the DC power supply voltage, u α ,u β It is the voltage in the αβ coordinate system;

[0086] The voltage in the dq coordinate system is obtained through the αβ-dq transformation:

[0087]

[0088] Where θ is the position of the motor rotor;

[0089] Then u e (k) has only a finite number of states, corresponding to x e (k+1) also has only a finite number of states.

[0090] Furthermore, in step S4, the designed loss function is:

[0091]

[0092] in,

[0093] Compared with the prior art, the present invention has the following advantages: This permanent magnet synchronous motor control method based on reinforcement learning and model predictive control uses two state-space expressions to describe multi-time-scale systems, and through reasonable equivalence and assumptions, the inner and outer loops can be decoupled for control; combining optimal control theory with integral reinforcement learning, it has a certain degree of adaptability to model errors and can provide the optimal reference signal under certain parameters; the use of model predictive control algorithm improves the performance of the inner loop, enabling the actual current to quickly track the reference signal. Attached Figure Description

[0094] Figure 1 This is a control principle diagram of the permanent magnet synchronous motor control method based on reinforcement learning and model predictive control in an embodiment of the present invention.

[0095] Figure 2 This is a topology diagram of the inverter in Embodiment 1 of the present invention;

[0096] Figure 3(a) is the convergence curve of the reinforcement learning P matrix in Embodiment 2 of the present invention;

[0097] Figure 3(b) shows the convergence curve of the reinforcement learning K matrix in Embodiment 2 of the present invention;

[0098] Figure 4 This is the motor speed curve under the permanent magnet synchronous motor control method based on reinforcement learning and model predictive control in Embodiment 2 of the present invention;

[0099] Figure 5 This is the motor speed curve when the inner loop uses PI control in Embodiment 2 of the present invention. Detailed Implementation

[0100] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0101] Example 1

[0102] This embodiment provides a technical solution: a permanent magnet synchronous motor control method based on reinforcement learning and model predictive control, including the following main contents:

[0103] Step 1

[0104] Define several operators:

[0105] Given a complex number U, real(U) denotes its real part and imag(U) denotes its imaginary part;

[0106] Given two row vectors Y1 and Y2, col(Y1,Y2) = [Y1 T Y1T ] T , It is the Kronecker product of Y1 and Y2;

[0107] Given a matrix It is the vectorization of M;

[0108] Given a matrix

[0109] Given vector

[0110] For any two vectors c1 and c2, and several time intervals [t0, t1], [t1, t2], ..., [t], ... s-1 ,t s Define the following two operators:

[0111]

[0112]

[0113] Where c1(t1) represents the value of vector c1 at time t1, c1(t2) represents the value of vector c1 at time t2, and so on.

[0114] For a three-phase permanent magnet synchronous motor, the stator voltage equation in the dq coordinate system is expressed as:

[0115]

[0116] Among them, u d ,u q These are the d-axis and q-axis voltages, respectively, i d i q These are the d-axis and q-axis currents, respectively, L d ,L q These are the d-axis and q-axis inductances, respectively; ω is the electric angular velocity of the motor; and ψ... f R is the permanent magnet flux linkage, and R is the stator resistance.

[0117] For surface-mounted three-phase permanent magnet synchronous motors, L d =L q L will be used to represent L in the following text. d ,L q Take i d =0 control to obtain the maximum current-to-torque ratio, the nonlinear dynamic equation of the motor is described as follows:

[0118]

[0119]

[0120] Where, np J is the number of pole pairs of the motor, J is the sum of the moments of inertia of the motor and the load, and T is the number of pole pairs of the motor. L It is the load torque.

[0121] Step Two

[0122] For the outer loop speed of the motor, a slow-time variable, it is assumed that the inner loop current can flawlessly track the reference signal, i.e., Motor speed is selected as the state variable, and q-axis reference current is used. As input, the tracking error is used as input. The mechanical and electromagnetic equations of the motor are combined to form the error tracking system, as described below:

[0123]

[0124]

[0125] Where x = ω, e = ω ref -ω, ω ref This is the desired motor speed;

[0126] To achieve good response performance in an error tracking system, it is necessary to comprehensively consider the system's input and output. The corresponding performance indicators are as follows:

[0127]

[0128] Here, Q and R are weight parameters selected as needed, representing smaller expected error and smaller expected input, respectively.

[0129] To comprehensively consider the above error tracking system, this performance index needs to be transformed into a standard LQR problem. Therefore, an auxiliary system is reconstructed, with the selected state variables and inputs as follows:

[0130]

[0131] The reconstructed state-space expression of the auxiliary system is:

[0132]

[0133] The corresponding performance metrics have been reconstructed as follows:

[0134]

[0135] in:

[0136]

[0137] Let K be the gain matrix that minimizes the performance index. Then the input can be expressed as:

[0138]

[0139] The corresponding algebraic Riccati equation is:

[0140]

[0141] Where P is the unique solution to the equation;

[0142] After obtaining the P matrix, the optimal feedback matrix can be expressed as:

[0143]

[0144] According to the literature (Kleinman D. On an iterative technique for Riccati equation computations[J]. IEEE Transactions on Automatic Control, 1968, 13(1): 114-115), the corresponding algebraic Riccati equation is solved by iterative method:

[0145]

[0146]

[0147] in, K k This represents the value of the matrix at the kth iteration;

[0148] Using the above formula, we have the following integral equation:

[0149]

[0150] Where δt is the sampling interval;

[0151] Using the previously defined operators, it can be reconstructed as follows:

[0152]

[0153] Further organized as follows:

[0154]

[0155] in:

[0156]

[0157]

[0158] The least squares solution to this equation is expressed as:

[0159]

[0160] The optimal control law, i.e., the gain matrix, under given parameters Q and R can be obtained through integral reinforcement learning algorithm, and the q-axis current reference signal can be obtained from it.

[0161] Step 3

[0162] For the inner current loop, select the d-axis and q-axis currents i d i q As state variables, the d-axis and q-axis voltages u d u q As input variables, the electromagnetic equations of the motor are expressed as:

[0163]

[0164]

[0165] In model predictive control algorithms, the motor speed ω is no longer a state variable, but is treated as a constant. The motor electromagnetic equations are discretized using the first-order Euler forward prediction formula:

[0166] x e (k+1)=A e x e (k)+B e u e (k)+E e

[0167] Where x e =[i d i q ] T ,u e =[u d ,u q ] T And there are:

[0168]

[0169] Among them, T s It is a control interval;

[0170] For a such Figure 2 For the three-phase two-level inverter with the topology shown, the inverter output voltage has only a finite number of states, which can be represented in the αβ coordinate system as follows:

[0171] U = u dc S

[0172]

[0173]

[0174]

[0175] Among them, u dc It is the DC power supply voltage, u α ,u β It is the voltage in the αβ coordinate system;

[0176] The voltage in the dq coordinate system is obtained through the αβ-dq transformation:

[0177]

[0178] Where θ is the position of the motor rotor;

[0179] Then u e (k) has only a finite number of states, corresponding to x e (k+1) also has only a finite number of states.

[0180] Step Four

[0181] The loss function is designed as follows:

[0182]

[0183] in, It is obtained from step two;

[0184] By minimizing the loss function and selecting the switching vector that minimizes the loss function to apply to the inverter, the motor current can be made to track the reference signal, thereby achieving the preset speed.

[0185] Example 2

[0186] This embodiment further illustrates Embodiment 1. First, exploratory noise is applied to the q-axis channel. The exploratory noise is in the form of a small sine wave. The LQR parameters are set to Q = 10 and R = 0.1. The inner loop uses MPC to obtain various data generated by the motor operation. Reinforcement learning is then performed to obtain the LQR gain matrix. The motor and load parameters are shown in Table 1. The reinforcement learning training iteration process is shown in Figures 3(a) and (b).

[0187] Table 1 Motor and Load Parameter Settings

[0188] Stator resistance 2.875Ω q-axis inductor 1.2mH d-axis inductance 3.4mH flux linkage constant 0.175Wb Extreme logarithm 4 Moment of inertia <![CDATA[0.008kg·m 2 ]]> Load torque 5 N·m

[0189] Substituting the gain matrix obtained from reinforcement learning into the control method of this invention, the resulting motor speed curve is as follows: Figure 4 As shown, the motor speed curve using the PI algorithm in the inner loop is as follows: Figure 5 As shown.

[0190] As can be seen from this embodiment, the gain matrix parameters obtained through reinforcement learning enable the motor to operate normally and achieve ideal performance. When the inner loop uses the MPC algorithm for motor driving, the motor's response speed is faster than that of the PI algorithm, and neither exhibits overshoot. Combining reinforcement learning with MPC achieves excellent motor speed regulation performance.

[0191] In summary, the permanent magnet synchronous motor control method based on reinforcement learning and model predictive control described in the above embodiments uses two state-space expressions to describe multi-time-scale systems. Through reasonable equivalence and assumptions, the inner and outer loops can be decoupled for control. By combining optimal control theory with integral reinforcement learning, it has a certain degree of adaptability to model errors and can provide the optimal reference signal under certain parameters. The application of model predictive control algorithm improves the performance of the inner loop, enabling the actual current to quickly track the reference signal.

[0192] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A control method for permanent magnet synchronous motors based on reinforcement learning and model predictive control, characterized in that, Includes the following steps: S1: Define the operator Define multiple operators; S2: Obtain the gain matrix The optimal control law, i.e. the gain matrix, under given parameters is obtained through an integral reinforcement learning algorithm; S3: Obtain voltage vector The inner loop uses a model predictive control algorithm to obtain inverter voltage vector drive motor; S4: Motor speed control By designing a loss function and minimizing it, a voltage vector that minimizes the loss function is applied to the inverter, enabling the motor current to track the reference signal and thus achieve the preset speed. In step S1, the defined operator is as follows: Given a complex number , It indicates that the actual department, Indicate its imaginary part; Given two row vectors , , , yes and The Kronecker product; Given a matrix , yes Vectorization; Given a matrix , ; Given vector , ; For any two vectors , and multiple time intervals Define the following two operators: ； in, Representative vector exist The value at time, Representative vector exist The value at time; Step S2 specifically includes the following sub-steps: S21: For the outer loop speed of the motor, a slow-time variable, it is assumed that the inner loop current can flawlessly track the reference signal, i.e., The motor speed is selected as the state variable, the q-axis reference current is selected as the input, and the tracking error is selected as the input. The motor mechanical equations and electromagnetic equations are combined to form the error tracking system, which is described as follows: ； in, , , , , , , , This is the desired motor speed; S22: Taking into account the input and output of the error tracking system, the corresponding design performance indicators are: ； in, These are weight parameters; S23: Transform the performance metrics in step S22 into a standard LQR problem, reconstruct an auxiliary system, and select the following state variables and inputs: ； S24: The reconstructed state-space expression of the auxiliary system is obtained as follows: ； The corresponding performance metrics have been reconstructed as follows: ； in: ； S25: Let the gain matrix that minimizes the performance index be... The input is then represented as: ； The corresponding algebraic Riccati equation is: ； in, It is the only solution to this equation; get After the matrix, the gain matrix It is expressed as follows: ； S26: The corresponding algebraic Riccati equation is solved using the iterative method: ； in, Representative gain matrix exist The value at the next iteration; The following integral equation is obtained: ； in, It is the sampling interval; S27: Reconstruct it using the operators defined in step S1 as follows: ； Further organized as follows: ； in: ； The least squares solution to this equation is expressed as: ； Then, the optimal control law, i.e., the gain matrix, under given parameters Q and R is obtained through an integral reinforcement learning algorithm, and the q-axis current reference signal is obtained from this. .

2. The permanent magnet synchronous motor control method based on reinforcement learning and model predictive control according to claim 1, characterized in that: For a three-phase permanent magnet synchronous motor, the stator voltage equation in the dq coordinate system is expressed as: ； in, These are the d-axis and q-axis voltages, respectively. These are the d-axis and q-axis currents, respectively. These are the d-axis and q-axis inductors, respectively. It is the electric angular velocity of the motor. It is a permanent magnet flux linkage.

3. The permanent magnet synchronous motor control method based on reinforcement learning and model predictive control according to claim 2, characterized in that: For surface-mounted three-phase permanent magnet synchronous motors , utilizing to replace ,take To achieve the maximum current-to-torque ratio, the nonlinear dynamic equation of the motor is described as follows: ； in, It is the number of pole pairs of the motor. It is the sum of the moments of inertia of the motor and the load. It is the load torque.

4. The permanent magnet synchronous motor control method based on reinforcement learning and model predictive control according to claim 3, characterized in that: In step S3, the specific processing procedure is as follows: S31: Select d-axis and q-axis currents for the inner current loop. , As state variables, the d-axis and q-axis voltages , As input variables, the electromagnetic equations of the motor are expressed as: ； S32: In the model predictive control algorithm, the electromagnetic equations of the motor are discretized using the first-order Euler forward prediction formula: ； in And there are: ； in, It is a control interval; S33: For a three-phase two-level inverter, the inverter output voltage has a finite number of states. Represented in coordinate system as: ； ； ； ； in, It is the DC power supply voltage. yes Voltage in a coordinate system; pass The -dq transformation yields the voltage in the dq coordinate system: ； in, It refers to the position of the motor rotor; but There are only a finite number of states, corresponding to There are only a finite number of states.

5. The permanent magnet synchronous motor control method based on reinforcement learning and model predictive control according to claim 4, characterized in that: In step S4, the designed loss function is: ； in, .

Images