Permanent magnet synchronous motor adaptive control method based on reinforcement learning

Abstract

The invention provides a permanent magnet synchronous motor adaptive control method based on reinforcement learning. Firstly, the system obtains state information of a motor in real time through a sensor, and selects an optimal action based on current state observation by using a deterministic strategy model constructed by a deep neural network. According to a strategy selection result, a corresponding control instruction is sent to the motor, the input voltage of the motor is adjusted, the above steps are executed repeatedly, interaction with the environment is carried out continuously, learning and strategy optimization are carried out continuously, a controller can adapt to the change and uncertainty of motor operation in a self-adaptive mode, the optimal control effect is achieved, and the expected operation state is achieved. Secondly, after the action is executed, the actual running effect of the motor is observed, and feedback information of the environment is obtained; and according to the environment feedback, calculating a reward signal for evaluating the quality of the current strategy. And finally, improving the deterministic strategy model according to the reward signal and the environment feedback by using a depth deterministic strategy gradient (PPO) algorithm.

Description

Technical Field

[0001] The present invention relates to the field of AC permanent magnet synchronous motor control, and in particular to a permanent magnet synchronous motor adaptive control method based on reinforcement learning. Background Art

[0002] In electro-hydraulic servo systems, permanent magnet synchronous motors often use a three-loop control structure, including current loop, speed loop and position loop. Traditional servo drives usually use PID controllers to achieve speed and current control, but PID controllers are only suitable for low-order, linear and time-invariant systems.

[0003] However, the electro-hydraulic servo system is a high-order, nonlinear and time-varying system, so the PID controller cannot achieve optimal control and cannot meet the system's requirements for high rigidity, high response and high precision. In order to ensure the rapid tracking of position and speed instructions, the electro-hydraulic servo controller needs to have fast adaptive capabilities. The adaptive control method can adjust the control strategy in real time to adapt to the high-order, nonlinear and time-varying characteristics of the system according to the dynamic characteristics of the system and environmental changes. Such an adaptive control method can achieve better control performance and meet the electro-hydraulic servo system's requirements for high rigidity, high response and high precision.

[0004] Therefore, in order to meet the needs of electro-hydraulic servo control systems, an adaptive control method that can adapt to high-order, nonlinear and time-varying characteristics is needed to achieve precise control and fast response of permanent magnet synchronous motors. Summary of the invention

[0005] To address the above-mentioned deficiencies, the present invention provides a permanent magnet synchronous motor adaptive control method based on reinforcement learning to improve the precise control and rapid response of the permanent magnet synchronous motor.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a permanent magnet synchronous motor adaptive control method based on reinforcement learning, comprising the following steps:

[0007] The state information of the motor is obtained in real time through sensors, and the strategy model built using the deep neural network model is used to select the optimal action based on the current state observation;

[0008] According to the action selected by the strategy model, the corresponding control command is sent to the motor to adjust the input voltage or current of the motor to achieve the desired operating state;

[0009] Calculate the reward value based on the feedback motor state information to evaluate the current strategy model;

[0010] A deep deterministic policy gradient algorithm is used to optimize the policy model based on the reward signal and motor state information.

[0011] The method of obtaining the state information of the motor in real time through the sensor and selecting the optimal action based on the current state observation using the strategy model constructed by the deep neural network specifically includes the following steps:

[0012] Use sensors to collect status information of permanent magnet synchronous motors in real time;

[0013] A deep neural network is used to build a policy model that accepts the current motor state information as input and outputs an optimal action; the input of the policy model includes the vector representation of the motor state:

[0014]

[0015] where ω e represents the current speed, W represents the target speed, U represents the d1 , U q1 Respectively represent the dq axis voltage at the previous moment, U d , U q Represent the dq axis voltage respectively, F represents the deep neural network model, which accepts the motor speed and current as input and outputs the dq axis voltage.

[0016] The action selected according to the strategy model sends the corresponding control instruction to the motor to adjust the input voltage or current of the motor to achieve the desired operating state, specifically:

[0017] Strategy selection: Through the deep neural network model, the optimal action or control instruction is selected according to the current motor state information;

[0018] Control instruction sending: according to the result of strategy selection, the corresponding control instruction is sent to the control system of the motor so that the input voltage or current information required by the motor is transmitted to the motor controller;

[0019] Motor input adjustment: When the motor controller receives a control instruction, it adjusts the input voltage or current of the motor according to the instruction.

[0020] The reward value is calculated based on the feedback motor state information to evaluate the current strategy model, as follows:

[0021]

[0022] Among them, r1(t) represents the reward value at time t, σ represents the acceptable range of the difference between the target speed and the current speed, and ω e represents the current speed, W represents the target speed, W1 and W2 represent the lower and upper limits of the speed respectively.

[0023] The reward values ​​are as follows:

[0024]

[0025] Among them, r2(t) represents the optimized reward value at time t, T is the total time of the entire iteration, and t is time t.

[0026] The method uses a deep deterministic policy gradient algorithm to optimize the policy model according to the reward signal and the motor state information, including the following steps:

[0027] 1) Initialize the weight parameter θ of the strategy model;

[0028] 2) Data collection: By executing motor actions, collect state observations s, actions a, reward values ​​r, and next state observations s′;

[0029] Among them, the state observation s includes: the current speed ω e , the d-axis voltage U at the last moment d1 , the q-axis voltage U at the previous moment q1 ; The action a includes: d-axis voltage U d , q-axis voltage U q ;

[0030] 3) Calculate the advantage estimate: Calculate the advantage estimate A of each state-action pair to measure the performance improvement of the current policy model relative to the previous iterative policy model. The formula is as follows:

[0031]

[0032] Where γ is the discount factor, δt(t) is the advantage estimation error at time t, and T is the total time of the entire iteration.

[0033] 4) Calculate the policy loss and value loss: Use the policy model to predict the action a′ based on the current state observation s, and calculate the action probability π(a′|s; θ);

[0034] Use the MSE loss function to calculate the policy loss L_policy: L_policy = MSE(a, a′); MSE stands for mean square error;

[0035] Calculate the predicted value V(s;θ) of the value network;

[0036] 5) Calculate PPO loss: Use advantage estimate A and policy loss L _ The policy calculates the PPO loss L_PPO:

[0037] L_PPO=min(ratio*A,L_policy*A)

[0038] Among them, ratio=exp(log(π(a|s;θ)-log(π(a′|s;θ)));

[0039] 6) Parameter update: According to the PPO loss, the weight parameter θ of the policy network is updated using the gradient descent method;

[0040] 7) Repeat steps 2) to 6) until the preset total time or number of training times is reached, or the performance of the strategy model meets the set requirements; the weight parameter θ obtained at this time is the optimal strategy model parameter to enable the motor output to reach the desired operating state.

[0041] A permanent magnet synchronous motor adaptive controller based on reinforcement learning, comprising:

[0042] A state sensing module is used to obtain the state information of the permanent magnet synchronous motor;

[0043] The state representation module is used to encode the acquired state information so as to input it into the neural network model;

[0044] Reinforcement learning algorithm module, used to build a neural network model as a policy model;

[0045] An action selection module, used to select a control action based on the output of the strategy model to be fed into the motor controller;

[0046] The motor controller module converts the selected control action into an actual control signal, optimizes the policy model based on the reward signal and motor state information using a deep deterministic policy gradient algorithm, and sends the optimized policy to the permanent magnet synchronous motor to achieve the desired motion;

[0047] The reward function module is used to calculate the reward value based on the feedback motor state information, which is used to evaluate the current strategy model.

[0048] The present invention has the following beneficial effects and advantages:

[0049] 1. First, this method can use deep neural networks to learn the dynamic characteristics of the motor system from a large amount of data without manually debugging parameters. This greatly simplifies the design and debugging process of the controller and reduces manpower investment and time costs.

[0050] 2. The adaptive controller achieves optimal control by applying a reinforcement learning algorithm, which enables the permanent magnet synchronous motor to reach the target speed more quickly and avoids the speed overshoot phenomenon that may occur in traditional PID control.

[0051] 3. The permanent magnet synchronous motor controller based on reinforcement learning also has a certain generalization ability, and can maintain good control performance even in different working environments and disturbances. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flow chart of the control algorithm of the present invention.

[0053] Figure 2 Flowchart for training the PPO model. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0055] A permanent magnet synchronous motor adaptive control method based on reinforcement learning comprises the following steps:

[0056] The motor status information is obtained in real time through sensors, and the optimal action is selected based on the current state observation and the deterministic strategy model built using a deep neural network.

[0057] According to the result of strategy selection, the corresponding control instructions are sent to the motor to adjust the input voltage or current of the motor to achieve the desired operating state.

[0058] After executing the action, observe the actual effect of the motor operation and obtain feedback information from the environment. Based on the environmental feedback, calculate the reward signal to evaluate the quality of the current strategy.

[0059] The deep deterministic policy gradient (PPO) algorithm is used to adjust the policy model based on the reward signal and environment feedback.

[0060] The state information of the motor is obtained in real time through the sensor, and based on the current state observation, the deterministic strategy model constructed by the deep neural network is used to select the optimal action, specifically:

[0061] Sensors obtain motor status information: Use sensors to collect relevant status information of permanent magnet synchronous motors in real time, such as current, speed, position, etc. This information can be used as input to build the state observation of the controller.

[0062] Build a deterministic policy model: Use a deep neural network to build a deterministic policy model that accepts the current motor state information as input and outputs an optimal action. The input of the deterministic policy model can include a vector representation of the motor state, as shown in the following formula.

[0063]

[0064] where ω e represents the current speed, W represents the target speed, U represents the d1 , U q1 Respectively represent the dq axis voltage at the previous moment, U d , U qRepresent the dq axis voltage respectively, and F represents the deep neural network model, which accepts the speed and current of the motor as input and outputs the dq axis voltage.

[0065] According to the result of the strategy selection, the corresponding control instruction is sent to the motor to adjust the input voltage or current of the motor to achieve the desired operating state, specifically:

[0066] Strategy selection: Through the deep neural network model, the optimal action or control instruction is selected based on the current state information of the motor. These actions or control instructions may include adjusting the input voltage or current of the motor to achieve the desired operating state.

[0067] Control command transmission: According to the result of strategy selection, the corresponding control command is sent to the motor control system. This can be achieved through the controller interface or communication protocol to pass the input voltage or current information required by the motor to the motor controller.

[0068] Motor input adjustment: After receiving the control command, the motor controller adjusts the input voltage or current of the motor according to the command. This may involve adjusting the parameters of the motor driver or using the control strategy inside the motor controller to make the motor output speed, position or other performance indicators reach the desired operating state.

[0069] After the action is executed, the actual effect of the motor operation is observed and feedback information from the environment is obtained. Based on the environmental feedback, the reward signal is calculated to evaluate the quality of the current strategy, specifically:

[0070] Taking into account the operation of the motor, the reward function is designed to encourage the motor to reach the target speed as quickly as possible and maintain it within a certain speed range. This article uses the following formula to calculate the reward value:

[0071]

[0072] Where r1(t) represents the reward value at time t, σ represents the acceptable range of the difference between the target speed and the current speed, and ω e represents the current speed, W represents the target speed, W1 and W2 represent the lower and upper limits of the speed respectively.

[0073] In order to make the PPO model reach the target speed quickly, this paper further designs the reward function to obtain the following reward function, as shown in the following formula

[0074]

[0075] Where T is the total iteration time and t is the time instant.

[0076] The deep deterministic policy gradient (PPO) algorithm is used to adjust the policy model according to the reward signal and environmental feedback, specifically:

[0077] 1. Initialize the policy network: Initialize the weight parameter θ of the policy network

[0078] 2. Data collection: Perform a sequence of actions in the motor control environment, collect state observations s, actions a, reward signals r, and next state observations s′.

[0079] 3. Calculate advantage estimate: Calculate the advantage estimate A for each state-action pair to measure the performance improvement of the current policy model over the old policy model. The formula is as follows

[0080]

[0081] where γ is the discount factor and δt(t) is the advantage estimate error at time t.

[0082] 4. Calculate policy loss and value loss: Use the policy network to predict action a′ based on the current state observation s and calculate the action probability π(a′|s; θ).

[0083] Use the MSE loss function to calculate the policy loss L_policy: L_policy = MSE(a, a′)

[0084] Calculate the predicted value V(s;θ) of the value network.

[0085] 5. Calculate PPO loss:

[0086] Compute the PPO loss L PPO using the advantage estimate A and the policy loss L_policy:

[0087] L_PPO=min(ratio*A,L_policy*A)

[0088] where ratio=exp(log(π(a|s;θ)-log(π(a′|s;θ)))

[0089] 6. Parameter update: Use gradient descent to update the weight parameter θ of the policy network;

[0090] 7. Repeat steps 2 to 6 until the preset total time or number of training times is reached, or the performance of the policy model meets the requirements.

[0091] During the training process, through the cycle of continuously collecting data, calculating advantage estimates, calculating losses, and updating parameters, the strategy model can gradually optimize and learn the optimal strategy for permanent magnet synchronous motor control.

[0092] The present invention provides a permanent magnet synchronous motor adaptive controller method based on reinforcement learning, such as Figure 1 As shown in the figure, this method applies reinforcement learning to permanent magnet synchronous motor control, realizes self-learning and self-tuning of system parameters by combining deep neural network and Actor-Critic algorithm, and achieves optimal control. First, the system obtains the state information of the motor in real time through sensors, and selects the optimal action based on the current state observation and the deterministic policy model constructed by deep neural network. According to the result of the policy selection, the corresponding control instruction is sent to the motor to adjust the input voltage or current of the motor to achieve the desired operating state. Secondly, after executing the action, the actual effect of the motor operation is observed and the feedback information of the environment is obtained. According to the feedback from the environment, the reward signal is calculated to evaluate the quality of the current strategy. Finally, the deep deterministic policy gradient (PPO) algorithm is used to learn and update the policy model according to the reward signal and environmental feedback. The above steps are repeated to continuously interact with the environment, learn and optimize the strategy, so that the controller can adaptively adapt to the changes and uncertainties of the motor operation and achieve the optimal control effect.

[0093] The training process of the PPO reinforcement learning model is as follows Figure 2 As shown:

[0094] 1. Initialize the motor model: Before training begins, the permanent magnet synchronous motor model needs to be initialized. This includes setting the physical parameters of the motor, such as inductance, resistance, etc., and determining the structure and parameters of the control system.

[0095] 2. Action selection: In each training step, the PPO model is used to select an action based on the current motor state observation. The action can be to adjust the input voltage or current of the motor to achieve a specific operating state. The process of selecting an action can be implemented by the deterministic policy network in the PPO algorithm.

[0096] 3. Start training: After selecting an action, apply the action to the motor control system so that the motor adjusts its input according to the selected action. Then, observe the performance of the motor in the new state and obtain reward signals and environmental feedback.

[0097] 4. Update PPO model parameters: Calculate strategy loss and value loss based on the collected reward signal and environment feedback;

[0098] Use gradient descent or other optimization algorithms to update the parameters of the PPO model. Specifically, the PPO loss is calculated based on the current state observation and the selected action, and the gradient of the loss function is calculated. Then, according to the direction of the gradient and the learning rate, the parameters of the PPO model are updated to minimize the loss function.

[0099] Repeat the above steps, by continuously selecting actions, training models, and updating parameters, the PPO model is gradually optimized, and it is able to select the optimal action to achieve adaptive control of the permanent magnet synchronous motor. During the training process, the total training time or number of trainings, as well as the convergence requirements of the control strategy, can be adjusted as needed.

[0100] The motor status information is obtained in real time through sensors, and the optimal action is selected based on the current state observation and the deterministic strategy model built using a deep neural network.

[0101] According to the result of strategy selection, the corresponding control instructions are sent to the motor to adjust the input voltage or current of the motor to achieve the desired operating state.

[0102] After executing the action, observe the actual effect of the motor operation and obtain feedback information from the environment. Based on the environmental feedback, calculate the reward signal to evaluate the quality of the current strategy.

[0103] The deep deterministic policy gradient (PPO) algorithm is used to adjust the policy model based on the reward signal and environment feedback.

[0104] A permanent magnet synchronous motor adaptive controller system based on reinforcement learning, comprising:

[0105] State sensing module: This module is used to obtain the state information of the permanent magnet synchronous motor, such as speed, position, current, etc. This information can be obtained using sensors or estimation algorithms.

[0106] State representation module: Appropriately represent and encode the acquired state information so that it can be input into the reinforcement learning algorithm. Common representation methods include using vectors, matrices, or images.

[0107] Reinforcement Learning Algorithm Module: This module contains the implementation of the reinforcement learning PPO algorithm, which can learn the optimal control strategy of the permanent magnet synchronous motor through interaction with the environment.

[0108] Action selection module: Based on the output of the reinforcement learning algorithm, it selects appropriate control actions, such as current, voltage or torque commands for the motor. These actions will be fed into the motor controller to achieve the desired control.

[0109] Motor Controller Module: This module is responsible for converting the selected control actions into actual control signals and sending them to the permanent magnet synchronous motor to achieve the desired motion.

[0110] Reward function module: This module defines a reward function for evaluating the action quality of the reinforcement learning algorithm. The reward function can be designed based on the performance indicators of the motor (such as tracking error, energy efficiency, etc.).

[0111] Environment Model Module: In some cases, an environment model can be used to approximate the dynamic characteristics of the motor system. This model can be used to generate virtual state and reward signals to accelerate the training process of reinforcement learning algorithms.

Claims

1. A permanent magnet synchronous motor adaptive control method based on reinforcement learning, characterized in that: The following steps are involved: The state information of the motor is obtained in real time through sensors, and the strategy model built using the deep neural network model is used to select the optimal action based on the current state observation; According to the action selected by the strategy model, the corresponding control command is sent to the motor to adjust the input voltage or current of the motor to achieve the desired operating state; Calculate the reward value based on the feedback motor state information to evaluate the current strategy model; The policy model is optimized based on the reward value and motor state information using a deep deterministic policy gradient algorithm.

2. The method for adaptive control of a permanent magnet synchronous motor based on reinforcement learning according to claim 1, characterized in that: The method of obtaining the state information of the motor in real time through the sensor and selecting the optimal action based on the current state observation using the strategy model constructed by the deep neural network specifically includes the following steps: Use sensors to collect status information of permanent magnet synchronous motors in real time; A deep neural network is used to build a policy model that accepts the current motor state information as input and outputs an optimal action; the input of the policy model includes the vector representation of the motor state: where ω e represents the current speed, W represents the target speed, U represents the d1 ,U q1 Respectively represent the dq axis voltage at the previous moment, U d ,U q They represent the d-axis and q-axis voltages respectively. F represents a deep neural network model, which accepts the motor speed and current as input and outputs the d-axis and q-axis voltages.

3. The method for adaptive control of a permanent magnet synchronous motor based on reinforcement learning according to claim 1, characterized in that: The action selected according to the strategy model sends the corresponding control instruction to the motor to adjust the input voltage or current of the motor to achieve the desired operating state, specifically: Strategy selection: Through the deep neural network model, the optimal action or control instruction is selected according to the current motor state information; Control instruction sending: according to the result of strategy selection, the corresponding control instruction is sent to the control system of the motor so that the input voltage or current information required by the motor is transmitted to the motor controller; Motor input adjustment: When the motor controller receives a control instruction, it adjusts the input voltage or current of the motor according to the instruction.

4. The method for adaptive control of a permanent magnet synchronous motor based on reinforcement learning according to claim 1, characterized in that: The reward value is calculated based on the feedback motor state information to evaluate the current strategy model, as follows: Among them, r1(t) represents the reward value at time t, σ represents the acceptable range of the difference between the target speed and the current speed, and ω e represents the current speed, W represents the target speed, W1 and W2 represent the lower and upper limits of the speed respectively.

5. The method for adaptive control of a permanent magnet synchronous motor based on reinforcement learning according to claim 1 or 4, characterized in that: The reward values ​​are as follows: Among them, r2(t) represents the optimized reward value at time t, T is the total time of the entire iteration, and t is the time.

6. The method for adaptive control of a permanent magnet synchronous motor based on reinforcement learning according to claim 1, characterized in that: The method uses a deep deterministic policy gradient algorithm to optimize the policy model according to the reward signal and the motor state information, including the following steps: 1) Initialize the weight parameter θ of the strategy model; 2) Data collection: By executing motor actions, collect state observations s, actions a, reward values ​​r, and next state observations s′; Among them, the state observation s includes: the current speed ω e , the d-axis voltage U at the previous moment d1 , the q-axis voltage U at the previous moment q1 ; The action a includes: d-axis voltage U d , q-axis voltage U q ; 3) Calculate the advantage estimate: Calculate the advantage estimate A of each state-action pair to measure the performance improvement of the current policy model relative to the previous iterative policy model. The formula is as follows: Where γ is the discount factor, δt(t) is the advantage estimation error at time t, and T is the total time of the entire iteration; 4) Calculate the policy loss and value loss: Use the policy model to predict the action a′ based on the current state observation s, and calculate the action probability π(a′|s; θ); Use the MSE loss function to calculate the policy loss L_policy: L_policy = MSE(a, a′); MSE stands for mean square error; Calculate the predicted value V(s;θ) of the value network; 5) Calculate PPO loss: Use advantage estimate A and policy loss L_policy to calculate PPO loss L_PPO: L_PPO=min(ratio*A,L_policy*A) Among them, ratio=exp(log(π(a|s;θ)-log(π(a′|s;θ))); 6) Parameter update: According to the PPO loss, the weight parameter θ of the policy network is updated using the gradient descent method; 7) Repeat steps 2) to 6) until the preset total time or number of training times is reached, or the performance of the strategy model meets the set requirements; the weight parameter θ obtained at this time is the optimal strategy model parameter to enable the motor output to reach the desired operating state.

7. A permanent magnet synchronous motor adaptive controller based on reinforcement learning, characterized in that: include: A state sensing module is used to obtain the state information of the permanent magnet synchronous motor; The state representation module is used to encode the acquired state information so as to input it into the neural network model; Reinforcement learning algorithm module, used to build a neural network model as a policy model; An action selection module, for selecting a control action based on the output of the strategy model to be sent to the motor controller; The motor controller module converts the selected control action into an actual control signal, optimizes the policy model based on the reward signal and motor state information using a deep deterministic policy gradient algorithm, and sends the optimized policy to the permanent magnet synchronous motor to achieve the desired motion; The reward function module is used to calculate the reward value based on the feedback motor state information, which is used to evaluate the current strategy model.

Images