A permanent magnet synchronous motor speed loop control method based on DDPG algorithm

Abstract

The application discloses a permanent magnet synchronous motor speed loop control method based on a DDPG algorithm, and comprises the following steps: constructing a motor vector control system to obtain the state of the permanent magnet synchronous motor during operation; constructing a DDPG intelligent controller, which comprises an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay pool, a first optimizer and a second optimizer, training the parameters of the Actor network and the Critic network to obtain a trained Actor network; inputting the state of the permanent magnet synchronous motor into the Actor network to output a quadrature-axis reference current; and transmitting the quadrature-axis reference current to the motor vector control system to control the speed of the permanent magnet synchronous motor in a wide speed regulation range according to the quadrature-axis reference current. The application realizes the fast response and high robustness control of the target rotating speed of the permanent magnet synchronous motor under the load mutation working condition.

Description

Technical Field

[0001] This invention relates to the field of permanent magnet synchronous motor technology, and in particular to a speed loop control method for permanent magnet synchronous motors based on the DDPG algorithm. Background Technology

[0002] Hybrid propulsion systems for aviation, as a key technology for achieving green aviation and improving aircraft energy efficiency, are gradually becoming the mainstream technology for the development of next-generation aero engines. In this system, the megawatt-class high-speed permanent magnet synchronous motor (PMSM), as the core hub of energy conversion, needs to achieve efficient bidirectional flow of electrical and mechanical energy within the wide flight envelope of the aero engine. However, the operating environment of aero hybrid engines is extremely harsh. The motor not only faces deep magnetic circuit saturation under megawatt-class high power and high-frequency losses and temperature rises caused by high-speed operation at tens of thousands of revolutions per minute, but also needs to cope with the severe load disturbances caused by flight condition switching (such as takeoff boost and in-flight restart). Under the combined influence of these highly nonlinear, highly time-varying, and multi-physics coupling factors, the motor parameters will drift significantly. Traditional vector control (FOC) or direct torque control (DTC) are mostly based on linearized model designs. When facing the complex nonlinear dynamic characteristics of aero hybrid systems, they often struggle to achieve both high dynamic response and steady-state accuracy across the entire flight envelope, limiting the overall efficiency and safety of the propulsion system.

[0003] To overcome the performance limitations of traditional linear control methods under complex operating conditions, research on introducing deep reinforcement learning into motor control has gradually emerged in recent years. However, most existing deep reinforcement learning-based motor control methods still employ indirect control architectures, that is, using reinforcement learning agents to adjust the parameters of traditional controllers such as PID or sliding mode controllers online, rather than directly generating control commands. This strategy of using deep reinforcement learning to optimize parameters enhances the adaptive capability of control parameters to some extent, but its control structure is still essentially limited by the fixed topology of traditional controllers, lacking end-to-end global optimization capabilities from environmental state perception to control command generation. Taking PID control as an example, its control law consists of a linear superposition of proportional, integral, and derivative terms, and the structure itself is difficult to characterize the significant nonlinear characteristics commonly found in motor systems, such as magnetic circuit saturation, strong time-varying parameters, and multi-physical coupling. Therefore, even with online parameter optimization through reinforcement learning, controllers based on the PID structure still have inherent limitations in function approximation capabilities, making it difficult to express complex nonlinear control laws. Under extreme operating conditions such as megawatt-level high-speed motors, this structural constraint further restricts the improvement of system dynamic performance, making it difficult to fully unleash the potential control performance. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose a speed loop control method for permanent magnet synchronous motors based on the DDPG algorithm. This method breaks the constraints of the traditional cascaded control structure and realizes the rapid response and high robustness control of permanent magnet synchronous motors to the target speed under sudden load changes.

[0005] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: This invention provides a speed loop control method for permanent magnet synchronous motors based on the DDPG algorithm, comprising the following steps: Step 1: Construct a motor vector control system and a motor operating environment. The motor vector control system obtains the state of the permanent magnet synchronous motor during operation based on the motor operating environment. Step 2: Construct the DDPG intelligent controller, including an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay pool, a first optimizer, and a second optimizer; train the parameters of the Actor network and the Critic network based on the Actor network, the Critic network, the target Actor network, the target Critic network, the experience replay pool, the first optimizer, and the second optimizer to obtain the trained Actor network; Step 3: Input the state of the permanent magnet synchronous motor into the trained Actor network in the DDPG intelligent controller. The Actor network outputs the quadrature axis reference current according to the current state. Step 4: The quadrature axis reference current output by the Actor network is transmitted to the motor vector control system, which controls the speed of the permanent magnet synchronous motor within a wide speed range based on the quadrature axis reference current.

[0006] Furthermore, step 1 specifically includes: Step 11: Construct a motor vector control system, including a permanent magnet synchronous motor, a three-phase inverter, a current sensor, a speed sensor, a Clark converter unit, a Park converter unit, an inverse Park converter unit, an SVPWM unit, and a current controller. Step 12: The speed sensor collects the real-time mechanical speed of the permanent magnet synchronous motor, and performs a difference calculation with the reference speed to obtain the speed error. After integration and differentiation, the speed error is used to obtain the integral error and the differential error. The speed error, the integral error, and the differential error are used as states to form a state space. The state space is constructed as follows: (1) The real-time mechanical speed of the permanent magnet synchronous motor is collected by a speed sensor installed on the shaft of the permanent magnet synchronous motor. The real-time mechanical speed is compared with the preset reference speed. The input subtractor performs an interpolation operation to obtain the speed error. ;in, ; (2) The speed error The input is integrated using a discrete-time integrator to obtain the integral of the error. ; (3) The speed error The input is differentiated using a discrete-time differentiator to obtain the differential error component. ; (4) The speed error Error integral quantity sum of error differentials Each value is normalized to map its range to the interval [-1, 1], and then they are concatenated in a fixed order to form a three-dimensional state. , as input to the DDPG intelligent controller.

[0007] Furthermore, the structure of the DDPG intelligent controller in step 2 includes: an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay pool, a first optimizer, and a second optimizer; the training process includes the following steps: Step 21: The Actor network receives the current state. Output a deterministic quadrature-axis reference current to obtain the current action. ; Step 22: Set the current action After acting on the motor's operating environment, an immediate reward r and the next state are obtained. ', Transform the state transition tuple ( , a , r, Stored in the experience replay pool; Step 23: Randomly draw state transition tuples from the experience replay pool in batches, and set the next state... 'Input the target Actor network to obtain the next action' ', then the next state 'and next steps' 'Input the target Critic network to calculate the target Q-value; set the current state...' and current action The Critic network is input to calculate the predicted Q value; the first optimizer calculates the mean square error between the predicted Q value and the target Q value as the loss function, and updates the parameters of the Critic network along the gradient descent direction of the loss function. Step 24: The second optimizer calculates the gradient of the Critic network with respect to the action and backpropagates it to the Actor network. It combines the gradient of the Actor network with respect to the parameters to form the policy gradient and updates the parameters of the Actor network along the rising direction of the policy gradient. Step 25: Using a soft update mechanism, synchronize the parameters of the Actor network to the target Actor network at a preset ratio, and synchronize the parameters of the Critic network to the target Critic network at a preset ratio.

[0008] Furthermore, step 21 specifically includes: Step 211: Input the current state S into the hidden layer of the Actor network, and perform weight matrix multiplication, bias superposition and nonlinear activation function operations in sequence to extract the nonlinear dynamic characteristics of the permanent magnet synchronous motor. Step 212: Input the nonlinear dynamic features extracted from the hidden layer into the output layer. The output layer uses the Tanh activation function to generate standardized action values, and the numerical range of the standardized action values ​​is [-1, 1]. Step 213: Input the generated standardized action value into the physical amplitude scaling stage. This scaling stage multiplies the standardized action value by a preset maximum quadrature axis current limit, mapping it to a quadrature axis reference current that conforms to the maximum current constraint of the permanent magnet synchronous motor. As an action space, it enables end-to-end computation from the state space to the action space; Step 214: After superimposing the quadrature-axis reference current with Gaussian noise, the current action is obtained. The specific formula is as follows: = （S| ）+

[0009] in, Let S represent the Actor network, and S represent the current state. For the parameters of the Actor network, The injected Gaussian noise is denoted by t, which represents the current time.

[0010] Furthermore, the experience replay pool in step 22 is constructed as follows: Step 221: Pre-allocate a fixed-capacity experience replay pool in computer memory, with the capacity set to 100,000 to 1,000,000 state transition tuples; Step 222: After each action is executed by the DDPG intelligent controller, the current state S and the current action executed are recorded. The immediate reward r obtained from the current motor operating environment and the next state. These four elements combine to form a state transition tuple (S, a , r, '), and store the state transition tuple in the experience replay pool in chronological order; where: The current state ,in, The current speed error, This is the current integral of the error. This is the current differential error component; The current action Quadrature axis reference current ; The instant reward r takes the following mathematical form: ,in, for The weight, for The weight, for The weights; The next state ,in, For the next speed error, For the next error integral, This is the next differential error component; Step 223: When the experience replay pool is full, the newly added state transition tuples will overwrite the oldest state transition tuples according to the first-in-first-out principle.

[0011] Furthermore, the parameter update method for the Critic network in step 23 is as follows: Step 231: During the training phase, uniformly and randomly sample mini-batch state transition tuples from the experience replay pool; Step 232: For each sample in the batch data, the first... i +1 sample of state S i+1 Input target Actor network The first number is obtained through forward propagation. i Actions in +1 sample i+1 = (S) i+1 | );in, Represents the parameters of the target Actor network; Step 233, the first i +1 sample of state S i+1 and actions i+1 Input target Critic network , and combined with the first i Instant reward for each sample And discount factor γ, calculate the first The target Q value for each sample : = +γ (S) i+1 , (S) i+1 | )| ) Wherein, γ is a discount factor, with a value range of [0,1] (usually 0.99), used to weigh the weight of immediate rewards and future long-term returns. The larger γ is, the more the DDPG intelligent controller values ​​the long-term stability of motor operation. These represent the parameters of the target Critic network; The sample index represents the first element in the current batch. One sample; This represents the target Critic network, used to evaluate the value of the next state and the next action.

[0012] Step 234, the first i The state S in each sample i and actions i Inputting the Critic network yields the first... The predicted Q value for each sample: ;in, These represent the parameters of the Critic network; Step 235: Calculate the mean square error between the predicted Q value and the target Q value as the loss function. L :

[0013] in, Indicates the number of samples in a small batch; Step 236: Update the parameters of the Critic network using the first optimizer along the gradient descent direction of the loss function, with the learning rate set between 0.0001 and 0.01.

[0014] Furthermore, the parameter update method for the Actor network in step 24 is as follows: Step 241: For each sample in the batch data, the first... The state in each sample Inputting the Actor network, the corresponding action is calculated through forward propagation. ; Step 242, change the status and actions The input is a Critic network. Forward propagation is used to calculate the Critic network's output value with respect to the action. Then, the automatic differentiation function of the deep learning framework is used to calculate the Critic network's output with respect to the action. gradient ; Step 243: Gradient The result is passed back to the Actor network and multiplied by the Actor network output regarding its network parameters. Jacobian matrix , obtain the policy gradient : ; Step 244: Update the parameters of the Actor network using the second optimizer along the ascending direction of the policy gradient, with the learning rate set between 0.0001 and 0.001.

[0015] Furthermore, the specific implementation of the soft update mechanism in step 25 is as follows: Step 251: Set a smoothing factor according to the preset ratio. The smoothing factor has a value range of 0.001 to 0.1. Step 252: After each training step, update the parameters of the target Critic network and the target Actor network using a moving average according to the following mathematical formula:

[0016]

[0017] in, The parameters of the target Critic network are... The parameters of the target Actor network.

[0018] Furthermore, step 4 specifically includes: Step 41: The quadrature-axis reference current output by the Actor network. Transmitted to the current controller; Step 42: The current sensor collects the three-phase stator current of the permanent magnet synchronous motor: I A I B I C The direct-axis feedback current I is obtained after passing through the Clark transformation unit and the Park transformation unit in sequence. d and quadrature axis feedback current I q The inputs are respectively sent to the current controller; Step 43: The current controller determines the current based on the quadrature axis reference current. With cross-axis feedback current I q The difference, direct-axis reference current With direct-axis feedback current I d The difference is used to output the direct-axis voltage command U. d and quadrature axis voltage command U q To the inverse Park transform unit; Step 44: The inverse Park transformation unit converts the direct-axis voltage command U... d and quadrature axis voltage command U q Converted to voltage components in a two-phase stationary coordinate system: and The output is sent to the SVPWM unit; Step 45: The SVPWM unit synthesizes a reference voltage vector based on the voltage components in the two-phase stationary coordinate system, generates a switching signal to drive the three-phase inverter, and the three-phase inverter controls the operation of the permanent magnet synchronous motor.

[0019] Furthermore, the transformation matrix of the Clark transformation unit is:

[0020] Among them, I A I B I C I is the three-phase stator current collected by the current sensor. α and I β The current in the two-phase stationary coordinate system output by the Clark transformation unit; The transformation matrix of the Park transformation unit is:

[0021] Where θ is the rotor position angle, I d and I q These are the direct-axis feedback current and quadrature-axis feedback current output from the Park transform unit, respectively. The transformation matrix of the inverse Park transformation unit is:

[0022] Among them, U d and U q These are the direct-axis voltage command and quadrature-axis voltage command output by the current controller, respectively. and These are the two-phase stationary coordinate system voltage components output by the inverse Park transformation unit.

[0023] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: This invention proposes an intelligent control solution based on the DDPG algorithm. First, relying on the constructed motor operating environment, the speed error, error integral, and error differential are used as state inputs and directly mapped to the continuous quadrature-axis reference current. This achieves a pure end-to-end direct mapping from system observation to the underlying control command, avoiding the response delay and error accumulation caused by multi-level controllers in series. This provides a more physically consistent end-to-end control strategy for motor control and avoids the jitter and accuracy loss caused by discretization of actions.

[0024] A DDPG intelligent controller update mechanism based on the DDPG algorithm is constructed. By introducing an experience replay pool to break down sample correlations, and employing a soft update mechanism to smoothly iterate the parameters of the target Actor network and the target Critic network, this overcomes the inherent defects of traditional deep learning algorithms that are prone to divergence when dealing with strongly coupled motor systems, as well as the policy oscillation problem caused by data distribution offset during training. This enables the invention to maintain high control accuracy and robustness when facing complex and variable operating conditions. In contrast, traditional methods such as sliding mode control and model predictive control either suffer from severe chattering in similar complex situations or are extremely dependent on accurate motor parameter models and have excessive computational burdens, making it difficult to balance control quality and real-time performance. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an execution flowchart of a speed loop control method for permanent magnet synchronous motors based on the DDPG algorithm provided in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the motor vector control system provided in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the structure of the DDPG intelligent controller provided in an embodiment of the present invention.

[0029] Figure 4 This is a flowchart of the online interaction and policy update algorithm based on DDPG provided in an embodiment of the present invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see Figures 1-4 The present invention provides a speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm, comprising the following steps: Step 1: Construct a motor vector control system and a motor operating environment (a reinforcement learning interactive environment conforming to Markov Decision Process (MDP)). The motor vector control system acquires the state of the permanent magnet synchronous motor (PMSM) during operation based on the motor operating environment. This study selects a permanent magnet synchronous motor (PMSM) vector control system as the core research object. Utilizing a high-fidelity motor simulation environment, a reinforcement learning interactive model conforming to Markov decision processes is constructed. A state space construction module is designed to simulate various operating conditions that may occur under complex scenarios. Key variables such as speed error, error integral, and error differential components during motor operation are integrated into a high-dimensional state vector. These state variables comprehensively replicate the nonlinear characteristics of the motor during dynamic processes, providing rich and diverse real-time observation data for the subsequent agent decision-making process based on the DDPG algorithm. This effectively simulates the response characteristics of the motor under load disturbances, laying a solid foundation for end-to-end intelligent control of the motor speed loop.

[0032] In this embodiment, step 1 specifically includes: Step 11: Construct a motor vector control system, including a permanent magnet synchronous motor, a three-phase inverter, a current sensor, a speed sensor, a Clark transformation unit, a Park transformation unit, an inverse Park transformation unit, an SVPWM (Space Vector Pulse Width Modulation) unit, and a current controller. Step 12: The speed sensor collects the real-time mechanical speed of the permanent magnet synchronous motor, and performs a difference calculation with the reference speed to obtain the speed error. After integration and differentiation, the speed error is used to obtain the integral error and the differential error. The speed error, the integral error, and the differential error are used as states to form a state space. The state space is constructed as follows: (1) The real-time mechanical speed of the permanent magnet synchronous motor is collected by a speed sensor installed on the shaft of the permanent magnet synchronous motor. The real-time mechanical speed is compared with the preset reference speed. The input subtractor performs an interpolation operation to obtain the speed error. ;in, ; (2) The speed error The input is integrated using a discrete-time integrator to obtain the integral of the error. ; (3) The speed error The input is differentiated using a discrete-time differentiator to obtain the differential error component. ; (4) The speed error Error integral quantity sum of error differentials Each value is normalized to map its range to the interval [-1, 1], and then they are concatenated in a fixed order to form a three-dimensional state. It serves as the input to the DDPG (Deep Deterministic Policy Gradient) intelligent controller.

[0033] Step 2: Construct the DDPG intelligent controller, including an Actor network (policy network), a Critic network (value network), a target Actor network, a target Critic network, an experience replay pool, a first optimizer, and a second optimizer; train the parameters of the Actor network and the Critic network based on the Actor network, the Critic network, the target Actor network, the target Critic network, the experience replay pool, the first optimizer, and the second optimizer to obtain the trained Actor network; In this embodiment, the structure of the DDPG intelligent controller in step 2 includes: an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay pool, a first optimizer, and a second optimizer; the training process includes the following steps: Step 21: The Actor network receives the current state. Output a deterministic quadrature-axis reference current to obtain the current action. Specifically, it includes: Step 211: Input the current state S into the hidden layer of the Actor network, and sequentially perform weight matrix multiplication, bias superposition, and nonlinear activation function (such as ReLU) operations to extract the nonlinear dynamic characteristics of the permanent magnet synchronous motor. Step 212: Input the nonlinear dynamic features extracted from the hidden layer into the output layer. The output layer uses the Tanh activation function to generate standardized action values, and the numerical range of the standardized action values ​​is [-1, 1]. Step 213: Input the generated standardized action value into the physical amplitude scaling stage. This scaling stage multiplies the standardized action value by a preset maximum quadrature axis current limit, mapping it to a quadrature axis reference current that conforms to the maximum current constraint of the permanent magnet synchronous motor. As an action space, it enables end-to-end computation from the state space to the action space; Step 214: After superimposing the quadrature-axis reference current with Gaussian noise, the current action is obtained. The specific formula is as follows: = （S| ）+

[0034] in, Let S represent the Actor network, and S represent the current state. For the parameters of the Actor network, The injected Gaussian noise is denoted by t, which represents the current time.

[0035] Step 22: Set the current action After acting on the motor's operating environment, an immediate reward r and the next state are obtained. ', Transform the state transition tuple ( , a , r, Stored in the experience replay pool; specifically including: Step 221: Pre-allocate a fixed-capacity experience replay pool in computer memory, with the capacity set to 100,000 to 1,000,000 state transition tuples; Step 222: After each action is executed by the DDPG intelligent controller, the current state S and the current action executed are recorded. The immediate reward r obtained from the current motor operating environment and the next state. These four elements combine to form a state transition tuple (S, , r, '), and store the state transition tuple in the experience replay pool in chronological order; where: The current state ,in, The current speed error, This is the current integral of the error. This is the current differential error component; The current action Quadrature axis reference current ; The instant reward r takes the following mathematical form: ,in, for The weight, for The weight, for The weights; The next state ,in, For the next speed error, For the next error integral, This is the next differential error component; Step 223: When the experience replay pool is full, the newly added state transition tuples will overwrite the oldest state transition tuples according to the first-in-first-out principle.

[0036] Step 23: Randomly draw state transition tuples from the experience replay pool in batches, and set the next state... 'Input the target Actor network to obtain the next action' ', then the next state 'and next steps' 'Input the target Critic network to calculate the target Q-value; set the current state...' and current action The Critic network is input to calculate the predicted Q-value; the first optimizer calculates the mean squared error between the predicted Q-value and the target Q-value as the loss function, and updates the parameters of the Critic network along the gradient descent direction of the loss function; specifically including: Step 231: During the training phase, uniformly and randomly sample mini-batch state transition tuples from the experience replay pool; Step 232: For each sample in the batch data, the first... i +1 sample of state S i+1 Input target Actor network The first number is obtained through forward propagation. i Actions in +1 sample i+1 = (S) i+1 | );in, Represents the parameters of the target Actor network; Step 233, the first i +1 sample of state S i+1 and actions i+1 Input target Critic network , and combined with the first i Instant reward for each sample And discount factor γ, calculate the first The target Q value for each sample : = +γ (S) i+1 , (S) i+1 | )| ) Wherein, γ is the discount factor, and its value ranges from [0,1]. These represent the parameters of the target Critic network; The sample index represents the first element in the current batch. One sample; Step 234, the first i The state S in each sample i and actions a i Inputting the Critic network yields the first... The predicted Q value for each sample: ;in, These represent the parameters of the Critic network; Step 235: Calculate the mean square error between the predicted Q value and the target Q value as the loss function. L :

[0037] in, Indicates the number of samples in a small batch; Step 236: Update the parameters of the Critic network using the first optimizer along the gradient descent direction of the loss function, with the learning rate set between 0.0001 and 0.01.

[0038] Step 24: The second optimizer calculates the gradient of the Critic network with respect to the action and backpropagates it to the Actor network. It then combines this gradient with the Actor network's own gradient with respect to the parameters to form the policy gradient, and updates the Actor network's parameters along the ascending direction of the policy gradient. Specifically, this includes: Step 241: For each sample in the batch data, the first... The state in each sample Inputting the Actor network, the corresponding action is calculated through forward propagation. ; Step 242, change the status and actions The input is a Critic network. Forward propagation is used to calculate the Critic network's output value with respect to the action. Then, the automatic differentiation function of the deep learning framework is used to calculate the Critic network's output with respect to the action. gradient ; Step 243: Gradient The result is passed back to the Actor network and multiplied by the Actor network output regarding its network parameters. Jacobian matrix , obtain the policy gradient : ; Step 244: Update the parameters of the Actor network using the second optimizer along the ascending direction of the policy gradient, with the learning rate set between 0.0001 and 0.001.

[0039] By using decorrelated batch data to synchronously update the Actor and Critic networks, temporal correlation of the data is eliminated, thereby improving the convergence stability of the algorithm. Step 25: Using a soft update mechanism, synchronize the parameters of the Actor network to the target Actor network at a preset ratio, and synchronize the parameters of the Critic network to the target Critic network at a preset ratio; specifically including: Step 251: Set a smoothing factor according to the preset ratio. The smoothing factor has a value range of 0.001 to 0.1. Step 252: After each training step, update the parameters of the target Critic network and the target Actor network using a moving average according to the following mathematical formula:

[0040]

[0041] in, The parameters of the target Critic network are... The parameters of the target Actor network.

[0042] A soft update mechanism is adopted, which introduces a smoothing factor to update the parameters of the target Actor network and the target Critic network by moving average. This ensures that the parameters of the target network change slowly and stably, thereby suppressing parameter oscillations during training and providing a stable learning target for the training process. This enhances the convergence stability of the algorithm under motor load disturbances and parameter changes.

[0043] Step 3: Input the state of the permanent magnet synchronous motor into the trained Actor network in the DDPG intelligent controller. The Actor network outputs the quadrature axis reference current according to the current state. By replacing the traditional speed loop with a well-trained Actor network, a fast response and highly robust control of the target speed can be achieved within a wide speed range.

[0044] Step 4: The quadrature-axis reference current output by the Actor network is transmitted to the motor vector control system, which controls the speed of the permanent magnet synchronous motor within a wide speed range based on the quadrature-axis reference current; specifically including: Step 41: The quadrature-axis reference current output by the Actor network. Transmitted to the current controller; Step 42: The current sensor collects the three-phase stator current of the permanent magnet synchronous motor: I A I B I C The direct-axis feedback current I is obtained after passing through the Clark transformation unit and the Park transformation unit in sequence. d and quadrature axis feedback current I q The inputs are respectively sent to the current controller; The transformation matrix of the Clark transformation unit is:

[0045] Among them, I A I B I C I is the three-phase stator current collected by the current sensor. α and I β The current in the two-phase stationary coordinate system output by the Clark transformation unit; The transformation matrix of the Park transformation unit is:

[0046] Where θ is the rotor position angle, I d and I q These are the direct-axis feedback current and quadrature-axis feedback current output from the Park transform unit, respectively. Step 43: The current controller determines the current based on the quadrature axis reference current. With cross-axis feedback current I q The difference, direct-axis reference current (Typically set to 0 to achieve torque decoupling control) and direct-axis feedback current I d The difference is calculated through internal PI (proportional-integral) adjustment, and the direct-axis voltage command U is output. d and quadrature axis voltage command U q To the inverse Park transform unit; Step 44: The inverse Park transformation unit converts the direct-axis voltage command U... d and quadrature axis voltage command U q Converted to voltage components in a two-phase stationary coordinate system: and The output is sent to the SVPWM unit; The transformation matrix of the inverse Park transformation unit is:

[0047] Among them, U d and U q These are the direct-axis voltage command and quadrature-axis voltage command output by the current controller, respectively. and These are the two-phase stationary coordinate system voltage components output by the inverse Park transformation unit.

[0048] Step 45: The SVPWM unit synthesizes a reference voltage vector based on the voltage components in the two-phase stationary coordinate system, generates a switching signal to drive the three-phase inverter, and the three-phase inverter controls the operation of the permanent magnet synchronous motor.

[0049] This invention designs a DDPG intelligent controller to replace the traditional speed loop. Specifically, it includes the topology design of the Actor and Critic networks in the intelligent agent architecture, as well as deep optimization of the reward function and network hierarchy. By iteratively training and updating the network parameters in the DDPG algorithm, it acquires adaptive disturbance rejection capability against sudden load disturbances during motor operation, thereby outputting a continuous and accurate quadrature-axis reference current. Specifically, the Actor network constructs an end-to-end mapping from the current state to the optimal control action, while the Critic network, based on a reward function that includes penalties for speed tracking accuracy and control action amplitude, quantitatively evaluates the current action, ensuring the dynamic stability and effectiveness of the control strategy.

[0050] This paper designs an online interaction and policy update algorithm based on DDPG (Driver-Driven Programming). Using a motor vector control system and a DDPG intelligent controller, it achieves information exchange between the deep reinforcement learning algorithm and the motor dynamics model. A deterministic policy gradient algorithm is employed, defining the state space as constructed from speed error, error integral, and error differential components, and using the quadrature-axis reference current as the action space. By introducing an experience replay pool to store state transition tuples, strong correlations between data are eliminated. Instead of the traditional hard replacement method, a soft update mechanism is used to synchronize the target network parameters, improving the algorithm's convergence stability under nonlinear conditions. Finally, various operating conditions are simulated, and load disturbances are randomly injected throughout the training process to achieve rapid response and precise control of the target speed.

[0051] This invention uses a high-precision mathematical model of a permanent magnet synchronous motor as the controlled object, and builds a reinforcement learning training environment on the Simulink platform, while retaining the inner-loop current controller to ensure underlying electrical safety. The construction, pre-training, and closed-loop integration with the vector control system of the speed loop intelligent agent module based on the DDPG algorithm are carried out systematically in the adapted computing environment, ensuring efficient system operation and online inference.

[0052] This invention mainly consists of a motor vector control system covering the current loop and a DDPG intelligent speed controller based on the DDPG algorithm, ensuring accurate control of the motor speed under high-speed, high-voltage and parameter drift environments.

[0053] The design and simulation experiments verified that the DDPG intelligent controller based on the DDPG algorithm can accurately track the target speed of the permanent magnet synchronous motor, and has good adaptive ability when facing complex working conditions such as sudden load changes and changes in rotational inertia. Compared with traditional PI control and conventional sliding mode control methods, it shows significant advantages in terms of dynamic recovery time and overshoot suppression when disturbances occur.

[0054] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm, characterized in that, Includes the following steps: Step 1: Construct a motor vector control system and a motor operating environment. The motor vector control system obtains the state of the permanent magnet synchronous motor during operation based on the motor operating environment. Step 2: Construct the DDPG intelligent controller, including an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay pool, a first optimizer, and a second optimizer; train the parameters of the Actor network and the Critic network based on the Actor network, the Critic network, the target Actor network, the target Critic network, the experience replay pool, the first optimizer, and the second optimizer to obtain the trained Actor network; Step 3: Input the state of the permanent magnet synchronous motor into the trained Actor network in the DDPG intelligent controller. The Actor network outputs the quadrature axis reference current according to the current state. Step 4: The quadrature axis reference current output by the Actor network is transmitted to the motor vector control system, which controls the speed of the permanent magnet synchronous motor within a wide speed range based on the quadrature axis reference current.

2. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 1, characterized in that, Step 1 specifically includes: Step 11: Construct a motor vector control system, including a permanent magnet synchronous motor, a three-phase inverter, a current sensor, a speed sensor, a Clark converter unit, a Park converter unit, an inverse Park converter unit, an SVPWM unit, and a current controller. Step 12: The speed sensor collects the real-time mechanical speed of the permanent magnet synchronous motor, and performs a difference calculation with the reference speed to obtain the speed error. After integration and differentiation, the speed error is used to obtain the integral error and the differential error. The speed error, the integral error, and the differential error are used as states to form a state space. The state space is constructed as follows: (1) The real-time mechanical speed of the permanent magnet synchronous motor is collected by a speed sensor installed on the shaft of the permanent magnet synchronous motor. The real-time mechanical speed is compared with the preset reference speed. The input subtractor performs an interpolation operation to obtain the speed error. ;in, ; (2) The speed error The input is integrated using a discrete-time integrator to obtain the integral of the error. ; (3) The speed error The input is differentiated using a discrete-time differentiator to obtain the differential error component. ; (4) The speed error Error integral quantity sum of error differentials Each value is normalized to map its range to the interval [-1, 1], and then they are concatenated in a fixed order to form a three-dimensional state. , as input to the DDPG intelligent controller.

3. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 1, characterized in that, The structure of the DDPG intelligent controller in step 2 includes: an Actor network, a Critic network, a target Actor network, a target Critic network, an experience replay pool, a first optimizer, and a second optimizer; the training process includes the following steps: Step 21: The Actor network receives the current state. Output a deterministic quadrature-axis reference current to obtain the current action. ; Step 22: Set the current action After acting on the motor's operating environment, an immediate reward r and the next state are obtained. ', Transform the state transition tuple ( , , r, Stored in the experience replay pool; Step 23: Randomly batch-sample state transition tuples from the experience replay pool, and set the next state... 'Input the target Actor network to obtain the next action' ', then move to the next state 'and next steps' 'Input the target Critic network to calculate the target Q-value; set the current state...' and current action The Critic network is input to calculate the predicted Q value; the first optimizer calculates the mean square error between the predicted Q value and the target Q value as the loss function, and updates the parameters of the Critic network along the gradient descent direction of the loss function. Step 24: The second optimizer calculates the gradient of the Critic network with respect to the action and backpropagates it to the Actor network. It combines the gradient of the Actor network with respect to the parameters to form the policy gradient and updates the parameters of the Actor network along the rising direction of the policy gradient. Step 25: Using a soft update mechanism, synchronize the parameters of the Actor network to the target Actor network at a preset ratio, and synchronize the parameters of the Critic network to the target Critic network at a preset ratio.

4. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 3, characterized in that, Step 21 specifically includes: Step 211: Input the current state S into the hidden layer of the Actor network, and perform weight matrix multiplication, bias superposition and nonlinear activation function operations in sequence to extract the nonlinear dynamic characteristics of the permanent magnet synchronous motor. Step 212: Input the nonlinear dynamic features extracted from the hidden layer into the output layer. The output layer uses the Tanh activation function to generate standardized action values, and the numerical range of the standardized action values ​​is [-1, 1]. Step 213: Input the generated standardized action value into the physical amplitude scaling stage. This scaling stage multiplies the standardized action value by a preset maximum quadrature axis current limit, mapping it to a quadrature axis reference current that conforms to the maximum current constraint of the permanent magnet synchronous motor. As an action space, it enables end-to-end computation from the state space to the action space; Step 214: After superimposing the quadrature-axis reference current with Gaussian noise, the current action is obtained. The specific formula is as follows: = （S| ）+ in, Let S represent the Actor network, and S represent the current state. For the parameters of the Actor network, The injected Gaussian noise is denoted by t, which represents the current time.

5. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 3, characterized in that, The experience replay pool in step 22 is constructed as follows: Step 221: Pre-allocate a fixed-capacity experience replay pool in computer memory, with the capacity set to 100,000 to 1,000,000 state transition tuples; Step 222: After each action is executed by the DDPG intelligent controller, the current state S and the current action executed are recorded. The immediate reward r obtained from the current motor operating environment and the next state. These four elements combine to form a state transition tuple (S, , r, '), and store the state transition tuple in the experience replay pool in chronological order; where: The current state ,in, The current speed error, This is the current integral of the error. This is the current differential error component; The current action Quadrature axis reference current ; The instant reward r takes the following mathematical form: ,in, for The weight, for The weight, for The weights; The next state ,in, For the next speed error, For the next error integral, This is the next differential error component; Step 223: When the experience replay pool is full, the newly added state transition tuples will overwrite the oldest state transition tuples according to the first-in-first-out principle.

6. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 3, characterized in that, The parameter update method for the Critic network in step 23 is as follows: Step 231: During the training phase, uniformly and randomly sample mini-batch state transition tuples from the experience replay pool; Step 232: For each sample in the batch data, the first... i +1 sample of state S i+1 Input target Actor network The first number is obtained through forward propagation. i Actions in +1 sample i+1 = (S) i+1 | );in, Represents the parameters of the target Actor network; Step 233, the first i +1 sample of state S i+1 and actions i+1 Input target Critic network and in combination with the first i Instant reward for each sample And discount factor γ, calculate the first The target Q value for each sample : = +g (S) i+1 , (S) i+1 | )| ) Wherein, γ is the discount factor, and its value ranges from [0,1]. These represent the parameters of the target Critic network; The sample index represents the first element in the current batch. One sample; Step 234, the first i The state S in each sample i and actions i Inputting the Critic network yields the first... The predicted Q value for each sample: ;in, These represent the parameters of the Critic network; Step 235: Calculate the mean square error between the predicted Q value and the target Q value as the loss function. L : in, Indicates the number of samples in a small batch; Step 236: Update the parameters of the Critic network using the first optimizer along the gradient descent direction of the loss function, with the learning rate set between 0.0001 and 0.

01.

7. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 3, characterized in that, The parameter update method for the Actor network in step 24 is as follows: Step 241: For each sample in the batch data, the first... The state in each sample Inputting the Actor network, the corresponding action is calculated through forward propagation. ; Step 242, change the status and actions The input is a Critic network. Forward propagation is used to calculate the Critic network's output value with respect to the action. Then, the automatic differentiation function of the deep learning framework is used to calculate the Critic network's output with respect to the action. gradient ; Step 243: Gradient The result is passed back to the Actor network and multiplied by the Actor network output regarding its network parameters. Jacobian matrix , obtain the policy gradient : ； Step 244: Update the parameters of the Actor network using the second optimizer along the ascending direction of the policy gradient, with the learning rate set between 0.0001 and 0.

001.

8. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 3, characterized in that, The specific implementation of the soft update mechanism in step 25 is as follows: Step 251: Set a smoothing factor according to the preset ratio. The smoothing factor has a value range of 0.001 to 0.

1. Step 252: After each training step, update the parameters of the target Critic network and the target Actor network using a moving average according to the following mathematical formula: in, The parameters of the target Critic network are... The parameters of the target Actor network.

9. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 2, characterized in that, Step 4 specifically includes: Step 41: The quadrature-axis reference current output by the Actor network. Transmitted to the current controller; Step 42: The current sensor collects the three-phase stator current of the permanent magnet synchronous motor: I A I B I C The direct-axis feedback current I is obtained after passing through the Clark transformation unit and the Park transformation unit in sequence. d and quadrature axis feedback current I q The inputs are respectively sent to the current controller; Step 43: The current controller determines the current based on the quadrature axis reference current. With cross-axis feedback current I q The difference, direct-axis reference current With direct-axis feedback current I d The difference is used to output the direct-axis voltage command U. d and quadrature axis voltage command U q To the inverse Park transform unit; Step 44: The inverse Park transformation unit converts the direct-axis voltage command U... d and quadrature axis voltage command U q Converted to voltage components in a two-phase stationary coordinate system: and The output is sent to the SVPWM unit; Step 45: The SVPWM unit synthesizes a reference voltage vector based on the voltage components in the two-phase stationary coordinate system, generates a switching signal to drive the three-phase inverter, and the three-phase inverter controls the operation of the permanent magnet synchronous motor.

10. The speed loop control method for a permanent magnet synchronous motor based on the DDPG algorithm as described in claim 9, characterized in that, The transformation matrix of the Clark transformation unit is: Among them, I A I B I C I is the three-phase stator current collected by the current sensor. α and I β The current in the two-phase stationary coordinate system output by the Clark transformation unit; The transformation matrix of the Park transformation unit is: Where θ is the rotor position angle, I d and I q These are the direct-axis feedback current and quadrature-axis feedback current output from the Park transform unit, respectively. The transformation matrix of the inverse Park transformation unit is: Among them, U d and U q These are the direct-axis voltage command and quadrature-axis voltage command output by the current controller, respectively. and These are the two-phase stationary coordinate system voltage components output by the inverse Park transformation unit.

Images