Permanent magnet synchronous motor MTPA control method based on end-to-end reinforcement learning

Abstract

The invention belongs to the field of motor control, and relates to a permanent magnet synchronous motor MTPA control method based on end-to-end reinforcement learning, in a motor control system, motor current and rotating speed collected in real time are constructed into a state vector, a trained intelligent agent is input for reasoning, and control voltage is output as an action vector. Finally, a voltage instruction is correspondingly generated to drive a motor; the network structure of the intelligent agent is constructed based on a TD3 algorithm, training data of the intelligent agent comprises random motor physical parameters and dynamic motor working condition rotating speed, iterative training of the intelligent agent is based on a dynamic reward function, and the intelligent agent is guided to converge automatically through the dynamic reward function. According to the method, a control strategy of directly mapping the rotating speed and the current state of the motor to the optimal voltage instruction can be obtained, so that rotating speed tracking is met, the maximum torque-current ratio working point is spontaneously converged, and the method has adaptive compensation capability for parameter nonlinear drift in motor operation.

Description

Technical Field

[0001] This invention belongs to the field of motor control and relates to a permanent magnet synchronous motor (MTPA) control method based on end-to-end reinforcement learning. Background Technology

[0002] Permanent magnet synchronous motors (PMSMs) have become the preferred drive core for electric vehicle traction, high-precision industrial robots, and aerospace servo applications due to their extremely high power density, wide speed range, and excellent dynamic response.

[0003] Currently, the mainstream control strategy in industry is a dual-closed-loop vector control architecture based on field-oriented control (FOC). This architecture decouples the stator three-phase AC current into direct-axis current (excitation component) and quadrature-axis current (torque component) in a synchronous rotating coordinate system through coordinate transformations: Clark and Park transformations. To minimize the stator current amplitude while generating the same electromagnetic torque, i.e., reducing motor copper losses and improving system efficiency, the control system must adopt the Maximum Torque Per Ampere (MTPA) strategy. The traditional implementation process is as follows: the speed loop outputs a torque command based on the speed error. This command passes through a dedicated MTPA calculation module, which typically allocates the optimal direct-axis current command and quadrature-axis current command based on formulas or lookup tables. This is then sent to the current loop PI controller for tracking control, and finally drives the inverter through SVPWM, ensuring that the current vector operates on the optimal efficiency trajectory.

[0004] Although the aforementioned FOC-based MTPA control scheme is technically mature, in practical high-performance application scenarios, this architecture still has the following significant limitations:

[0005] 1. High dependence on motor model parameters and difficulty in handling parameter drift during operation, leading to model mismatch and reduced energy efficiency: Traditional analytical formula methods are based on fixed mathematical models of motors. However, in practice, it is difficult to obtain accurate motor model parameters, and motors are highly nonlinear time-varying systems in actual operation. Due to factors such as magnetic saturation and temperature rise, motor parameters will deviate. Since traditional algorithms cannot perceive these dynamic changes in parameters in real time, the calculated optimal operating point will deviate from the actual MTPA trajectory.

[0006] 2. Existing lookup table methods require lengthy offline bench tests, and the data cannot adapt to motor aging or manufacturing variations: Offline calibration is cumbersome, and lookup table methods are difficult to adapt to complex operating conditions. To avoid complex real-time calculations, offline lookup table methods are often used in engineering. This requires bench tests during the development phase to create current / torque mapping tables, resulting in a large workload and long cycle for calibration. In addition, the pre-made tables are static and only correspond to specific motor conditions, such as specific temperatures. When faced with complex actual operating conditions such as motor aging, inconsistent manufacturing tolerances, or drastic changes in the external environment, the fixed table data cannot be adaptively adjusted, leading to a significant reduction in control performance and a lack of versatility and robustness. Summary of the Invention

[0007] To address the aforementioned technical problems in the existing technology, this invention proposes a permanent magnet synchronous motor (MTPA) control method based on end-to-end reinforcement learning, the specific technical solution of which is as follows:

[0008] A permanent magnet synchronous motor (MTPA) control method based on end-to-end reinforcement learning is proposed. In the motor control system, the real-time acquired motor current and speed are used to construct a state vector, which is then input into a trained agent for inference. The output control voltage is used as an action vector, and finally, a voltage command is generated to drive the motor. The network structure of the agent is constructed based on the TD3 algorithm. The training data of the agent includes random motor physical parameters and dynamic motor operating conditions and speeds. The iterative training of the agent is based on a dynamic reward function, which guides the agent to automatically converge.

[0009] Furthermore, the training data is obtained by designing a PMSM control system simulation model. The PMSM control system simulation model performs uniformly distributed random sampling of the stator resistance, right-angle shaft inductance, and permanent magnet flux linkage of the motor physical parameters within a preset floating range of nominal values. For obtaining the motor operating speed, it simulates and sets randomly changing speed commands and randomly abrupt load torque.

[0010] Furthermore, the network structure of the intelligent agent specifically includes a policy network and a value network. The policy network adopts a fully connected feedforward neural network structure, which takes a state vector as input and outputs an action vector. The value network processes the nonlinear combination of the state vector and the action vector through a two-stream merging structure to evaluate the output of the policy network.

[0011] Furthermore, the policy network receives an 8-dimensional state vector, which is then processed through two hidden layers containing 64 and 32 neurons respectively, with ReLU activation functions, for feature extraction. Finally, the output layer's Tanh activation function directly maps the vector to normalized direct-axis and quadrature-axis voltages of the motor, thus obtaining the action vector. The definition is as follows:

[0012] S t =[E ω ,ω ref ,∫E ω dt,I d ,I q ,∫I d dt,U d (t-1),U q (t-1)] T ,

[0013] Wherein, velocity error E ω The reference speed ω reflects the deviation between the current speed and the target speed. ref Provide prior information on the operating speed, the speed error integral ∫E ω dt endows the agent with the ability to remember historical errors, and the direct-axis current I d The direct-axis current integral ∫I reflects the excitation state of the motor. d dt is used to eliminate the static error of decoupling control, and the quadrature-axis current I q Reflecting the motor's current torque output and load status, the control voltage U at the previous moment... d (t-1),U q (t-1) provides information on the temporal continuity of control actions.

[0014] Furthermore, the value network contains two independent input paths, one of which processes the state vector and the other processes the action vector. After being encoded by a fully connected layer, the two are fused by element-wise addition. The fused features are then processed by a hidden layer to output a scalar Q value.

[0015] Furthermore, during the iterative training process, the agent updates the value network and the policy network. The update of the value network incorporates noise ∈, which follows a truncated normal distribution, when calculating the target action. The expression is as follows:

[0016]

[0017] In the formula, The smoothed target motion vector for the next time step, containing direct-axis and quadrature-axis reference voltage commands, S i+1 Let μ' represent the state vector of the i-th sample at the next time step, μ' represent the target actor network, and θ represent the target policy network. μ′ Represents the weight parameters of the target policy network; Let ∈ be a subtotal normal distribution, and let the amplitude of ∈ be restricted to the interval [-c,c].

[0018] Calculate the target Q value:

[0019]

[0020] In the formula, y i Let Q be the target Q-value for the i-th sample, which serves as the supervision label for network updates. j ' is the target value network, R i For the agent in state S i Perform action A i The immediate reward value obtained afterward is calculated using the aforementioned reward function, where γ is the discount factor. This represents the weight parameters of the j-th target value network;

[0021] The loss function of the value network is defined as mean squared error. The loss of the two-stream merging structure network is calculated and backpropagation is performed.

[0022] The policy network is updated with the goal of maximizing the Q value, and a delayed update strategy is adopted, that is, the policy network is updated once after the value network is updated d times.

[0023] Furthermore, the reward function combines tracking error reward, MTPA optimization reward, and action penalty reward. When the control system detects a large transient speed error, the reward function takes the speed tracking target as the main objective and guides the agent to output the maximum torque to eliminate the error. When the speed error converges to the threshold range, the agent is guided to take the current minimization target as the main objective.

[0024] Furthermore, the tracking error reward expression is as follows:

[0025]

[0026] In the formula, (E ω ) 2 The term guides the agent to quickly reduce the current error, (∫E ω dt) 2 This term is used to enable the agent to adjust its strategy to eliminate steady-state error. The term is used to suppress motor speed overshoot and oscillation, w p w i w d These are the weight coefficients for the corresponding items.

[0027] Furthermore, the MTPA optimization reward expression is as follows:

[0028]

[0029] In the formula, I q For quadrature axis current, I d For direct-axis current; β(e t ) is an adaptive coefficient used to adjust the proportion of MTPA optimization reward in the reward function according to different operating conditions, where et Let α be the rotational speed error, e be the natural constant, and α be the value of β(e). t The sensitivity coefficient of )

[0030] The MTPA-optimized reward-guided agent automatically converges to the operating point with the minimum current under a fixed output torque.

[0031] Furthermore, the action penalty reward expression is as follows:

[0032] Indicates the voltage command at the current moment. Voltage command from the previous moment The Euclidean distance, where w act For R action The weighting coefficients.

[0033] The beneficial effects of this invention include:

[0034] 1. This invention solves the problems of cumbersome and poor versatility in traditional calibration work, and gets rid of parameter dependence and tedious table lookup. That is, it does not require precise motor inductance and flux linkage parameters, nor does it require offline calibration to make table lookup. Instead, it constructs a reward function that includes current amplitude penalty and uses reinforcement learning to adaptively optimize, guiding the agent to automatically lock the MTPA operating point under unknown parameters, which significantly reduces development and calibration costs.

[0035] 2. This invention solves the problem that traditional MTPA relies on accurate models and has poor resistance to nonlinear disturbances. The simulation includes a complex dynamic environment with internal parameter perturbations and external operating condition changes. It has strong robustness in dealing with parameter perturbations and can effectively cope with the drift of inductance and resistance parameters caused by magnetic saturation, temperature rise or aging. This allows the system to adjust the strategy in real time and maintain optimal control performance even under nonlinear operating conditions and model mismatch.

[0036] 3. This invention introduces an error-based dynamic weighting mechanism, which automatically reduces the energy efficiency reward weight during the dynamic process of the system and automatically increases the energy efficiency reward weight during the steady-state process. This solves the contradiction between rapid dynamic response and steady-state energy efficiency optimization, and realizes global collaborative optimization of dynamic and steady-state performance. It not only ensures rapid following during startup and sudden changes, but also automatically minimizes current.

[0037] 4. The neural network of this invention directly establishes a nonlinear mapping from the motor state to the orthogonal axis reference voltage, realizing end-to-end direct control from state observation to voltage output, which significantly simplifies the control logic and system structure. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the policy network structure in this embodiment;

[0039] Figure 2This is a schematic diagram of the value network structure in this embodiment;

[0040] Figure 3 This is a schematic diagram of the adaptive coefficient in the MTPA optimization reward function of this embodiment;

[0041] Figure 4 This is a schematic diagram of a permanent magnet synchronous motor (MTPA) control method based on end-to-end reinforcement learning in this embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0043] This invention discloses a method for controlling a permanent magnet synchronous motor (MTPA) based on end-to-end reinforcement learning, comprising:

[0044] Step S1: Construct a generalization training environment based on domain randomization.

[0045] A PMSM control system model is built in the simulation platform. To endow the agent with generalization and versatility to adapt to individual differences in different motors and resist parameter drift during operation, this step introduces domain randomization technology to construct a dynamic training scenario with time-varying parameters and severe operating conditions:

[0046] Global randomization of physical parameter domain: Abandoning the traditional fixed-parameter training model, in the initialization phase of each training episode, randomization is performed on the core physical parameters of the motor: stator resistance R. s Right-angle axis inductance L d / L q Permanent magnet flux linkage ψ f The algorithm performs uniformly distributed random sampling within ±15% of the nominal value. This strategy forces the agent to learn the fundamental decoupled mapping between current, speed state, and control voltage, rather than relying on memorizing specific motor parameter combinations to obtain high rewards. This allows the trained policy network to adapt to individual motors with different physical characteristics and exhibits good robustness to resistance changes caused by temperature rise or inductance fluctuations caused by magnetic saturation during actual operation.

[0047] Dynamic operating condition simulation: Set randomly varying speed commands and randomly abrupt load torque. Simulate the complete process of the motor running from zero speed to high speed, as well as harsh operating conditions such as sudden load increases / decreases during steady-state operation. Train the agent to achieve smooth transitions between different states, ensuring control accuracy during full operating condition switching.

[0048] Step S2: Construct an agent network structure based on the TD3 algorithm.

[0049] The network structure of an intelligent agent includes a policy network (Actor) and a value network (Critic).

[0050] Policy networks: such as Figure 1 As shown, a fully connected feedforward neural network structure is adopted. The network receives an 8-dimensional state vector, which is processed through two hidden layers containing 64 and 32 neurons respectively, with ReLU activation functions, for feature extraction. Finally, the output layer's Tanh activation function directly maps the vector to a normalized direct-axis voltage U. d and quadrature axis voltage U q .

[0051] Value networks: such as Figure 2 As shown, a two-stream merging structure is employed to handle the nonlinear combination of states and actions. The network contains two independent input paths: one processing the state vector and the other processing the action vector. Both are encoded by a fully connected layer and then fused element-wise. The fused features are then processed by subsequent hidden layers, ultimately outputting a scalar Q-value used to evaluate the effectiveness of the current control strategy.

[0052] The agent's input state vector Defined as:

[0053] S t =[E ω ,ω ref ,∫E ω dt,I d ,I q ,∫I d dt,U d (t-1),U q (t-1)] T ,

[0054] Wherein, velocity error E ω The reference speed ω reflects the deviation between the current rotational speed and the target rotational speed, and is the core feedback quantity for speed tracking tasks. ref Providing prior information about the operating conditions, and considering the significant differences in back electromotive force characteristics of the motor at low and high speeds, introducing a reference speed can help the agent distinguish between different operating conditions. Speed ​​error integral ∫E ω dt endows the agent with the ability to remember historical errors. When there is a deviation in the neural network output, the integral term will continue to accumulate, forcing the agent to adjust the output until the steady-state error is zero. Direct-axis current I d Reflecting the motor's excitation state, the intelligent agent needs to implement decoupling control based on this signal to ensure the maximum torque-to-current ratio. Direct-axis current integral ∫I d Similarly, dt utilizes the integral principle to forcibly eliminate the static error of decoupling control, ensuring the accuracy of magnetic field orientation. Quadrature-axis current I qIt reflects the current torque output and load status of the motor, helping the intelligent agent to perceive the dynamic load of the system. The control voltage U at the previous moment... d (t-1),U q (t-1) provides information on the temporal continuity of control actions. By combining the current state with the previous action, the agent can avoid drastic changes in output voltage, thereby suppressing high-frequency oscillations.

[0055] The agent's output action vector Defined as:

[0056] A t =[U d (t),U q (t)] T ,

[0057] U d (t),U q (t) represents the control voltages for the direct axis and quadrature axis, respectively.

[0058] Step S3: Iterative training based on the dynamic reward function.

[0059] To guide the agent to simultaneously satisfy the three objectives of velocity tracking, current decoupling, and control smoothness, a single-step reward function R is defined. t as follows:

[0060] R t =R track +R MTPA +R action +R safety ,

[0061] Among them, R track This is a reward for tracking error in motor speed, used to ensure the dynamic and steady-state performance of the system; R MTPA The reward for MTPA optimization targeting motor current is a core component for achieving high-efficiency control; R action The action penalty reward for the control voltage is used to suppress drastic fluctuations in the control quantity. The specific mathematical expression is as follows:

[0062]

[0063] (E ω ) 2 The term guides the agent to quickly reduce the current error, (∫E ω dt) 2 This forces the agent to adjust its strategy to completely eliminate steady-state error. This term is used to suppress speed overshoot and oscillation, and improve the damping characteristics of the system. p w i w d These are the weight coefficients for the corresponding items.

[0064]

[0065] The intelligent agent is guided to automatically converge to the operating point with the minimum current under the load, i.e., the MTPA point, when the output torque is constant.

[0066] β(e t ) represents the adaptive coefficient, such as Figure 3 As shown, the proportion of MTPA optimization reward in the reward function is adjusted according to different operating conditions, where e t Let α be the rotational speed error, e be the natural constant, and α be the value of β(e). t The sensitivity coefficient of ) determines β(e) t Sensitivity. β(e) t In error e t When the value is large, it approaches 0, and the error e t The value approaches 1 in hours. When a large transient speed error is detected in the system, β(e) t Automatic decay ensures that the reward function prioritizes speed tracking, guaranteeing the agent outputs maximum torque to quickly eliminate errors. When the system reaches steady state and the speed error converges to a threshold range, β(e) t Smooth recovery allows the current minimization objective to regain dominance. This mechanism enables the agent to achieve adaptive search with steady-state accuracy without sacrificing dynamic bandwidth.

[0067]

[0068] Where w act For R action The weighting coefficient, which penalizes the voltage command at the current moment. Voltage command from the previous moment The Euclidean distance. Minimizing this term makes the control voltage waveform more continuous and smooth, reducing motor noise and losses.

[0069] Current threshold reward R safety :

[0070]

[0071] When the motor current exceeds the safety threshold I max At that time, a huge negative reward C is given. penalty It also terminates the current round early, guiding the agent to explore within safe boundaries.

[0072] Iterative training process: Randomly sample N data points from the ExperienceReplayBuffer, each data point containing S... i A i,R i ,S i+1 The following network updates will be performed:

[0073] Value network update:

[0074] To smooth the target policy, noise ∈ that follows a truncated normal distribution is added when calculating the target action, as expressed below:

[0075]

[0076] in, The smoothed target motion vector for the next time step, containing direct-axis and quadrature-axis reference voltage commands, S i+1 Let μ' represent the state vector of the i-th sample at the next time step, μ' represent the target actor network, and θ represent the target policy network. μ′ Represents the weight parameters of the target policy network; Let ∈ be a subtotal normal distribution, and let the amplitude of ∈ be restricted to the interval [-c, c].

[0077] Calculate the target Q value:

[0078]

[0079] Among them, y i Let R be the target Q-value of the i-th sample, and let R be the supervision label for network updates. i For the agent in state S i Perform action A i The immediate reward value obtained afterward is calculated using the aforementioned reward function, where γ is the discount factor. This represents the weight parameters of the j-th target value network.

[0080] Loss function of value network Defined as mean squared error:

[0081]

[0082] Where N represents the total number of data samples in the mini-batch sampling, i is the sample index in the current batch, i = 1 to N, and y i Let Q be the target Q value for the i-th sample calculated in the previous step. j (·) represents the j-th online value network function. Let represent the parameters of the j-th online value network, where j=1 and j=2 correspond to the first and second online value networks, respectively. During training, the system calculates the loss of each network separately, performs backpropagation, minimizes the loss function using gradient descent, and independently updates their respective weight parameters. and To enable the updating of the value network.

[0083] Policy Network Update: The goal of the policy network is to maximize the Q-value. A delayed update strategy is adopted, meaning the policy network is updated only once after the value network has been updated d times. Its loss function is J(θ). μ ) is defined as:

[0084]

[0085] Q1(·) represents the first online value network, which calculates the Q-value of actions generated by the current policy network and acts as a judge in this step to evaluate the quality of the actions. μ(·) is the online policy network, i.e., the current controller model, and θ μ For the weight parameters to be updated in the online policy network, The weight parameters of the first online value network are used in this formula for calculation. Keep it fixed, only update θ μ Perform backpropagation and minimize J(θ) using gradient descent. μ That is, gradient ascent maximizes the Q-value and updates the policy network parameters θ. μ This completes the update of the policy network.

[0086] Step S4: Policy network extraction and code deployment migrates the trained agent to the actual control system.

[0087] Extract the policy network from the agent; this network is the final controller.

[0088] The policy network is converted into embedded code and deployed to the motor controller chip.

[0089] like Figure 4 As shown, in actual operation, the controller collects current and speed in real time, constructs a state vector, inputs it into the intelligent agent network for inference, and directly generates voltage commands to drive the motor, realizing intelligent control under all operating conditions.

[0090] In summary, this invention constructs an end-to-end intelligent control architecture integrating speed control, current decoupling, and energy efficiency optimization. It utilizes the nonlinear decision-making capabilities of reinforcement learning to replace the cascaded structure of speed loop PI, MTPA calculation module, and current loop PI in traditional vector control. Its core innovative mechanism lies in the fact that the agent, guided by a reward function oriented towards minimizing stator current amplitude, autonomously explores in a fully randomized simulation environment without requiring motor mathematical model parameters such as inductance and flux linkage. The agent can learn a control strategy that directly maps motor speed and current state to the optimal voltage command, thereby achieving speed tracking while spontaneously converging to the maximum torque-to-current ratio operating point, and possessing adaptive compensation capabilities for nonlinear parameter drift during motor operation.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A permanent magnet synchronous motor (MTPA) control method based on end-to-end reinforcement learning, characterized in that: In the motor control system, the real-time collected motor current and speed are used to construct a state vector, which is then input into a trained agent for reasoning. The output control voltage is used as an action vector, and finally, a voltage command is generated to drive the motor. The network structure of the agent is built based on the TD3 algorithm. The training data of the agent includes random motor physical parameters and dynamic motor operating conditions and speed. The iterative training of the agent is based on a dynamic reward function, which guides the agent to converge automatically.

2. The MTPA control method for a permanent magnet synchronous motor as described in claim 1, characterized in that: The training data is obtained by designing a PMSM control system simulation model. The PMSM control system simulation model randomly samples the stator resistance, right-angle shaft inductance, and permanent magnet flux linkage of the motor physical parameters in a uniform distribution within a preset floating range of the nominal values. For obtaining the motor operating speed, it simulates and sets randomly changing speed commands and randomly abrupt load torque.

3. The MTPA control method for a permanent magnet synchronous motor as described in claim 1, characterized in that: The network structure of the intelligent agent specifically includes a policy network and a value network. The policy network adopts a fully connected feedforward neural network structure, which takes a state vector as input and outputs an action vector. The value network processes the nonlinear combination of the state vector and the action vector through a two-stream merging structure to evaluate the output of the policy network.

4. The MTPA control method for permanent magnet synchronous motors as described in claim 3, characterized in that: The policy network receives an 8-dimensional state vector, which is then processed through two hidden layers containing 64 and 32 neurons respectively, with ReLU activation functions, for feature extraction. Finally, the output layer's Tanh activation function directly maps the vector to normalized direct-axis and quadrature-axis motor voltages, thus obtaining the action vector. The definition is as follows: S t =[E ω ,ω ref ,∫E ω dt,I d ,I q ,∫I d dt,U d (t-1),U q (t-1)] T , Wherein, velocity error E ω The reference speed ω reflects the deviation between the current speed and the target speed. ref Provide prior information on the operating speed, the speed error integral ∫E ω dt endows the agent with the ability to remember historical errors, and the direct-axis current I d The direct-axis current integral ∫I reflects the excitation state of the motor. d dt is used to eliminate the static error of decoupling control, and the quadrature-axis current I q Reflecting the motor's current torque output and load status, the control voltage U at the previous moment... d (t-1),U q (t-1) provides information on the temporal continuity of control actions.

5. The MTPA control method for a permanent magnet synchronous motor as described in claim 3, characterized in that: The value network contains two independent input paths. One path processes the state vector, and the other path processes the action vector. After being encoded by a fully connected layer, the two are fused by element-wise addition. The fused features are then processed by a hidden layer to output a scalar Q value.

6. The MTPA control method for a permanent magnet synchronous motor as described in claim 3, characterized in that: During iterative training, the agent updates the value network and the policy network. The value network update incorporates noise ∈, which follows a truncated normal distribution, when calculating the target action. The expression is as follows: In the formula, The smoothed target motion vector for the next time step, containing direct-axis and quadrature-axis reference voltage commands, S i+1 Let μ' represent the state vector of the i-th sample at the next time step, μ' represent the target policy network (TargetActor), and θ represent the state vector of the i-th sample at the next time step. μ′ Represents the weight parameters of the target policy network; Let ∈ be a subtotal normal distribution, and let the amplitude of ∈ be restricted to the interval [-c,c]. Calculate the target Q value: In the formula, y i Let Q be the target Q-value for the i-th sample, which serves as the supervision label for network updates. j ' is the target value network, R i For the agent in state S i Perform action A i The immediate reward value obtained afterward is calculated using the aforementioned reward function, where γ is the discount factor. This represents the weight parameters of the j-th target value network; The loss function of the value network is defined as mean squared error. The loss of the two-stream merging structure network is calculated and backpropagation is performed. The policy network is updated with the goal of maximizing the Q value, and a delayed update strategy is adopted, that is, the policy network is updated once after the value network is updated d times.

7. The MTPA control method for a permanent magnet synchronous motor as described in claim 1, characterized in that: The reward function combines tracking error reward, MTPA optimization reward, and action penalty reward. When the control system detects a large transient speed error, the reward function takes the speed tracking target as the main objective and guides the agent to output the maximum torque to eliminate the error. When the speed error converges to the threshold range, the agent is guided to take the current minimization target as the main objective.

8. The MTPA control method for a permanent magnet synchronous motor as described in claim 7, characterized in that: The tracking error reward expression is: In the formula, (E ω ) 2 The term guides the agent to quickly reduce the current error, (∫E ω dt) 2 This term is used to enable the agent to adjust its strategy to eliminate steady-state error. The term is used to suppress motor speed overshoot and oscillation, w p w i w d These are the weight coefficients for the corresponding items.

9. The MTPA control method for a permanent magnet synchronous motor as described in claim 7, characterized in that: The MTPA optimization reward expression is as follows: In the formula, I q For quadrature axis current, I d For direct-axis current; β(e t ) is an adaptive coefficient used to adjust the proportion of MTPA optimization reward in the reward function according to different operating conditions, where e t Let α be the rotational speed error, e be the natural constant, and α be the value of β(e). t The sensitivity coefficient of ) The MTPA-optimized reward-guided agent automatically converges to the operating point with the minimum current under a fixed output torque.

10. The MTPA control method for a permanent magnet synchronous motor as described in claim 7, characterized in that: The action penalty reward expression is: Indicates the voltage command at the current moment. Voltage command from the previous moment The Euclidean distance, where w act For R action The weighting coefficients.

Images