Permanent magnet synchronous motor rotating speed stability control method based on reinforcement learning

Abstract

The invention discloses a permanent magnet synchronous motor rotation speed stability control method based on reinforcement learning. The method comprises the following specific steps: collecting real-time operation data of a motor, and carrying out feed-forward compensation control of the permanent magnet synchronous motor and stabilization of rotation speed fluctuation of the motor based on a reinforcement learning model; strategy migration from a single working condition to multiple working conditions and from a single motor to multiple motors is carried out based on migration learning, and multi-working-condition and multi-motor rotating speed fluctuation suppression is achieved; integrating the designed reinforcement learning model and the strategy migration method into the motor control system, and evaluating and optimizing the system performance. According to the method, the feedforward control method based on the reinforcement learning model is adopted, the fluctuation of the rotating speed of the motor can be suppressed on the premise of not accurately modeling a motor servo system, and meanwhile, a powerful tool is provided for the uncertainty and parameter change robustness in the system and the stable control of the rotating speed of the permanent magnet synchronous motor.

Description

Technical Field

[0001] The present invention belongs to the technical field of motor control, and in particular relates to a permanent magnet synchronous motor speed stabilization control method based on reinforcement learning. Background Art

[0002] As energy reforms proceed, motors play a key role in energy supply, consumption, technology and system. Especially in the field of new energy, advanced motor technology is of great significance in improving motor efficiency and reducing energy consumption. The intelligentization and precise control technology of motors support the efficient operation of the energy industry.

[0003] Permanent magnet synchronous motors are usually simplified and modeled as ideal models in controller design due to their nonlinear and strong coupling characteristics. Under this background, factors such as time-varying motor parameters, motor manufacturing defects, and installation process errors will lead to the transmission and amplification of modeling errors, thereby forming internal disturbances of permanent magnet synchronous motors. On the other hand, inverter nonlinear characteristics, cable impedance asymmetry, and external load fluctuations constitute external disturbances of permanent magnet synchronous motors. The combined effect of the two may lead to increased motor losses, speed fluctuations, increased noise, and even further cause oscillation and instability of the entire system. High-performance permanent magnet synchronous motor drive systems should have excellent dynamic steady-state performance, including no static error, low overshoot, and anti-disturbance characteristics. Speed ​​stability directly reflects the technical level of the permanent magnet synchronous motor drive system and determines the market competitiveness of the product.

[0004] Existing technologies based on sliding mode control and iterative learning control require known information such as motor current, voltage, and speed, and rely on accurate modeling and observation. They are difficult to implement in the absence of domain expert knowledge, cannot overcome the contradiction between high performance and system jitter under some working conditions, and have poor generalization.

[0005] Therefore, technicians in this field are committed to developing a permanent magnet synchronous motor speed stabilization control method based on reinforcement learning to adapt to different working conditions and environmental changes and achieve more flexible and intelligent control response. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art, the purpose of the present invention is to provide a permanent magnet synchronous motor speed stabilization control method based on reinforcement learning. It can suppress the motor speed fluctuations under multiple working conditions without accurately modeling the motor servo system, targeting the speed instability caused by factors such as processing defects and various non-ideal and non-uniform loads in the driver hardware.

[0007] To achieve the above purpose, the specific technical solution of the present invention is as follows: The method for controlling the speed stability of a permanent magnet synchronous motor based on reinforcement learning comprises the following specific steps: S1. Obtain the real-time operation data of the motor, and perform feedforward compensation control of the permanent magnet synchronous motor and smooth the motor speed fluctuation based on the reinforcement learning model; S2. Based on transfer learning, the strategy is migrated from a single working condition to multiple working conditions and from a single motor to multiple motors to achieve speed fluctuation suppression in multiple working conditions and multiple motors; S3. Integrate the designed reinforcement learning model and policy transfer method into the motor control system, and evaluate and optimize the system performance.

[0008] Furthermore, the specific operations of step S1 are: S11. Acquire real-time operation data of the motor based on the sensor device; S12. The collected data will be preprocessed and feature extracted for subsequent control algorithms; S13. Based on the collected real-time operation data of the motor, a reinforcement learning model is designed, where the reinforcement learning model includes a state space, an action space, and a reward function; S14. Use the proximal strategy optimization method to train the designed reinforcement learning model. During the training process, the motor speed fluctuation is used as a feedback signal to continuously optimize the model parameters so that the model can accurately learn the feedforward compensation control strategy.

[0009] Furthermore, in step S11, the real-time operation data of the motor Including motor direct-axis current, motor quadrature-axis current, motor speed, motor mechanical angle, and motor servo system speed loop PI output.

[0010] Furthermore, in step S13, the theoretical basis of the reinforcement learning model is the Markov decision process, that is, in a random process, the probability distribution of the next state depends only on the current state and is independent of the past state; the scenario of reinforcement learning is usually that the agent interacts with the environment in a series of time steps, and at each time step, the agent accepts the current state. , select actions based on the current strategy , perform the action Afterwards get reward And enter the next state , the goal of reinforcement learning is to maximize the cumulative reward obtained after several time steps from the initial state.

[0011] Furthermore, in step S14, the proximal strategy optimization algorithm is a reinforcement learning method based on the actor-critic framework. (In this invention, it is the current running data of the motor, which has been normalized and dimensionless) and probability to select the appropriate action (In the present invention, it is the q-axis current compensation value, which has been normalized), interacts with the environment, and obtains the next state and rewards (In this invention, it is an evaluation index of motor speed fluctuation); the critic calculates the value function of the current state and the value function of the next state , Indicates that in a specific Next, from the state At the beginning, the expected return that the algorithm can obtain is calculated, and then the time difference error is obtained (TD error) is as follows: , The optimization goals are as follows: , in: , , Among them, the time difference error It reflects the difference between the actual immediate reward and the expected reward (i.e., the estimate of the current state value function) under the current strategy. This difference is used to update the value function to make it closer to the true value, thereby improving the performance of the strategy. Represents the empirical mean of a finite batch of samples. In machine learning and statistics, the empirical mean is a method of estimating the expected value, which approximates the expected value of the entire data set by averaging the samples; is the updated policy parameter, is the strategy parameter before updating, The representative strategy network parameters are , the input observation state is When , the output action is , Similarly; is a parameter of the clip function, which describes a truncation function that keeps the input value at and between; At the same time, the advantage function is calculated by generalized advantage estimation (GAE) : , in, Used to measure the relative performance of actions and reduce the variance of policy gradients. It is dimensionless. , is the decay coefficient, which is used to characterize the importance of future rewards relative to current rewards, and is usually in the range of 0.95-0.99; is a hyperparameter between 0 and 1, used to characterize, When GAE is simplified to single-step time difference error, When GAE becomes the weighted sum of all future time difference errors.

[0012] Furthermore, in step S2, the strategy migration method involves transfer reinforcement learning based on reward scaling, dynamically calculating the standard deviation of the sum of rolling discounted rewards, and then standardizing the current reward.

[0013] Furthermore, the transfer reinforcement learning applies the knowledge and models learned in the source domain to the target domain by utilizing the similarities between data, domains and tasks, thereby improving or completing the task learning effect in the target domain. The domain is the subject of transfer learning, including data and the probability distribution of generated data. A sample data in the domain includes input Output and its probability distribution , so a field can be represented as: ; In transfer learning, the domain being transferred is called the source domain, and the domain to be learned is called the target domain. Given the source domain and target domain , when there is at least a difference in feature space, label space or probability distribution, using source domain data to learn the prediction function on the target domain, transfer learning can be formalized as: , In the formula, is the objective function of machine learning, The hypothesis space it satisfies is, Indicates that for all In Find the expected value, The cross entropy is often used as the loss function in classification tasks, while the minimum mean square error is usually used as the loss function in regression problems.

[0014] Further, using and They represent the Markov decision process of the source domain and target domain of reinforcement learning respectively, and the optimal strategies in the source domain and target domain are recorded as and , the strategy space is recorded as and , the task of transfer reinforcement learning is to find the mapping in the strategy space , so that: .

[0015] Compared with the prior art, the present invention has the following advantages: (1) The present invention adopts a feedforward control method based on reinforcement learning model, which can suppress the motor speed fluctuation without the need to accurately model the motor servo system. At the same time, it is robust to the uncertainty and parameter changes in the system, providing a powerful tool for the stable speed control of permanent magnet synchronous motors.

[0016] (2) The present invention adopts a transfer reinforcement learning method based on reward scaling to realize the strategy transfer of a single model under multiple working conditions and multiple motors, thereby improving the generalization of the speed stability control method.

[0017] The concept, specific structure and technical effects of the present invention will be further described below to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a diagram of the reinforcement learning interaction process of the present invention; Figure 2 The Actor-Critic framework of the present invention; Figure 3 It is a flowchart of the proximal strategy optimization algorithm of the present invention; Figure 4 This is a block diagram of transfer reinforcement learning of the present invention; Figure 5 It is the reward convergence diagram of the strategy migration of the present invention; Figure 6 The diagrams of the migration effect of the present invention are as follows: (a) is a diagram of the migration to 500 rpm speed control effect; (b) is a diagram of the migration to 2000 rpm speed control effect; Figure 7 The training result diagram of the present invention, (a) is a comparison of strategy transfer training results on motors with different parameters, and (b) is a comparison diagram of direct training results; Figure 8 This is the speed control effect at 500 rpm in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] The present invention introduces reinforcement learning model and transfer learning into the speed stability control of permanent magnet synchronous motor, and uses proximal strategy optimization method, reward scaling, transfer reinforcement learning and other technical means to carry out research on the speed stability control of permanent magnet synchronous motor. It is not only beneficial to improve the accuracy and stability of speed control of permanent magnet synchronous motor, and is suitable for real-time control requirements in various dynamic environments, but also can reduce the learning cost of the system in a new environment, accelerate the convergence speed of the algorithm, and improve the control efficiency and performance.

[0021] The following first introduces the theoretical basis, data model and algorithm involved in the present invention: A permanent magnet synchronous motor speed stabilization control method based on reinforcement learning, the specific steps are as follows: (I) Obtain the real-time operation data of the motor, and perform feedforward compensation control of the permanent magnet synchronous motor and smooth the motor speed fluctuation based on the reinforcement learning model.

[0022] (1) Obtain the real-time operation data of the motor based on the sensor equipment. These real-time operation data include the motor direct-axis current, motor quadrature-axis current, motor speed, motor mechanical angle, and motor servo system speed loop PI output.

[0023] (2) Perform batch normalization and other preprocessing and feature extraction on the collected data for use in subsequent control algorithms.

[0024] (3) Based on the collected real-time motor data, a reinforcement learning model is designed, which includes state space, action space and reward function.

[0025] like Figure 1 As shown in Figure 1, the theoretical basis of the reinforcement learning model is the Markov decision process: in a random process, the probability distribution of the next state depends only on the current state and has nothing to do with the past state. The scenario of reinforcement learning is usually that the agent interacts with the environment in a series of time steps. At each time step, the agent accepts the current state. , select actions based on the current strategy , perform the action Afterwards get reward And enter the next state , the goal of RL is to maximize the cumulative reward obtained after several time steps from the initial state.

[0026] (4) The designed reinforcement learning model is trained using the proximal strategy optimization method. During the training process, the motor speed fluctuation is used as a feedback signal to continuously optimize the model parameters so that the model can accurately learn the feedforward compensation control strategy.

[0027] The proximal policy optimization algorithm is a reinforcement learning method based on the actor-critic framework, which combines the policy iteration and value iteration methods in reinforcement learning. Figure 2 As shown. The actor (Actor) according to the current state (The current running data of the motor, which has been normalized and dimensionless) and probability to select the appropriate action (is the q-axis current compensation value, which has been normalized), interacts with the environment, and obtains the next state and rewards (An evaluation index for motor speed fluctuation); The critic calculates the value function of the current state and the value function of the next state , Indicates that in a specific Next, from the state At the beginning, the expected return that the algorithm can obtain. Then the time difference error is calculated (TD error): , Among them, the time difference error It reflects the difference between the actual immediate reward and the expected reward (i.e., the estimate of the current state value function) under the current strategy. This difference is used to update the value function to make it closer to the true value, thereby improving the performance of the strategy.

[0028] The proximal policy optimization algorithm is a reinforcement learning algorithm derived from the Actor-Critic method and can be well applied to tasks involving continuous control, such as Figure 3 As shown. Traditional Actor-Critic algorithms usually face the problem of being unable to set an appropriate learning rate. In the later stages of training, a small change in the policy network parameters may also lead to a significant change in policy behavior. The proximal policy optimization algorithm introduces importance sampling and gradient clipping to limit policy improvements, making the model training more stable. Its optimization goals are as follows: , in: , , Among them, the time difference error It reflects the difference between the actual immediate reward and the expected reward (i.e., the estimate of the current state value function) under the current strategy. This difference is used to update the value function to make it closer to the true value, thereby improving the performance of the strategy. Represents the empirical mean of a finite batch of samples. In machine learning and statistics, the empirical mean is a method of estimating the expected value, which approximates the expected value of the entire data set by averaging the samples. is the updated policy parameter, is the strategy parameter before updating, The representative strategy network parameters are , the input observation state is When , the output action is , Same reason. is a parameter of the clip function, which describes a truncation function that keeps the input value at and between.

[0029] At the same time, the advantage function is calculated by generalized advantage estimation (GAE) : , in, Used to measure the relative performance of actions and reduce the variance of policy gradients. It is dimensionless. , is the decay coefficient, which is used to characterize the importance of future rewards relative to current rewards, and is usually in the range of 0.95-0.99; is a hyperparameter between 0 and 1, used to characterize, When GAE is simplified to single-step time difference error, When GAE becomes the weighted sum of all future time difference errors.

[0030] (ii) Based on transfer learning, the strategy is migrated from a single operating condition to multiple operating conditions and from a single motor to multiple motors to achieve speed fluctuation suppression in multiple operating conditions and multiple motors.

[0031] The policy transfer method involves transfer reinforcement learning based on reward scaling, dynamically calculating the standard deviation of the sum of rolling discounted rewards, and then normalizing the current reward. The application of reward scaling in PPO usually involves the process of calculating the advantage function, which is used to estimate the superiority of a state-behavior pair, that is, how good the state-behavior is relative to the average level. When calculating Advantage, reward scaling can adjust the original reward signal.

[0032] Transfer reinforcement learning uses the similarities between data, domains, and tasks to apply the knowledge and models learned in the source domain to the target domain, thereby improving or completing the task learning effect in the target domain. The domain is the subject of transfer learning, including data and the probability distribution of generated data. A sample data in the domain includes input Output and its probability distribution , so a field can be represented as: In transfer learning, the domain being transferred is called the source domain, and the domain to be learned is called the target domain. and target domain , when there is at least a difference in feature space, label space or probability distribution, using source domain data to learn the prediction function on the target domain, transfer learning can be formalized as: , In the formula, is the objective function of machine learning, The hypothesis space it satisfies is, Indicates that for all In Find the expected value, The cross entropy is often used as the loss function in classification tasks, while the minimum mean square error is usually used as the loss function in regression problems.

[0033] use and They represent the Markov decision process of the source domain and target domain of reinforcement learning respectively, and the optimal strategies in the source domain and target domain are recorded as and , the strategy space is recorded as and , the task of transfer reinforcement learning is to find the mapping in the strategy space , so that: .

[0034] Based on the concept of transfer learning, the present invention can design a strategy migration method from a single working condition to multiple working conditions based on reward scaling. The method first adopts a model that has been trained and converged through reinforcement learning under specific parameter motors and specific speed conditions. The model can accurately capture the optimal control strategy under specific working conditions. When faced with new working conditions, the training results of the above model are used for fine-tuning, that is, some parameters of the model are retrained by adjusting the optimizer used for model update to adapt to the new working conditions, thereby effectively suppressing speed fluctuations under multiple working conditions. Fine-tuning is a transfer learning strategy that adjusts some or all of the parameters of the pre-trained model so that the model can adapt to the specific data characteristics in the new task. The reward convergence diagram and motor control effect after migration are shown in the figure. Figure 5 and Figure 6 shown.

[0035] The present invention further proposes a method for migrating from a single-motor control strategy to a multi-motor control strategy. This method is similar to the principle of operating condition migration, and aims to migrate the control strategy that has been trained and optimized on a single motor with specific parameters to a motor with different parameters. During implementation, the parameters of the trained single-motor control model are fine-tuned to enable it to adapt to the specific needs of motors with different parameters. This fine-tuning process is similar to the strategy adjustment in operating condition migration, but in this case, the adjustment is to adapt to changes in motor parameters rather than changes in operating conditions. In this way, the control strategy trained on one motor can be effectively migrated to another motor with different parameters, thereby achieving speed fluctuation suppression of the multi-motor system. The control effect is as follows: Figure 7 As shown in Figure 2. This method not only improves the applicability and flexibility of the control strategy, but also reduces the need to redevelop and train control models for motors with different parameters, thereby reducing development costs and time. Through this strategy migration, efficient control of multi-motor systems can be achieved, maintaining system stability and performance even when there are differences in motor parameters.

[0036] The speed fluctuation suppression effect can be evaluated by the L2 norm: , In the formula, is the actual speed sequence, is the target speed sequence, and The actual speed and the target speed are The smaller the L2 norm is, the smaller the speed fluctuation is.

[0037] It can be understood that the present application relates to a training method for a speed stabilization control strategy for a motor under specific parameter configuration and specific speed conditions. The method adopts reinforcement learning technology, and uses the initial model parameters obtained through the training process (i.e., the parameters when the model performance is close to 80% of the convergence score) as the starting point of transfer learning. Specifically, the method involves freezing the weights of the first few layers of the model so that these weights can be applied to the training of the same motor under different speed conditions, or to the training process of motors with different parameters. Through this strategy, the convergence speed of the reinforcement learning model can be significantly accelerated, and the risk of the model falling into a local optimal solution can be effectively reduced, thereby improving the generalization ability and adaptability of the model.

[0038] Compared with directly performing batch normalization on the rewards, the reward scaling method adopted in the present invention is to dynamically calculate the standard deviation of the rolling discounted reward sum, and then standardize the current reward. By using the rolling discounted sum, it is more adaptive and can more flexibly adapt to changes in reward signals in different tasks or environments.

[0039] (iii) Integrate the designed reinforcement learning model and strategy transfer method into the motor control system to ensure that it can seamlessly connect with the existing motor control system, evaluate and optimize the system performance, and ensure that it can effectively suppress speed fluctuations under various working conditions.

[0040] In a specific embodiment, the motor control system includes STM32G431- IHM08, the STM32G431 chip is based on the Arm® Cortex®-M4 32-bit RISC core, the main frequency can reach up to 170MHz, it has a high-order timer to support four-channel PWM with multiplexing, and a built-in coordinate rotation digital computer calculation unit (Coordinate Rotation Digital Computer, CORDIC); X-Nucleo-IHM08M1 is a three-phase brushless DC motor / permanent magnet synchronous motor drive expansion board based on STL220N6F7 STripFET™ F7 power MOSFET.

[0041] The switching frequency is set to 32kHz, and the high-frequency task is executed once every two PWM cycles. The medium-frequency task cycle is 1ms. The current sampling selects three current sampling to achieve precise current control. The DC power supply provides a bus voltage of 26.9V. The edge computing module is the NVIDIA Jetson TX2 module. The Ubuntu 18.04 system is used. The neural network environment based on pytorch and the reinforcement learning environment based on the gym library are built through the archiconda virtual environment toolbox. The speed control effect obtained by training is as follows: Figure 8 .

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A permanent magnet synchronous motor speed stabilization control method based on reinforcement learning, characterized in that: The specific steps of the method are as follows: S1. Obtain the real-time operation data of the motor, and perform feedforward compensation control of the permanent magnet synchronous motor and smooth the motor speed fluctuation based on the reinforcement learning model; S2. Based on transfer learning, the strategy is migrated from a single working condition to multiple working conditions and from a single motor to multiple motors to achieve speed fluctuation suppression in multiple working conditions and multiple motors; S3. Integrate the designed reinforcement learning model and policy transfer method into the motor control system, and evaluate and optimize the system performance.

2. The method for controlling the speed of a permanent magnet synchronous motor based on reinforcement learning according to claim 1 is characterized in that: The specific operations of step S1 are: S11. Acquire real-time operation data of the motor based on the sensor device; S12. The collected data will be preprocessed and feature extracted for subsequent control algorithms; S13. Based on the collected real-time operation data of the motor, a reinforcement learning model is designed, where the reinforcement learning model includes a state space, an action space, and a reward function; S14. Use the proximal strategy optimization method to train the designed reinforcement learning model. During the training process, the motor speed fluctuation is used as a feedback signal to continuously optimize the model parameters so that the model can accurately learn the feedforward compensation control strategy.

3. The method for controlling the speed of a permanent magnet synchronous motor based on reinforcement learning according to claim 2 is characterized in that: In the step S11, the real-time operation data of the motor includes the direct-axis current of the motor, the quadrature-axis current of the motor, the motor speed, the motor mechanical angle, and the speed loop PI output of the motor servo system.

4. The method for controlling the speed of a permanent magnet synchronous motor based on reinforcement learning according to claim 2 is characterized in that: In step S13, the theoretical basis of the reinforcement learning model is the Markov decision process, that is, in a random process, the probability distribution of the next state depends only on the current state and has nothing to do with the past state; The scenario of reinforcement learning is usually that the agent interacts with the environment in a series of time steps. At each time step, the agent accepts the current state , select actions based on the current strategy , perform the action Afterwards get reward And enter the next state , the goal of reinforcement learning is to maximize the cumulative reward obtained after several time steps from the initial state.

5. The method for controlling the speed stability of a permanent magnet synchronous motor based on reinforcement learning according to claim 4 is characterized in that: In step S14, the proximal strategy optimization algorithm is a reinforcement learning method based on the Actor-Critic framework. and probability to choose the appropriate action , interact with the environment, and obtain the next state and rewards ; Critic calculates the value function of the current state and the value function of the next state , Indicates that in a specific Next, from the state At the beginning, the expected return that the algorithm can obtain is calculated, and then the time difference error is obtained as follows: ， The optimization goals are as follows: ， in: ， ， Among them, the time difference error Reflects the difference between the actual immediate reward and the expected reward under the current strategy; represents the empirical mean of a finite batch of samples; is the updated policy parameter, is the strategy parameter before updating, The representative strategy network parameters are , the input observation state is When , the output action is , Similarly; is a parameter of the clip function, which describes a truncation function that keeps the input value at and between; At the same time, the advantage function is calculated by the generalized advantage estimate GAE : ， in, Used to measure the relative performance of actions and reduce the variance of policy gradients. It is dimensionless. , is the decay coefficient, which is used to characterize the importance of future rewards relative to current rewards, and is usually in the range of 0.95-0.99; is a hyperparameter between 0 and 1, used to characterize, When GAE is simplified to single-step time difference error, When GAE becomes the weighted sum of all future time difference errors.

6. The method for controlling the speed stability of a permanent magnet synchronous motor based on reinforcement learning according to claim 5 is characterized in that: In step S2, the strategy migration method involves transfer reinforcement learning based on reward scaling, dynamically calculating the standard deviation of the sum of rolling discounted rewards, and then standardizing the current reward.

7. The method for controlling the speed stability of a permanent magnet synchronous motor based on reinforcement learning according to claim 6 is characterized in that: The transfer reinforcement learning applies the knowledge and models learned in the source domain to the target domain by utilizing the similarities between data, domains and tasks, thereby improving or completing the task learning effect in the target domain. The domain is the main body of transfer learning, including data and the probability distribution of generated data. A sample data in the domain includes input Output and its probability distribution , so a field can be represented as: ; In transfer learning, the domain being transferred is called the source domain, and the domain to be learned is called the target domain. Given the source domain and target domain , when there is at least a difference in feature space, label space or probability distribution, using source domain data to learn the prediction function on the target domain, transfer learning can be formalized as: ， In the formula, is the objective function of machine learning, The hypothesis space it satisfies is, Indicates that for all In Find the expected value, The cross entropy is often used as the loss function in classification tasks, while the minimum mean square error is usually used as the loss function in regression problems.

8. The method for controlling the speed stability of a permanent magnet synchronous motor based on reinforcement learning according to claim 7 is characterized in that: use and They represent the Markov decision process of the source domain and target domain of reinforcement learning respectively, and the optimal strategies in the source domain and target domain are recorded as and , the strategy space is recorded as and , the task of transfer reinforcement learning is to find the mapping in the strategy space , so that: 。

Images