A parameter optimization method for active disturbance rejection controller based on two-stage training reinforcement learning

Abstract

A method for optimizing parameters of an auto-disturbance rejection controller based on two-stage training reinforcement learning belongs to the field of computer system control technology. The present invention solves the problems of low parameter adjustment efficiency, inability to guarantee optimal parameters, and reliance on initial parameter selection when using an auto-disturbance rejection controller in existing methods. The two-stage training method based on pre-training and advanced reinforcement learning of the present invention designs a reward function based on speed error in the pre-training stage, and designs a segmented reward mechanism based on error threshold in the advanced training process, divides the speed tracking process of the motor into different intervals, guides the training of the intelligent agent with different rewards in different intervals, decouples performance indicators, effectively avoids local optimality of parameters, and realizes optimal parameters of the auto-disturbance rejection controller. The method of the present invention can be applied to parameter optimization of the auto-disturbance rejection controller.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer system control, and in particular relates to a parameter optimization method for an auto-disturbance rejection controller based on two-stage training reinforcement learning. Background Art

[0002] Permanent magnet synchronous motors (PMSMs) are widely used due to their high power density and conversion efficiency. Improving the dynamic response of the speed loop is crucial for enhancing the control performance of PMS motors. Active disturbance rejection control (ADRC), due to its interference rejection capabilities and model-free nature, has the potential to be applied to the speed loop control of PMS motors.

[0003] However, ADRC controllers have numerous parameters, coupled with weak sensitivity. Parameter adjustment often relies on a crude model of identification, often requiring manual tuning based on experience. This approach is time-consuming and difficult to achieve satisfactory control performance. Many researchers have combined artificial intelligence algorithms with ADRC controllers, using parameter tuning to adaptively output control strategies. These include using heuristic algorithms such as ACO (Ant Colony Optimization), DBO (Dung Beetle Optimization), and PSO (Particle Swarm Optimization) to combine ADRC controllers. These algorithms leverage their learning and iterative capabilities to optimize parameters and achieve better controller output performance. However, these heuristic algorithms are typically constructed based on experience, resulting in instability when dealing with complex real-world problems and lacking a guarantee of finding the optimal solution. Other researchers have used neural networks as parameter optimizers for ADRC controllers. For example, networks based on RBF (Radial Basis Function Networks) learn nonlinear mapping relationships within the system to dynamically adjust parameters, or employ backpropagation neural networks for tuning. However, due to their high sensitivity to initial weights, neural networks suffer from over-reliance on the initial parameter selection.

[0004] In summary, the existing methods using ADRC still have problems such as low parameter adjustment efficiency, inability to ensure optimal parameters, and dependence on initial parameter selection. Therefore, it is very necessary to propose a new method for optimizing the parameters of the ADRC to solve the above problems and improve the control performance of the ADRC. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems of low parameter adjustment efficiency, inability to ensure optimal parameters, and reliance on initial parameter selection when using an active disturbance rejection controller in existing methods, and to propose an active disturbance rejection controller parameter optimization method based on two-stage training reinforcement learning.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for optimizing the parameters of an active disturbance rejection controller based on two-stage training reinforcement learning, the method specifically comprising the following steps:

[0007] Step 1: Establish an active disturbance rejection controller model and a permanent magnet synchronous motor model;

[0008] Step 2: Build a deep reinforcement learning model, using the established permanent magnet synchronous motor model as the agent's interactive environment and the parameters to be optimized of the active disturbance rejection controller as the agent's action space.

[0009] Step 3: Design a reward function based on speed error, and use the reward function based on speed error and the actor-critic algorithm to pre-train the deep reinforcement learning model until the reward function converges, thus obtaining a pre-trained deep reinforcement learning model.

[0010] Step 4: Use the pre-trained deep reinforcement learning model parameters as the initial parameters, design a piecewise reward function based on the error threshold, and use the piecewise reward function to perform advanced training on the deep reinforcement learning model until the piecewise reward function converges, thus obtaining a deep reinforcement learning model with advanced training.

[0011] Step 5: Use the deep reinforcement learning model trained in step 4 to output the parameters of the active disturbance rejection controller, and use the output of the active disturbance rejection controller as the input of the speed loop of the permanent magnet synchronous motor model to achieve control of the speed loop.

[0012] Furthermore, the active disturbance rejection controller model is a first-order active disturbance rejection controller model, and the active disturbance rejection controller model includes three parts: a tracking differential unit, an extended state observation unit, and a nonlinear state feedback unit.

[0013] Furthermore, the working process of the tracking differential unit is:

[0014]

[0015] Among them, ω * is the reference speed, z 11 To track the output signal of the differential unit, represents z 11 The first-order derivative of , r, a1 and δ1 are hyperparameters, and r, a1 and δ1 are the parameters to be optimized for the tracking differential unit, and fal is a nonlinear function;

[0016]

[0017] Here, |·| represents the absolute value, and sign(·) represents the sign function.

[0018] Furthermore, the working process of the expansion state observation unit is as follows:

[0019]

[0020] Among them, z 21 and z 22 is the state variable output by the extended state observation unit, ω is the observation speed fed back by the permanent magnet synchronous motor model, represents z 21 The first derivative of represents z 22 The first-order derivative of , u is the output of the ADRC, b is the output gain, β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 are all hyperparameters, and β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 is the parameter to be optimized of the expanded state observation unit;

[0021]

[0022]

[0023] Furthermore, the working process of the nonlinear state feedback unit is:

[0024]

[0025] Among them, u0 is the intermediate variable of the output parameter, β3, a3 and δ3 are all hyperparameters, and β3, a3 and δ3 are the parameters to be optimized of the nonlinear state feedback unit;

[0026]

[0027] Furthermore, the state space of the agent is:

[0028] observation=[e,ω * ,ω,i d ,i q , T E , z 22 ] (12)

[0029] Among them, observation is the state space, e is the error between the reference velocity and the observed velocity, i d is the d-axis output current, i qis the q-axis output current i q , T E is the output torque.

[0030] Furthermore, the reward function based on speed error is:

[0031] R pre =|ω * -ω| (14)

[0032] Among them, R pre is the reward function value based on the speed error.

[0033] Furthermore, the design is based on a piecewise reward function with an error threshold, specifically:

[0034] Step 4.1: According to the current interactive step speed error e cc , the speed error of the previous interaction step e la and the error threshold ε, to determine the state interval of the speed tracking control process of the permanent magnet synchronous motor;

[0035] Step 42: Design a reward function for each state interval.

[0036] Furthermore, the step 4-1 is specifically as follows:

[0037] When e cc >ε and e la When ≥ε, the speed tracking control process of the permanent magnet synchronous motor is in the response or disturbance range;

[0038] When e cc ≤ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor is in the steady-state range;

[0039] When e cc ≤ε and e la When ≥ε, the speed tracking control process of the permanent magnet synchronous motor begins to enter the steady-state range;

[0040] When e cc >ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state range and enters the disturbance or response range.

[0041] Furthermore, the step 42 is specifically as follows:

[0042] In the response interval, the reward function R re for:

[0043]

[0044] Among them, t re and ere are the duration and speed error of the response interval, respectively, δ is a non-zero number with an arbitrarily small absolute value, α1 and α2 are the normalization coefficients of the performance indicators;

[0045] In the perturbation interval, the reward function R dis for:

[0046] R dis =-r′-α3·|e dis |-α4·t dis (16)

[0047] Among them, e dis and t dis are the velocity error and duration of the disturbance interval, r′ is a constant, α3 and α4 are the normalization coefficients of the performance indicators;

[0048] In the steady-state range, the reward function R st for:

[0049]

[0050] Among them, α5 is the standardization coefficient of the performance index, e st is the speed error during steady state.

[0051] The beneficial effects of the present invention are:

[0052] The training method of the present invention, based on pre-training and advanced reinforcement learning, can shorten the training time and improve the efficiency of iterative optimization. During the advanced training process, a segmented reward mechanism based on an error threshold is used to divide the speed tracking process of the motor into different intervals. Different rewards are used to guide the training of the intelligent agent in different intervals, decouple performance indicators, effectively avoid local optimality of parameters, and achieve optimal parameters of the active disturbance rejection controller.

[0053] The present invention optimizes the parameters of the ADRC by designing reinforcement learning, which can achieve better dynamic and steady-state performance. Compared with existing methods, the method of the present invention does not need to rely on initial parameter selection and does not require manual parameter adjustment, and has the advantage of high parameter adjustment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a method for optimizing parameters of an auto-disturbance rejection controller based on two-stage training reinforcement learning according to the present invention;

[0055] Figure 2 Schematic diagram of the structure of the active disturbance rejection controller model established by the present invention;

[0056] Figure 3 This is a training flow chart of advanced reinforcement learning based on pre-training in the present invention;

[0057] Figure 4 Flowchart of the actor-critic algorithm for the deterministic strategy of the present invention;

[0058] Figure 5a Schematic diagram of the error threshold-based segmented reward mechanism designed for this invention;

[0059] Figure 5b for Figure 5a A magnified view of the switching point region;

[0060] Figure 6 A diagram of the interior permanent magnet synchronous motor experimental platform used in the present invention;

[0061] Figure 7a This is a diagram showing the speed change during the motor startup process using an experience-based manually tuned ADRC controller.

[0062] Figure 7b This is a diagram showing the speed change during motor startup for an ADRC controller that only performs pre-training reinforcement learning optimization.

[0063] Figure 7c A diagram showing the speed change during the motor startup process of the active disturbance rejection controller of the present invention;

[0064] Figure 8a This is a diagram showing the speed change of the motor under the switching speed condition using an experience-based manually tuned ADRC.

[0065] Figure 8b The speed change diagram of the motor under the switching speed condition for the active disturbance rejection controller that only performs pre-training reinforcement learning optimization;

[0066] Figure 8c A diagram showing the speed change of the motor of the active disturbance rejection controller of the present invention under the switching speed condition;

[0067] Figure 9a The motor speed change diagram of the automatic disturbance rejection controller manually adjusted based on experience under sudden load disturbance conditions;

[0068] Figure 9b This is the motor speed change diagram of the ADRC under sudden load disturbance conditions, which only uses pre-trained reinforcement learning optimization;

[0069] Figure 9c The figure is a diagram showing the motor speed change of the active disturbance rejection controller of the present invention under sudden load disturbance conditions. DETAILED DESCRIPTION

[0070] Specific implementation method 1: Combination Figure 1 and Figure 3This embodiment describes a method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning, and the method specifically includes the following steps:

[0071] Step 1: Establish an active disturbance rejection controller model and a permanent magnet synchronous motor model (any permanent magnet synchronous motor model can be used, that is, the method of the present invention is applicable to any permanent magnet synchronous motor model);

[0072] Step 2: Build a deep reinforcement learning model, using the established permanent magnet synchronous motor model as the agent's interactive environment and the parameters to be optimized of the active disturbance rejection controller as the agent's action space.

[0073] Step 3: Design a reward function based on speed error, and use the reward function based on speed error and the actor-critic algorithm to pre-train the deep reinforcement learning model until the reward function converges, thus obtaining a pre-trained deep reinforcement learning model.

[0074] Step 4: Use the pre-trained deep reinforcement learning model parameters as the initial parameters, design a piecewise reward function based on the error threshold, and use the piecewise reward function to perform advanced training on the deep reinforcement learning model until the piecewise reward function converges, thus obtaining a deep reinforcement learning model with advanced training.

[0075] Step 5: Use the deep reinforcement learning model trained in step 4 to output the parameters of the active disturbance rejection controller, and use the output of the active disturbance rejection controller as the input of the speed loop of the permanent magnet synchronous motor model to achieve control of the speed loop.

[0076] The agent is trained using the actor-critic algorithm, which optimizes actions through the environment and state, and then explores and improves the optimized actions based on the rewards obtained from the actions. The agent interacts with the environment continuously until the rewards converge. The agent can then output optimized control parameters based on the current state. Figure 4 As shown in Figure 2, the update process of network parameters during training is:

[0077] The actor-critic algorithm mainly includes four parts: actor behavior network, critic comment network, target network and experience replay. The target network includes actor target network and critic target network. The actor behavior network serves as the main policy network, outputting corresponding policy actions based on the current state; the critic comment network serves as the main value network, used to evaluate and optimize the value of the policy. In order to prevent overestimation of the policy, two value networks are used; the actor target network copies the parameters of the actor behavior network, and the critic target network copies the parameters of the critic comment network, which are used to provide stable target values ​​during training. Experience replay stores the experience data obtained from the interaction of intelligent agents into the experience pool, and updates the main network parameters by randomly sampling the experience pool. The main network parameters are updated using the gradient descent method, and the target network is updated using a delayed update strategy; the details are as follows:

[0078]

[0079]

[0080]

[0081]

[0082] in, and are the parameters of the main value network and the main strategy network respectively, and are the parameters of the target value network and the target policy network, Q m , Q t The value of the action evaluation by the main value network and the target value network, τ, is the delayed update coefficient and is less than 1, which helps to reduce the fluctuation of value and smooth the parameter changes of the network. y is the evaluation value of the target network. The goal is to minimize the difference between the evaluation value of the target network and the main network for the current action, thereby maximizing the evaluation value of the value network. t’ and s t’+1 are the states at time t′ and time t′+1, respectively, m and π t Represent the strategies of the main strategy network and the target strategy network respectively, a t’ and are the actions output by the main policy network and the target policy network respectively, i represents the i-th sample, and N represents the total number of samples.

[0083] Update the parameters of the main network in real time The target network's parameters are provided by the main network and are updated less frequently than the main network. Pre-training enables the agent to acquire a certain degree of generalization ability in basic tasks. Pre-training also helps to increase the agent's training speed, providing a foundation for subsequent advanced training.

[0084] After advanced training, the intelligent agent further improved its response speed and anti-interference ability for motor control, and can output optimized parameters in real time in more complex tasks to achieve precise control of the motor speed loop.

[0085] Specific implementation method 2: Combination Figure 2 This embodiment differs from the first embodiment in that the ADRC model is a first-order ADRC model, which includes three parts: a tracking differentiator (TD), an extended state observer (ESO), and a nonlinear state inverse feedback (NLSEF).

[0086] Other steps and parameters are the same as those in the first embodiment.

[0087] The tracking differential unit (TDU) arranges the transition process for the system's reference signal and extracts the differential signal to mitigate the conflict between overshoot and fast response. The extended state observation unit (ESU), the core of the active disturbance rejection controller, analyzes the system's state variables and estimates the total disturbance. The nonlinear state feedback unit (NSFU) analyzes the outputs of the TDU and ESU and uses a nonlinear control law to adjust the output error between them and compensate for the total disturbance.

[0088] The present invention aims at the speed loop control of a permanent magnet synchronous motor. By adjusting the parameters in the active disturbance rejection controller, the controller observes the state of the motor system and adaptively outputs the optimal control law.

[0089] Specific embodiment three: This embodiment differs from specific embodiment one or two in that the working process of the tracking differential unit is as follows:

[0090]

[0091] Among them, ω * is the reference speed, z 11 To track the output signal of the differential unit, represents z 11 The first-order derivative of , r, a1 and δ1 are hyperparameters, and r, a1 and δ1 are the parameters to be optimized for the tracking differential unit, and fal is a nonlinear function;

[0092]

[0093] Here, |·| represents the absolute value, and sign(·) represents the sign function.

[0094] Other steps and parameters are the same as those in the first or second embodiment.

[0095] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the working process of the expansion state observation unit is as follows:

[0096]

[0097] Among them, z 21 and z 22 is the state variable output by the extended state observation unit, ω is the observation speed fed back by the permanent magnet synchronous motor model, represents z 21 The first derivative of represents z 22 The first-order derivative of , u is the output of the active disturbance rejection controller (used to control the speed loop of the permanent magnet synchronous motor), b is the output gain, β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 are all hyperparameters, and β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 is the parameter to be optimized of the expanded state observation unit;

[0098]

[0099]

[0100] The other steps and parameters are the same as those in the first to third embodiments.

[0101] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that the working process of the nonlinear state feedback unit is as follows:

[0102]

[0103] Among them, u0 is the intermediate variable of the output parameter, β3, a3 and δ3 are all hyperparameters, and β3, a3 and δ3 are the parameters to be optimized of the nonlinear state feedback unit;

[0104]

[0105] The other steps and parameters are the same as those in the first to fourth embodiments.

[0106] The value ranges of the filter factor δ and the nonlinear factor a in the tracking differential unit, the expanded state observation unit and the nonlinear state feedback unit are all between 0 and 1.

[0107] Specific embodiment 6: This embodiment differs from any one of specific embodiments 1 to 5 in that the state space of the agent is:

[0108] observation=[e,ω * ,ω,i d ,i q , T E , z 22 ] (12)

[0109] Among them, observation is the state space, e is the error between the reference velocity and the observed velocity, i d is the d-axis output current, i q is the q-axis output current i q , T E is the output torque.

[0110] The other steps and parameters are the same as those in the first to fifth embodiments.

[0111] The intelligent agent in reinforcement learning outputs the optimized parameters of the ADRC, applies the control law output by the ADRC to the environment, and the environment feeds back the observed state, evaluates the state, and gives a reward for the current action.

[0112] e is used to evaluate the speed tracking capability under the current control strategy, i d 、i q With torque T E Used to help the intelligent agent understand the electrical and dynamic characteristics of the motor system, 22 As a disturbance observer in the active disturbance rejection controller, it enables the intelligent agent to better understand the uncertainties such as load changes and environmental disturbances of the system.

[0113] The action space of the agent is 12 parameters. Adding noise can improve the exploration ability of the agent. The action space can be expressed as:

[0114] action=π(s|θ π )+noise (13)

[0115] Specific embodiment 7: This embodiment differs from any one of specific embodiments 1 to 6 in that the reward function based on speed error is:

[0116] Rpre =|ω * -ω| (14)

[0117] Among them, R pre is the reward function value based on the speed error.

[0118] The other steps and parameters are the same as those in the first to sixth embodiments.

[0119] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the design is a segmented reward mechanism based on error threshold (SRMET), specifically:

[0120] Step 4.1: According to the current interactive step speed error e cc , the speed error of the previous interaction step e la and the error threshold ε, to determine the state interval of the speed tracking control process of the permanent magnet synchronous motor;

[0121] Step 42: Design a reward function for each state interval.

[0122] The other steps and parameters are the same as those in the first to seventh embodiments.

[0123] Specific embodiment 9: This embodiment differs from specific embodiments 1 to 8 in that step 4 is specifically as follows:

[0124] When e cc >ε and e la When ≥ε, the speed tracking control process of the permanent magnet synchronous motor is in the response or disturbance range;

[0125] When e cc ≤ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor is in the steady-state range;

[0126] When e cc ≤ε and e la When ≥ε, the speed tracking control process of the permanent magnet synchronous motor begins to enter the steady-state range;

[0127] When e cc >ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state range and enters the disturbance or response range.

[0128] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0129] In the present invention, the value of ε can be set to 1. If the speed error e of the current interactive step is cc From the steady-state interval to e cc >ε and e cc >ε′, the value of ε′ is 100, then the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the response interval, and the pointer p is set to 0; if the speed error e of the current interactive step is cc From the steady-state range to ε′>e cc >ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the disturbance interval, and the pointer p is set to 1. cc >ε and e la When ≥ε, the interval is determined based on the value of pointer p in the most recent interaction process. If pointer p is 0, the current interval is the response interval; if pointer p is 1, the current interval is the disturbance interval.

[0130] Specific implementation method ten: Combination Figure 5a and Figure 5b This embodiment is described. The difference between this embodiment and the first embodiment of the embodiment 1 to the ninth embodiment is that the step 42 is specifically as follows:

[0131] In the response interval, the reward function R re for:

[0132]

[0133] Among them, t re and e re are the duration and speed error of the response interval, respectively, 6 is a non-zero number with an arbitrarily small absolute value, α1 and α2 are the normalization coefficients of the performance indicators;

[0134] Setting a reward function for the response interval can specifically guide the agent to optimize the motor's rapid response capability;

[0135] In the perturbation interval, the reward function R dis for:

[0136] R dis =-r′-α3·|e dis |-α4·t dis (16)

[0137] Among them, e dis and t dis are the velocity error and duration of the disturbance interval, r′ is a constant, α3 and α4 are the normalization coefficients of the performance indicators;

[0138] When entering the disturbance zone, the agent is penalized for leaving the steady state, and the agent is trained to improve its anti-disturbance capability.

[0139] In the steady-state range, the reward function R st for:

[0140]

[0141] Among them, α5 is the standardization coefficient of the performance index, e st is the speed error during steady state.

[0142] Reward R by Design st To encourage the agent to continue to maintain this state.

[0143] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0144] This embodiment sets different reward functions in different state intervals, which helps to optimize the control performance of the motor in a targeted manner, such as response speed, anti-interference ability, etc.

[0145] Experimental part

[0146] The effectiveness of the present invention is illustrated by the experimental results below. The trained network parameters are imported into the permanent magnet synchronous motor system. The permanent magnet synchronous motor experimental platform is as follows: Figure 6 The system parameters are shown in Table 1. In order to verify the improvement of the motor speed loop control performance by the method of the present invention, the performance indicators of the manual active disturbance rejection controller based on experience (method 1), the active disturbance rejection controller with only pre-training reinforcement learning optimization (method 2), and the active disturbance rejection controller combining pre-training and advanced reinforcement learning optimization method (the present invention) are compared and analyzed.

[0147] Table 1

[0148]

[0149] A. Startup performance

[0150] Figure 7a The startup waveform of the manual ADRC based on experience is shown. Figure 7b The startup waveform of the ADRC with only pre-training reinforcement learning optimization is shown. Figure 7c The startup waveforms of an active disturbance rejection controller combining pre-training and advanced reinforcement learning optimization are shown. It can be seen that Method 2 improves the control system's response speed and significantly reduces overshoot compared to Method 1, but the method of the present invention offers faster response speed and smaller overshoot.

[0151] B. Switching speed performance

[0152] Figure 8a The switching speed waveform of the manually adjusted ADRC based on experience is shown. Figure 8bThe switching speed waveform of the ADRC optimized by pre-training reinforcement learning is shown. Figure 8c The switching speed waveform of the ADRC combining pre-training and advanced reinforcement learning optimization methods is shown. It can be seen that the method of the present invention still has the best performance among the three in terms of overshoot and response speed.

[0153] C. Performance under load disturbance

[0154] Figure 9a The load disturbance waveform of the ADRC is shown when a sudden load (8Nm) is applied. Figure 9b The load disturbance waveform of the ADRC with only pre-trained reinforcement learning optimization is shown when a sudden load (8Nm) is applied. Figure 9c The load disturbance waveforms of the active disturbance rejection controller using a combination of pre-training and advanced reinforcement learning optimization are shown when a sudden load (8 Nm) is applied. It can be seen that method 1 has the worst anti-disturbance capability under external load disturbances, while the proposed method has the smallest maximum speed fluctuation and the shortest disturbance recovery time.

[0155] The above experimental results show that the optimization method of the present invention has excellent control performance.

[0156] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A parameter optimization method for an auto-disturbance rejection controller based on two-stage training reinforcement learning, characterized in that: The method specifically comprises the following steps: Step 1: Establish an active disturbance rejection controller model and a permanent magnet synchronous motor model; Step 2: Build a deep reinforcement learning model, using the established permanent magnet synchronous motor model as the agent's interactive environment and the parameters to be optimized of the active disturbance rejection controller as the agent's action space. Step 3: Design a reward function based on speed error, and use the reward function based on speed error and the actor-critic algorithm to pre-train the deep reinforcement learning model until the reward function converges, thus obtaining a pre-trained deep reinforcement learning model. Step 4: Use the pre-trained deep reinforcement learning model parameters as the initial parameters, design a piecewise reward function based on the error threshold, and use the piecewise reward function to perform advanced training on the deep reinforcement learning model until the piecewise reward function converges, thus obtaining a deep reinforcement learning model with advanced training. Step 5: Use the deep reinforcement learning model trained in step 4 to output the parameters of the active disturbance rejection controller, and use the output of the active disturbance rejection controller as the input of the speed loop of the permanent magnet synchronous motor model to achieve control of the speed loop.

2. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 1, characterized in that: The active disturbance rejection controller model is a first-order active disturbance rejection controller model, which includes three parts: a tracking differential unit, an extended state observation unit and a nonlinear state feedback unit.

3. The method for optimizing parameters of an auto-disturbance rejection controller based on two-stage training reinforcement learning according to claim 2, characterized in that: The working process of the tracking differential unit is as follows: Among them, ω * is the reference speed, z 11 To track the output signal of the differential unit, represents z 11 The first-order derivative of , r, a1 and δ1 are hyperparameters, and r, a1 and δ1 are the parameters to be optimized for the tracking differential unit, and fal is a nonlinear function; Here, |·| represents the absolute value, and sign(·) represents the sign function.

4. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 3 is characterized in that: The working process of the extended state observation unit is as follows: Among them, z 21 and z 22 is the state variable output by the extended state observation unit, ω is the observation speed fed back by the permanent magnet synchronous motor model, represents z 21 The first derivative of represents z 22 The first-order derivative of , u is the output of the ADRC, b is the output gain, β 21 ,a 21 ,δ 21 ,β 22 ,a 22 and δ 22 are all hyperparameters, and β 21 ,a 21 ,δ 21 ,β 22 ,a 22 and δ 22 is the parameter to be optimized of the expanded state observation unit; 5. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 4 is characterized in that: The working process of the nonlinear state feedback unit is as follows: Among them, u0 is the intermediate variable of the output parameter, β3, a3 and δ3 are all hyperparameters, and β3, a3 and δ3 are the parameters to be optimized of the nonlinear state feedback unit; 6. The method for optimizing parameters of an auto-disturbance rejection controller based on two-stage training reinforcement learning according to claim 5, characterized in that: The state space of the agent is: observation=[e,ω * ,ω,i d ,i q ,T E ,z 22 ] (12) Among them, observation is the state space, e is the error between the reference velocity and the observed velocity, i d is the d-axis output current, i q is the q-axis output current i q , T E is the output torque.

7. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 6, characterized in that: The reward function based on speed error is: R pre =|ω * -ω| (14) Among them, R pre is the reward function value based on the speed error.

8. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 7, characterized in that: The design is based on a piecewise reward function with an error threshold, specifically: Step 4.1: According to the current interactive step speed error e cc , the speed error of the previous interaction step e la and the error threshold ε, to determine the state interval of the speed tracking control process of the permanent magnet synchronous motor; Step 42: Design a reward function for each state interval.

9. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 8, characterized in that: The step 4-1 is specifically as follows: When e cc >ε and e la When ≥ε, the speed tracking control process of the permanent magnet synchronous motor is in the response or disturbance range; When e cc ≤ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor is in the steady-state range; When e cc ≤ε and e la When ≥ε, the speed tracking control process of the permanent magnet synchronous motor begins to enter the steady-state range; When e cc >ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state range and enters the disturbance or response range.

10. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 9, characterized in that: The step 42 is specifically as follows: In the response interval, the reward function R re for: Among them, t re and e re are the duration and speed error of the response interval, respectively, δ is a non-zero number with an arbitrarily small absolute value, α1 and α2 are the normalization coefficients of the performance indicators; In the perturbation interval, the reward function R dis for: R dis =-r′-α3·|e dis |-α4·t dis (16) Among them, e dis and t dis are the velocity error and duration of the disturbance interval, r′ is a constant, α3 and α4 are the normalization coefficients of the performance indicators; In the steady-state range, the reward function R st for: Among them, α5 is the standardization coefficient of the performance index, e st is the speed error during steady state.

Images