Active-disturbance-rejection controller parameter optimization method based on two-stage training reinforcement learning
The self-immune controller parameters are optimized through the reinforcement learning method of two-stage training, which solves the problems of low parameter adjustment efficiency and dependence on initial parameter selection in the prior art, and achieves better control performance and parameter optimization.
Patent Information
- Application Number
- CN202411049808.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The existing self-immune controller has low parameter adjustment efficiency, cannot guarantee optimal parameters, and relies on initial parameter selection, resulting in insufficient control performance.
Using reinforcement learning method based on two-stage training, by establishing a self-immunity controller model and a permanent magnet synchronous motor model, a reward function based on speed error is designed for pre-training and advanced training of segmented reward function, and optimizing self-immunity Controller parameters.
The training time is shortened, it improves iterative optimization efficiency, avoids local optimal parameters, and achieves better dynamic and steady-state performance. It does not rely on initial parameter selection and has high parameter adjustment efficiency.
Smart Images

Figure CN118963136A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer system control, and in particular relates to a method for optimizing parameters of an auto-disturbance rejection controller based on two-stage training reinforcement learning. Background Art
[0002] Permanent magnet synchronous motors are widely used because of their high power density and conversion efficiency. Improving the dynamic response capability of the speed loop is very important for improving the control performance of permanent magnet synchronous motors. Due to its anti-interference ability and model-free characteristics, ADRC has the potential to be applied to the speed loop control of permanent magnet synchronous motors.
[0003] However, there are many parameters in the ADRC, and there is a coupling relationship between the parameters and the sensitivity is weak. Parameter adjustment usually relies on the rough model of identification, that is, the manual adjustment method based on experience, which is time-consuming and difficult to obtain satisfactory control performance. Many researchers combine artificial intelligence algorithms with ADRCs, and use parameter tuning to adaptively output control strategies, including using heuristic algorithms such as ACO (Ant Colony Optimization), DBO (Dung Beetle Optimization) and PSO (Particle Swarm Optimization) to combine ADRCs, optimize parameters through the learning and iteration capabilities of the algorithms, and obtain controller outputs with better performance. However, heuristic algorithms are usually constructed based on experience, and are unstable when dealing with complex practical problems, and cannot guarantee the optimal solution. Some researchers also use neural networks as parameter optimizers for ADRCs, such as using a network based on RBF (Radial basis function network) to learn the nonlinear mapping relationship in the system to dynamically adjust parameters, or using back propagation neural networks for tuning. However, due to the high sensitivity of neural networks to initial weights, there is a problem of over-reliance on initial parameter selection.
[0004] In summary, when the existing methods use the ADRC, there are still problems such as low parameter adjustment efficiency, inability to guarantee optimal parameters, and dependence on initial parameter selection. Therefore, it is necessary to propose a new method for optimizing the parameters of the ADRC to solve the above problems and improve the control performance of the ADRC. Summary of the invention
[0005] The purpose of the present invention is to solve the problems of low parameter adjustment efficiency, inability to guarantee optimal parameters and reliance on initial parameter selection when an auto-disturbance rejection controller is used in existing methods, and a parameter optimization method for an auto-disturbance rejection controller based on two-stage training reinforcement learning is proposed.
[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for optimizing the parameters of an active disturbance rejection controller based on two-stage training reinforcement learning, the method specifically comprising the following steps:
[0007] Step 1: Establish an active disturbance rejection controller model and a permanent magnet synchronous motor model;
[0008] Step 2: Establish a deep reinforcement learning model, use the established permanent magnet synchronous motor model as the interactive environment of the intelligent agent, and use the parameters to be optimized of the ADRC as the action space of the intelligent agent;
[0009] Step 3: Design a reward function based on speed error, use the reward function based on speed error and the actor-critic algorithm to pre-train the deep reinforcement learning model, and stop pre-training when the reward function converges to obtain a pre-trained deep reinforcement learning model;
[0010] Step 4: Use the pre-trained deep reinforcement learning model parameters as initial parameters, design a piecewise reward function based on the error threshold, and use the piecewise reward function to perform advanced training on the deep reinforcement learning model until the piecewise reward function converges, and stop training to obtain an advanced trained deep reinforcement learning model;
[0011] Step 5: Use the deep reinforcement learning model trained in step 4 to output the parameters of the ADRC, and use the output of the ADRC as the input of the speed loop of the permanent magnet synchronous motor model to achieve control of the speed loop.
[0012] Furthermore, the active disturbance rejection controller model is a first-order active disturbance rejection controller model, and the active disturbance rejection controller model includes three parts: a tracking differential unit, an extended state observation unit and a nonlinear state feedback unit.
[0013] Furthermore, the working process of the tracking differential unit is:
[0014]
[0015] Among them, ω * is the reference speed, z 11 To track the output signal of the differential unit, Indicates z 11 The first-order derivative of , r, a1 and δ1 are hyperparameters, and r, a1 and δ1 are the parameters to be optimized of the tracking differential unit, and fal is a nonlinear function;
[0016]
[0017] Among them, |·| means taking the absolute value, and sign(·) means the sign function.
[0018] Furthermore, the working process of the expansion state observation unit is:
[0019]
[0020] Among them, z 21 and z 22 is the state variable output by the extended state observation unit, ω is the observed speed fed back by the permanent magnet synchronous motor model, Indicates z 21 The first derivative of Indicates z 22 The first-order derivative of , u is the output of the ADRC, b is the output gain, β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 are all hyperparameters, and β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 is the parameter to be optimized of the extended state observation unit;
[0021]
[0022]
[0023] Furthermore, the working process of the nonlinear state feedback unit is:
[0024]
[0025] Among them, u0 is the intermediate variable of the output parameter, β3, a3 and δ3 are all hyperparameters, and β3, a3 and δ3 are the parameters to be optimized of the nonlinear state feedback unit;
[0026]
[0027] Furthermore, the state space of the agent is:
[0028] observation = [e, ω * ,ω,i d ,i q , T E , z 22 ] (12)
[0029] Among them, observation is the state space, e is the error between the reference speed and the observed speed, i d is the d-axis output current, i qis the q-axis output current i q , T E is the output torque.
[0030] Furthermore, the reward function based on speed error is:
[0031] R pre =|ω * -ω| (14)
[0032] Among them, R pre is the reward function value based on the speed error.
[0033] Furthermore, the design is based on a piecewise reward function of an error threshold, specifically:
[0034] Step 41: According to the current interactive step speed error e cc , the speed error of the previous interactive step e la and an error threshold ε, to determine the state interval in which the speed tracking control process of the permanent magnet synchronous motor is located;
[0035] Step 42: Design a reward function for each state interval.
[0036] Furthermore, the step 4-1 is specifically as follows:
[0037] When e cc >ε and e la >ε, the speed tracking control process of the permanent magnet synchronous motor is in the response or disturbance range;
[0038] When e cc ≤ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor is in the steady-state range;
[0039] When e cc ≤ε and e la When >ε, the speed tracking control process of the permanent magnet synchronous motor begins to enter the steady-state range;
[0040] When e cc >ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the disturbance or response interval.
[0041] Furthermore, the step 42 is specifically as follows:
[0042] In the response interval, the reward function R re for:
[0043]
[0044] Among them, t re and ere are the duration and speed error of the response interval, respectively, δ is a non-zero number with an arbitrarily small absolute value, α1 and α2 are the standardized coefficients of the performance indicators;
[0045] In the disturbance interval, the reward function R dis for:
[0046] R dis =-r′-α3·|e dis |-α4·t dis (16)
[0047] Among them, e dis and t dis are the velocity error and duration of the disturbance interval, respectively, r′ is a constant, α3 and α4 are the normalization coefficients of the performance indicators;
[0048] In the steady-state range, the reward function R st for:
[0049]
[0050] Among them, α5 is the standardized coefficient of the performance index, e st is the speed error during steady state.
[0051] The beneficial effects of the present invention are:
[0052] The training method based on pre-training and advanced reinforcement learning of the present invention can shorten the training time and improve the efficiency of iterative optimization. In the process of advanced training, a segmented reward mechanism based on an error threshold is used to divide the speed tracking process of the motor into different intervals, and different rewards are used to guide the training of the intelligent agent in different intervals, decouple performance indicators, effectively avoid local optimal parameters, and achieve optimal parameters of the anti-disturbance control controller.
[0053] The present invention optimizes the parameters of the ADRC by designing reinforcement learning, which can achieve better dynamic and steady-state performance. Compared with the existing methods, the method of the present invention does not need to rely on initial parameter selection, does not need to manually adjust the parameters, and has the advantage of high parameter adjustment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flow chart of a method for optimizing parameters of an auto-disturbance rejection controller based on two-stage training reinforcement learning of the present invention;
[0055] Figure 2 A schematic diagram of the structure of the active disturbance rejection controller model established by the present invention;
[0056] Figure 3 This is a training flow chart of advanced reinforcement learning based on pre-training in the present invention;
[0057] Figure 4 A flowchart of the actor-critic algorithm for the deterministic strategy of the present invention;
[0058] Figure 5a Schematic diagram of the error threshold-based segmented reward mechanism designed for the present invention;
[0059] Figure 5b for Figure 5a A magnified view of the switching point region;
[0060] Figure 6 A diagram of an experimental platform for a built-in permanent magnet synchronous motor used in the present invention;
[0061] Figure 7a This is a diagram showing the speed change during the motor startup process of the manually adjusted anti-disturbance controller based on experience;
[0062] Figure 7b The speed change diagram of the motor during the startup process of the ADRC that only performs pre-training reinforcement learning optimization;
[0063] Figure 7c A diagram showing the speed variation during the motor startup process of the active disturbance rejection controller of the present invention;
[0064] Figure 8a This is the speed change diagram of the motor under the switching speed condition of the automatic anti-disturbance controller manually adjusted based on experience;
[0065] Figure 8b The speed change diagram of the motor under the switching speed condition for the ADRC that only performs pre-training reinforcement learning optimization;
[0066] Figure 8c A diagram showing the speed variation of the motor of the active disturbance rejection controller of the present invention under the switching speed condition;
[0067] Figure 9a The motor speed change diagram of the automatic disturbance rejection controller manually adjusted based on experience under sudden load disturbance conditions;
[0068] Figure 9b The motor speed change diagram of the ADRC under sudden load disturbance conditions with only pre-trained reinforcement learning optimization;
[0069] Fig.9c The figure is a diagram showing the motor speed change of the active disturbance rejection controller of the present invention under sudden load disturbance conditions. DETAILED DESCRIPTION
[0070] Specific implementation method 1: Combination Figure 1 and Figure 3The present embodiment describes a method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning, and the method specifically comprises the following steps:
[0071] Step 1: Establish an active disturbance rejection controller model and a permanent magnet synchronous motor model (any permanent magnet synchronous motor model can be used, that is, the method of the present invention is applicable to any permanent magnet synchronous motor model);
[0072] Step 2: Establish a deep reinforcement learning model, use the established permanent magnet synchronous motor model as the interactive environment of the intelligent agent, and use the parameters to be optimized of the ADRC as the action space of the intelligent agent;
[0073] Step 3: Design a reward function based on speed error, use the reward function based on speed error and the actor-critic algorithm to pre-train the deep reinforcement learning model, and stop pre-training when the reward function converges to obtain a pre-trained deep reinforcement learning model;
[0074] Step 4: Use the pre-trained deep reinforcement learning model parameters as initial parameters, design a piecewise reward function based on the error threshold, and use the piecewise reward function to perform advanced training on the deep reinforcement learning model until the piecewise reward function converges, and stop training to obtain an advanced trained deep reinforcement learning model;
[0075] Step 5: Use the deep reinforcement learning model trained in step 4 to output the parameters of the ADRC, and use the output of the ADRC as the input of the speed loop of the permanent magnet synchronous motor model to achieve control of the speed loop.
[0076] The agent is trained using the actor-critic algorithm, which optimizes actions through the environment and state, and then explores and improves the optimized actions based on the rewards obtained from the actions. The agent interacts with the environment continuously until the rewards converge, and the agent can output optimized control parameters based on the current state. Figure 4 As shown in Figure 2, the update process of network parameters during training is:
[0077] The actor-critic algorithm mainly includes four parts: Actor behavior network, Critic comment network, target network and experience replay. The target network includes Actor target network and Critic target network. The Actor behavior network serves as the main policy network and outputs corresponding policy actions based on the current state; the Critic comment network serves as the main value network, which is used to evaluate and optimize the value of the policy. In order to prevent overestimation of the policy, two value networks are used; the Actor target network copies the parameters of the Actor behavior network, and the Critic target network copies the parameters of the Critic comment network, which are used to provide stable target values during training. Experience replay stores the experience data obtained from the interaction of the intelligent agents into the experience pool, and updates the main network parameters by randomly sampling the experience pool. The main network parameters are updated using the gradient descent method, and the target network is updated using the delayed update strategy; the details are as follows:
[0078]
[0079]
[0080]
[0081]
[0082] in, and are the parameters of the main value network and the main strategy network, and are the parameters of the target value network and the target strategy network, respectively, Q m , Q t is the value of the action evaluation by the main value network and the target value network, τ is the delayed update coefficient and is less than 1, which helps to reduce the fluctuation of value and smooth the parameter changes of the network. y is the evaluation value of the target network, and the goal is to minimize the difference between the evaluation of the target network and the main network for the current action, so as to maximize the evaluation value of the value network. t’ and t’+1 are the states at time t′ and time t′+1, respectively, π m and π t Represent the strategies of the main strategy network and the target strategy network respectively, a t’ and They are the actions output by the main policy network and the target policy network respectively, i represents the i-th sample, and N represents the total number of samples.
[0083] Update the parameters of the main network in real time The parameters of the target network are provided by the main network, and the update frequency is lower than that of the main network. Pre-training enables the agent to have a certain generalization ability in basic tasks, and pre-training helps to improve the training speed of the agent, providing a foundation for subsequent advanced training.
[0084] After advanced training, the intelligent agent further improved its response speed and anti-interference ability for motor control, and can output optimized parameters in real time in more complex tasks to achieve precise control of the motor speed loop.
[0085] Specific implementation method 2: Combination Figure 2 The present embodiment is different from the first embodiment in that the ADRC model is a first-order ADRC model, and the ADRC model includes three parts: a tracking differentiator (TD), an extended state observer (ESO), and a nonlinear state inverse feedback (NLSEF).
[0086] The other steps and parameters are the same as those in the first embodiment.
[0087] The function of the tracking differential unit is to arrange the transition process for the reference signal of the system and extract the differential signal to alleviate the contradiction between overshoot and fast response. The extended state observation unit is the core of the ADRC, which is used to analyze the state variables of the system and estimate the total disturbance of the system. The nonlinear state feedback unit analyzes the output of the tracking differential unit and the extended state observation unit, and uses the nonlinear control law to adjust the output error between the two to compensate for the total disturbance.
[0088] The present invention aims at the speed loop control of a permanent magnet synchronous motor and adjusts the parameters in an anti-disturbance controller so that the controller observes the state of the motor system and outputs an optimal control law adaptively.
[0089] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the working process of the tracking differential unit is as follows:
[0090]
[0091] Among them, ω * is the reference speed, z 11 To track the output signal of the differential unit, Indicates z 11 The first-order derivative of , r, a1 and δ1 are hyperparameters, and r, a1 and δ1 are the parameters to be optimized of the tracking differential unit, and fal is a nonlinear function;
[0092]
[0093] Among them, |·| means taking the absolute value, and sign(·) means the sign function.
[0094] The other steps and parameters are the same as those in the first or second embodiment.
[0095] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the working process of the expansion state observation unit is as follows:
[0096]
[0097] Among them, z 21 and z 22 is the state variable output by the extended state observation unit, ω is the observed speed fed back by the permanent magnet synchronous motor model, Indicates z 21 The first derivative of Indicates z 22 The first-order derivative of , u is the output of the ADRC (used to control the speed loop of the permanent magnet synchronous motor), b is the output gain, β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 are all hyperparameters, and β 21 , a 21 , δ 21 , β 22 , a 22 and δ 22 is the parameter to be optimized of the extended state observation unit;
[0098]
[0099]
[0100] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0101] Specific implementation mode 5: This implementation mode is different from any one of the specific implementation modes 1 to 4 in that the working process of the nonlinear state feedback unit is as follows:
[0102]
[0103] Among them, u0 is the intermediate variable of the output parameter, β3, a3 and δ3 are all hyperparameters, and β3, a3 and δ3 are the parameters to be optimized of the nonlinear state feedback unit;
[0104]
[0105] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0106] The value ranges of the filter factor δ and the nonlinear factor a in the tracking differential unit, the extended state observation unit and the nonlinear state feedback unit are all between 0 and 1.
[0107] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the state space of the agent is:
[0108] observation = [e, ω * ,ω,i d ,i q , T E , z 22 ] (12)
[0109] Among them, observation is the state space, e is the error between the reference speed and the observed speed, i d is the d-axis output current, i q is the q-axis output current i q , T E is the output torque.
[0110] The other steps and parameters are the same as those in Specific Implementations 1 to 5.
[0111] The intelligent agent in reinforcement learning outputs the optimized parameters of the ADRC, applies the control law output by the ADRC to the environment, and the environment feeds back the observed state and evaluates the state, giving a reward for the current action.
[0112] e is used to evaluate the speed tracking capability under the current control strategy, i d 、i q With torque T E Used to help the intelligent agent understand the electrical and dynamic characteristics of the motor system, 22 As a disturbance observer in the ADRC, it enables the agent to better understand uncertainties such as load changes and environmental disturbances in the system.
[0113] The action space of the agent is 12 parameters. Adding noise can improve the exploration ability of the agent. The action space can be expressed as:
[0114] action=π(s|θ π )+noise (13)
[0115] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the reward function based on speed error is:
[0116] Rpre =|ω * -ω| (14)
[0117] Among them, R pre is the reward function value based on the speed error.
[0118] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0119] Specific implementation eight: This implementation differs from one of the specific implementations one to seven in that the design is a segmented reward mechanism based on error threshold (SRMET), specifically:
[0120] Step 41: According to the current interactive step speed error e cc , the speed error of the previous interactive step e la and an error threshold ε, to determine the state interval in which the speed tracking control process of the permanent magnet synchronous motor is located;
[0121] Step 42: Design a reward function for each state interval.
[0122] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0123] Specific implementation method 9: This implementation method is different from specific implementation methods 1 to 8 in that the step 41 is specifically as follows:
[0124] When e cc >ε and e la >ε, the speed tracking control process of the permanent magnet synchronous motor is in the response or disturbance range;
[0125] When e cc ≤ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor is in the steady-state range;
[0126] When e cc ≤ε and e la When >ε, the speed tracking control process of the permanent magnet synchronous motor begins to enter the steady-state range;
[0127] When e cc >ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the disturbance or response interval.
[0128] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0129] In the present invention, the value of ε can be set to 1. If the speed error e of the current interactive step is cc From the steady state interval to e cc >ε and e cc >ε′, the value of ε′ is 100, then the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the response interval, and the pointer p is set to 0; if the speed error e of the current interactive step is cc From the steady-state interval to ε′>e cc >ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the disturbance interval, and the pointer p is set to 1. cc >ε and e la When ≥ε, the interval is determined according to the value of pointer p in the most recent interaction process. If pointer p is 0, the current interval is the response interval; if pointer p is 1, the current interval is the disturbance interval.
[0130] Specific implementation method ten: Combination Figure 5a and Figure 5b This embodiment is different from the first to ninth embodiments in that the step 42 is as follows:
[0131] In the response interval, the reward function R re for:
[0132]
[0133] Among them, t re and e re are the duration and speed error of the response interval, respectively, 6 is a non-zero number with an arbitrarily small absolute value, α1 and α2 are the standardized coefficients of the performance index;
[0134] Setting the reward function for the response interval can specifically guide the agent to optimize the motor's rapid response capability;
[0135] In the disturbance interval, the reward function R dis for:
[0136] R dis =-r′-α3·|e dis |-α4·t dis (16)
[0137] Among them, e dis and t dis are the velocity error and duration of the disturbance interval, respectively, r′ is a constant, α3 and α4 are the normalization coefficients of the performance indicators;
[0138] When entering the disturbance zone, the agent is penalized for leaving the steady state, and the agent is trained to improve the anti-disturbance capability of the motor.
[0139] In the steady-state range, the reward function R st for:
[0140]
[0141] Among them, α5 is the standardized coefficient of the performance index, e st is the speed error during steady state.
[0142] Reward R by Design st To encourage the agent to continue to maintain this state.
[0143] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0144] This embodiment sets different reward functions in different state intervals, which helps to optimize the control performance of the motor in a targeted manner, such as response speed, anti-interference ability, etc.
[0145] Experimental Section
[0146] The effectiveness of the present invention is illustrated by experimental results. The trained network parameters are introduced into the permanent magnet synchronous motor system. The permanent magnet synchronous motor experimental platform is as follows: Figure 6 As shown, the parameters of the system are shown in Table 1. In order to verify the improvement of the motor speed loop control performance by the method of the present invention, the performance indicators under the control of the experience-based manual ADRC (method one), the ADRC that only performs pre-training reinforcement learning optimization (method two) and the ADRC that combines pre-training and advanced reinforcement learning optimization methods (the present invention) are compared and analyzed.
[0147] Table 1
[0148]
[0149] A. Startup performance
[0150] Figure 7a The startup waveform of the manually adjusted ADRC based on experience is shown. Figure 7b The startup waveform of the ADRC that only performs pre-training reinforcement learning optimization is shown. Figure 7c The startup waveform of the ADRC combined with pre-training and advanced reinforcement learning optimization method is shown. It can be seen that compared with method 1, the response speed of the control system has been improved to a certain extent, and the overshoot has been significantly reduced, but the method of the present invention has a faster response speed and a smaller overshoot.
[0151] B. Switching speed performance
[0152] Figure 8a The switching speed waveform of the manually adjusted ADRC based on experience is shown. Figure 8bThe switching speed waveform of the ADRC optimized by pre-training reinforcement learning is shown. Figure 8c The switching speed waveform of the ADRC combining pre-training and advanced reinforcement learning optimization method is shown. It can be seen that the method of the present invention still has the best performance among the three in terms of overshoot and response speed.
[0153] C. Performance under load disturbance
[0154] Figure 9a The load disturbance waveform of the manually adjusted ADRC based on experience is shown when a sudden load (8Nm) is applied. Figure 9b The load disturbance waveform of the ADRC optimized by pre-trained reinforcement learning is shown when a sudden load (8Nm) is applied. Fig.9c The load disturbance waveform of the self-disturbance rejection controller combined with pre-training and advanced reinforcement learning optimization method is shown when a sudden load (8Nm) is applied. It can be seen that under external load disturbance, the anti-disturbance ability of method one is the worst, the maximum speed fluctuation of the present invention is the smallest, and the disturbance recovery time is the shortest.
[0155] The above experimental results show that the optimization method of the present invention has excellent control performance.
[0156] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A parameter optimization method for an auto-disturbance rejection controller based on two-stage training reinforcement learning, characterized in that: The method specifically comprises the following steps: Step 1: Establish an active disturbance rejection controller model and a permanent magnet synchronous motor model; Step 2: Establish a deep reinforcement learning model, use the established permanent magnet synchronous motor model as the interactive environment of the intelligent agent, and use the parameters to be optimized of the ADRC as the action space of the intelligent agent; Step 3: Design a reward function based on speed error, use the reward function based on speed error and the actor-critic algorithm to pre-train the deep reinforcement learning model, and stop pre-training when the reward function converges to obtain a pre-trained deep reinforcement learning model; Step 4: Use the pre-trained deep reinforcement learning model parameters as initial parameters, design a piecewise reward function based on the error threshold, and use the piecewise reward function to perform advanced training on the deep reinforcement learning model until the piecewise reward function converges, and stop training to obtain an advanced trained deep reinforcement learning model; Step 5: Use the deep reinforcement learning model trained in step 4 to output the parameters of the ADRC, and use the output of the ADRC as the input of the speed loop of the permanent magnet synchronous motor model to achieve control of the speed loop.
2. According to claim 1, a method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning is characterized in that: The active disturbance rejection controller model is a first-order active disturbance rejection controller model, and the active disturbance rejection controller model includes three parts: a tracking differential unit, an extended state observation unit and a nonlinear state feedback unit.
3. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 2 is characterized in that: The working process of the tracking differential unit is: Among them, ω * is the reference speed, z 11 To track the output signal of the differential unit, Indicates z 11 The first-order derivative of , r, a1 and δ1 are hyperparameters, and r, a1 and δ1 are the parameters to be optimized of the tracking differential unit, and fal is a nonlinear function; Among them, |·| means taking the absolute value, and sign(·) means the sign function.
4. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 3 is characterized in that: The working process of the extended state observation unit is as follows: Among them, z 21 and z 22 is the state variable output by the extended state observation unit, ω is the observed speed fed back by the permanent magnet synchronous motor model, Indicates z 21 The first derivative of Indicates z 22 The first-order derivative of , i is the output of the ADRC, b is the output gain, β 21 ,a 21 ,δ 21 ,β 22 ,a 22 and δ 22 are all hyperparameters, and β 21 ,a 21 ,δ 21 ,β 22 ,a 22 and δ 22 is the parameter to be optimized of the extended state observation unit; 5. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 4 is characterized in that: The working process of the nonlinear state feedback unit is: Among them, u0 is the intermediate variable of the output parameter, β3, a3 and δ3 are all hyperparameters, and β3, a3 and δ3 are the parameters to be optimized of the nonlinear state feedback unit; 6. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 5, characterized in that: The state space of the agent is: observation=[e,ω * ,ω,i d ,i q ,T E ,z 22 ] (12) Among them, observation is the state space, e is the error between the reference velocity and the observed velocity, i d is the d-axis output current, i q is the q-axis output current i q , T E is the output torque.
7. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 6 is characterized in that: The speed error-based reward function is: R pre =|ω * -ω| (14) Among them, R pre is the reward function value based on the speed error.
8. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 7 is characterized in that: The design is based on a piecewise reward function with an error threshold, specifically: Step 41: According to the current interactive step speed error e cc , the speed error of the previous interactive step e la and an error threshold ε, to determine the state interval in which the speed tracking control process of the permanent magnet synchronous motor is located; Step 42: Design a reward function for each state interval.
9. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 8, characterized in that: The step 4-1 is specifically as follows: When e cc >ε and e la >ε, the speed tracking control process of the permanent magnet synchronous motor is in the response or disturbance range; When e cc ≤ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor is in the steady-state range; When e cc ≤ε and e la When >ε, the speed tracking control process of the permanent magnet synchronous motor begins to enter the steady-state range; When e cc >ε and e la When ≤ε, the speed tracking control process of the permanent magnet synchronous motor leaves the steady-state interval and enters the disturbance or response interval.
10. The method for optimizing parameters of an active disturbance rejection controller based on two-stage training reinforcement learning according to claim 9, characterized in that: The step 42 is specifically as follows: In the response interval, the reward function R re for: Among them, t re and e re are the duration and speed error of the response interval, respectively, δ is a non-zero number with an arbitrarily small absolute value, α1 and α2 are the standardized coefficients of the performance indicators; In the disturbance interval, the reward function R dis for: R dis =-r′-α3·|e dis |-α4·t dis (16) Among them, e dis and t dis are the velocity error and duration of the disturbance interval, respectively, r′ is a constant, α3 and α4 are the normalization coefficients of the performance indicators; In the steady-state range, the reward function R st for: Among them, α5 is the standardized coefficient of the performance index, e st is the speed error during steady state.
Citation Information
Patent Citations
Auto-disturbance rejection controller parameter automatic adjustment system for vector control of permanent magnet synchronous motor
CN111211718A
Heavy haul train reinforcement learning control method and system
CN112193280A
Active-disturbance-rejection controller parameter optimization method based on deep reinforcement learning
CN115097736A
Unmanned ship trajectory tracking control method based on Actor-Credit-Advantage network
CN115793455A
Asteroid flexible detector intelligent control method based on deep reinforcement learning SAC algorithm
CN116400589A
Cited By
Comprehensive energy system planning optimization method and system based on deep reinforcement learning
CN119444502A
A self-anti-interference parameter self-optimization control method and system based on reinforcement learning
CN122710579A
