PID controller parameter self-tuning method based on reinforcement learning

Abstract

The invention relates to the technical field of controller parameter tuning, and discloses a PID controller parameter self-tuning method based on reinforcement learning, comprising a PID controller parameter self-tuning system, the PID controller parameter self-tuning system comprises an environment module, a reinforcement learning agent module and a controller module, the environment module is in signal connection with the controller module through the reinforcement learning agent module, and the environment module comprises a controlled object, a sensor and an actuator. According to the PID controller parameter self-tuning method based on reinforcement learning, an environment module and a reinforcement learning agent module which are provided with a PID controller parameter self-tuning system are arranged, and a controlled object, a sensor and an actuator of the environment module can be utilized, so that data of the PID controller parameter self-tuning system can be conveniently measured and collected; and by utilizing the state, action and reward of the reinforcement learning agent module, the robustness of the system can be conveniently improved by utilizing reinforcement learning, and the function of improving the parameter self-tuning control performance of the PID controller is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of controller parameter tuning, and specifically to a PID controller parameter self-tuning method based on reinforcement learning. Background Art

[0002] There are many types of motor controllers, among which the widely used one is the motor PID controller. The PID controller is a controller widely used in industrial control, which is used to automatically adjust the output of the system to make it reach the desired set value. The self-tuning of PID controller parameters is a technology used to automatically adjust the proportional, integral, and differential parameters of the PID controller to achieve the best control performance.

[0003] According to the publication number CN103888044A, a parameter self-tuning method for a fuzzy PID controller is disclosed, which includes the following steps: according to the basic domains and fuzzy domains of the deviation signal, deviation change rate signal, proportional parameter, differential parameter, and integral parameter in the control system.

[0004] Adopting the above technical solution, the learning process does not require iteration and has the characteristic of being "extremely" fast. The data required for the learning process is significantly less than that of traditional neural networks. At the same time, compared with the support vector machine that is also suitable for small sample data, the accuracy of the established model has been greatly improved. However, in the above technical solution, the parameter self-tuning method of the fuzzy PID controller cannot use reinforcement learning to improve the robustness of the system, and cannot automatically optimize the parameters of the PID controller, reducing the control performance of the PID controller parameter self-tuning, thereby increasing the steady-state error and prolonging the response time, and cannot be applied to various different controlled objects and control tasks, and requires re-designing the control strategy, thereby reducing the applicability of the PID controller parameter self-tuning method. Summary of the Invention

[0005] The purpose of the present invention is to provide a PID controller parameter self-tuning method based on reinforcement learning to solve the problems raised in the above background art.

[0006] To solve the above technical problems, the present invention provides the following technical solution: A PID controller parameter self-tuning method based on reinforcement learning, including a PID controller parameter self-tuning system. The PID controller parameter self-tuning system includes an environment module, a reinforcement learning agent module, and a controller module. The environment module is signal-connected to the controller module through the reinforcement learning agent module. The environment module includes a controlled object, a sensor, and an actuator. Based on the controlled object, sensor, and actuator of the environment module, data parameters of the system to be controlled, the output of the system, and the input for adjusting the controlled object can be detected.

[0007] The reinforcement learning agent module includes state, action, and reward. Based on the state, action, and reward of the reinforcement learning agent module, data parameter control can be performed on the current state of the system, the change of PID controller parameters, and the evaluation of control effects.

[0008] The controller module includes proportional, integral, and derivative. Based on the proportional, integral, and derivative of the controller module, self-tuning control of PID controller parameters can be conveniently performed.

[0009] Preferably, the controlled object includes a temperature control system or a motor speed control system, and is used to detect the system that needs to be controlled based on the controlled object.

[0010] Through the controlled object of the environment module, the self-tuning method of the PID controller parameters can be conveniently applied to different control systems, improving the applicability of the self-tuning method of the PID controller parameters.

[0011] Preferably, the sensor includes a temperature sensor or a speed sensor, and is used to measure the output of the system based on the sensor.

[0012] By using the sensor of the environment module, the output parameters of the controlled system can be conveniently measured, thus facilitating the use of the self-tuning method of the PID controller parameters.

[0013] Preferably, the actuator includes a heater or a motor, and is used to adjust the input of the controlled object based on the actuator.

[0014] Adjusting the input of the controlled object by using the actuator, thus facilitating the self-tuning of its parameters by using the PID controller.

[0015] Preferably, the state includes measured value, set value, error, and its rate of change, and is used to measure the current state of the system based on the state.

[0016] Measuring the current state of the system by the state of the reinforcement learning agent module can measure the measured value, set value, error, and its rate of change of the system, facilitating the use of the self-tuning system of the PID controller parameters.

[0017] Preferably, the action includes the adjustment of proportional gain Kp, integral time Ti, and derivative time Td, and is used to measure the change of PID controller parameters based on the action.

[0018] By using the action of the reinforcement learning agent module, the proportional gain Kp, integral time Ti, and derivative time Td can be adjusted, thus facilitating the measurement of the change of PID controller parameters.

[0019] Preferably, the reward is a reward function that can be defined according to the performance index of the system, and is used to evaluate the control effect based on the reward.

[0020] By evaluating the performance of the system, the reward value is calculated. The reward function is usually based on the magnitude of the error, the response speed, and the steady-state error index.

[0021] Preferably, the ratio is the proportional gain Kp, and based on the ratio, it is used to adjust the control output according to the current error.

[0022] By comparing the ratio of the controller module with the parameter error of the system initialization PID controller, the control output is adjusted.

[0023] Preferably, the integral is the integral time Ti, which is used to eliminate the steady-state error based on the integral, and the derivative is the derivative time Td, which is used to predict the future error change based on the derivative.

[0024] Through the integral and derivative of the controller module, the steady-state error of the PID controller parameters can be eliminated, and the future error change of the PID controller can be predicted.

[0025] A method for self-tuning the parameters of a PID controller based on reinforcement learning includes the following steps:

[0026] S1: First, initialize the parameters of the PID controller of the system, and determine the environment, state space, action space, and reward function of reinforcement learning;

[0027] S2: Measure the current state of the system through a sensor, including the output of the controlled object, the set value, and the error;

[0028] S3: The reinforcement learning agent selects an action according to the current state, that is, adjusts the parameters of the PID controller. The method of selecting an action can be a policy-based method or a policy gradient-based method;

[0029] S4: Apply the adjusted PID parameters to the controller. The controller calculates the control output according to the new parameters and adjusts the input of the controlled object;

[0030] S5: Evaluate the performance of the system, calculate the reward value. The reward function is usually based on the magnitude of the error, the response speed, and the steady-state error index, and feedback the reward value to the reinforcement learning agent;

[0031] S6: The reinforcement learning agent updates its policy according to the reward value, optimizes the selection of PID parameters. The optimization process can be to update the Q value through Q-learning, or to update the policy parameters through the policy gradient method;

[0032] S7: The system continuously repeats the above steps to gradually optimize the parameters of the PID controller and improve the control performance.

[0033] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0034] First, the present invention is provided with an environment module and a reinforcement learning agent module for the PID controller parameter self-tuning system. It can utilize the controlled object, sensors, and actuators of the environment module. The controlled object is used to detect the system to be controlled, the sensors are used to measure the output of the system, and the actuators are used to adjust the input of the controlled object. Thus, it is convenient to measure and collect the data of the PID controller parameter self-tuning system. By using the state, action, and reward of the reinforcement learning agent module, the current state of the system can be measured through the measured value, set value, error, and its change rate of the state. By the proportional gain, integral time, and derivative time of the action, it is convenient to adjust the proportional gain, integral time, and derivative time of the PID controller to measure the change of the PID controller parameters. And by defining the reward function according to the system performance index through the reward to evaluate the control effect, it is thus possible to conveniently use reinforcement learning to improve the robustness of the system, playing a role in improving the control performance of the PID controller parameter self-tuning.

[0035] Second, the present invention is provided with a controller module for the PID controller parameter self-tuning system. It can utilize the proportional, integral, and derivative of the controller module. The proportional gain is used to adjust the control output according to the current error, the integral time is used to eliminate the steady-state error, and the derivative time is used to predict the future error change. The reinforcement learning agent updates its policy according to the reward value, optimizing the selection of PID parameters. The optimization process can be to update the Q value through Q-learning or to update the policy parameters through the policy gradient method. Finally, the system continuously repeats the above steps to gradually optimize the parameters of the PID controller, improving the control performance, playing a role in further improving the control performance of the PID controller parameter self-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is the system module diagram of the PID controller parameter self-tuning system of the present invention;

[0037] Figure 2 is the flow tree diagram of the PID controller parameter self-tuning method based on reinforcement learning of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0039] Embodiment 1

[0040] Please refer toFigure 1 and Figure 2 , including a PID controller parameter self-tuning system. The PID controller parameter self-tuning system includes an environment module, a reinforcement learning agent module, and a controller module. The environment module is signal-connected to the controller module through the reinforcement learning agent module. The environment module includes a controlled object, a sensor, and an actuator. Based on the controlled object, sensor, and actuator of the environment module, data parameters of the system to be controlled, the output of the system, and the input for adjusting the controlled object can be detected. By setting the environment module of the PID controller parameter self-tuning system, the controlled object, sensor, and actuator of the environment module can be utilized. The controlled object is used to detect the system to be controlled. The sensor is used to measure the output of the system. The actuator is used to adjust the input of the controlled object. Thus, it is convenient to measure and collect the data of the PID controller parameter self-tuning system, which plays a role in improving the control performance of the PID controller parameter self-tuning.

[0041] The controlled object includes a temperature control system or a motor speed control system. Based on the controlled object being used to detect the system to be controlled, through the controlled object of the environment module, it is convenient to apply the PID controller parameter self-tuning method to different control systems, improving the applicability of the PID controller parameter self-tuning method.

[0042] The sensor includes a temperature sensor or a speed sensor. A temperature sensor is a sensor that converts a temperature variable into a standardized output signal that can be transmitted. A speed sensor is a sensor that measures linear velocity and angular velocity. Based on the sensor being used to measure the output of the system, by using the sensor of the environment module, it is convenient to measure the output parameters of the controlled system, thus facilitating the use of the PID controller parameter self-tuning method.

[0043] The actuator includes a heater or a motor. Based on the actuator being used to adjust the input of the controlled object, by using the actuator to adjust the input of the controlled object, it is convenient to utilize the PID controller for self-tuning of its parameters.

[0044] The specific implementation of this embodiment is as follows: When in use, when it is necessary to use the PID controller parameter self-tuning method based on reinforcement learning to self-tune the system parameters using a PID controller, the controlled object, sensor, and actuator of the environment module are utilized. The controlled object is used to detect the system to be controlled. Through the controlled object of the environment module, it is convenient to apply this PID controller parameter self-tuning method to different control systems, improving the applicability of this PID controller parameter self-tuning method. The sensor is used to measure the output of the system. By using the sensor of the environment module, it is convenient to measure the output parameters of the controlled system, thus facilitating the use of this PID controller parameter self-tuning method. The actuator is used to adjust the input of the controlled object. By using the actuator to adjust the input of the controlled object, it is convenient to self-tune its parameters using a PID controller, and further convenient to measure and collect the data of the PID controller parameter self-tuning system.

[0045] Embodiment Two

[0046] Please refer to Figure 1 and Figure 2 , the reinforcement learning agent module includes state, action, and reward. Based on the state, action, and reward of the reinforcement learning agent module, data parameter control can be performed on the current state of the system, the change of PID controller parameters, and the evaluation of control effects. By setting the reinforcement learning agent module of the PID controller parameter self-tuning system and using the state, action, and reward of the reinforcement learning agent module, the current state of the system can be measured through the measured value, set value, error, and their change rates of the state. Through the proportional gain, integral time, and derivative time of the action, it is convenient to adjust the proportional gain, integral time, and derivative time of the PID controller to measure the change of PID controller parameters. And through the reward, a reward function is defined according to the performance index of the system to evaluate the control effect, so that it is convenient to use reinforcement learning to improve the robustness of the system, playing a role in improving the control performance of this PID controller parameter self-tuning.

[0047] The state includes the measured value, set value, error, and their change rates. Based on the state for measuring the current state of the system, the state of the reinforcement learning agent module measures the current state of the system, and can measure the measured value, set value, error, and their change rates of the system, facilitating the use of this PID controller parameter self-tuning system.

[0048] The action includes the adjustment of the proportional gain Kp, integral time Ti, and derivative time Td. Based on the action for measuring the change of PID controller parameters, using the action of the reinforcement learning agent module, the proportional gain Kp, integral time Ti, and derivative time Td can be adjusted, thus facilitating the measurement of the change of PID controller parameters.

[0049] The reward is capable of defining a reward function based on the performance metrics of the system. The reward is used to evaluate the control effect. By evaluating the performance of the system, the reward value is calculated. The reward function is usually based on the magnitude of the error, the response speed, and the steady-state error metric.

[0050] The specific implementation of this embodiment is as follows: When in use, when it is necessary to use the PID controller parameter self-tuning method based on reinforcement learning to self-tune the system parameters using a PID controller, by utilizing the state, action, and reward of the reinforcement learning agent module, the current state of the system can be measured through the measured value, set value, error, and its rate of change of the state. The state of the reinforcement learning agent module measures the current state of the system, and can measure the measured value, set value, error, and its rate of change of the system, facilitating the use of this PID controller parameter self-tuning system. Through the proportional gain, integral time, and derivative time of the action, it is convenient to adjust the proportional gain, integral time, and derivative time of the PID controller to measure the change of the PID controller parameters. By utilizing the action of the reinforcement learning agent module, the proportional gain Kp, integral time Ti, and derivative time Td can be adjusted, thus facilitating the measurement of the change of the PID controller parameters. And through the reward, a reward function is defined based on the performance metrics of the system to evaluate the control effect. By evaluating the performance of the system, the reward value is calculated. The reward function is usually based on the magnitude of the error, the response speed, and the steady-state error metric, thereby being able to conveniently utilize reinforcement learning to improve the robustness of the system.

[0051] Embodiment Three

[0052] Please refer to Figure 1 and Figure 2 ., the controller module includes proportional, integral, and derivative. Based on the proportional, integral, and derivative of the controller module, it is convenient to perform self-tuning control on the PID controller parameters. By setting the controller module of the PID controller parameter self-tuning system, the proportional, integral, and derivative of the controller module can be utilized. The proportional gain is used to adjust the control output according to the current error, the integral time is used to eliminate the steady-state error, and the derivative time is used to predict the future error change. Then the reinforcement learning agent updates its policy according to the reward value, optimizing the selection of PID parameters. The optimization process can be to update the Q value through Q-learning or update the policy parameters through the policy gradient method. Finally, the system continuously repeats the above steps, gradually optimizing the parameters of the PID controller and improving the control performance, playing a role in further improving the control performance of this PID controller parameter self-tuning.

[0053] The proportionality is the proportional gain Kp. Based on the proportionality, it is used to adjust the control output according to the current error. By comparing with the parameter error of the system's initialized PID controller through the proportionality of the controller module, the control output is adjusted.

[0054] The integral is the integral time Ti, which is used to eliminate the steady-state error based on integration. The derivative is the derivative time Td, which is used to predict the future error change based on differentiation. Through the integration and differentiation of the controller module, the steady-state error of the PID controller parameters can be eliminated, and the future error change of the PID controller can be predicted.

[0055] The specific implementation of this embodiment is as follows: When in use, when it is necessary to use the PID controller parameter self-tuning method based on reinforcement learning to self-tune the system parameters using the PID controller, the proportional, integral, and differential of the controller module are used. The proportional gain is used to adjust the control output according to the current error. The proportional of the controller module is compared with the parameter error of the system initialization PID controller, so as to adjust the control output. The integral time is used to eliminate the steady-state error, and the derivative time is used to predict the future error change. The reinforcement learning agent updates its policy according to the reward value. Through the integration and differentiation of the controller module, the steady-state error of the PID controller parameters can be eliminated, and the future error change of the PID controller can be predicted, optimizing the selection of PID parameters. The optimization process can be to update the Q value through Q-learning or update the policy parameters through the policy gradient method. Finally, the system continuously repeats the above steps to gradually optimize the PID controller parameters and improve the control performance.

[0056] A PID controller parameter self-tuning method based on reinforcement learning includes the following steps:

[0057] S1: First, initialize the parameters of the PID controller in the system, and determine the environment, state space, action space, and reward function of reinforcement learning;

[0058] S2: Measure the current state of the system through sensors, including the output, set value, and error of the controlled object;

[0059] S3: The reinforcement learning agent selects an action according to the current state, that is, adjusts the parameters of the PID controller. The method of selecting an action can be a policy-based method or a policy gradient-based method;

[0060] S4: Apply the adjusted PID parameters to the controller, and the controller calculates the control output according to the new parameters and adjusts the input of the controlled object;

[0061] S5: Evaluate the performance of the system, calculate the reward value. The reward function is usually based on the magnitude of the error, response speed, and steady-state error index, and feedback the reward value to the reinforcement learning agent;

[0062] S6: The reinforcement learning agent updates its policy according to the reward value, optimizes the selection of PID parameters. The optimization process can be to update the Q value through Q-learning or update the policy parameters through the policy gradient method;

[0063] S7: The system continuously repeats the above steps to gradually optimize the parameters of the PID controller and improve the control performance.

[0064] The working principle of the present invention is as follows: When in use, when it is necessary to use the PID controller parameter self-tuning method based on reinforcement learning to self-tune the system parameters using a PID controller, the controlled object, sensor, and actuator of the environment module are utilized. The controlled object is used to detect the system to be controlled. Through the controlled object of the environment module, it is convenient to apply this PID controller parameter self-tuning method to different control systems, improving the applicability of this PID controller parameter self-tuning method. The sensor is used to measure the output of the system. By using the sensor of the environment module, it is convenient to measure the output parameters of the controlled system, thus facilitating the use of this PID controller parameter self-tuning method. The actuator is used to adjust the input of the controlled object. By using the actuator to adjust the input of the controlled object, it is convenient to self-tune the parameters of the PID controller for it, and further convenient to measure and collect the data of the PID controller parameter self-tuning system. When in use, when it is necessary to use the PID controller parameter self-tuning method based on reinforcement learning to self-tune the system parameters using a PID controller, the state, action, and reward of the reinforcement learning agent module are utilized. The current state of the system can be measured through the measured value, set value, error, and its change rate of the state. The state of the reinforcement learning agent module measures the current state of the system, and can measure the measured value, set value, error, and its change rate of the system, facilitating the use of this PID controller parameter self-tuning system. Through the proportional gain, integral time, and derivative time of the action, it is convenient to adjust the proportional gain, integral time, and derivative time of the PID controller to measure the change of the PID controller parameters. By using the action of the reinforcement learning agent module, the proportional gain Kp, integral time Ti, and derivative time Td can be adjusted, thus facilitating the measurement of the change of the PID controller parameters. And the reward is used to define the reward function based on the performance index of the system to evaluate the control effect. By evaluating the performance of the system, the reward value is calculated. The reward function is usually based on the magnitude of the error, response speed, and steady-state error index, thus facilitating the use of reinforcement learning to improve the robustness of the system. When in use, when it is necessary to use the PID controller parameter self-tuning method based on reinforcement learning to self-tune the system parameters using a PID controller, the proportional, integral, and derivative of the controller module are utilized. The proportional gain is used to adjust the control output according to the current error. By comparing the proportional of the controller module with the parameter error of the initialized PID controller of the system, the control output is adjusted. The integral time is used to eliminate the steady-state error, and the derivative time is used to predict the future error change. The reinforcement learning agent updates its policy according to the reward value. Through the integral and derivative of the controller module, the steady-state error of the PID controller parameters can be eliminated, and the future error change of the PID controller can be predicted, optimizing the selection of PID parameters. The optimization process can be to update the Q value through Q-learning, or to update the policy parameters through the policy gradient method. Finally, the system continuously repeats the above steps to gradually optimize the parameters of the PID controller.Improve control performance.,

[0065] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or device.

[0066] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A PID controller parameter self-tuning method based on reinforcement learning, comprising a PID controller parameter self-tuning system, characterized in that: The PID controller parameter self-tuning system includes an environment module, a reinforcement learning agent module and a controller module, wherein the environment module is connected to the controller module by signal through the reinforcement learning agent module, and the environment module includes a controlled object, a sensor and an actuator. The controlled object, the sensor and the actuator based on the environment module can perform data parameter detection on the system to be controlled, the output of the system and the input of the controlled object; The reinforcement learning agent module includes a state, an action and a reward, and based on the state, the action and the reward of the reinforcement learning agent module, data parameter control can be performed on the current state of the system, the change of the PID controller parameters and the evaluation control effect; The controller module includes proportion, integration and differentiation. Based on the proportion, integration and differentiation of the controller module, it is convenient to perform self-tuning control on the PID controller parameters.

2. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The controlled object includes a temperature control system or a motor speed control system, and the controlled object is used to detect a system that needs control.

3. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The sensor includes a temperature sensor or a speed sensor, based on which the sensor is used to measure the output of the system.

4. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The actuator includes a heater or a motor, and the actuator is used to adjust an input of a controlled object.

5. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The state includes a measured value, a set value, an error and a change rate thereof, and is used to measure the current state of the system based on the state.

6. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The actions include adjustments to the proportional gain (Kp), integral time (Ti), and derivative time (Td) based on changes to the measured PID controller parameters.

7. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The reward is capable of defining a reward function according to a performance indicator of the system, and the reward is used to evaluate the control effect.

8. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The ratio is a proportional gain (Kp) based on which the control output is adjusted according to the current error.

9. The method for self-tuning PID controller parameters based on reinforcement learning according to claim 1, characterized in that: The integral is an integral time (Ti) based on which a steady-state error is eliminated, and the differential is a differential time (Td) based on which a future error change is predicted.

10. A PID controller parameter self-tuning method based on reinforcement learning according to any one of claims 1 to 9, characterized in that: The specific steps include: S1: First, the system initializes the parameters of the PID controller and determines the environment, state space, action space and reward function of reinforcement learning; S2: Measure the current state of the system through sensors, including the output, set value and error of the controlled object; S3: The reinforcement learning agent selects an action based on the current state, that is, adjusts the parameters of the PID controller. The method of selecting the action can be a policy-based method or a policy gradient-based method; S4: Apply the adjusted PID parameters to the controller, which calculates the control output based on the new parameters and adjusts the input of the controlled object; S5: Evaluate the performance of the system and calculate the reward value. The reward function is usually based on the error size, response speed and steady-state error index, and feeds back the reward value to the reinforcement learning agent. S6: The reinforcement learning agent updates its strategy based on the reward value and optimizes the selection of PID parameters. The optimization process can be to update the Q value through Q-learning or to update the policy parameters through the policy gradient method. S7: The system repeats the above steps continuously, gradually optimizing the parameters of the PID controller and improving the control performance.

Images