PID dynamic setting motor control system based on DDPG reinforcement learning

Abstract

The application provides a PID dynamic setting motor control system based on DDPG reinforcement learning, which comprises a DDPG controller and a PID controller; the input of the DDPG controller is a target value, an actual value and a deviation value, wherein the target value is given externally, the actual value is fed back by the motor, and the deviation value is the difference between the target value and the actual value; the DDPG controller outputs an action signal to the PID controller; the PID controller is provided with two signal input ends, which respectively input the action signal and the deviation value; and the output end of the PID controller is connected with the motor to control the operation of the motor. The DDPG-DIP control system can dynamically set the action parameters of the PID controller by the DDPG controller, so that the system converges to the target as soon as possible and the control error between the actual value of the motor and the target value is quickly eliminated.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of motor PID control, and particularly relates to a PID dynamic setting motor control system based on DDPG reinforcement learning. BACKGROUND

[0002] Brush motor has the characteristics of large torque, simple speed control, and excellent low-speed torque performance, and thus has an irreplaceable position in many application scenarios. In the control of the output torque, speed and position of a brush DC motor, stability is an important indicator for measuring the control effect. In many use cases, the control of motor speed and angle not only has accuracy requirements, but also requires the above control to quickly converge under various external disturbances, so that the motor speed and angle always match the target instruction, and the deviation or fluctuation needs to be minimized.

[0003] The output control of a brush DC motor usually adopts a feedback plus error correction algorithm, and common error correction algorithms include PID control, fuzzy control, neural network control, etc. However, these control algorithms often need parameter setting, rule optimization or artificial training to optimize the control performance in terms of accuracy and dynamics. However, the above optimization requires high technical ability and experience of the technicians. At the same time, for a conventional PID control system, the parameters Kp, Ki and Kd in the system are set as constants, and thus must be compromised when setting to take into account the positive and negative effects, which may cause a significant decline in control performance when the actual operating conditions of the motor exceed the preset range during optimization. SUMMARY

[0004] In view of the deficiencies in the prior art, the application provides a PID dynamic setting motor control system based on DDPG reinforcement learning (referred to as DDPG-PID control system), which solves the optimization problem of the PID control system by dynamically setting the PID system parameters according to the operating conditions.

[0005] The application achieves the above technical purpose through the following technical means.

[0006] A PID dynamic setting motor control system based on DDPG reinforcement learning, comprising a DDPG controller and a PID controller, the input of the DDPG controller being a target value, an actual value and an error value, the DDPG controller outputting an action signal to the PID controller, the PID controller being provided with two signal input ends for inputting the action signal and the error value respectively, the output end of the PID controller being connected with a motor to control the operation of the motor, wherein the target value is given by an external source, the actual value is fed back by the motor, and the error value is the difference between the target value and the actual value.

[0007] Further, the DDPG controller comprises an intelligent module, an observation module, a reward module and a termination module, the observation module, the reward module and the termination module are respectively used for generating observation signals, reward signals and termination signals and outputting to the intelligent module, and the intelligent module is used for generating action signals and outputting to the PID controller.

[0008] Further, the observation module is provided with two input ends for inputting actual values and deviation values respectively, and the observation signal comprises three components, namely actual values, deviation values and accumulated values of the deviation.

[0009] Further, the accumulated value is cleared and restarted to accumulate at the beginning of each round of training of the deep learning neural network of the intelligent module or when the target value input by the DDPG controller is changed.

[0010] Further, the reward module is provided with three signal input ends for inputting deviation values, target values and termination signals respectively, and the generation process of the reward signal comprises three steps.

[0011] Step 1: Calculate the deviation range according to the deviation value and the target value, give a reward when the deviation range <1%, and the reward value is 10; give a penalty when the deviation range ≥1%, and the penalty value is -1;

[0012] Step 2: Give a penalty when the termination signal is "true", and the penalty value is -100;

[0013] Step 3: Add the results of Step 1 and Step 2 to obtain the reward signal.

[0014] Further, the input of the termination module is the actual value, and the termination module outputs the corresponding termination signal according to whether the actual value is out of limit, wherein when the actual value is out of limit, the termination signal is "true", and when the actual value is not out of limit, the termination signal is "false".

[0015] Further, the intelligent module is configured with an Actor network and a Critic network.

[0016] Further, the Actor network comprises an input layer, a fully connected layer A, a linear rectifier function layer and a fully connected layer B from top to bottom, wherein the input size of the input layer is 3, the output size of the fully connected layer A is 10, the output size of the fully connected layer B is 3, and the tanh activation function is adopted in the linear rectifier function layer.

[0017] Further, the Critic network comprises a state subgraph, an action subgraph and a common subgraph, the state subgraph comprises in sequence from top to bottom an input layer with an input size of 3, a fully connected layer with an output size of 25, a linear rectifier function layer and a fully connected layer with an output size of 50, the action subgraph comprises in sequence from top to bottom an input layer with an input size of 3 and a fully connected layer with an output size of 50, and the common subgraph interfaces the outputs of the state subgraph and the action subgraph, the common subgraph comprises in sequence from top to bottom an addition layer, a linear rectifier function layer and a fully connected layer.

[0018] Further, the linear rectifier function layer in the state subgraph adopts a leakyRelu activation function, and the linear rectifier function layer in the common subgraph adopts a Relu activation function.

[0019] The present application has the following beneficial effects:

[0020] (1) The present application provides a DDPG-PID control system, which can be used for brush DC motor control, wherein the DDPG controller is used to dynamically set the action parameters, that is, Kp, Ki and Kd can dynamically change with the system state, wherein Kp and Ki are as large as possible when deviating from the target, and Kd is as small as possible, so that the system converges to the target as soon as possible, and the control error between the actual value and the target value of the motor is quickly eliminated; and after approaching the target, Kp and Ki are reduced in time, and Kd is increased, so as to avoid overshoot and enhance the stability of the system. The present application effectively improves the control effect of the PID controller based on the above-mentioned mode. In addition, due to the differences in the final application scenarios and driving loads of each DC brush motor, and the characteristics of the motor itself, such as nonlinearity, time-varying uncertainty and inertia hysteresis, the traditional PID control system is difficult to meet the requirements of precision control. The DDPG-PID control system of the present application can perform targeted reinforcement learning training according to each different application scenario, and therefore has strong environmental adaptability.

[0021] (2) In a deep neural network control system with supervised learning, the training process usually requires a lot of human intervention. The DDPG-PID control system of the present application uses reinforcement learning technology, and during training, the observation signal of the motor running state and the reward signal for judging the goodness of the output action are obtained, and self-trial and exploration are performed, so that the trial and exploration process does not require human intervention, and the labor cost of the training process can be greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Fig. 1 is a structure diagram of the DDPG-PID motor control system of the present application;

[0023] Figure 2 Fig. 2 is a structure diagram of the Actor network in the DDPG controller of the present application;

[0024] Figure 3 This is a diagram of the Critic network structure in the DDPG controller of this invention;

[0025] Figure 4 This is a diagram showing the training and testing results of the DDPG-PID controller system of the present invention;

[0026] Figure 5 The graph shows the comparative training test results using different observation signals;

[0027] Figure 6 The graph shows the comparative training and testing results using different network structures. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein similar or identical reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] I. Plan

[0030] like Figure 1 The DDPG-PID control system shown includes a DDPG controller and a PID controller. The DDPG controller comprises an intelligent module, an observation module, a reward module, and a termination module. "DDPG" stands for Deep Deterministic Policy Gradient, a type of reinforcement learning algorithm.

[0031] The control quantities of a DC motor currently include its position, speed, and current. Therefore, the input of the control system of this invention corresponds to the externally given target values ​​of motor position, motor speed, or motor current (hereinafter referred to as target values). Simultaneously, the actual measured values ​​of motor position, motor speed, or motor current (hereinafter referred to as actual values) are input to the control system as feedback signals. These actual values ​​are acquired and fed back by the motor and its drive circuit. The difference between the target value and the actual value is the deviation value (hereinafter referred to as deviation value). Therefore, the input signals of the control system of this invention include three quantities: target value, actual value, and deviation value. For example, when the control target is the motor current magnitude, the input consists of the target current value, the actual current value, and the current deviation value.

[0032] The observation module is provided with two signal input ends for inputting actual value and deviation value respectively, and generates corresponding observation signals according to the input signals and outputs the observation signals to the intelligent module. The observation module firstly performs accumulation operation on the deviation value, and based on multiplexing technology, the actual value, the deviation value and the accumulated value of the deviation are combined into one observation signal, that is, the observation signal contains the actual value, the deviation value and the accumulated value of the deviation.

[0033] The accumulation operation on the deviation value can be expressed as D t = D t-1 + P t , wherein D t is the accumulated value of the deviation at the tth time, D t-1 is the accumulated value of the deviation at the (t-1)th time, and P t is the input deviation value at the tth time. The observation module is provided with a corresponding storage unit for storing and recording the accumulated value of the deviation D t-1 . The initial accumulated value of the deviation D0 is 0; wherein during the training of the deep learning neural network, the accumulated value recorded by the storage unit is cleared at the beginning of each round of training; and during the actual application after the training of the deep learning neural network is completed, the accumulated value recorded by the storage unit is cleared and then re-accumulated according to whether the target value is changed, wherein the accumulated value is always accumulated when the target value is unchanged, and the accumulated value is cleared and then re-accumulated when the target value is changed.

[0034] The reward module is provided with three signal input ends for inputting deviation value, target value and termination signal respectively, and generates corresponding reward signals according to the three input signals and outputs the reward signals to the intelligent module. The specific generation logic of the reward signal is as follows:

[0035] 1) According to the comparison between the deviation value and the target value, when the deviation range is <1%, a reward is given, and the reward value is 10; when the deviation range is ≥1%, a punishment is given, and the punishment value is -1.

[0036] 2) When the termination signal is "true", a punishment is given, and the punishment value is -100.

[0037] 3) The results of 1) and 2) are added to obtain the reward signal.

[0038] The input of the termination module is the actual value, and the termination module outputs corresponding termination signals according to the actual value, and the termination signals are sent to the reward module and the intelligent module respectively. The termination signal has two values of "true" and "false", which are selected according to whether the actual value is out of limit, wherein when the actual value exceeds the limit (set by man), for example, the measured motor speed exceeds the upper limit of the motor speed or is lower than the lower limit of the motor speed, the termination signal is "true", and vice versa, when the actual value is within the limit, the termination signal is "false".

[0039] The intelligent module generates corresponding action signals for the PID controller based on the input observation signal, reward signal, and termination signal. These action signals are the three parameters of the PID controller: the proportional gain Kp, the integral time constant Ki, and the derivative time constant Kd. Of course, before being used in a production environment, the intelligent module needs to be trained using a training set.

[0040] The intelligent module is equipped with a Critic network and an Actor network; for details, please refer to the 2016 ICLR (International Conference on Learning Representations) paper "Continuous Control With Deep Reinforcement Learning".

[0041] like Figure 2 The Actor network shown consists of, from top to bottom, an input layer (input size 3), a fully connected layer (output size 10), a linear rectified function layer (using the tanh activation function), and another fully connected layer (output size 3). The network's input is a three-element vector consisting of the actual value, the bias value, and the accumulated bias value. Its outputs are action signals Kp, Ki, and Kd.

[0042] like Figure 3 The Critic network shown comprises three subgraphs: a state subgraph, an action subgraph, and a common subgraph. The state subgraph, from top to bottom, consists of an input layer (input size 3), a fully connected layer (output size 25), a linear rectified function layer (using the Leaky ReLU activation function), and a fully connected layer (output size 50). The action subgraph, from top to bottom, consists of an input layer (input size 3) and a fully connected layer (output size 50). The common subgraph consists of three neural network layers: an additive layer, a linear rectified function layer (using the ReLU activation function), and a fully connected layer (output size 1). The state subgraph receives the 3-element vector (consisting of the actual value, the deviation value, and the accumulated deviation) from the observation signal generation module. The action subgraph receives the 3-element vector (consisting of Kp, Ki, and Kd) from the Actor network. The additive layer receives the outputs of the state and action subgraphs, and the common subgraph outputs the value evaluation Q for the action. The value evaluation Q is used to input the corresponding loss function to optimize the neural network; the setting of the loss function in the intelligent module and how to optimize the network are all existing technologies, which can be found in "Continuous Control With Deep Reinforcement Learning", so they will not be elaborated here.

[0043] The PID controller is a prior art PID controller, which is provided with two signal input ends for inputting action signals and deviation values respectively. Compared with the prior art PID controller in which the parameters Kp, Ki and Kd are fixed values, the parameters of the PID controller in the application are dynamically adjusted by the DDPG controller, and on the basis of the given Kp, Ki and Kd parameters, the PID controller generates a corresponding motor control strategy according to the input deviation value, and finally outputs to the motor (and its driving circuit) in the form of an occupancy signal, so as to realize precise control of the motor.

[0044] II. Test

[0045] The DDPG-PID control system is trained by building a training environment simulating a brush DC motor. Taking 200 steps of training per round as an example, according to the reward generation logic of the reward module, the maximum reward value of each step is 10, and the reward value obtained in each round of training is the sum of the reward values obtained in each step in the round, so the maximum reward value of each round of training is 2000; in order to reduce the influence of the action output noise of the DDPG controller, the average reward value is taken every 20 rounds as an evaluation index. In addition, the critic network of the current network part in the intelligent module also gives a value evaluation Q at each step in the training, and the value obtained by the control system in each round is the sum of the values obtained in each step in the round, and the value obtained in each round in the training is also an important evaluation index.

[0046] As Figure 4 The training test results of the DDPG-PID control system of the application are shown in the figures, and as shown in the figures, after about 800 training rounds, the average reward value obtained by the system can be stably close to 2000, that is, it can reach 99% of the maximum reward value, meeting the training requirements.

[0047] III. Comparative Example

[0048] As Figure 5 and Figure 6 The training test results of the control system in the comparative example are shown in the figures, wherein Figure 5 The control system used in the comparative example has three observation signals selected as actual values, deviation values and derivatives of the deviation, and after more than 900 training rounds, the reward value and the value obtained by the control system still do not show a convergence trend; Figure 6 The control system used in the comparative example has a linear rectifier function layer in the state subgraph of the Critic network in the intelligent module, which uses Sigmoid as the activation function, and also after more than 900 training rounds, the reward value and the value obtained by the control system still do not show a convergence trend; thus, the two control systems of the comparative example cannot meet the training requirements.

[0049] Since there are many observation signals and corresponding mathematical processing methods that can be adopted, and there are many structures and activation functions of commonly used artificial neural networks, and even the same neural network has great differences in depth and width, which will cause great differences in function fitting effect. Therefore, there are many specific setting schemes under the framework of DDPG-PID, and the comparative tests listed in Figure 5 、 Figure 6 The test results show that most of the schemes cannot meet the control requirements, and the related test results show that either the convergence cannot be achieved at all, or even if the convergence can be achieved, the final reward value is only 30% to 80% of the maximum reward value, so the control requirements of rapidity, high precision and low overshoot cannot be met.

[0050] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0051] The present application is not limited to the above-mentioned embodiments, and any obvious improvements, replacements or modifications made by those skilled in the art without departing from the essential content of the present application shall fall within the protection scope of the present application.

Claims

1. A PID dynamic setting motor control system based on DDPG reinforcement learning, characterized in that: The DDPG controller and the PID controller are included, the input of the DDPG controller is the target value, the actual value and the deviation value, the DDPG controller outputs the action signal to the PID controller, the PID controller is provided with two signal input ends, the action signal and the deviation value are input respectively, the output end of the PID controller is connected with the motor, and the motor is controlled to run;The target value is given by the outside, the actual value is fed back by the motor, and the deviation value is the difference between the target value and the actual value; The DDPG controller includes an intelligent module, an observation module, a reward module and a termination module, the observation module, the reward module and the termination module are used for generating observation signals, reward signals and termination signals respectively and outputting to the intelligent module, and the intelligent module is used for generating an action signal and outputting to the PID controller; The input of the termination module is the actual value, and the termination module outputs the corresponding termination signal according to whether the actual value is out of limit, wherein when the actual value is out of limit, the termination signal is "true", and when the actual value is not out of limit, the termination signal is "false"; The reward module is provided with three signal input ends, and the deviation value, the target value and the termination signal are input respectively, and the generation process of the reward signal includes three steps: Step 1, the deviation range is calculated according to the deviation value and the target value, when the deviation range is <1%, the reward is given, and the reward value is 10;When the deviation range is ≥1%, the punishment is given, and the punishment value is-1; Step 2, when the termination signal is "true", the punishment is given, and the punishment value is-100; Step 3, the results of step 1 and step 2 are added to obtain the reward signal; The intelligent module is configured with an Actor network and a Critic network; The Actor network includes an input layer, a full connection layer A, a linear rectifier function layer and a full connection layer B from top to bottom, wherein the input size of the input layer is 3, the output size of the full connection layer A is 10, the output size of the full connection layer B is 3, and the tanh activation function is used in the linear rectifier function layer; The Critic network includes a state subgraph, an action subgraph and a common subgraph, the state subgraph includes an input layer with an input size of 3, a full connection layer with an output size of 25, a linear rectifier function layer and a full connection layer with an output size of 50 from top to bottom, the action subgraph includes an input layer with an input size of 3 and a full connection layer with an output size of 50 from top to bottom, and the common subgraph connects the outputs of the state subgraph and the action subgraph, and the common subgraph includes an addition layer, a linear rectifier function layer and a full connection layer from top to bottom;The leakyRelu activation function is used in the linear rectifier function layer in the state subgraph, and the Relu activation function is used in the linear rectifier function layer in the common subgraph.

2. The control system of claim 1, wherein: The observation module is provided with two input ends, and the actual value and the deviation value are input respectively, and the observation signal includes three components, which are the actual value, the deviation value and the accumulated value of the deviation.

3. The control system of claim 2, wherein: When the deep learning neural network of the intelligent module starts each round of training, or when the target value input by the DDPG controller is changed, the accumulated value is cleared and starts to accumulate again.

Images