PID parameter tuning method and device based on PILCO reinforcement learning transfer PID control

Abstract

The application discloses a PID parameter adjusting method and device based on PILCO reinforcement learning migration PID control, and relates to the technical field of control architecture migration. The method comprises the following steps: obtaining target interaction data when an intelligent agent interacts with a control environment based on a PILCO reinforcement learning algorithm; calculating state errors, error integrals and error derivatives in the target interaction data according to expected states of a controlled object; and applying a recursive least square method combined with a forgetting factor to identify PID control parameters of a PID controller. The application uses the recursive least square method combined with the forgetting factor to perform parameter estimation, and migrates and converts offline learning results of PILCO into PID control, without changing the PILCO reinforcement learning algorithm and the PID controller architecture, so that the problems that the PILCO reinforcement learning algorithm has high calculation complexity and poor expansion in a high-dimensional state space and is difficult to be deployed online are solved.

Description

Technical Field

[0001] This application relates to the field of control architecture transfer technology, and in particular to a PID parameter tuning method and apparatus based on PILCO reinforcement learning transfer of PID control. Background Technology

[0002] PID control remains the mainstream control method in industry due to its simple structure, strong robustness, and low computational complexity. Commonly used PID parameter tuning methods include manual trial and error, model-based optimization, and the Ziegler-Nichols method. Manual trial and error is flexible and intuitive, requiring no mathematical model, but it is time-consuming and relies on experience. It is difficult and inefficient for complex systems (such as nonlinear or large time-delay systems). Model-based optimization designs PID parameters by using pole placement and frequency response matching methods on the system transfer function. This method is theoretically rigorous and can guarantee stability, but real-world systems often have nonlinearities and disturbances, making it difficult to establish an accurate mathematical model. The Ziegler-Nichols method calculates initial parameters using empirical formulas based on critical proportional gain and critical oscillation period, which can quickly obtain preliminary parameters. However, this method requires reaching the critical oscillation state, and the parameters are often too aggressive, which may lead to safety hazards.

[0003] PILCO is a reinforcement learning algorithm based on a probabilistic model. It establishes a probabilistic model of the system through Gaussian processes and optimizes the policy using Bayesian inference. Gaussian processes can explicitly model dynamic noise and parameter perturbations, predict state distributions through probabilistic models, and learn effective policies with only a small number of interactions. It has the advantages of outstanding ability to handle uncertainty and high data utilization efficiency. However, the time complexity of Gaussian processes is O(N³), and the computational cost increases dramatically in high-dimensional state-action spaces. The high computational complexity and poor scalability in high-dimensional state spaces make it difficult to deploy PILCO algorithm online. Summary of the Invention

[0004] The purpose of this application is to provide a PID parameter tuning method and apparatus based on PILCO reinforcement learning transfer PID control, which can be used to transfer offline reinforcement learning to engineering PID control, and realize the control architecture transfer without changing the controller structure.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a PID parameter tuning method based on PILCO reinforcement learning transfer PID control, including:

[0007] The target interaction data is derived from the PILCO reinforcement learning algorithm when the agent interacts with the control environment. The target interaction data refers to the interaction data when the state of the controlled object converges, obtained by the agent with the optimal policy parameters and the interaction between the agent with the current optimal policy parameters and the controlled object. The action data and state data of the controlled object in the target interaction data are respectively denoted as transition action data and transition state data.

[0008] Calculate the state error, error integral, and error derivative at each time step based on the expected state of the controlled object and the transition state data in the target interaction data;

[0009] Based on the state error, error integral, and error derivative at each moment, the PID control parameters of the PID controller corresponding to the controlled object are identified by applying the recursive least squares method combined with the forgetting factor.

[0010] Secondly, this application provides a PID parameter tuning system based on PILCO reinforcement learning transfer PID control, comprising:

[0011] The target interaction data acquisition module is used to obtain target interaction data when the agent interacts with the control environment based on the PILCO reinforcement learning algorithm. The target interaction data refers to the interaction data when the state of the controlled object converges, obtained by the agent with the optimal policy parameters and the interaction between the agent with the current optimal policy parameters and the controlled object. The action data and state data of the controlled object in the target interaction data are respectively denoted as transition action data and transition state data.

[0012] The state error calculation module is used to calculate the state error, error integral, and error derivative at each moment based on the expected state of the controlled object and the transition state data in the target interaction data.

[0013] The parameter identification module is used to identify the PID control parameters of the PID controller corresponding to the controlled object based on the state error, error integral, and error derivative at each time step, using the recursive least squares method combined with the forgetting factor.

[0014] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described PID parameter tuning method based on PILCO reinforcement learning transfer PID control.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described PID parameter tuning method based on PILCO reinforcement learning transfer PID control.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described PID parameter tuning method based on PILCO reinforcement learning transfer PID control.

[0017] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0018] This application provides a PID parameter tuning method and apparatus based on PILCO reinforcement learning transfer PID control. To take into account the advantages of PILCO reinforcement learning algorithm's outstanding ability to handle uncertainty, high data utilization efficiency, and PID control's simple structure and strong robustness, this application proposes a control architecture transfer method suitable for PILCO reinforcement learning algorithm to PID control. During the parameter tuning process, there is no need to change the PILCO reinforcement learning algorithm and PID controller architecture. By combining the recursive least squares method of forgetting factor, the offline learning results of PILCO are transformed into PID control, which solves the problem of high computational complexity and poor scalability of high-dimensional state space of PILCO reinforcement learning algorithm, making it difficult to deploy online. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an application environment diagram of a PID parameter tuning method based on PILCO reinforcement learning transfer PID control in one embodiment of this application;

[0021] Figure 2 A flowchart illustrating a PID parameter tuning method based on PILCO reinforcement learning transfer PID control, provided as an embodiment of this application;

[0022] Figure 3 A schematic diagram illustrating the technical concept of a PID parameter tuning method based on PILCO reinforcement learning transfer PID control provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram comparing the action output of the PILCO reinforcement learning algorithm and the output of the identification PID controller provided in an embodiment of this application;

[0024] Figure 5 A schematic diagram showing the comparison results of inverted pendulum control provided in an embodiment of this application;

[0025] Figure 6A schematic diagram of the functional modules of a PID parameter tuning device based on PILCO reinforcement learning transfer PID control, provided for another embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The PID parameter tuning method based on PILCO reinforcement learning transfer PID control provided in this application embodiment is specifically a control architecture transfer method suitable for transferring PID control from the PILCO reinforcement learning algorithm. It can be applied to, for example... Figure 1 The application environment is shown. The terminal communicates with the server via a network. The data storage system stores the data the server needs to process. The data storage system can be set up independently, integrated into the server, or located in the cloud or on another server. The terminal can send the target interaction data between the agent and the control environment, derived from the PILCO reinforcement learning algorithm, to the server. After receiving the target interaction data, the server calculates the state error, error integral, and error derivative at each time step based on the desired state of the controlled object and the transition state data in the target interaction data. Based on the state error, error integral, and error derivative at each time step, the server uses a recursive least squares method combined with a forgetting factor to identify the PID control parameters of the corresponding PID controller for the controlled object. The server can then feed back the obtained PID control parameters to the terminal. Furthermore, in some embodiments, the PID parameter tuning method based on PILCO reinforcement learning transfer PID control can also be implemented independently by the server or the terminal. For example, the terminal can directly obtain the target interaction data of the control environment of the controlled object and the intelligent agent based on the PILCO reinforcement learning algorithm to perform PID parameter tuning based on PILCO reinforcement learning transfer PID control. Alternatively, the server can obtain the target interaction data of the control environment of the controlled object and the intelligent agent based on the PILCO reinforcement learning algorithm from the data storage system and perform PID parameter tuning based on PILCO reinforcement learning transfer PID control.

[0030] The terminal can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0031] In one exemplary embodiment, such as Figure 2 and Figure 3 As shown, a PID parameter tuning method based on PILCO reinforcement learning transfer PID control is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 The following steps, 101 to 103, are used as an example to illustrate the process of using a server in the example.

[0032] Step 101: Based on the PILCO reinforcement learning algorithm, obtain the target interaction data when the agent interacts with the control environment; the target interaction data refers to the interaction data when the state of the controlled object converges, obtained by the agent corresponding to the optimal policy parameters and the interaction between the agent and the controlled object corresponding to the current optimal policy parameters; the action data and state data of the controlled object in the target interaction data are respectively denoted as the transition action data and the transition state data.

[0033] Step 102: Calculate the state error, error integral, and error derivative at each time step based on the expected state of the controlled object and the migration state data in the target interaction data.

[0034] Step 103: Based on the state error, error integral, and error derivative at each moment, the recursive least squares method combined with the forgetting factor is applied to identify the PID control parameters of the PID controller corresponding to the controlled object.

[0035] To balance the advantages of PILCO reinforcement learning algorithm (its outstanding ability to handle uncertainty, high data utilization efficiency, and PID control's simple structure and strong robustness) by implementing steps 101 to 103 above, this application proposes a control architecture migration method suitable for transferring PILCO reinforcement learning algorithm to PID control. During parameter tuning, neither the PILCO reinforcement learning algorithm nor the PID controller architecture needs to be changed. After the PILCO reinforcement learning algorithm is optimized, parameter estimation is performed using the recursive least squares method combined with the forgetting factor, transforming the offline PILCO learning results into PID control. This solves the problems of high computational complexity and poor high-dimensional state space scalability of the PILCO reinforcement learning algorithm, making online deployment difficult. This application can be used to migrate offline reinforcement learning to engineering PID control, achieving control architecture migration without changing the controller structure, and has the advantages of simple algorithm and easy engineering implementation.

[0036] In another exemplary embodiment of this application, step 101, which involves obtaining target interaction data between the agent and the control environment based on the PILCO reinforcement learning algorithm, specifically includes:

[0037] (1-1) Randomly set the policy parameters of the agent. An agent is a machine learning model that takes the state of the controlled object as input and the actions of the controlled object as output, such as an RBP neural network. The policy parameters of the agent refer to the model parameters (hyperparameters) of the machine learning model.

[0038] (1-2) Interact the agent corresponding to the current policy parameters with the controlled object to obtain the current interaction state data and action data.

[0039] In this application, taking the control of an inverted pendulum as an example, an inverted pendulum control environment is created and initialized. The state variables include the pendulum deflection angle and the pendulum deflection angular velocity, and the action (control variable) is the motor torque. The intelligent agent and the inverted pendulum control environment interact to generate state data and action data.

[0040] (1-3) Construct a training dataset for the Gaussian probability model based on the current interaction state data and action data.

[0041] The input-output data pairs in the training dataset are represented as follows: in, It includes status and control information. The state variable x is the deflection angle of the lever, and the control variable u is the input torque, Δ. t For the difference variable Δ t =x t -x t-1 +ε. ε represents artificially introduced data noise, used to equivalently establish the uncertainty of the data.

[0042] (1-4) Use the current training dataset to train a Gaussian probability model and obtain the trained Gaussian probability model. Figure 3 The Gaussian model in this context is the Gaussian probability model.

[0043] Gaussian process fitting is performed on the input and output data in the training dataset to train a Gaussian probability model.

[0044] (1-5) Calculate the value function using the currently trained Gaussian probability model.

[0045] The trained Gaussian probability model replaces the dynamic model of the controlled object, such as the dynamic model of an inverted pendulum. The value function is calculated based on the trained Gaussian probability model. Where, c(x) t ) is a cost function given by the user, and the successor states are x1,...,xT The probability distribution can be obtained through recursive calculation:

[0046] p(x t )=∫∫p(x t |x t-1 )p(u t-1 |x t-1 )p(x t-1 )dx t-1 du t-1 ,t=1,...,T

[0047] In the formula, p(x) t p(x) represents the probability distribution of the state of the controlled object at time t. t |x t-1 p(u) represents the probability of obtaining the current state given that the state at the previous time step occurred. t-1 |x t-1 p(x) represents the probability of taking the corresponding action given the state at the previous time step. t-1 ) represents the probability distribution of the state of the controlled object at time t-1, and these data are generated by a Gaussian probability model. T represents the total time step of the interaction process.

[0048] (1-6) Calculate the gradient of the value function with respect to the current policy parameters, and update the agent's current policy parameters based on the gradient.

[0049] A gradient-based strategy search method is adopted, which enables... Where Π represents the policy space corresponding to all policy parameters. π refers to the policy network, i.e., the agent. * This refers to the optimal policy network, and the agent corresponding to the optimal policy parameters. x0 represents the initial state. In the scenario of an inverted pendulum, the initial state is that the pendulum is only affected by gravity and is vertically downward. This refers to optimizing the strategy parameters across the parameter space corresponding to all strategy parameters, minimizing the calculated value of the value function under those strategy parameters. For example, the strategy parameters of a PID controller are set as follows: K... p K i K d The strategy space is K p ∈[0,5], K i ∈[0,0.5], K d ∈[0,1], optimize the PID parameters in the policy parameter space to minimize the calculated value of the value function. Since the cost function is typically defined as the deviation between the actual state and the desired state, the better the control performance, the smaller the calculated cost function value.

[0050] Value function The gradient calculation formula for the policy parameter ψ is:

[0051]

[0052] Equivalent to This indicates taking the average. With c(μ) t ) are equivalent. π ψ This refers to the control strategy, i.e., the intelligent agent (RBF neural network).

[0053] When c(x) t Once given, and The mean μ of the current step can be obtained through analysis. t ∑ covariance t Compared with the mean μ in the previous step t-1 ∑ covariance t-1 And it is related to the policy parameter ψ of the current step. and The formula for calculation is:

[0054]

[0055] Where, μ t and μ t-1 Let Σ represent the mean of the Gaussian probability model output, indicating the states at input time t and time t-1, respectively; t and Σ t-1 This represents the variance of the Gaussian probability model output, indicating the states at input time t and time t-1.

[0056] By summing the gradients of all steps, we can obtain the gradient of the value function with respect to the policy parameters. Then, we can use the conjugate gradient method to obtain the optimized policy parameters.

[0057] (1-7) Interact between the agent corresponding to the updated policy parameters and the controlled object to obtain convergence judgment interaction data.

[0058] (1-8) Determine whether the state of the controlled object (Gaussian probability model) corresponding to the current convergence judgment interaction data has converged, and obtain the first judgment result. Figure 3 The "whether it has converged" judgment step refers to the interaction between the agent and the Gaussian probability model, which achieves control convergence under the Gaussian probability model.

[0059] (1-9) If the first judgment result is negative, then set the updated policy parameter to the current policy parameter and return to step (1-6) "Calculate the gradient of the value function with respect to the current policy parameter".

[0060] (1-10) If the first judgment result is yes, then the current optimal policy parameters of the agent are obtained, and the agent corresponding to the current optimal policy parameters interacts with the control environment of the controlled object to obtain extended interaction data.

[0061] The optimal controller parameters under the Gaussian probability model are learned, and the updated controller interacts with the environment, outputting new control and state information to correct the Gaussian regression model. This process of Gaussian modeling, policy updating, and environment interaction is repeated until reinforcement learning ends and the control curve converges (e.g., the inverted pendulum angle is vertically upward). The control and state information generated from the last interaction are then output. In the PILCO reinforcement learning algorithm, the agent's role is equivalent to that of the controller.

[0062] (1-11) Determine whether reinforcement learning has been completed (whether the state of the controlled object (dynamic model) in the current expanded interaction data has converged), and obtain the second judgment result. Here, state convergence refers to the interaction between the agent and the dynamic model, which achieves the convergence of the real dynamic model.

[0063] (1-12) If the second judgment result is negative, the current expanded interaction data is added to the current training dataset, and the process returns to step (1-4) "Train the Gaussian probability model using the current training dataset to obtain the trained Gaussian probability model".

[0064] (1-13) If the second judgment result is yes, then output the extended interaction data of the last interaction, i.e. the target interaction data.

[0065] In the initial stages of simulation, the number of generated random policies and the number of input-output data pairs obtained after interaction with the environment are both limited. Therefore, the Gaussian probability model obtained from Gaussian process modeling differs from the actual dynamic model and is not entirely accurate. Consequently, control convergence under the Gaussian probability model is not equivalent to convergence under the dynamic model. As the optimization iterations proceed and new input-output data pairs are added, the Gaussian probability model will increasingly approximate the dynamic model, ultimately achieving convergence of the agent's control curves under both the Gaussian probability model and the dynamic model.

[0066] In another exemplary embodiment of this application, in step 103, based on the state error, error integral, and error derivative at each moment, the recursive least squares method is applied, combined with the parameter estimation method of the forgetting factor, to identify the PID control parameters of the PID controller corresponding to the controlled object, specifically including:

[0067] (3-1) Set the initial values ​​of the forgetting factor and the covariance matrix, and calculate the gain matrix, update parameter estimation matrix and covariance matrix corresponding to each time step based on the state error, error integral and error derivative at each time step using the recursive least squares method; the forgetting factor is used to calculate the gain matrix.

[0068] The state error, error integral, and error derivative at each moment are expressed as follows:

[0069]

[0070] In the formula, φ t Refers to the state error, error integral, and error derivative at time t; e t This represents the state error at time t; Let represent the integral of the error at time t; e represents the derivative of the error at time t; t-1 ΔT represents the state error at time t-1; ΔT represents the simulation step size.

[0071] The formulas for calculating the gain matrix, the updated parameter estimation matrix, and the covariance matrix are as follows:

[0072]

[0073] In the formula, K t Represents the gain matrix at time t; γ t and γ t-1 Let P represent the update parameter estimation matrices at time t and time t-1, respectively; t and P t-1 Let u represent the covariance matrices at time t and time t-1, respectively; λ represents the forgetting factor; u t This represents the action data of the controlled object at time t in the migration action data.

[0074] (3-2) Identify the PID parameters at each time step based on the gain matrix, updated parameter estimation matrix, and covariance matrix at each time step. in, These represent differential gain, integral gain, and proportional gain, respectively.

[0075] (3-3) The output of the PID controller is determined based on the PID parameters corresponding to each time step.

[0076] (3-4) Calculate the sum of the deviations between the output of the PID controller and the corresponding migration action data at each time point.

[0077] Calculate and identify the deviation between the PID controller output and the actual control quantity (the action data from the last interaction) and in, u tThis represents the action data at time t in the migration action data; φ t Refers to the state error, error integral, and error derivative at time t; γ t This represents the updated parameter estimation matrix at time t; here, T represents the total time step in the last interaction data. The superscript T indicates transpose.

[0078] (3-5) Determine whether the deviation exceeds the preset threshold and obtain the third judgment result.

[0079] (3-6) If the third judgment result is yes, then adjust the initial value of the forgetting factor λ and the covariance matrix P0 = δ. -1 I, δ -1 I is the initial covariance matrix. Then return to the step "Calculate the gain matrix, update parameter estimation matrix and covariance matrix corresponding to each time step based on the state error, error integral and error derivative at each time step using the recursive least squares method".

[0080] The selection rules for the covariance matrix P and the forgetting factor λ are as follows: a smaller covariance matrix P results in slower but more stable convergence, while a larger value leads to faster convergence but may cause oscillations. A smaller forgetting factor allows for faster tracking of PID parameter changes but is more sensitive to noise; a larger forgetting factor results in a smoother PID parameter change curve. Parameter adjustments are made based on the identified PID controller output and the actual control quantity until the deviation and e are minimized. d The threshold requirement is met, and the output identifies the PID parameters that satisfy the constraints.

[0081] (3-7) If the third judgment result is negative, then the PID parameter corresponding to the last moment is taken as the finally identified PID parameter.

[0082] Figure 4 This is a schematic diagram comparing the action output in the last interaction data based on the PILCO reinforcement learning algorithm with the output of the identified PID controller. Figure 5 The comparison results of the inverted pendulum control are shown. Figure 4 and Figure 5 This demonstrates that a PID controller can effectively control an inverted pendulum.

[0083] In this application, the environment is first created and initialized. Gaussian process fitting is performed using the input-output data generated from the initial interaction to train a Gaussian probability model. Then, the optimal controller parameters (referring to the agent's policy parameters) under this Gaussian probability model are obtained through learning. The learned parameters are then applied to the actual or simulated system, iterated multiple times until effective control is achieved. Next, the system error, error integral, and error derivative at each time step are calculated based on the state output results. Finally, the corresponding PID control parameters are identified using a parameter estimation method based on recursive least squares combined with a forgetting factor. More importantly, by changing the initial value of the covariance matrix and the forgetting factor, both fixed PID and time-varying PID parameters can be identified, making it applicable to a wide range of scenarios.

[0084] The method presented in this application is beneficial for solving the control architecture migration problem between reinforcement learning algorithms and PID control. It leverages the advantages of the PILCO reinforcement learning algorithm, such as its outstanding ability to handle uncertainty and high data utilization efficiency, to obtain the control strategy offline (the aforementioned last interaction data). Through recursive least squares method combined with a forgetting factor, the corresponding PID control parameters are identified, achieving effective control of the controlled object. Furthermore, this method does not require changes to the PILCO reinforcement learning algorithm and PID controller architecture during design, making it easily implementable in engineering.

[0085] This application also provides an application scenario in which the above-described PID parameter tuning method based on PILCO reinforcement learning transfer PID control is applied. Specifically, the PID parameter tuning method based on PILCO reinforcement learning transfer PID control provided in this embodiment can be applied to the control scenario of an inverted pendulum, including a data acquisition stage, a PID parameter identification stage, and an inverted pendulum control stage. The data acquisition stage is used to obtain the target interaction data when the control environment of the inverted pendulum interacts with the intelligent agent based on the PILCO reinforcement learning algorithm. The PID parameter identification stage is used to identify the PID parameters of the inverted pendulum PID controller based on the target interaction data. The inverted pendulum control stage is used to assign the identified PID parameters to the actual inverted pendulum PID controller to control the inverted pendulum in the actual scenario. The PID parameter tuning method based on PILCO reinforcement learning transfer PID control provided in this embodiment belongs to the data acquisition stage and the PID parameter identification stage.

[0086] Aircraft are prone to loss of control and crashing when control parameters are poorly set. In such cases, relying on manual parameter tuning based on experience often increases the time and economic cost of tuning. The PID parameter tuning method based on PILCO reinforcement learning transfer of PID control proposed in this application can also be extended to attitude control applications such as quadcopter and fixed-wing aircraft, reducing tuning risks and time costs. Taking a fixed-wing aircraft as the controlled object as an example, the states are pitch angle, roll angle, and yaw angle, and the actions are elevator yaw angle, rudder yaw angle, and aileron yaw angle; taking a quadcopter as an example, the states are pitch angle, roll angle, and yaw angle, and the actions are the propeller speeds of the four rotors.

[0087] Based on the same inventive concept, this application also provides a PID tuning device for implementing the PID tuning method based on PILCO reinforcement learning transfer PID control described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the PID tuning device based on PILCO reinforcement learning transfer PID control provided below can be found in the limitations of the PID tuning method based on PILCO reinforcement learning transfer PID control described above, and will not be repeated here.

[0088] In one exemplary embodiment, such as Figure 6 As shown, a PID parameter tuning device based on PILCO reinforcement learning transfer PID control is provided, comprising:

[0089] The target interaction data acquisition module M1 is used to obtain target interaction data when the agent interacts with the control environment based on the PILCO reinforcement learning algorithm. The target interaction data refers to the interaction data when the state of the controlled object converges, obtained by the interaction between the agent corresponding to the optimal policy parameters and the controlled object. The action data and state data of the controlled object in the target interaction data are respectively denoted as migration action data and migration state data.

[0090] The state error calculation module M2 is used to calculate the state error, error integral, and error derivative at each moment based on the expected state of the controlled object and the migration state data in the target interaction data.

[0091] The parameter identification module M3 is used to identify the PID control parameters of the PID controller corresponding to the controlled object based on the state error, error integral, and error derivative at each time step, using the recursive least squares method combined with the forgetting factor.

[0092] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores PID parameter tuning data for PILCO reinforcement learning transfer PID control. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a PID parameter tuning method for PILCO reinforcement learning transfer PID control.

[0093] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0094] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0095] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0096] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0097] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0098] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A PID parameter tuning method based on PILCO reinforcement learning transfer PID control, characterized in that, include: The target interaction data when the agent interacts with the control environment is obtained based on the PILCO reinforcement learning algorithm. The target interaction data refers to the interaction data obtained when the state of the controlled object converges, which is obtained by the interaction between the agent corresponding to the optimal policy parameters and the controlled object. The action data and state data of the controlled object in the target interaction data are respectively denoted as migration action data and migration state data; Calculate the state error, error integral, and error derivative at each time step based on the expected state of the controlled object and the transition state data in the target interaction data; Based on the state error, error integral, and error derivative at each moment, the PID control parameters of the PID controller corresponding to the controlled object are identified by applying the recursive least squares method combined with the forgetting factor. Specifically, the target interaction data obtained based on the PILCO reinforcement learning algorithm when the agent interacts with the control environment includes: Randomly set the policy parameters of the agent; The agent corresponding to the current policy parameters interacts with the control environment to obtain the current interaction state data and action data; The training dataset for constructing a Gaussian probability model is based on the current interaction state data and action data; Train a Gaussian probability model using the current training dataset to obtain the trained Gaussian probability model; Calculate the value function using the currently trained Gaussian probability model; Calculate the gradient of the value function with respect to the current policy parameters, and update the agent's current policy parameters based on the gradient; The agent corresponding to the updated policy parameters interacts with the control environment to obtain convergence judgment interaction data. Determine whether the state of the controlled object corresponding to the current convergence interaction data has converged, and obtain the first judgment result; If the first judgment result is negative, then set the updated policy parameters to the current policy parameters and return to the step "Calculate the gradient of the value function with respect to the current policy parameters"; If the first judgment result is yes, then the current optimal policy parameters of the agent are obtained, and the agent corresponding to the current optimal policy parameters interacts with the control environment of the controlled object to obtain extended interaction data. Determine whether reinforcement learning is currently complete, and obtain a second judgment result; whether reinforcement learning is currently complete refers to whether the state of the controlled object in the current expanded interaction data has converged. If the second judgment result is negative, the current expanded interaction data is added to the current training dataset, and the process returns to the step "Train the Gaussian probability model using the current training dataset to obtain the trained Gaussian probability model". If the second judgment result is yes, then output the extended interaction data of the last interaction, i.e. the target interaction data; Specifically, based on the state error, error integral, and error derivative at each moment, the recursive least squares method combined with the forgetting factor is applied to identify the PID control parameters of the PID controller corresponding to the controlled object, including: Set initial values ​​for the forgetting factor and covariance matrix, and calculate the gain matrix, update parameter estimation matrix, and covariance matrix for each time step using the recursive least squares method based on the state error, error integral, and error derivative at each time step; the forgetting factor is used to calculate the gain matrix. The PID parameters corresponding to each time step are identified based on the gain matrix, updated parameter estimation matrix, and covariance matrix at each time step. The output of the PID controller is identified based on the PID parameters corresponding to each time step. Calculate the sum of the deviations between the output of the PID controller and the corresponding migration action data at each time step; The system determines whether the deviation exceeds a preset threshold and arrives at a third judgment result. If the third judgment result is yes, then adjust the initial values ​​of the forgetting factor and covariance matrix, and return to the step "Calculate the gain matrix, update the parameter estimation matrix and covariance matrix corresponding to each time step based on the state error, error integral and error derivative at each time step"; If the third judgment result is negative, then the PID parameter corresponding to the last moment is taken as the final identified PID parameter.

2. The PID parameter tuning method based on PILCO reinforcement learning transfer PID control according to claim 1, characterized in that, The state error, error integral, and error derivative at each time step are expressed as follows: In the formula, Refers to the state error, error integral, and error derivative at time t; This represents the state error at time t; Let represent the integral of the error at time t; This represents the error derivative at time t; This represents the state error at time t-1; This indicates the simulation step size.

3. The PID parameter tuning method based on PILCO reinforcement learning transfer PID control according to claim 2, characterized in that, The formulas for calculating the gain matrix, the updated parameter estimation matrix, and the covariance matrix are as follows: In the formula, Let represent the gain matrix at time t; and Let represent the update parameter estimation matrices at time t and time t-1, respectively; and Let represent the covariance matrices at time t and time t-1, respectively; Indicates the forgetting factor; This represents the action data of the controlled object at time t in the migration action data.

4. The PID parameter tuning method based on PILCO reinforcement learning transfer PID control according to claim 3, characterized in that, The formula for calculating the sum of deviations is: In the formula, Indicates the deviation and sum; This represents the action data at time t in the migration action data; Refers to the state error, error integral, and error derivative at time t; Let represent the update parameter estimation matrix at time t; T represents the total time step in the last interaction data.

5. A PID parameter tuning system based on PILCO reinforcement learning transfer PID control, characterized in that, include: The target interaction data acquisition module is used to obtain target interaction data when the agent interacts with the control environment based on the PILCO reinforcement learning algorithm. The target interaction data refers to the interaction data obtained when the state of the agent corresponding to the agent with the optimal policy parameters is converged and the agent with the controlled object with the current optimal policy parameters is at its optimal state. The action data and state data of the controlled object in the target interaction data are respectively denoted as migration action data and migration state data; Specifically, the target interaction data obtained based on the PILCO reinforcement learning algorithm when the agent interacts with the control environment includes: Randomly set the policy parameters of the agent; The agent corresponding to the current policy parameters interacts with the control environment to obtain the current interaction state data and action data; The training dataset for constructing a Gaussian probability model is based on the current interaction state data and action data; Train a Gaussian probability model using the current training dataset to obtain the trained Gaussian probability model; Calculate the value function using the currently trained Gaussian probability model; Calculate the gradient of the value function with respect to the current policy parameters, and update the agent's current policy parameters based on the gradient; The agent corresponding to the updated policy parameters interacts with the control environment to obtain convergence judgment interaction data. Determine whether the state of the controlled object corresponding to the current convergence interaction data has converged, and obtain the first judgment result; If the first judgment result is negative, then set the updated policy parameters to the current policy parameters and return to the step "Calculate the gradient of the value function with respect to the current policy parameters"; If the first judgment result is yes, then the current optimal policy parameters of the agent are obtained, and the agent corresponding to the current optimal policy parameters interacts with the control environment of the controlled object to obtain extended interaction data. Determine whether reinforcement learning is currently complete, and obtain a second judgment result; whether reinforcement learning is currently complete refers to whether the state of the controlled object in the current expanded interaction data has converged. If the second judgment result is negative, the current expanded interaction data is added to the current training dataset, and the process returns to the step "Train the Gaussian probability model using the current training dataset to obtain the trained Gaussian probability model". If the second judgment result is yes, then output the extended interaction data of the last interaction, i.e. the target interaction data; The state error calculation module is used to calculate the state error, error integral, and error derivative at each moment based on the expected state of the controlled object and the transition state data in the target interaction data. The parameter identification module is used to identify the PID control parameters of the PID controller corresponding to the controlled object based on the state error, error integral, and error derivative at each moment, by applying the recursive least squares method combined with the forgetting factor. Specifically, based on the state error, error integral, and error derivative at each moment, the recursive least squares method combined with the forgetting factor is applied to identify the PID control parameters of the PID controller corresponding to the controlled object, including: Set initial values ​​for the forgetting factor and covariance matrix, and calculate the gain matrix, update parameter estimation matrix, and covariance matrix for each time step using the recursive least squares method based on the state error, error integral, and error derivative at each time step; the forgetting factor is used to calculate the gain matrix. The PID parameters corresponding to each time step are identified based on the gain matrix, updated parameter estimation matrix, and covariance matrix at each time step. The output of the PID controller is identified based on the PID parameters corresponding to each time step. Calculate the sum of the deviations between the output of the PID controller and the corresponding migration action data at each time step; The system determines whether the deviation exceeds a preset threshold and arrives at a third judgment result. If the third judgment result is yes, then adjust the initial values ​​of the forgetting factor and covariance matrix, and return to the step "Calculate the gain matrix, update the parameter estimation matrix and covariance matrix corresponding to each time step based on the state error, error integral and error derivative at each time step"; If the third judgment result is negative, then the PID parameter corresponding to the last moment is taken as the final identified PID parameter.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the PID parameter tuning method for PILCO reinforcement learning transfer PID control as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the PID parameter tuning method based on PILCO reinforcement learning transfer PID control as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the PID parameter tuning method based on PILCO reinforcement learning transfer PID control as described in any one of claims 1-4.

Images