A rl-based three-phase separator parameter setting method

Abstract

The present application relates to the technical field of industrial control, and specifically to a three-phase separator parameter setting method based on RL, which specifically comprises the following steps: 1, constructing a value function and determining the parameters of the value function; 2, constructing a reinforcement learning system and initializing the reinforcement learning system, and selecting a baseline controller; 3, constructing an operation reward function and determining a set reward; 4, iteratively training the reinforcement learning system by means of the operation reward function, the set reward, the baseline controller and the value function until the number of iterations reaches a set requirement, and updating the parameters of the reinforcement learning system; and 5, using the trained reinforcement learning system to estimate the parameters of the three-phase separator. The present application proposes a new artificial intelligence parameter tuning method to improve the system response of the three-phase separator and provides a new idea and method for parameter tuning of the three-phase separator.

Description

Technical Field

[0001] This invention relates to the field of industrial control technology, and in particular to a method for setting parameters of a three-phase separator based on RL. Background Technology

[0002] Three-phase separators are used in the pretreatment of food waste to separate oil, water, and solid residue. Traditional PID control algorithms for three-phase separators require multiple manual adjustments to the control parameters, relying entirely on the engineer's personal experience. Furthermore, they need to be readjusted when materials or operating conditions change, resulting in high technical barriers and time-consuming processes.

[0003] In recent years, artificial intelligence technology has been widely applied in the field of control systems, among which reinforcement learning algorithms are an effective optimization method. Standard reinforcement learning involves a learning agent that interacts with the environment. The agent represents the decision-making mechanism, while the environment typically represents the object. The goal of the agent is to find the optimal policy to optimize long-term cumulative rewards through interaction with the environment, which is typically characterized by Markov decision processes. It has been widely applied in control systems, complex decision-making, signal processing, and other fields, achieving excellent results. Therefore, there is an urgent need for a method to set the parameters of a three-phase separator using reinforcement learning algorithms. Summary of the Invention

[0004] This invention provides a parameter setting method for a three-phase separator based on RL, in order to solve the technical problem that the traditional PID control algorithm for three-phase separators is manually set and relies entirely on the personal experience of engineers.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] This invention provides a method for setting parameters of a three-phase separator based on RL, specifically including the following steps:

[0007] S1. Construct the Q-value function and determine its parameters;

[0008] S2. Construct a reinforcement learning system, which includes an Actor network μ. θ (s k ), Critic network Q(s) k a k w), Actor target network μ θ′ (s k+1 ) and Critic target network Q(s k+1 μ θ′ (s k+1 ), w′), and initialize the four networks, and select the baseline controller.

[0009] S3. Construct the runtime reward function RR(t) and determine the reward settings;

[0010] S4. Using the running reward function RR(t), set the reward and baseline controller. The Q-value function is used to iteratively train the reinforcement learning system until the number of iterations reaches the set requirement. The parameters of the reinforcement learning system are then updated to obtain the trained reinforcement learning system.

[0011] S5. Use the trained reinforcement learning system to predict the parameters of the three-phase separator.

[0012] Furthermore, the Actor network μ θ (s k Used to select an action based on the current state;

[0013] Critic network Q(s) k a k w) is used to evaluate the current Q-value function;

[0014] Actor target network μ θ′ (s k+1 This is used to select the action that maximizes the next Q-value function;

[0015] Critic target network Q(s) k+1 μ θ′ (s k+1 ), w′) is used to evaluate the optimal g-value function for the next state.

[0016] Furthermore, the runtime reward function RR(t) in S3 is as follows:

[0017]

[0018] Where t represents the time step; s represents a certain state; e s This represents the tracking error in state s.

[0019] Furthermore, step S4 specifically includes the following steps:

[0020] S41. Initialize the reward function RR(t) to 0, observe state s, and utilize the Actor network μ. θ (s k Select action a, where a = clip(μ θ (s k )+ò,a low a high); ò represents random noise; the clip(.) function is used to control the elements in an array within a given range, where the upper and lower boundaries of the range to be controlled are given. low a high These are the lower boundary and the upper boundary, respectively.

[0021] S42, The parameters (K) of reinforcement learning systems p , τ I , τ D During iterative training, if the reward function RR(t) is greater than the set reward (i.e., RR(t) > λR), then... bmk , where λ represents the base reward multiple, and λ≥1; R bmk If the baseline reward is indicated, then the parameters in the three-phase separator PID are replaced with the parameters in the baseline controller, and then the process proceeds to S43.

[0022] S43. Determine if the current time step is greater than the set time step T. If yes, end the iterative training and obtain the trained reinforcement learning system; otherwise, use the Actor target network μ. θ′ (s k+1 Simulate the next action and update the running reward function RR(t) corresponding to the next action. Update the state s to the state s′, calculate the reward data r of the running reward function RR(t) corresponding to the next action, update the parameters (s, a, r, s′) to the buffer of the three-phase separator PID, update the parameters of the reinforcement learning system, increment the current time step by 1 and enter S42.

[0023] Furthermore, updating the parameters of the reinforcement learning system in step S43 specifically includes the following steps:

[0024] S431. Randomly sample parameter B from the buffer of the three-phase separator PID, B = {(s, a, r, s′)}, and the number of samples is |B|.

[0025] S432. Calculate the target y for each sample in parameter B using the reward data r obtained in S43.

[0026] S433. Based on the target y obtained in S432, calculate the Critic network Q(s). k a k The gradient of w is calculated, and the difference between the evaluation value and the expected value is minimized using the gradient descent algorithm; the difference is used to update the parameters w of the Critic network by maximizing the cumulative expected return;

[0027] S434, Calculate the Actor network μ θ (s kThe gradient of ) is calculated, and the gradient ascent algorithm is used to maximize the cumulative expected return to update the Actor network μ. θ (s k The parameter θ;

[0028] S435, Regarding the Actor target network μ θ′ (s k+1 ) and Critic target network Q(s k+1 μ θ′ (s k+1 The parameters w′ and θ′ of the policy are updated as follows:

[0029] w′←ρw′+(1-ρ)w, θ′←ρθ′+(1-ρ)θ;

[0030] Where ρ represents the learning rate, and ρ∈(0,1).

[0031] Furthermore, the specific calculation process for the target y in S432 is as follows:

[0032]

[0033] Further, in S433, the Critic network Q(s) is calculated. k a k The gradient of w is expressed by the following formula:

[0034]

[0035] Furthermore, in S434, the Actor network μ is calculated. θ (s k The gradient of ) is expressed by the following formula:

[0036]

[0037] The beneficial effects of this invention are:

[0038] This invention explicitly considers the closed-loop stability of the entire reinforcement learning-based parameter tuning process. A novel scenario tuning framework is proposed, allowing closed-loop operations under selected PID parameters, where the actor and Critic networks are updated once at the end of each training session. To ensure closed-loop stability during tuning, a conservative but stable baseline PID controller is used to initialize training, with the resulting reward serving as the benchmark score. Once the running reward exceeds the benchmark score, the lower-level controller is replaced by the baseline controller as an early correction to prevent instability. Layer normalization is used to standardize the inputs of each layer in the actor and Critic networks to overcome policy saturation at action boundaries, ensuring convergence to the optimum. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the architecture of the present invention;

[0040] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0041] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0043] Reference Figure 1 and Figure 2 This application provides a method for setting parameters of a three-phase separator based on RL, specifically including the following steps:

[0044] S1. Construct the Q-value function and determine its parameters;

[0045] S2. Construct a reinforcement learning system, which includes an Actor network μ. θ (s k ), Critic network Q(s) k a k w), Actor target network μ θ′ (s k+1 ) and Critic target network Q(s k+1 μ θ′ (s k+1 ), w′), and initialize the four networks, and select the baseline controller. Baseline controller These are the optimal parameters based on human experience.

[0046] For the Actor target network μ θ′ (s k+1To encourage the agent to explore the PID parameter space initially, we added Gaussian noise after the tanh function to promote exploration at the start of reinforcement learning training. The output action of the reinforcement learning system is the next set of PID parameters to be deployed to the closed-loop system.

[0047] S3. Construct the runtime reward function RR(t) and determine the reward settings;

[0048] S4. Using the running reward function RR(t), set the reward and baseline controller. The Q-value function is used to iteratively train the reinforcement learning system until the number of iterations reaches the set requirement. The parameters of the reinforcement learning system are then updated to obtain the trained reinforcement learning system.

[0049] S5. Use the trained reinforcement learning system to predict the parameters of the three-phase separator.

[0050] In some embodiments, the Actor network μ θ (s k Used to select an action based on the current state;

[0051] Critic network Q(s) k a k w) is used to evaluate the current Q-value function;

[0052] Actor target network μ θ′ (s k+1 This is used to select the action that maximizes the next Q-value function;

[0053] Critic target network Q(s) k+1 μ θ′ (s k+1 ), w′) is used to evaluate the optimal Q-value function for the next state.

[0054] In some embodiments, the running reward function RR(t) in S3 is specifically as follows:

[0055]

[0056] Where t represents the time step; s represents a certain state; e s This represents the tracking error in state s.

[0057] In some embodiments, S4 specifically includes the following steps:

[0058] S41. Initialize the reward function RR(t) to 0, observe state s, and utilize the Actor network μ. θ (s kSelect action a, a = clip(μ) θ (s k )+ò,a low a high ); where ò represents random noise; the clip() function is used to control the elements in an array within a given range, where the upper and lower boundaries of the range to be controlled are given, a low a high These are the lower and upper bounds, respectively.

[0059] S42, The parameters (K) of reinforcement learning systems p , τ I , τ D During iterative training, if the reward function RR(t) is greater than the set reward (i.e., RR(t) > λR), then... bmk , where λ represents the base reward multiple, and λ≥1; R bmk If the baseline reward is indicated, then the parameters in the three-phase separator (PID) are replaced with the parameters in the baseline controller, and then the process proceeds to S43.

[0060] Specifically, at time t, if the running reward exceeds the baseline reward: RR(t) > λR bmk If λ ≥ 1, it indicates that the performance of the controller on the three-phase separator is even worse than that of the reference controller. This is highly likely to occur due to controller instability. Based on these observations, the supervisor replaced the basic parameters with the baseline parameters. This allows for early correction of the closed-loop response before actual instability occurs. The remaining closed-loop operations will be performed under the baseline controller. Due to the subsequent transients, the final total reward will be significantly worse than the baseline. This final reward will be used as the relevant reward for the deployed PID parameters. The poorer reward value can inform the reinforcement learning system to avoid exploring the vicinity in subsequent trials.

[0061] S43. Determine if the current time step is greater than the set time step T. If yes, end the iterative training and obtain the trained reinforcement learning system; otherwise, use the Actor target network μ. θ′ , (s k+1Simulate the next action and update the corresponding reward function RR(t). Update state s to state s′ and calculate the reward data r of the reward function RR(t) for the next action. Update the parameters (s, a, r, s′) to the buffer of the three-phase separator PID controller to update the parameters of the reinforcement learning system. For clarity, each change in the parameters of the PID controller is considered as one reinforcement learning parameter update. Use k∈{0, 1, ..., K} to represent the k-th update step, where K represents the maximum number of training sets. Define the operation step as a time step in the closed-loop operation, denoted as t∈{0, 1, ..., T}, where T is the set time step.

[0062] Increment the current time step by 1 and proceed to S42.

[0063] In this method, the closed-loop three-phase separator with PID control is considered as the environment. The environment outputs the trajectory of the process variables throughout the entire closed-loop operation. After the k-th closed-loop operation, the process state space is defined as:

[0064]

[0065] Among them, y k This represents the control output data during the k-th training iteration; u represents the control output data at time T1 during the k-th iteration of training; k This represents the control input data for the k-th training iteration; This represents the control input data at time T1 during the k-th training iteration;

[0066] Consider the case where the setpoint is constant. The action space of the reinforcement learning system consists of the PID parameters of the three-phase separator:

[0067] a = [K] p , τ I , τ D ]

[0068] At the end of each loop, the reinforcement learning system receives information from the environment, updates its parameters, and then passes a set of PID parameters to the controller. The closed-loop system then runs the next loop under the new PID controller.

[0069] In some embodiments, updating the parameters of the reinforcement learning system in step S43 specifically includes the following steps:

[0070] S431. Randomly sample parameter B from the buffer of the three-phase separator PID, B = {(s, a, r, s′)}, and the number of samples is |B|.

[0071] S432. Calculate the target y for each sample in parameter B using the reward data r obtained in S43.

[0072] S433. Based on the target y obtained in S432, calculate the Critic network Q(s). k a k The gradient of w is calculated, and the difference between the evaluation value and the expected value is minimized using the gradient descent algorithm; the difference is used to update the parameters w of the Critic network by maximizing the cumulative expected return;

[0073] S434, Calculate the Actor network μ θ (s k The gradient of ) is calculated, and the gradient ascent algorithm is used to maximize the cumulative expected return to update the Actor network μ. θ (s k The parameter θ;

[0074] S435, Regarding the Actor target network μ θ′ (s k+1 ) and Critic target network Q(s k+1 μ θ′ (s k+1 The parameters w′ and θ′ of the policy are updated as follows:

[0075] w′←ρw′+(1-ρ)w, θ′←ρθ′+(1-ρ)θ;

[0076] Where ρ represents the learning rate, and ρ∈(0,1), usually taking the value 0.005;

[0077] In some embodiments, the target y calculation process in S432 is as follows:

[0078]

[0079] In some embodiments, the Critic network Q(s) is calculated in S433. k a k The gradient of w is expressed by the following formula:

[0080]

[0081] In some embodiments, the Actor network μ is calculated in S434. θ (s k The gradient of ) is expressed by the following formula:

[0082]

[0083] This invention explicitly considers the closed-loop stability of the entire reinforcement learning-based parameter tuning process. Specifically, it proposes a novel scenario tuning framework that allows closed-loop operation under selected PID parameters, where the Actor network μ... θ (s k ) and Critic network Q(s k a k The Actor network (μ) is updated once at the end of each training session. To ensure closed-loop stability during the adjustment process, a conservative but stable baseline PID controller is used to initialize training, and the resulting reward is used as the baseline score. Once the running reward exceeds the baseline score, the underlying controller is replaced by the baseline controller as an early correction to prevent instability. Layer normalization is used to normalize the Actor network μ. θ (s k ) and the Critic network Q(s k a k The input of each layer (w) is used to overcome the policy saturation problem at the action boundary to ensure convergence to the optimum.

[0084] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for setting parameters of a three-phase separator based on RL, characterized in that, Specifically, it includes the following steps: S1. Construct the Q-value function and determine its parameters; S2. Construct a reinforcement learning system, which includes an Actor network. Critic Network Actor Target Network and Critic target network The four networks were initialized, and the baseline controller was selected. ; S3. Construct and run the reward function And determine the rewards to be set; S4. Using the runtime reward function Setting rewards, baseline controller The Q-value function is used to iteratively train the reinforcement learning system until the number of iterations reaches the set requirement. The parameters of the reinforcement learning system are then updated to obtain the trained reinforcement learning system. S5. Use the trained reinforcement learning system to predict the parameters of the three-phase separator; The running reward function in S3 Specifically as follows: Where t represents the time step; s represents a certain state; s represents the tracking error in state s; S4 specifically includes the following steps: S41, Running reward function Perform initialization, that is =0, observe state s, and utilize the Actor network. Select Action ,in ; It is random noise; clip The `.` function is used to control the elements in an array within a given range, specifying the upper and lower boundaries of the range to be controlled. , These are the lower boundary and the upper boundary, respectively. S42, Allow the reinforcement learning system to adjust parameters The training process is iterative; if the reward function is run during training... Greater than the set reward, that is ,in This indicates the multiple of the base reward, and ; If the baseline reward is indicated, then the parameters in the three-phase separator PID are replaced with the parameters in the baseline controller, and then the process proceeds to S43. S43. Determine if the current time step is greater than the set time step T. If yes, end the iterative training and obtain the trained reinforcement learning system; otherwise, use the Actor target network. Simulate the next action and update the runtime reward function corresponding to the next action. Update state s to state . And calculate the runtime reward function corresponding to the next action. Reward data r， Update parameters The parameters of the reinforcement learning system are updated in the buffer area of ​​the three-phase separator PID, the current time step is incremented by 1, and then the process enters S42.

2. The method for setting parameters of a three-phase separator according to claim 1, characterized in that, The Actor network Used to select an action based on the current state; Critic Network Used to assess the current Value function; Actor Target Network Used to select the next step to maximize. The action of the value function; Critic target network The best for assessing the next state Value function.

3. The method for setting parameters of a three-phase separator according to claim 2, characterized in that, The steps for updating the parameters of the reinforcement learning system in S43 are as follows: S431. Randomly sample parameters from the buffer of the three-phase separator PID. B , The number of samples is ; S432, Using the reward data obtained in S43 r Calculation parameters B The target y for each sample; S433. Based on the target y obtained in S432, calculate the Critic network. The gradient is calculated, and the difference between the evaluated value and the expected value is minimized using the gradient descent algorithm; the difference is then used to update the parameters of the Critic network by maximizing the cumulative expected return. ; S434, Computing the Actor Network The gradient is calculated, and the gradient ascent algorithm is used to maximize the cumulative expected return to update the Actor network. parameters ; S435, For the Actor target network and Critic target network parameters and current strategy parameters The update is as follows: ； in, Represents the learning rate, and (0,1).

4. The method for setting parameters of a three-phase separator according to claim 3, characterized in that, The specific calculation process for the target y in S432 is as follows: 。 5. The method for setting parameters of a three-phase separator according to claim 4, characterized in that, The Critic network is calculated in S433. The gradient is expressed by the following formula: 。 6. The method for setting parameters of a three-phase separator according to claim 5, characterized in that, The Actor network is calculated in S434. The gradient is expressed by the following formula: 。

Images