An intelligent valve control method based on double-delay deep deterministic policy gradient

Abstract

The application relates to an intelligent valve control method based on a double-delay deep deterministic policy gradient. The method comprises the following steps: first, establishing a regulating valve mathematical model, determining a valve transfer function, and building a valve simulation environment; then, based on the valve simulation environment, a double-delay deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model; finally, PID valve control is realized based on the optimized and adjusted parameters. Compared with the traditional method, the PID controller optimized and adjusted by the TD3 algorithm significantly reduces the steady-state error of the system and significantly improves the accuracy of target tracking. Through adaptive adjustment of reinforcement learning, the system can more accurately match the reference target, reduce error accumulation, and improve the stability of precision control. Compared with manual adjustment, the TD3 algorithm can continuously optimize the controller parameters according to the real-time feedback of the system, so that the control system has stronger adaptability and self-adjusting ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of valve control, and particularly relates to an intelligent valve control method based on a double-delay deep deterministic policy gradient. BACKGROUND

[0002] As a key component in process control systems, the main function of the valve positioner is to accurately adjust the opening of the valve to ensure that the process parameters are stable within the set range. In traditional valve control, the PID (Proportion-Integral-Derivative) controller is widely used due to its simple algorithm and fast response. However, the PID controller still has the following shortcomings in actual industrial applications. First, the parameter adjustment is complex. The parameters of the traditional PID controller usually rely on experience adjustment or trial-and-error method, which is difficult to achieve global optimization in complex and variable working conditions. Second, the robustness is insufficient. Nonlinearity, time-varying and uncertainty are common in valve systems, making it difficult for the PID controller with fixed parameters to adapt to fluctuations in working conditions, which can easily lead to system overshoot or oscillation. Third, the dynamic response is poor. In scenarios with high precision and high response speed requirements, the traditional PID control is difficult to quickly respond to load changes, affecting the overall efficiency of the system.

[0003] To overcome the above problems, in recent years, intelligent control technology based on data driving has gradually become a research hotspot. In particular, reinforcement learning (RL) as a technology that can optimize control strategies through self-learning in dynamic and complex environments shows significant potential. In the prior art, there have been studies attempting to apply reinforcement learning to the field of industrial control, but there are still the following problems. First, the algorithm lacks sufficient combination with the physical model. Most studies only verify the effectiveness of the reinforcement learning algorithm in an offline simulation environment, without constructing a detailed mathematical model for a specific physical system, resulting in a disconnection between the algorithm and the actual working conditions. Second, the learning efficiency is low. Traditional reinforcement learning algorithms such as Q-learning and DDPG (Deep Deterministic Policy Gradient) have slow convergence speed in high-dimensional continuous action space, making it difficult to meet real-time control requirements. Finally, there is a lack of parameter generalization. Existing studies have not fully addressed the parameter adaptability of reinforcement learning algorithms in various working conditions, resulting in poor performance in non-training environments.

[0004] Therefore, in the related art, there is an urgent need for a way to solve the problems of parameter adjustment difficulty, insufficient control performance, and limited response speed of traditional valve positioners in control. SUMMARY

[0005] Therefore, it is necessary to provide an intelligent valve control method based on a double-delay deep deterministic policy gradient to solve the problems of parameter adjustment difficulty, insufficient control performance and limited response speed of a traditional valve positioner.

[0006] In a first aspect, the application provides an intelligent valve control method based on a double-delay deep deterministic policy gradient. The method comprises:

[0007] establishing a control valve mathematical model, determining a valve transfer function, and building a valve simulation environment;

[0008] based on the valve simulation environment, using a double-delay deep deterministic policy gradient algorithm to optimize and adjust the parameters of the control valve mathematical model;

[0009] based on the optimized and adjusted parameters, implementing PID valve control.

[0010] Optionally, in an embodiment of the application, the establishment of the control valve mathematical model and the determination of the valve transfer function comprise:

[0011] respectively establishing a valve stem mathematical model, a valve pneumatic actuator mathematical model and a pilot electric / gas proportional valve mathematical model, and determining the valve transfer function through Laplace transform.

[0012] Optionally, in an embodiment of the application, the establishment of the control valve mathematical model and the determination of the valve transfer function further comprise:

[0013] establishing a valve feedback link mathematical model and constructing a valve feedback link transfer function.

[0014] Optionally, in an embodiment of the application, the optimization and adjustment of the parameters of the control valve mathematical model based on the valve simulation environment and using the double-delay deep deterministic policy gradient algorithm comprise:

[0015] initializing the valve simulation environment and an agent, the agent comprising a policy network and a value network;

[0016] determining an action in the policy network according to a current state, each action corresponding to different parameters;

[0017] executing the action, evaluating a corresponding reward by the value network, and adjusting the action based on the reward.

[0018] Optionally, in an embodiment of the application, the value network comprises two independent Q value networks for evaluating the value of a state-action pair.

[0019] Optionally, in an embodiment of the present application, the adjusting the action based on the reward evaluated by the value network comprises:

[0020] After evaluating the action value by two independent Q-value networks, the policy network is updated with delay.

[0021] In a second aspect, the present application further provides an intelligent valve control device based on double-delay deep deterministic policy gradient. The device comprises:

[0022] a mathematical model building module, configured to build a regulating valve mathematical model, determine a valve transfer function, and build a valve simulation environment;

[0023] a parameter optimization module, configured to optimize and adjust parameters of the regulating valve mathematical model based on the valve simulation environment and using a double-delay deep deterministic policy gradient algorithm;

[0024] a PID valve control module, configured to implement PID valve control based on the optimized and adjusted parameters.

[0025] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes steps of the method in each of the above embodiments.

[0026] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of the method in each of the above embodiments.

[0027] The above-mentioned intelligent valve control method based on double-delay deep deterministic policy gradient first establishes a regulating valve mathematical model, determines the valve transfer function, and builds a valve simulation environment; then, based on the valve simulation environment, the double-delay deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model; finally, the PID valve control is realized based on the optimized and adjusted parameters. Compared with the traditional method, the PID controller optimized and adjusted by the TD3 algorithm significantly reduces the steady-state error of the system and significantly improves the accuracy of target tracking. Through the adaptive adjustment of reinforcement learning, the system can more accurately match the reference target, reduce error accumulation, and improve the stability of precision control. The controller based on the TD3 algorithm shows significant performance optimization when facing dynamic changes, and the dynamic response of the control system is more rapid and stable. Due to the introduction of delay update and target action noise, the system's overshoot is greatly reduced, the response speed is faster, and the common overshoot problem in traditional control methods is avoided, improving the stability and responsiveness of the system. In the case of system parameter changes or external disturbances, the TD3 optimized controller can maintain stable control performance and exhibit strong adaptability. Through online adjustment and adaptation of the reinforcement learning algorithm, the system can automatically respond to complex environmental changes, enhancing the robustness of the control system. The TD3 algorithm can automatically learn and optimize the parameters of the PI controller according to the actual working conditions, without human intervention, simplifying the traditional manual adjustment process. Compared with manual adjustment, the TD3 algorithm can continuously optimize the controller parameters based on real-time feedback of the system, making the control system have stronger adaptability and self-adjustment ability, thereby improving the overall work efficiency and intelligent level. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Application environment diagram for an embodiment of an intelligent valve control method based on double-delay deep deterministic policy gradient;

[0029] Figure 2 Flowchart of an embodiment of an intelligent valve control method based on double-delay deep deterministic policy gradient;

[0030] Figure 3 Schematic diagram of a valve simulation environment model in an embodiment;

[0031] Figure 4 Schematic diagram of a regulating valve structure in an embodiment;

[0032] Figure 5 Schematic diagram of a valve rod force analysis in an embodiment;

[0033] Figure 6 Schematic diagram of fluid flow in an embodiment;

[0034] Figure 7 Fig. 9 is a schematic diagram of an embodiment of a double-delay deep deterministic policy gradient algorithm for optimizing adjustment parameters;

[0035] Figure 8 Fig. 10 is a schematic diagram of an embodiment of results of a comparative test;

[0036] Figure 9 Fig. 11 is a schematic diagram of an embodiment of a structure of an intelligent valve control device based on a double-delay deep deterministic policy gradient;

[0037] Figure 10 Fig. 12 is a schematic diagram of an embodiment of an internal structure of a computer device. DETAILED DESCRIPTION

[0038] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0039] An embodiment of the present application provides an intelligent valve control method based on a double-delay deep deterministic policy gradient, which can be applied to an application environment as shown in Figure 1 . The terminal communicates with the server through a network. The data storage system can store data required to be processed by the server. The data storage system can be integrated on the server, or placed on a cloud or other network server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0040] In an embodiment, as shown in Figure 2 , an intelligent valve control method based on a double-delay deep deterministic policy gradient is provided. Taking the server in Figure 1 as an example, the method includes the following steps:

[0041] S201: Establish a regulating valve mathematical model, determine a valve transfer function, and build a valve simulation environment.

[0042] In the embodiments of the present application, first, the single-cylinder action valve of the intelligent valve positioner is taken as the research object, the system model of each part of the regulating valve is established, and the corresponding mathematical model transfer functions G1, G3 and G3 are derived from the model of the regulating valve established by the physical formula, so as to obtain the valve transfer function G(s) = G1*G3*G3. Then, the simulation environment model of the controlled object is built by using MATLAB Simulink. As shown in Figure 3 , it is a schematic diagram of the valve simulation environment model, and the target of the control system is to keep the opening value of the valve matching the reference value. As shown in Figure 4 , it is a structure diagram of the regulating valve, which is a pneumatic thin-film straight stroke single-cylinder action valve.

[0043] Specifically, in one embodiment of the present application, the establishment of the mathematical model of the regulating valve and the determination of the valve transfer function include:

[0044] The mathematical model of the valve rod, the mathematical model of the valve pneumatic actuator and the mathematical model of the pilot electric / gas proportional valve are respectively established, and the valve transfer function is determined by Laplace transform.

[0045] In one embodiment of the present application, first, as shown in Figure 5 , the force analysis of the valve rod movement process can be considered as a spring constant damping system under the action of pneumatic pressure, FQ is the gas source thrust, Fm is the gravity, Fk is the spring force, Fr is the fluid disturbance force and inertial force, and Ff is the friction. First, it is assumed that the entire valve rod system is a rigid body and has no elastic deformation; the heat exchange between the valve rod and the filler is not considered in the working process of the valve rod; only the displacement in the up-down direction is generated in the movement process of the valve rod, and the lateral displacement is ignored; the friction generated between the valve rod and the filler is considered as a linear constant damping force. When the gas source gas is filled into the cylinder of the pneumatic valve, a downward pneumatic thrust is generated, and the valve rod starts to move towards the specific target valve position under the action of the pneumatic thrust, so that the valve is opened to a specific opening. Assuming that the displacement of the valve rod is X(t), the pneumatic thrust of the gas source is FQ; the spring stiffness is k, and the spring force Fk of the spring on the valve rod is kx; the valve rod and its attached parts (including the weight of the spring) are regarded as a mass block with a mass M, and according to the force analysis diagram, it can be known that the gravity Fm = Mg; the damping coefficient of the system is α, the sum of the static friction and the dynamic friction between the valve rod and the filler is Ff = FJ + FD, and the remaining forces such as the fluid disturbance force and the inertial force are temporarily ignored, and the motion differential equation of the system is established:

[0046]

[0047] In the research process of the valve rod movement system, the motion differential equation can be written as

[0048]

[0049] wherein M, a, k are constants, and Laplace transform is made to both sides of the equation

[0050]

[0051] And from the physical formula FQ = P / S, wherein P is the output pressure of the pneumatic actuator, and S is the cross-sectional area of the spool, the transfer function of the mathematical model of the valve stem movement can be approximated as

[0052]

[0053] wherein m is the mass of the valve stem and its attached components, affecting the inertia of the system, i.e. the acceleration response of the valve stem under force; a is the damping coefficient of the system, reflecting the damping force generated during the movement of the valve stem due to friction, etc., affecting the rate of change of the speed of the system; k is the spring stiffness coefficient, representing the restoring force of the spring, determining the size of the restoring force exerted by the spring when the valve stem deviates from the equilibrium position, affecting the stability and recovery ability of the system; s is the Laplace transform variable, used to represent the behavior of the system in the frequency domain; and S is the cross-sectional area of the spool.

[0054] For the mathematical model of the valve pneumatic actuator, the valve pneumatic actuator can be regarded as a closed pneumatic resistance-capacitance link. With different valve opening requirements, the microprocessor adjusts the required air pressure charged into the valve cylinder by controlling the current signal, and the output pressure of the air source will change with the change of the control signal. The input and output air intake speed is determined by the time constant T1 of the pneumatic resistance-capacitance link, wherein T1 = R·C, R = a·L / S, wherein a is the damping coefficient of the system, S is the cross-sectional area of the valve stem, L is the length of the valve stem that can move in the air chamber, and C is the air capacity of the air path. If the input pressure of the pneumatic actuator is represented by P1, and the output pressure of the pneumatic actuator is represented by P2, then the differential equation of the mathematical model of the valve pneumatic actuator can be written as:

[0055]

[0056] The Laplace transform is made to both ends of the above equation, and considering that the valve pneumatic actuator has a certain hysteresis to the output of the air source, the transfer function can be obtained as:

[0057]

[0058] wherein K is the amplification coefficient, A is the equivalent cross-sectional area of the valve stem, C is the air capacity of the air path, C = β2·V, β2 represents the air capacity coefficient of the air path, and V represents the volume of the air chamber; T1 is the time constant of the pneumatic resistance-capacitance link, T2 = R·C, R is the air resistance of the air path, β1 represents the air resistance coefficient of the air path, a represents the air path cross-sectional area, l is the effective length of the air cavity, C represents the volume of the air cavity (air capacity), which determines the gas storage capacity of the pneumatic system; ts is the lag time, which represents the response delay of the pneumatic actuator, that is, the delay time from when the system receives an input signal to when the actual output starts to change.

[0059] For the mathematical model of the pilot electric / pneumatic proportional valve, the straight-travel pneumatic regulating valve, when the valve microprocessor sends out a control current signal, the pneumatic actuator starts to act after receiving the control signal, and stops moving until the valve stem moves to a specific position, and then waits for the action command signal again. During the movement of the valve stem, it is mainly affected by two factors: the structure of the valve manufacturing and assembly, the state of the medium and its medium flow. By analyzing the two main factors, the corresponding mathematical model is established.

[0060] As shown in Figure 6 , first establish a fluid flow sketch, assume that in Δt this period of time, fluid flows from the area S1 to S2, the flow rate through the pipe is Q, the density of the fluid is ρ, the fluid flow rate through the cross section is v, and the start and end positions of the fluid from one cross section S1 to another cross section S2 are recorded as L1 and L2 respectively. According to the momentum theorem, the following equation can be obtained:

[0061]

[0062] First, differentiate both ends of the above equation simultaneously, and finally it can be seen that there are two different types of forces acting on the fluid. The liquid dynamic force caused by the change of momentum during the stable flow of the liquid is generally called steady-state liquid dynamic force Fw, and the liquid dynamic force caused by the change of momentum during the flow through the valve core is generally called transient liquid dynamic force FS, and FZ = FW+FS. Secondly, it is assumed that during the movement of the valve core, the influence of environmental conditions is negligible, the flow direction of the fluid is consistent and directional, and there is no fluid leakage during the flow process under ideal conditions, and the valve core system dynamics differential equation is established according to the gas state equation, as shown below:

[0063]

[0064] Where x1 represents the compressed length, x0 represents the length of the spring before compression, F1 represents the pressure generated after the valve enters the membrane head, F0 represents the pressure generated by the valve output, Fm represents the system friction, α represents the air path damping coefficient of the valve, and k represents the spring elastic coefficient. After Laplace transformation of both ends, a second-order oscillation link is obtained.

[0065] The pneumatic regulating valve movement process is complex, and it is difficult to obtain its accurate mathematical model. The obtained mathematical model is only an approximate model under ideal state. According to the approximate mathematical model, a transfer function composed of typical control links can be obtained, which can be approximately expressed as

[0066]

[0067] Wherein, k represents the spring elastic coefficient, ω represents the angular velocity of gear rotation, and ξ represents the damping ratio, which describes the damping characteristics of the system. The damping ratio affects the response speed and stability of the system. If ξ < 1, the system is under-damped, and oscillation is exhibited; if ξ = 1, the system is critically damped, and the response is fast and no oscillation; if ξ > 1, the system is over-damped, and the response is slow and no oscillation.

[0068] In addition, in an embodiment of the present application, the establishing a regulating valve mathematical model and determining a valve transfer function further comprises:

[0069] A valve feedback link mathematical model is established, and a valve feedback link transfer function is constructed.

[0070] In an embodiment of the present application, the working mode of the valve positioner in the valve position adjustment process belongs to a closed-loop control system, and a control feedback link is inevitably present. The valve positioner feedback control link includes two aspects of valve front and valve back relative differential pressure feedback and valve position feedback. The pipeline pressure difference is a pressure difference formed by a pressure transmitter by measuring the pressures before and after the pipeline valve, and the measured and calculated pressure difference is fed back to the valve main control chip. Secondly, the valve position sensor sends the specific position signal data of the valve to the valve main control chip after A / D conversion, and the detection of the valve position needs to convert the straight stroke of the valve rod into an angular stroke that can be detected by a precision gear. Assuming that the precision gear rotates without clearance and sliding, the angular velocity of rotation is ω, the radius of rotation is r, and the linear velocity of the gear is v. Accordingly, the following can be obtained:

[0071] ω·r

[0072] Similarly, the Laplace transform is performed on both ends of the formula, and the transfer function for constructing the valve feedback link mathematical model is obtained as follows:

[0073]

[0074] S203: Based on the valve simulation environment, a double-delay deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model.

[0075] In this embodiment, a Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is trained within a Simulink simulation environment based on a built valve simulation environment to optimize the parameters of the control valve mathematical model. The optimization objective is to minimize performance indicators. These indicators include system error, steady-state error, overshoot, and response time, with error and response time being the most important.

[0076] In one embodiment of the present application, optimizing and adjusting the parameters of the regulating valve mathematical model using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment includes:

[0077] S301: Initialize the valve simulation environment and agent, wherein the agent includes a policy network and a value network.

[0078] S303: Determine an action in the policy network according to the current state, where each action corresponds to different parameters.

[0079] S305: Execute the action, evaluate the corresponding reward by the value network, and adjust the action based on the reward.

[0080] In one embodiment of the present application, Figure 7 As shown, we first initialize the environment and agent, creating a valve control system simulation environment using Simulink. The environment's inputs are the valve error and its integral (observation space), and its output is the PID controller's gain (action space). The observation space is where the reinforcement learning environment receives action signals from the agent and generates observation signals in response to those actions. This is a collection of environmental information that describes the agent's observable state and is typically used to represent the state of the environment.

[0081] The TD3 algorithm is used to create the reinforcement learning agent. The agent consists of two main parts: the policy network and the value network. Among them, the policy network selects the gain of the PID controller according to the current environment state (error and error integral), and the value network evaluates the pros and cons of the selected action. Then training, the agent constantly interacts with the environment, and at each step the agent selects an action (PID gain) according to the current state, and then adjusts its strategy according to the feedback of the environment. After the agent performs an action each time, the environment returns a reward. When the error is lower than the scalar reward signal, the reward is positive, otherwise it is negative. The goal of the agent is to gradually optimize the parameters of the PID controller through repeated trials, reduce the system error, and maximize the reward. Training will stop when the average reward of the agent reaches a certain threshold. To terminate the training episode and simulation, specify a logical signal to the isdone input port of the module. It should be noted that during the training process, in order to avoid too large action changes, some noise is added to the target policy, so that the learning process is more stable. At the same time, the agent breaks the correlation between data by replaying previous experiences, improving learning effect.

[0082] In an embodiment of the present application, the value network includes two independent Q value networks for evaluating the value of state-action pairs.

[0083] In an embodiment of the present application, the adjusting the action based on the reward evaluated by the value network comprises:

[0084] After evaluating the value of the action by the two independent Q value networks, the policy network is updated with a delay.

[0085] In an embodiment of the present application, the value network includes two independent Q value networks, and the agent evaluates the value of the action through the two Q networks. At the same time, the use of delayed updates ensures the stability of training, that is, after the two independent Q value networks evaluate the value of the action, the policy network is updated with a delay.

[0086] In an embodiment of the present application, the policy network, the value network, the target network and the experience replay buffer are first initialized. Specifically, the policy network (π) is initialized to generate a deterministic policy, that is, to output a determined action according to the current state. The value network is initialized, that is, two independent Q value networks Q1(s, a) and Q2(s, a) are initialized to evaluate the value of state-action pairs, where s is the current state and a is the action response. The target network is initialized, that is, a target network is copied for each Q value network and policy network (i.e. Q 1′ , Q 2′and the target networks are used to compute target values and delayed updates in training. An experience replay buffer is initialized to store the interaction data of the agent, including state, action, reward and next state. Then data is collected, the interaction data is sampled from the simulation environment (s t ,a t ,r t ,s t+1 ) and stored in the experience replay buffer. Where s t is the current state, a t is the action based on the current state, r t is the reward based on the current action, s t+1 is the next state based on the current action. The action is selected according to the current policy network (π). Then, the Q value is updated, and the Q network is updated twice each time. The target Q network is used to calculate the target Q value of the next state, and the formula is:

[0087]

[0088] Where y is the target Q value, r t is the reward, γ is the discount factor, Q'1 and Q'2 are the target Q networks, and min selects the smaller value of the two Q values to reduce the problem of overestimation of Q values, is the target action, is the target action noise, and the noise is added to avoid the agent overusing the action with a higher Q value estimate.

[0089] At the same time, the current Q value error is calculated, and the parameters of the Q network are updated using the gradient descent method to minimize the error by calculating the error of the current Q network. The error formula is as follows:

[0090]

[0091] Then, the policy network is updated with a delay, and the policy network is updated every C times of Q value update. The value of the policy is evaluated using the current Q value network, and the parameters of the policy network are updated according to the policy gradient. The goal of the update is to maximize the value estimated by the Q value function, that is:

[0092]

[0093] Then, the target network is updated with a delay, and the parameters of the target network (Q 1′ , Q 2′ and ') are updated every D times of policy network update. The update of the target network uses soft update:

[0094] θ' = τθ + (1 - τ)θ'

[0095] where τ is the learning rate of soft update, which is usually set to a small value.

[0096] The above steps are repeated until the training reaches a predetermined maximum number of steps or the performance converges.

[0097] S205: Implement PID valve control based on the optimized adjusted parameters.

[0098] In the embodiments of the present application, after the training is completed, the performance of the simulation agent in the environment is drawn to draw the step response curve, and the performance of the controller optimized by reinforcement learning is compared with that of the traditional PID controller. The PID controller is tuned using Simulink Control SystemTuner app as a control group, and the performance differences of the controller tuned by TD3 in terms of steady-state error, overshoot and response time are compared. The final result is shown in Figure 8 The abscissa represents the time of control, and the ordinate represents the result of control. As can be seen from the figure, the RL line takes significantly shorter time to reach the target result, and there is no steady-state error, and it is stuck around a target value without change, and finally it also only appears once overshoot to reach the target. In summary, the trained model has the advantages of basically no steady-state error, less overshoot, short response time, etc.

[0099] In the above-mentioned intelligent valve control method based on double-delay deep deterministic policy gradient, first, a regulating valve mathematical model is established, the valve transfer function is determined, and a valve simulation environment is built. Then, based on the valve simulation environment, the double-delay deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model. Finally, PID valve control is realized based on the optimized and adjusted parameters. Compared with traditional methods, the PID controller optimized and adjusted by the TD3 algorithm significantly reduces the steady-state error of the system and significantly improves the accuracy of target tracking. Through adaptive adjustment of reinforcement learning, the system can more accurately match the reference target, reduce error accumulation, and improve the stability of precision control. The controller based on the TD3 algorithm shows significant performance optimization when facing dynamic changes, and the dynamic response of the control system is more rapid and stable. Due to the introduction of delay update and target action noise, the system's overshoot is significantly reduced, the response speed is faster, and the overshoot problem common in traditional control methods is avoided, improving the stability and responsiveness of the system. In the case of system parameter changes or external disturbances, the TD3 optimized controller can maintain stable control performance and exhibit strong adaptability. Through online adjustment and adaptation of the reinforcement learning algorithm, the system can automatically respond to complex environmental changes, enhancing the robustness of the control system. The TD3 algorithm can automatically learn and optimize the parameters of the PI controller according to the actual working conditions, without human intervention, simplifying the traditional manual adjustment process. Compared with manual adjustment, the TD3 algorithm can continuously optimize the controller parameters based on real-time feedback from the system, making the control system more adaptable and self-adjusting, thereby improving overall work efficiency and intelligent level.

[0100] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0101] Based on the same inventive concept, the embodiments of the present application further provide a double-delay deep deterministic policy gradient based intelligent valve control device for implementing the double-delay deep deterministic policy gradient based intelligent valve control method. The implementation scheme for solving problems provided by the device is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more double-delay deep deterministic policy gradient based intelligent valve control device embodiments provided below can be referred to the limitations of the double-delay deep deterministic policy gradient based intelligent valve control method described above, which will not be repeated here.

[0102] In one embodiment, as shown in Figure 9 a double-delay deep deterministic policy gradient based intelligent valve control device 900 is provided, comprising a mathematical model building module 901, a parameter optimization module 903 and a PID valve control module 905, wherein:

[0103] The mathematical model building module 901 is configured to establish a regulating valve mathematical model, determine a valve transfer function, and build a valve simulation environment.

[0104] The parameter optimization module 903 is configured to optimize and adjust parameters of the regulating valve mathematical model based on the valve simulation environment and using a double-delay deep deterministic policy gradient algorithm.

[0105] The PID valve control module 905 is configured to implement PID valve control based on the optimized and adjusted parameters.

[0106] In one embodiment of the present application, the mathematical model building module is further configured to:

[0107] establish a valve stem mathematical model, a valve pneumatic actuator mathematical model and a pilot electric / gas proportional valve mathematical model respectively, and determine a valve transfer function through Laplace transform.

[0108] In one embodiment of the present application, the mathematical model building module is further configured to:

[0109] establish a valve feedback link mathematical model and construct a valve feedback link transfer function.

[0110] In one embodiment of the present application, the parameter optimization module is further configured to:

[0111] initialize a valve simulation environment and an agent, wherein the agent comprises a policy network and a value network;

[0112] determine an action in the policy network according to a current state, and each action corresponds to different parameters;

[0113] Actions are performed, and the corresponding rewards are evaluated by the value network, which adjusts the actions based on the rewards.

[0114] In one embodiment of the present application, the value network includes two independent Q-value networks for evaluating the value of state-action pairs.

[0115] In one embodiment of the present application, the parameter optimization module is further configured to:

[0116] After evaluating the action value through two independent Q-value networks, the policy network is updated with a delay.

[0117] Each module in the aforementioned intelligent valve control device based on a dual-delay deep deterministic policy gradient can be implemented in whole or in part via software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0118] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an intelligent valve control method based on a double-delay deep deterministic policy gradient is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0119] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0120] In an embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0121] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0122] In an embodiment, a computer program product is provided, comprising a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0125] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0126] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An intelligent valve control method based on double-delayed deep deterministic policy gradient, characterized in that: The method comprises: Establish a mathematical model of the control valve, determine the valve transfer function, and build a valve simulation environment; Based on the valve simulation environment, a double-delayed deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model; Implementing PID valve control based on the optimized and adjusted parameters; The establishing of the regulating valve mathematical model and determining the valve transfer function comprises: The mathematical models of valve stem, valve pneumatic actuator and pilot electric / air proportional valve are established respectively, and the valve transfer function is determined by Laplace transform derivation. The establishing of the regulating valve mathematical model and determining the valve transfer function further comprises: Establish a mathematical model of the valve feedback link and construct the transfer function of the valve feedback link; The optimizing and adjusting the parameters of the regulating valve mathematical model using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment includes: Initializing a valve simulation environment and an agent, wherein the agent includes a policy network and a value network; In the policy network, actions are determined based on the current state, and each action corresponds to different parameters; An action is performed, and the corresponding reward is evaluated by the value network, which adjusts the action based on the reward.

2. The intelligent valve control method based on double-delayed deep deterministic policy gradient according to claim 1 is characterized in that: The value network includes two independent Q-value networks for evaluating the value of state-action pairs.

3. The intelligent valve control method based on double-delayed deep deterministic policy gradient according to claim 2 is characterized in that: The reward corresponding to the value network evaluation and the action of adjusting the reward based on the reward include: After evaluating the action value through two independent Q-value networks, the policy network is updated with a delay.

4. An intelligent valve control device based on double-delayed deep deterministic policy gradient, characterized in that: The device comprises: Mathematical model building module, used to establish the mathematical model of the control valve, determine the valve transfer function, and build the valve simulation environment; A parameter optimization module, configured to optimize and adjust the parameters of the regulating valve mathematical model using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment; A PID valve control module, configured to implement PID valve control based on the optimized and adjusted parameters; The establishing of the regulating valve mathematical model and determining the valve transfer function comprises: The mathematical models of valve stem, valve pneumatic actuator and pilot electric / air proportional valve are established respectively, and the valve transfer function is determined by Laplace transform derivation. The establishing of the regulating valve mathematical model and determining the valve transfer function further comprises: Establish a mathematical model of the valve feedback link and construct the transfer function of the valve feedback link; The optimizing and adjusting the parameters of the regulating valve mathematical model using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment includes: Initializing a valve simulation environment and an agent, wherein the agent includes a policy network and a value network; In the policy network, actions are determined based on the current state, and each action corresponds to different parameters; An action is performed, and the corresponding reward is evaluated by the value network, which adjusts the action based on the reward.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images