Intelligent valve control method based on double-delay depth deterministic strategy gradient

Abstract

The invention relates to an intelligent valve control method based on a double-delay depth deterministic strategy gradient. The method comprises the steps that firstly, a regulating valve mathematical model is established, a valve transfer function is determined, and a valve simulation environment is established; then, on the basis of the valve simulation environment, parameters of the regulating valve mathematical model are optimized and adjusted through a double-delay depth deterministic strategy gradient algorithm; and finally, PID valve control is realized based on the optimized and adjusted parameters. Compared with the traditional method, the PID controller optimized and set by using the TD3 algorithm has the advantages that the steady-state error of the system is remarkably reduced, and the target tracking precision is remarkably improved. Through self-adaptive adjustment of reinforcement learning, the system can be matched with a reference target more accurately, error accumulation is reduced, and the stability of precision control is improved. Compared with manual adjustment, the TD3 algorithm can continuously optimize controller parameters according to real-time feedback of the system, so that the control system has higher adaptability and self-adjustment capability.

Description

Technical Field

[0001] The present application relates to the field of valve control technology, and in particular to an intelligent valve control method based on double-delay deep deterministic policy gradient. Background Art

[0002] As a key component in the process control system, the main function of the valve positioner is to accurately adjust the valve opening to ensure that the process parameters are stable within the set range. In traditional valve control, PID (proportional-integral-differential) controllers are widely used due to their simple algorithms and fast response speed. However, PID controllers still have the following shortcomings in actual industrial applications. First, parameter adjustment is complex. The parameters of traditional PID controllers usually rely on empirical adjustment or trial and error, which makes it difficult to achieve global optimization under complex and changeable working conditions. Second, the robustness is insufficient. Nonlinearity, time-varying and uncertainty are common in valve systems, making it difficult for fixed-parameter PID control to adapt to operating fluctuations and easily lead to system overshoot or oscillation. Third, the dynamic response is poor. In scenarios with high requirements for high precision and high response speed, traditional PID control is difficult to respond quickly to load changes, affecting the overall efficiency of the system.

[0003] In order to overcome the above problems, data-driven intelligent control technology has gradually become a research hotspot in recent years. In particular, reinforcement learning (RL), as a technology that can optimize control strategies through self-learning in dynamic and complex environments, shows significant potential. In the prior art, there have been studies that have attempted to apply reinforcement learning to the field of industrial control, but there are still the following problems. First, the algorithm is not well integrated with the physical model. Most studies only verify the effectiveness of the reinforcement learning algorithm in an offline simulation environment, without building a detailed mathematical model for the specific physical system, resulting in the algorithm being out of touch with the actual working conditions. Second, the learning efficiency is low. Traditional reinforcement learning algorithms such as Q-learning and DDPG (deep deterministic policy gradient) converge slowly in high-dimensional continuous action spaces, making it difficult to meet real-time control requirements. Finally, there is insufficient parameter generalization. Existing studies have failed to fully solve the parameter adaptability problem of reinforcement learning algorithms under various working conditions, resulting in poor performance in non-training environments.

[0004] Therefore, in the related art, there is an urgent need for a method that can solve the problems of difficult parameter adjustment, insufficient control performance and limited response speed in the control of traditional valve positioners. Summary of the invention

[0005] Based on this, it is necessary to provide an intelligent valve control method based on double-delay deep deterministic strategy gradient to address the above technical problems, which can solve the problems of difficult parameter adjustment, insufficient control performance and limited response speed of traditional valve positioners in control.

[0006] In a first aspect, the present application provides an intelligent valve control method based on a double-delayed deep deterministic policy gradient. The method comprises:

[0007] Establish a mathematical model of the control valve, determine the valve transfer function, and build a valve simulation environment;

[0008] Based on the valve simulation environment, a double-delayed deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the mathematical model of the regulating valve;

[0009] PID valve control is implemented based on the optimized and adjusted parameters.

[0010] Optionally, in one embodiment of the present application, establishing a mathematical model of the regulating valve and determining the valve transfer function includes:

[0011] The mathematical models of valve stem, valve pneumatic actuator and pilot electric / pneumatic proportional valve are established respectively, and the valve transfer function is determined by Laplace transform derivation.

[0012] Optionally, in one embodiment of the present application, the step of establishing a mathematical model of the regulating valve and determining the valve transfer function further comprises:

[0013] A mathematical model of the valve feedback link is established, and the transfer function of the valve feedback link is constructed.

[0014] Optionally, in one embodiment of the present application, the optimizing and adjusting the parameters of the regulating valve mathematical model by using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment includes:

[0015] Initializing a valve simulation environment and an agent, wherein the agent includes a policy network and a value network;

[0016] In the policy network, actions are determined based on the current state, and each action corresponds to different parameters;

[0017] Actions are performed, and the corresponding rewards are evaluated by the value network, which adjusts the actions based on the rewards.

[0018] Optionally, in one embodiment of the present application, the value network includes two independent Q-value networks for evaluating the value of state-action pairs.

[0019] Optionally, in one embodiment of the present application, the reward corresponding to the value network evaluation, and the action of adjusting the reward based on the reward include:

[0020] After evaluating the action value through two independent Q-value networks, the policy network is updated with a delay.

[0021] In a second aspect, the present application also provides an intelligent valve control device based on double-delayed deep deterministic policy gradient. The device comprises:

[0022] Mathematical model building module, used to establish the mathematical model of the control valve, determine the valve transfer function, and build the valve simulation environment;

[0023] A parameter optimization module, used to optimize and adjust the parameters of the mathematical model of the regulating valve using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment;

[0024] The PID valve control module is used to implement PID valve control based on the optimized and adjusted parameters.

[0025] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the methods described in the above embodiments.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in each of the above embodiments are implemented.

[0027] The above-mentioned intelligent valve control method based on double-delay deep deterministic policy gradient first establishes a regulating valve mathematical model, determines the valve transfer function, and builds a valve simulation environment; then, based on the valve simulation environment, the double-delay deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model; finally, PID valve control is realized based on the optimized and adjusted parameters. Compared with the traditional method, the PID controller optimized and adjusted by the TD3 algorithm significantly reduces the steady-state error of the system and significantly improves the accuracy of target tracking. Through the adaptive adjustment of reinforcement learning, the system can match the reference target more accurately, reduce error accumulation, and improve the stability of precision control. The controller based on the TD3 algorithm shows significant performance optimization in the face of dynamic changes, and the dynamic response of the control system is faster and smoother. Due to the introduction of delayed update and target action noise, the overshoot of the system is greatly reduced, the response speed is faster, the overshoot problem common in traditional control methods is avoided, and the stability and responsiveness of the system are improved. When the system parameters change or are disturbed by external factors, the TD3 optimized controller can maintain stable control performance and show strong adaptability. Through the online adjustment and adaptation of the reinforcement learning algorithm, the system can automatically respond to complex environmental changes and enhance the robustness of the control system. The TD3 algorithm can automatically learn and optimize the parameters of the PI controller according to the actual working conditions without manual intervention, simplifying the traditional manual adjustment process. Compared with manual adjustment, the TD3 algorithm can continuously optimize the controller parameters according to the real-time feedback of the system, making the control system more adaptable and self-adjusting, thereby improving the overall work efficiency and intelligence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An application environment diagram of an intelligent valve control method based on double-delay deep deterministic policy gradient in one embodiment;

[0029] Figure 2 A schematic flow chart of an intelligent valve control method based on double-delayed deep deterministic policy gradient in one embodiment;

[0030] Figure 3 A schematic diagram of a valve simulation environment model in one embodiment;

[0031] Figure 4 A schematic diagram of a control valve structure diagram in one embodiment;

[0032] Figure 5 A schematic diagram of a valve stem force analysis in one embodiment;

[0033] Figure 6 is a simplified schematic diagram of fluid flow in one embodiment;

[0034] Figure 7 A schematic diagram of optimizing and adjusting parameters of a double-delayed deep deterministic policy gradient algorithm in one embodiment;

[0035] Figure 8 A schematic diagram of the results of a comparative test in an embodiment;

[0036] Fig. 9 It is a structural block diagram of an intelligent valve control device based on double-delayed deep deterministic policy gradient in one embodiment;

[0037] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The embodiment of the present application provides an intelligent valve control method based on double-delayed deep deterministic policy gradient, which can be applied to Figure 1 In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server, or it can be placed on the cloud or other network servers. Among them, the terminal can be but not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0040] In one embodiment, Figure 2 As shown in FIG, a smart valve control method based on double-delay deep deterministic policy gradient is provided. The method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0041] S201: Establish a mathematical model of the control valve, determine the valve transfer function, and build a valve simulation environment.

[0042] In the embodiment of the present application, firstly, the single cylinder acting valve of the intelligent valve positioner is taken as the research object, the system model of each part of the regulating valve is established, and the corresponding mathematical model transfer functions G1, G3, G3 are derived from the established model of the regulating valve through the physical formula, and the valve transfer function is obtained, G(s) = G1*G3*G3. Then, MATLAB Simulink is used to build the simulation environment model of the controlled object. Figure 3 As shown in Figure 1, it is a schematic diagram of the valve simulation environment model. The goal of the control system is to keep the valve opening value matching the reference value. Figure 4 The figure shows the structure diagram of the regulating valve, which is a pneumatic diaphragm straight-stroke single-cylinder acting valve.

[0043] Specifically, in one embodiment of the present application, establishing a mathematical model of the regulating valve and determining the valve transfer function includes:

[0044] The mathematical models of valve stem, valve pneumatic actuator and pilot electric / pneumatic proportional valve are established respectively, and the valve transfer function is determined by Laplace transform derivation.

[0045] In one embodiment of the present application, first, Figure 5 As shown in the figure, the force analysis of the valve stem movement process can be considered as a spring constant damping system under the action of pneumatic pressure conditions, FQ is the gas source thrust, Fm is gravity, Fk is the spring force, Fr is the fluid disturbance force and inertia force, and Ff is the friction force. First, it is assumed that the entire valve stem system is a rigid body and has no elastic deformation; the heat exchange between the valve stems is not considered during the working process; the valve stem will only produce an upward and downward displacement during the movement, and the lateral displacement is ignored; the friction force generated by the valve stem and the packing is regarded as a linear constant damping force. When the gas source gas is filled into the cylinder of the pneumatic valve, a downward pneumatic thrust will be generated, and the valve stem will begin to move towards the specific target valve position under the action of the pneumatic thrust, so that the valve will open to a specific opening. Assume that the displacement of the valve stem is X(t), the pneumatic thrust of the air source is FQ; the spring stiffness is the coefficient k, then the spring force Fk=kx on the valve stem; the valve stem and its accessories (including the weight of the spring) are regarded as a mass block of mass M, combined with the force analysis diagram, it can be seen that the gravity Fm=Mg; suppose the damping coefficient of the system is α, the sum of the static friction and dynamic friction between the valve stem and the packing is Ff=FJ+FD, and the other forces such as fluid disturbance force and inertia force are always recorded as Fr and temporarily ignored, and the system is established to obtain the differential equation of motion:

[0046]

[0047] In the process of studying the valve stem motion system, the motion differential equation can be written as

[0048]

[0049] Among them, M, α, and k are all constants. Laplace transform is performed on both sides of the equation to obtain

[0050]

[0051] According to the physical formula FQ = P / S, where P is the output pressure of the pneumatic actuator and S is the cross-sectional area of ​​the valve core movement, the transfer function of the valve stem movement mathematical model can be approximated as

[0052]

[0053] Among them, m is the mass of the valve stem and its accessories, which affects the inertia of the system, that is, the acceleration response of the valve stem under force; α is the damping coefficient of the system, which reflects the damping force generated by friction and other reasons during the movement of the valve stem, and affects the velocity change rate of the system; k is the spring stiffness coefficient, which represents the restoring force of the spring, determines the size of the restoring force applied by the spring when the valve stem deviates from the equilibrium position, and affects the stability and recovery ability of the system; s is the Laplace transform variable, which is used to represent the behavior of the system in the frequency domain; S is the cross-sectional area of ​​the valve core movement.

[0054] For the mathematical model of valve pneumatic actuator, the valve pneumatic actuator can be regarded as a closed pneumatic RC link. With different valve position opening requirements, the microprocessor adjusts the air pressure of the air source that needs to be filled into the valve cylinder through the control current signal, and the output pressure of the air source will change with the change of the control signal. The frequency of the input and output air intake is determined by the time constant T1 of the pneumatic RC link, where T1 = R·C, R = α·L / S, where α is the damping coefficient of the system, S is the cross-sectional area of ​​the valve stem, L is the length that the valve stem can move in the air cavity, and C is the air volume of the air path. If the input pressure of the pneumatic actuator is represented by P1, and the output pressure of the pneumatic actuator is represented by P2, the mathematical model differential equation of the valve pneumatic actuator can be approximately written as:

[0055]

[0056] Perform Laplace transform on both ends of the above equation, and consider that the valve pneumatic actuator has a certain hysteresis in the output of the gas source, and the transfer function is:

[0057]

[0058] Where K is the amplification factor, A is the equivalent cross-sectional area of ​​the valve stem, C is the gas volume of the gas path, C = β2·V, β2 represents the gas volume coefficient of the gas path, V represents the volume of the gas cavity; T1 is the time constant of the pneumatic resistance-capacitance link, T2 = R·C, R is the air resistance of the air path, β1 represents the air resistance coefficient of the air path, a represents the cross-sectional area of ​​the air path, l is the effective length of the air cavity, and C represents the volume of the air cavity (air capacity), which determines the air storage capacity of the pneumatic system; ts is the lag time, which represents the response delay of the pneumatic actuator, that is, the delay time from the system receiving the input signal to the actual output starting to change.

[0059] For the mathematical model of pilot electric / pneumatic proportional valve, straight stroke pneumatic control valve, when the valve microprocessor sends a control current signal, the pneumatic actuator receives the control signal and starts to move until the valve stem moves to a specific position and stops moving, waiting to receive the action command signal again before moving again. The movement of the valve stem is mainly affected by two factors: the valve manufacturing assembly structure, the medium state and the medium flow rate. By analyzing the two main factors, the corresponding mathematical model is established.

[0060] like Figure 6 As shown, a simplified diagram of fluid flow is first established. Assume that during the period of Δt, the fluid flows from the pipe surface with an area of ​​S to the pipe surface S2, the flow rate through the pipe is Q, the density of the fluid is ρ, the fluid velocity flowing through the cross section is v, and the starting and ending positions of the fluid from one cross section S1 to another cross section S2 are recorded as L1 and L2 respectively. According to the momentum theorem, the following equation can be obtained:

[0061]

[0062] First, differentiate both ends of the above equation at the same time. Finally, it can be seen that there are two different types of forces acting on the fluid. Generally, the fluid force caused by the change in momentum when the liquid flows steadily is called the steady-state fluid force Fw, and the fluid force caused by the change in momentum when flowing through the valve core is generally called the transient fluid force FS. FZ = FW + FS. Secondly, it is assumed that during the movement of the valve core, the influence of environmental conditions is negligible, the flow direction of the fluid is consistent and directional, and there is no leakage of the fluid during the flow process. Under ideal conditions, the valve core system dynamics differential equation is established based on the gas state equation, as shown below:

[0063]

[0064] Among them, x1 represents the compressed length, x0 represents the length of the spring before it is compressed, F1 represents the pressure generated after the valve enters the diaphragm head, F0 represents the pressure generated by the valve output, Fm represents the system friction, α represents the valve gas path damping coefficient, and k represents the spring elastic coefficient. Performing Laplace transformation on both ends will result in a second-order oscillation link.

[0065] The movement process of the pneumatic control valve is complex, and it is difficult to obtain its accurate mathematical model. The obtained mathematical model is only an approximate model under an ideal state. Based on its approximate mathematical model, the transfer function composed of typical control links can be obtained. This transfer function can be approximately expressed as

[0066]

[0067] Among them, k represents the spring elastic coefficient, ω represents the angular velocity of the gear rotation, and ξ represents the damping ratio, which describes the damping characteristics of the system. The damping ratio affects the response speed and stability of the system. If ξ<1, the system is underdamped and exhibits oscillation; if ξ=1, the system is critically damped, responds quickly and has no oscillation; if ξ>1, the system is overdamped, responds slowly and has no oscillation.

[0068] In addition, in one embodiment of the present application, establishing a mathematical model of the regulating valve and determining the valve transfer function further comprises:

[0069] A mathematical model of the valve feedback link is established, and the transfer function of the valve feedback link is constructed.

[0070] In one embodiment of the present application, the working mode of the valve positioner during the valve position adjustment process belongs to a closed-loop control system, and there must be a control feedback link. The feedback control link of the valve positioner includes two aspects: the relative differential pressure feedback before and after the valve and the valve position feedback. The pipeline pressure difference is the pressure difference formed by the pressure transmitter by measuring the pressure before and after the pipeline valve, and the pressure difference obtained by the measurement and calculation is fed back to the valve main control chip; secondly, the valve position sensor also sends the valve specific position signal data to the valve main control chip after A / D conversion, and the detection of the valve position requires the straight stroke valve stem movement to be converted into an angular stroke that can be detected by precision gears. Assuming that the precision gear is a gapless and non-slip rotation, its angular velocity is set to ω, the radius of rotation is set to r, and the linear velocity of the gear is set to v, based on this:

[0071] ω·r

[0072] Similarly, Laplace transform is performed on both ends of the equation, and the transfer function of the mathematical model of the valve feedback link is obtained as follows:

[0073]

[0074] S203: Based on the valve simulation environment, a double-delayed deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model.

[0075] In the embodiment of the present application, in the Simulink simulation environment, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is trained based on the constructed valve simulation environment to optimize the parameters of the control valve mathematical model, and the optimization goal is to minimize the performance index. Among them, the performance index includes the system error, steady-state error, overshoot and response time, among which the error and response time are the two most important points.

[0076] In one embodiment of the present application, the optimizing and adjusting the parameters of the regulating valve mathematical model using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment includes:

[0077] S301: Initialize the valve simulation environment and agent, wherein the agent includes a policy network and a value network.

[0078] S303: Determine an action in the policy network according to the current state, and each action corresponds to different parameters.

[0079] S305: Execute the action, evaluate the corresponding reward by the value network, and adjust the action based on the reward.

[0080] In one embodiment of the present application, Figure 7 As shown, first initialize the environment and agent, and use Simulink to create a valve control system simulation environment. The input of the environment is the valve error and the integral of the error (observation space), and the output is the gain of the PID controller (action space). The observation space is the reinforcement learning environment receiving action signals from the agent and generating observation signals in response to these actions. This is a collection of environmental information that can be observed by the agent, usually used to represent the state of the environment.

[0081] Use the TD3 algorithm to create a reinforcement learning agent. The agent consists of two main parts: a policy network and a value network. The policy network selects the gain of the PID controller based on the current state of the environment (error and error integral), and the value network evaluates the quality of the selected action. After training, the agent interacts with the environment continuously. At each step, the agent selects an action (PID gain) based on the current state, and then adjusts its policy based on the feedback from the environment. Each time the agent performs an action, the environment returns a reward. The reward is positive when the error is lower than the scalar reward signal, otherwise it is negative. The goal of the agent is to gradually optimize the parameters of the PID controller through repeated trials, reduce the system error, and maximize the reward. Training stops when the average reward of the agent reaches a certain threshold. To terminate the training episode and simulation, specify a logical signal to the isdone input port of the module. It should be noted that during the training process, in order to avoid excessive action changes, some noise is added to the target policy to make the learning process more stable. At the same time, the agent breaks the correlation between data by replaying previous experience to improve the learning effect.

[0082] In one embodiment of the present application, the value network includes two independent Q-value networks for evaluating the value of state-action pairs.

[0083] In one embodiment of the present application, the reward corresponding to the value network evaluation and the action of adjusting the reward based on the reward include:

[0084] After evaluating the action value through two independent Q-value networks, the policy network is updated with a delay.

[0085] In one embodiment of the present application, the value network includes two independent Q-value networks, and the agent evaluates the value of the action through the two Q networks. At the same time, delayed updates are used to ensure training stability, that is, after every two independent Q-value networks evaluate the value of the action, the policy network is updated with a delay.

[0086] In one embodiment of the present application, the policy network, value network, target network and experience replay buffer are first initialized. Specifically, the policy network (π) is initialized to generate a deterministic policy, that is, to output a certain action according to the current state. The value network is initialized, that is, two independent Q value networks are initialized, namely Q1(s, a) and Q2(s, a), which are used to evaluate the value of the state-action pair, where s is the current state and a is the action response. The target network is initialized, that is, the target network (i.e., Q2) is copied for each Q value network and policy network. 1′ , Q 2′and π'), which are used to calculate the target value and perform delayed updates during training. Initialize the experience replay buffer to store the agent's interaction data, including state, action, reward, and next state. Then collect data and sample interaction data (s t ,a t ,r t ,s t+1 ) and stores it in the experience replay buffer. t is the current state, a t is the action based on the current state, r t is the reward based on the current action, s t+1 is the next state based on the current action. The action is selected according to the current policy network (π). After that, the Q value is updated, and the Q network is updated twice each time. The target Q value of the next state is calculated using the target Q network, and the calculation formula is:

[0087]

[0088] Among them, y is the target Q value, r t is the reward, γ is the discount factor, Q′1 and Q′2 are the target Q networks, min selects the smaller of the two Q values ​​to reduce the problem of overestimation of the Q value, is the target action, is the target action noise, and the noise is added to prevent the agent from over-estimating actions with higher Q values.

[0089] At the same time, the current Q value error is calculated. By calculating the error of the current Q network, the parameters of the Q network are updated using the gradient descent method to minimize the error. The error formula is as follows:

[0090]

[0091] After that, the policy network is updated with a delay, and the policy network is updated after every C Q value updates. The value of the policy is evaluated using the current Q value network, and the parameters of the policy network are updated according to the policy gradient. The goal of the update is to maximize the value estimated by the Q value function, that is:

[0092]

[0093] After that, the target network is updated with a delay. After each D policy network updates, the target network is updated with a delay (Q 1′ , Q 2′ The target network is updated using soft update:

[0094] θ′=τθ+(1-τ)θ′

[0095] Among them, τ is the learning rate of soft update, which is usually set to a small value.

[0096] Repeat the above steps until the training reaches the predetermined maximum number of steps or the performance converges.

[0097] S205: Implementing PID valve control based on the optimized and adjusted parameters.

[0098] In the embodiment of the present application, after the training is completed, the performance of the simulated agent in the environment is plotted, and the step response curve is compared to the performance of the controller optimized by reinforcement learning and the traditional PID controller. The PID controller tuned by Simulink Control SystemTuner app is used as the control group to compare the performance differences of the controller tuned by TD3 in terms of steady-state error, overshoot, and response time. The final results are as follows Figure 8 As shown. The horizontal axis represents the control time, while the vertical axis represents the control result. As can be seen from the figure, the time required for the RL line to reach the target result is obviously shorter, and there is no steady-state error. It is trapped around a target value and remains unchanged. In the end, it only has one overshoot before reaching the target. In summary, it shows that the trained model has basically no steady-state error, less overshoot, and short response time.

[0099] In the above-mentioned intelligent valve control method based on double-delay deep deterministic policy gradient, first, a regulating valve mathematical model is established, the valve transfer function is determined, and a valve simulation environment is built; then, based on the valve simulation environment, the double-delay deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the regulating valve mathematical model; finally, PID valve control is realized based on the optimized and adjusted parameters. Compared with the traditional method, the PID controller optimized and adjusted by the TD3 algorithm significantly reduces the steady-state error of the system and significantly improves the accuracy of target tracking. Through the adaptive adjustment of reinforcement learning, the system can match the reference target more accurately, reduce error accumulation, and improve the stability of precision control. The controller based on the TD3 algorithm shows significant performance optimization in the face of dynamic changes, and the dynamic response of the control system is faster and smoother. Due to the introduction of delayed update and target action noise, the overshoot of the system is greatly reduced, the response speed is faster, the overshoot problem common in traditional control methods is avoided, and the stability and responsiveness of the system are improved. When the system parameters change or are disturbed by external factors, the TD3 optimized controller can maintain stable control performance and show strong adaptability. Through the online adjustment and adaptation of the reinforcement learning algorithm, the system can automatically respond to complex environmental changes and enhance the robustness of the control system. The TD3 algorithm can automatically learn and optimize the parameters of the PI controller according to the actual working conditions without manual intervention, simplifying the traditional manual adjustment process. Compared with manual adjustment, the TD3 algorithm can continuously optimize the controller parameters according to the real-time feedback of the system, making the control system more adaptable and self-adjusting, thereby improving the overall work efficiency and intelligence level.

[0100] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0101] Based on the same inventive concept, the embodiment of the present application also provides a smart valve control device based on dual-delay deep deterministic policy gradient for implementing the above-mentioned smart valve control method based on dual-delay deep deterministic policy gradient. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the smart valve control device based on dual-delay deep deterministic policy gradient provided below can refer to the above limitations on a smart valve control method based on dual-delay deep deterministic policy gradient, and will not be repeated here.

[0102] In one embodiment, Fig. 9 As shown, an intelligent valve control device 900 based on double-delayed deep deterministic policy gradient is provided, comprising: a mathematical model building module 901, a parameter optimization module 903 and a PID valve control module 905, wherein:

[0103] The mathematical model building module 901 is used to establish a mathematical model of the regulating valve, determine the valve transfer function, and build a valve simulation environment.

[0104] The parameter optimization module 903 is used to optimize and adjust the parameters of the regulating valve mathematical model based on the valve simulation environment by using a double-delayed deep deterministic policy gradient algorithm.

[0105] The PID valve control module 905 is used to implement PID valve control based on the optimized and adjusted parameters.

[0106] In one embodiment of the present application, the mathematical model building module is also used to:

[0107] The mathematical models of valve stem, valve pneumatic actuator and pilot electric / pneumatic proportional valve are established respectively, and the valve transfer function is determined by Laplace transform derivation.

[0108] In one embodiment of the present application, the mathematical model building module is also used to:

[0109] A mathematical model of the valve feedback link is established, and the transfer function of the valve feedback link is constructed.

[0110] In one embodiment of the present application, the parameter optimization module is further used to:

[0111] Initializing a valve simulation environment and an agent, wherein the agent includes a policy network and a value network;

[0112] In the policy network, actions are determined based on the current state, and each action corresponds to different parameters;

[0113] Actions are performed, and the corresponding rewards are evaluated by the value network, which adjusts the actions based on the rewards.

[0114] In one embodiment of the present application, the value network includes two independent Q-value networks for evaluating the value of state-action pairs.

[0115] In one embodiment of the present application, the parameter optimization module is further used to:

[0116] After evaluating the action value through two independent Q-value networks, the policy network is updated with a delay.

[0117] Each module in the above-mentioned intelligent valve control device based on double-delay deep deterministic policy gradient can be implemented in whole or in part by software, hardware and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0118] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an intelligent valve control method based on a double-delay deep deterministic policy gradient is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0119] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0121] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0122] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An intelligent valve control method based on double-delayed deep deterministic policy gradient, characterized in that: The method comprises: Establish a mathematical model of the control valve, determine the valve transfer function, and build a valve simulation environment; Based on the valve simulation environment, a double-delayed deep deterministic policy gradient algorithm is used to optimize and adjust the parameters of the mathematical model of the regulating valve; PID valve control is implemented based on the optimized and adjusted parameters.

2. According to claim 1, a smart valve control method based on double-delayed deep deterministic policy gradient is characterized in that: The establishing of the regulating valve mathematical model and determining the valve transfer function comprises: The mathematical models of valve stem, valve pneumatic actuator and pilot electric / pneumatic proportional valve are established respectively, and the valve transfer function is determined by Laplace transform derivation.

3. The intelligent valve control method based on double-delayed deep deterministic policy gradient according to claim 2 is characterized in that: The establishing of the regulating valve mathematical model and determining the valve transfer function further comprises: A mathematical model of the valve feedback link is established, and the transfer function of the valve feedback link is constructed.

4. The intelligent valve control method based on double-delayed deep deterministic policy gradient according to claim 1 is characterized in that: The method of optimizing and adjusting the parameters of the regulating valve mathematical model by using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment includes: Initializing a valve simulation environment and an agent, wherein the agent includes a policy network and a value network; In the policy network, actions are determined based on the current state, and each action corresponds to different parameters; Actions are performed, and the corresponding rewards are evaluated by the value network, which adjusts the actions based on the rewards.

5. The intelligent valve control method based on double-delayed deep deterministic policy gradient according to claim 4 is characterized in that: The value network includes two independent Q-value networks for evaluating the value of state-action pairs.

6. The intelligent valve control method based on double-delayed deep deterministic policy gradient according to claim 5 is characterized in that: The reward corresponding to the value network evaluation, and the action of adjusting the reward based on the reward include: After evaluating the action value through two independent Q-value networks, the policy network is updated with a delay.

7. An intelligent valve control device based on double-delayed deep deterministic policy gradient, characterized in that: The device comprises: Mathematical model building module, used to establish the mathematical model of the control valve, determine the valve transfer function, and build the valve simulation environment; A parameter optimization module, used to optimize and adjust the parameters of the mathematical model of the regulating valve using a double-delayed deep deterministic policy gradient algorithm based on the valve simulation environment; The PID valve control module is used to implement PID valve control based on the optimized and adjusted parameters.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images