Adaptive control method, device and equipment of vacuum regulating valve and storage medium

Abstract

The present application relates to the technical field of pressure control, and particularly relates to an adaptive control method, device and equipment of a vacuum regulating valve and a storage medium, wherein the method constructs a reinforcement learning framework based on a deep deterministic policy gradient algorithm, uses a mathematical model as a simulation environment for training, and automatically generates a control agent; in the training process, a reward rate function and a round termination condition are used to dynamically optimize the policy, which replaces the parameter setting process of a traditional PID relying on artificial experience, significantly reduces the complexity and time consumption of parameter configuration; after deployment, real-time pressure information is collected to construct a real-time state vector, which is input into the control agent to directly output an opening degree adjustment instruction; the real-time state feedback and the dynamic decision mechanism of the control agent can independently cope with changes in airflow characteristics or external interference compared with a PID with fixed parameters, do not need to manually reset parameters, effectively reduce overshoot, accelerate response speed, suppress oscillation, and improve adaptive ability and operation stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pressure control, and particularly relates to an adaptive control method, device and equipment of a vacuum regulating valve and a storage medium. BACKGROUND

[0002] As a key core component of semiconductor equipment, the vacuum regulating valve is mainly used for realizing key functions such as on-off control, flow regulation and pressure precise regulation of the cavity gas path pipeline, and has the advantages of fast response speed and high pressure control precision, and has become an indispensable key component in the fields of high-end manufacturing such as semiconductor manufacturing, new display panels and solar photovoltaics.

[0003] In the technical field of pressure control of the vacuum regulating valve, the traditional proportional-integral-derivative (PID) control algorithm is widely used due to its simple structure and easy engineering implementation. However, in the actual application process, due to the significant nonlinear characteristics of the internal pressure change of the process cavity, the accurate mathematical modeling of the pressure control system faces severe challenges. When the conventional PID control algorithm is applied to the vacuum regulating valve system, it is difficult to meet the control requirements of high precision and high stability. Specifically, the pressure control system is prone to have a large overshoot, response delay and sustained oscillation phenomenon, and it is difficult to realize precise regulation of pressure under the condition of dynamic change of gas flow.

[0004] In addition, the parameter setting process of the existing PID control algorithm is complex and time-consuming, and the parameters are usually fixed after setting, which cannot be adjusted in real time online according to the changes of gas flow characteristics or external disturbances, resulting in significant deficiencies in the adaptive ability and running stability of the system.

[0005] It can be seen that the prior art still needs to be improved and improved. SUMMARY

[0006] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide an adaptive control method for a vacuum regulating valve, which effectively solves the inherent defects of traditional control technology and provides a high-precision, high-stability and strong-adaptive control scheme for pressure adjustment of the vacuum regulating valve.

[0007] The first aspect of the present application provides a self-adaptive control method of a vacuum regulating valve, comprising: obtaining control demand parameters and hardware configuration parameters of the vacuum regulating valve; constructing a mathematical model of the vacuum regulating valve based on the hardware configuration parameters and an ideal gas state equation; defining a state vector and designing a reward function and a round termination condition based on the control demand parameters; constructing a reinforcement learning framework based on a deep deterministic policy gradient algorithm; taking the constructed mathematical model as a simulation environment, running the reinforcement learning framework to perform training according to the defined state vector, the designed reward function and the round termination condition, and generating and deploying a control agent when a preset iteration stop condition is met; obtaining real-time pressure information to construct a real-time state vector, inputting the real-time state vector into the control agent, and adjusting the opening degree of the vacuum regulating valve based on the output result of the control agent.

[0008] Optionally, in the first implementation manner of the first aspect of the present application, the obtaining of the control demand parameters and the hardware configuration parameters of the vacuum regulating valve comprises: obtaining control demand parameters of the vacuum regulating valve, the control demand parameters comprising a pressure control range, a pressure control precision and a target cavity pressure; obtaining hardware configuration parameters of the vacuum regulating valve, the hardware configuration parameters comprising a valve plate diameter, a cavity volume and a gas type; determining a molar mass and a gas constant based on the gas type.

[0009] Optionally, in the second implementation manner of the first aspect of the present application, the constructing of the mathematical model of the vacuum regulating valve based on the hardware configuration parameters and the ideal gas state equation comprises: deriving a pressure change rate formula for characterizing the relationship between the gas inflow, the gas outflow and the pressure change based on the ideal gas state equation; introducing a calibration constant to construct a mapping relationship between the motor step number and the pressure change rate, and deriving a step pressure correlation formula; constructing the mathematical model of the vacuum regulating valve based on the pressure change rate formula and the step pressure correlation formula, the mathematical model being used to calculate the gas outflow.

[0010] Optionally, in the third implementation manner of the first aspect of the present application, the defining of the state vector and the designing of the reward function and the round termination condition based on the control demand parameters comprise: defining a state vector, the state vector comprising a pressure error, an integral of the pressure error and a current cavity pressure, the pressure error being calculated based on the current cavity pressure and the target cavity pressure; designing a reward function according to the pressure error, a pressure control range and a pressure control precision; setting a round termination condition according to the current cavity pressure and the pressure control precision.

[0011] Optionally, in a fourth implementation form of the first aspect of the present application, the deep deterministic policy gradient algorithm is used to build the reinforcement learning framework, which comprises: building an Actor network and a Critic network, the input end of the Actor network is configured to receive a state vector, the input end of the Critic network is configured to receive the state vector and a control action output by the Actor network, and the action value output by the Critic network is configured to provide a gradient basis for parameter optimization of the Actor network; configuring an experience replay pool configured to store experience data groups generated in the training process; and configuring a parameter update rule of the Actor network and the Critic network to complete the construction of the reinforcement learning framework, wherein the Critic network updates the network parameters by minimizing a loss function, and the Actor network updates the network parameters based on the action value gradient output by the Critic network.

[0012] Optionally, in a fifth implementation form of the first aspect of the present application, the constructed mathematical model is used as a simulation environment, and the reinforcement learning framework is run for training according to the defined state vector, the designed return rate function and the round termination condition, and a control agent is generated and deployed when a preset iteration stop condition is met, which comprises:

[0013] The constructed mathematical model is configured as a simulation environment, and the reinforcement learning framework is run for training according to the defined state vector, the designed return rate function and the round termination condition; in the process of each training round, the sum of reward values corresponding to each training step in the training round is calculated in real time to obtain the cumulative reward corresponding to each training round; a control agent is generated when the number of training rounds reaches 4000 rounds or the average cumulative reward of 100 consecutive rounds is greater than or equal to 800; the performance of the generated control agent is verified based on a preset test group, and if all test groups meet the preset verification indicators, the control agent is deployed.

[0014] Optionally, in a sixth implementation form of the first aspect of the present application, the real-time pressure information comprises a real-time cavity pressure; a real-time pressure error is calculated based on the real-time cavity pressure and a target cavity pressure, and a real-time pressure error integral is obtained by time integration of the real-time pressure error; a real-time state vector is constructed based on the real-time cavity pressure, the real-time pressure error and the real-time pressure error integral, and the real-time state vector is input into the control agent; the number of steps of the stepping motor is adjusted based on the output result of the control agent, and then the opening angle of the valve plate is adjusted to change the gas flow area, so as to adjust the gas outflow, and finally the real-time cavity pressure is stabilized to the target cavity pressure.

[0015] The second aspect of the present application provides an adaptive control device of a vacuum regulating valve, comprising: an acquisition module configured to acquire control demand parameters and hardware configuration parameters of the vacuum regulating valve; a first construction module configured to construct a mathematical model of the vacuum regulating valve based on the hardware configuration parameters and an ideal gas state equation; a definition module configured to define a state vector and design a reward function and a round termination condition based on the control demand parameters; a second construction module configured to construct a reinforcement learning framework based on a deep deterministic policy gradient algorithm; a training module configured to use the constructed mathematical model as a simulation environment, run the reinforcement learning framework to perform training according to the defined state vector, the designed reward function and the round termination condition, and generate and deploy a control agent when a preset iteration stop condition is met; and an adjustment module configured to acquire real-time pressure information to construct a real-time state vector, input the real-time state vector into the control agent, and adjust an opening degree of the vacuum regulating valve based on an output result of the control agent.

[0016] The third aspect of the present application provides an adaptive control device of a vacuum regulating valve, comprising: a memory and at least one processor, wherein the memory stores instructions; and the at least one processor invokes the instructions in the memory to enable the adaptive control device of the vacuum regulating valve to perform each step of the adaptive control method of the vacuum regulating valve according to any one of the above aspects.

[0017] The fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and the instructions are executed by a processor to implement each step of the adaptive control method of the vacuum regulating valve according to any one of the above aspects.

[0018] In the technical solution of the present application, the reinforcement learning framework is constructed based on the deep deterministic policy gradient algorithm, the mathematical model is used as a simulation environment for training to automatically generate a control agent; the reward function and the round termination condition are used to dynamically optimize the strategy during the training process, which replaces the parameter setting process of the traditional PID relying on artificial experience, overcomes the control deviation caused by the simplification of the traditional model, and reduces the complexity and time consumption of parameter configuration; after deployment, the real-time state vector is constructed by real-time acquisition of pressure information, and the opening degree adjustment instruction is directly output by inputting the control agent; the real-time state feedback and the dynamic decision mechanism of the control agent can autonomously cope with the airflow characteristic change or external interference compared with the PID with fixed parameters, without the need for manual re-parameter setting, effectively reducing overshoot, accelerating response speed, suppressing oscillation, and improving adaptive ability and operation stability. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The adaptive control method of the vacuum regulating valve provided by the embodiments of the present application has a logic flowchart;

[0020] Figure 2 A structural schematic diagram of an adaptive control device of a vacuum regulating valve provided by an embodiment of the present application is shown in the figure.

[0021] Figure 3 A structural schematic diagram of an adaptive control device of a vacuum regulating valve provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0022] The present application provides an adaptive control method, device and equipment of a vacuum regulating valve and a storage medium. In the present application, the terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-mentioned figures are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the term "includes" or "has" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] For the sake of understanding, the specific flow of an embodiment of the present application is described below. Please refer to Figure 1 An embodiment of the adaptive control method of a vacuum regulating valve in the present application includes:

[0024] 101. Obtain control requirement parameters and hardware configuration parameters of a vacuum regulating valve.

[0025] In this embodiment, by obtaining the control requirement parameters and the hardware configuration parameters, actual scene basis is provided for subsequent mathematical modeling, control agent training and actual deployment, so as to avoid that the control scheme is separated from engineering application.

[0026] 102. Based on the hardware configuration parameters and the ideal gas state equation, a mathematical model of the vacuum regulating valve is constructed.

[0027] In this embodiment, the constructed mathematical model establishes the quantitative correlation between the control action (motor step number), flow change and pressure change, and provides a simulation environment for control agent training.

[0028] 103. Define a state vector, and design a reward rate function and a round termination condition based on the control requirement parameters.

[0029] In the embodiment, the state vector is configured to control the perception input of the agent, the reward rate function and the round termination condition jointly constitute a learning rule of the agent, and the agent is guided to learn in the direction of achieving accurate pressure control and safe operation through predefinition and design.

[0030] 104. Constructing a reinforcement learning framework based on a deep deterministic policy gradient algorithm;

[0031] In the embodiment, the constructed reinforcement learning framework is used as a decision-making subject of the agent, so that the agent can autonomously output an optimal control action according to an input state vector; the deep deterministic policy gradient algorithm (DDPG algorithm) is combined with the nonlinear pressure control scene of the vacuum regulating valve, online self-tuning of parameters is realized, and manual intervention is not required; the DDPG algorithm can process the nonlinear scene of continuous pressure change and continuous motor step adjustment, thereby effectively avoiding system overshoot, overshoot and oscillation.

[0032] 105. The constructed mathematical model is used as a simulation environment, the reinforcement learning framework is run for training according to the defined state vector, the designed reward rate function and the round termination condition, and the agent is generated and deployed when a preset iteration stop condition is met.

[0033] In the embodiment, the agent masters the pressure control ability through simulation training, that is, by connecting the mathematical model to the reinforcement learning framework.

[0034] 106. Real-time pressure information is acquired to construct a real-time state vector, the real-time state vector is input to the agent, and the opening degree of the vacuum regulating valve is adjusted based on the output result of the agent.

[0035] The application discloses an adaptive control method for a vacuum regulating valve. A reinforcement learning framework is constructed based on a deep deterministic policy gradient algorithm, a mathematical model is used as a simulation environment for training, and an agent is automatically generated. During the training process, the strategy is dynamically optimized through a reward rate function and a round termination condition, which replaces the parameter setting process of a traditional PID relying on manual experience, overcomes the control deviation caused by simplification of a traditional model, and reduces the complexity and time consumption of parameter configuration. After deployment, real-time pressure information is collected to construct a real-time state vector, which is input to the agent to directly output an opening degree adjustment instruction. Real-time state feedback and a dynamic decision-making mechanism of the agent can autonomously cope with changes in airflow characteristics or external interference without the need for manual parameter resetting, effectively reducing overshoot, speeding up response, suppressing oscillation, and improving adaptive ability and operation stability.

[0036] In the embodiment, the control demand parameters and the hardware configuration parameters of the vacuum regulating valve are acquired, including:

[0037] 201、obtaining control requirement parameters of the vacuum regulating valve, the control requirement parameters including a pressure control range, a pressure control precision and a target cavity pressure;

[0038] In the embodiment, the pressure control range is set to 0-500 based on the safety threshold of the process cavity and process requirements, the pressure control precision is set to a pressure error ≤ 0.2% of the target pressure, and the target cavity pressure is dynamically set by a user according to a specific process step; the control requirement parameters are used to clearly define the core requirements of the process on pressure control, to delimit control targets and performance indicators, and to avoid that the trained control agent does not meet the actual application requirements.

[0039] 202、obtaining hardware configuration parameters of the vacuum regulating valve, the hardware configuration parameters including a valve plate diameter, a cavity volume and a gas type;

[0040] 203、determining a molar mass and a gas constant based on the gas type;

[0041] In the embodiment, the valve plate diameter (d) is a core hardware parameter of the vacuum regulating valve, directly affects the calculation of the gas flow area (A), and needs to be obtained by measurement or equipment manual; the cavity volume (V) is the internal volume of the process cavity, is a key parameter of the pressure change rate formula, and needs to be determined by geometric calculation or actual measurement; the semiconductor process commonly used gases include nitrogen, argon, oxygen and the like, the molar mass (M) and the gas constant (R) of different gases are different, and need to be determined according to the actual process gas by querying a standard data table or by experimental test; the mathematical model constructed based on the actual hardware parameters can accurately reflect the device characteristics, so that the control scheme trained by the control agent is more suitable for the actual device, and the control effect after deployment is improved. In the embodiment of the application, the mathematical model of the vacuum regulating valve is constructed based on the hardware configuration parameters and the ideal gas state equation, including: 301、deduced based on the ideal gas state equation, a pressure change rate formula for representing the relationship between the gas inflow, the gas outflow and the pressure change;

[0042] In the embodiment, the pressure change rate formula is expressed as:

[0043]

[0044] In the embodiment, the pressure change rate formula is expressed as:

[0045] ;

[0046] wherein, is the gas inflow, is the gas outflow, is the vacuum cavity volume, ​​Molar mass of the gas flowing through the vacuum regulating valve, Gas constant, Temperature, in practical applications, temperature Can be regarded as constant, therefore, the rate of change of pressure Mainly by the amount of gas inflow And the amount of gas outflow Determined.

[0047] 302, introduce calibration constant, build the mapping relationship between motor step and the rate of change of pressure, derived step pressure correlation formula;

[0048] In this embodiment, the opening angle of the valve plate of the vacuum regulating valve is controlled by the rotation step number of the stepper motor, the rotation step number More, the opening angle of the valve plate The larger, the gas flow area A is larger; through experimental calibration, the motor step number And the rate of change of pressure Proportional relationship, that is, the step pressure correlation formula:

[0049] ;

[0050] Where, Calibration constant, related to motor step angle, valve plate transmission mechanism, cavity characteristics, need to be calibrated by no-load test, such as adjusting the motor step number, record the rate of change of pressure, fitting to get The value; the role of the step pressure correlation formula is to convert the abstract pressure regulation demand into specific motor control instruction, to provide clear direction for the action output of the control agent.

[0051] 303, based on the pressure change rate formula and the step pressure correlation formula, build a mathematical model of the vacuum regulating valve, the mathematical model is used to calculate the amount of gas outflow;

[0052] In this embodiment, the calculation formula of the amount of gas outflow As follows:

[0053] ;

[0054] In the formula, The butterfly valve flow coefficient is provided by the valve manufacturer or measured by experiment; The gas flow area, The pressure before the valve of the vacuum control valve, The pressure after the valve of the vacuum control valve.

[0055] The calculation formula of the gas flow area :

[0056] ;

[0057] In the formula, The diameter of the valve plate of the vacuum regulating valve. This refers to the opening and closing angle of the vacuum regulating valve. With motor steps The relationship is determined by the step angle and transmission ratio of the stepper motor;

[0058] After integration, the mathematical model can be based on the number of motor steps. Calculate the circulation area Gas outflow rate Then, the pressure change is calculated using the pressure change rate formula. Ultimately, the real-time cavity pressure is obtained, achieving a complete simulation between action and pressure.

[0059] In this embodiment, by integrating the classical thermodynamic model with the hardware characteristics of the vacuum regulating valve, the problem of the disconnect between control actions and pressure changes in traditional modeling is solved. There is no need to build a complex nonlinear accurate model. By using a modeling method that correlates the ideal gas equation of state with hardware characteristics, both modeling accuracy and engineering feasibility are taken into account, thus adapting to the nonlinear pressure changes of the process cavity. The constructed mathematical model can accurately simulate the chain reaction between actions, flow rate and pressure, so that the control strategy learned by the control agent in the simulation can be directly transferred to the actual equipment, reducing the deviation between training and deployment.

[0060] In this embodiment of the invention, defining the state vector and designing the rate-of-return function and round termination condition based on the control requirement parameters includes:

[0061] 401. Define a state vector, which includes pressure error, pressure error integral and current cavity pressure, wherein the pressure error is calculated based on the current cavity pressure and the target cavity pressure;

[0062] In this embodiment, the state vector Its composition is:

[0063]

[0064]

[0065] ;

[0066] in, Pressure error reflects the instantaneous deviation between the current cavity pressure and the target cavity pressure; It is the integral of the error, that is, the accumulated pressure error over a period of time, to avoid the control agent from making frequent adjustments due to short-term fluctuations; The current cavity pressure is used to determine whether the safety boundary is touched to avoid pressure out of range; represents each moment, T is the matrix transpose, and represents the vector format; The target cavity pressure set by the user.

[0067] 402. Design the reward function according to the pressure error, the pressure control range and the pressure control accuracy;

[0068] In this embodiment, the setting logic of the reward function is to encourage the control agent to maintain high-precision pressure control by positive rewards, to punish slight deviations by small negative rewards, and to strongly avoid safety risks by large negative rewards. Through the difference between rewards and punishments, the control agent forms the behavior logic of prioritizing safety and then pursuing accuracy.

[0069] The expression form of the reward function is:

[0070] ;

[0071] When the pressure error is less than 0.2% of the target pressure, a positive reward is given, otherwise a negative reward is given. In addition, when the current cavity pressure is not within the pressure control range, a larger punishment reward is applied.

[0072] 403. Set the round termination condition according to the current cavity pressure and the pressure control accuracy;

[0073] In this embodiment, the round termination condition is used to avoid invalid simulation data caused by pressure out of range, and to simulate the emergency protection logic of the actual device to ensure that the trained control agent has a safety control consciousness. The specific form of the round termination condition is:

[0074] ;

[0075] When the current cavity pressure is less than 0 or greater than 500, , the system sends a round termination signal; if the current round training is terminated, record the cumulative reward of this round. Since it contains a large negative reward, the cumulative reward will be significantly reduced. The control agent learns from the experience of this round that it is important to avoid pressure out of range, and will actively avoid such behavior in subsequent training.

[0076] In the embodiment, the termination condition is used as a hard constraint to directly terminate the current training round, thereby avoiding invalid training; the reward rate function is used as a soft guide to guide the control agent to actively learn safe behavior through a reward and punishment mechanism, thereby reducing invalid exploration and shortening the training period; the termination condition and the reward rate function are combined to form a dual safety mechanism of passive stop loss and active avoidance, specifically, through the dual mechanism of large negative reward and termination condition, the control agent establishes a safety-first awareness in the early stage of training, and after deployment, can effectively avoid system pressure out of range, thereby protecting equipment and process safety.

[0077] In the embodiment, the deep deterministic policy gradient algorithm is used to construct a reinforcement learning framework, which includes:

[0078] 501. An Actor network and a Critic network are constructed, the input end of the Actor network is configured to receive a state vector, the input end of the Critic network is configured to receive a state vector and a control action output by the Actor network, and the action value output by the Critic network is configured to provide a gradient basis for parameter optimization of the Actor network;

[0079] In the embodiment, the Actor network is used as an action executor to output an optimal control action according to a current state; the network structure of the Actor network includes: an input layer (3 neurons corresponding to 3 components of the state vector , then connected to a hidden layer (2-3 layers, 64-128 neurons in each layer, and the activation function uses a ReLU function), and finally connected to an output layer (1 neuron corresponding to the number of steps of the stepper motor );

[0080] The Critic network is used as an action evaluator to quantify the advantages and disadvantages of the action output by the Actor network, and provide a judgment basis for parameter optimization of the Actor network; the network structure of the Critic network specifically includes: an input layer (4 neurons, 3 corresponding to the state vector , and 1 corresponding to the action output by the Actor ), then connected to a hidden layer (2-3 layers, 64-128 neurons in each layer, and the activation function uses a ReLU function), and finally connected to an output layer (1 neuron corresponding to the action value ).

[0081] 502. An experience replay pool is configured, and the experience replay pool is configured to store experience data groups generated in the training process;

[0082] In the embodiment, after the Actor network outputs an action , the system generates an experience data group of "state-action-reward-next state" ; the experience replay pool is used to store experience data sets in the training process, avoid training shock caused by data correlation, and improve the generalization ability of the control agent; small batches of data are randomly sampled from the experience replay pool during training ) are represented as , as training data of online networks, wherein may be , , , …

[0083] 503, configure the parameter update rule of the Actor network and the Critic network to complete the construction of the reinforcement learning framework, the Critic network realizes network parameter update by minimizing the loss function, and the Actor network completes network parameter update based on the action value gradient output by the Critic network;

[0084] In this embodiment, by establishing an iterative optimization mechanism of the double network, the Actor network can continuously output better actions, the Critic network can continuously improve the judgment accuracy, and finally a stable control strategy is formed; each network includes two versions of online network and target network, and the parameters of the target network lag behind the online network update;

[0085] The update rule of the Critic network is: , and the specific process is explained as:

[0086] The control agent performs the first step action, obtains the immediate reward , and observes the next state , adopts the Actor target network , and outputs the next action based on , wherein is the parameter of the Actor target network; the Critic target network is adopted, and the future action value is calculated based on , wherein is the parameter of the Critic target network, and finally the target action value is calculated based on the update rule of the Critic network, wherein is a discount factor, generally taking a value range of 0~1, used to balance the weight of immediate reward and future reward, the closer to 1, the more the control agent values the future reward, and the closer to 0, the more the control agent values the immediate reward;

[0087] The Critic online network predicts the first Action value of step By minimizing the loss function: The parameters of the Critic online network are updated using gradient descent The prediction of the Critic online network is increasingly close to the target action value ; wherein, is the average coefficient of the small batch samples, is the mean square error term of a single sample;

[0088] The Actor network update rule is: based on the action value output by the Critic online network, calculate the value gradient : The specific process is explained as follows:

[0089] Collect samples from the experience replay pool, for each sample , output the action using the Actor online network; calculate the action using the Critic online network The gradient of to to clarify the optimization direction of the action; calculate the gradient of the parameters of the Actor online network to the action to clarify the adjustment mode of the parameters; multiply the gradient of to and the gradient of the parameters to the action , and then take the average of the B samples to get the value gradient ; finally, update the parameters of the Actor online network through gradient ascent , so that the parameter update of the Actor online network is in the direction of maximizing the action value, that is, the action output by the Actor next time can obtain a higher value.

[0090] In this embodiment, the control action (motor step number) of the vacuum regulating valve and the pressure change are both continuous variables, so it constitutes a continuous action space control problem; since the DDPG algorithm is a reinforcement learning algorithm specially designed for continuous action space, compared with discrete action space algorithms (such as DQN), it is more suitable for this application scenario; the Actor-Critic double network architecture forms a closed loop mechanism of execution, judgment and optimization, so that the control agent can continuously iterate and optimize the control action, gradually approaching the optimal strategy, thereby continuously improving the pressure control precision.

[0091] In the embodiment of the present application, the constructed mathematical model is configured as a simulation environment, and the reinforcement learning framework is run according to the defined state vector, the designed return rate function and the round termination condition to perform training, and when a preset iteration stop condition is met, a control agent is generated and deployed, including:

[0092] 601, configure the constructed mathematical model as a simulation environment, and run the reinforcement learning framework according to the defined state vector, the designed return rate function and the round termination condition to perform multi-round training on the reinforcement learning framework;

[0093] In the embodiment, before training the reinforcement learning framework, training initialization is performed. Specifically, a pressure value in the range of 0-500 is randomly selected as an initial pressure, and a pressure value in the range of 0-500 is randomly selected as a target pressure. Then, the parameters of the Actor and Critic networks are initialized, and the experience replay pool is emptied.

[0094] 602, in the process of each training round, the sum of the reward values corresponding to each training step in the training round is calculated in real time to obtain the cumulative reward corresponding to each training round;

[0095] In the embodiment, a training round refers to a complete process from the initial pressure to the target pressure (error ≤0.2%) or triggering the round termination signal condition through the control agent adjusting multiple times. A training round is composed of a series of consecutive training steps. In the execution process of a single training round, the immediate reward value corresponding to each training step in the training round is recorded in real time , the reward values of all training steps are summed up to obtain the total sum, which is the cumulative reward corresponding to the training round. The cumulative reward is used to evaluate the training effect of the training round.

[0096] A training round is a complete pressure control process of the control agent from a specific initial pressure to a target pressure. Starting from a specific initial pressure (selected by random uniform sampling) of the process cavity, the control agent adjusts the pressure by outputting actions multiple times until any of the following conditions is met:

[0097] Forward termination condition: when the pressure error between the positive cavity pressure and the target pressure is ≤0.2%, and the accuracy is maintained for 5 consecutive training steps;

[0098] Abnormal termination condition: triggering the round termination signal, such as the safety boundary of the cavity pressure <0 or >500, or the number of training steps exceeding the preset upper limit of 200 steps.

[0099] 603、when the number of training rounds reaches 4000 rounds or the average cumulative reward of 100 consecutive rounds is greater than or equal to 800, a control agent is generated;

[0100] In this embodiment, after any preset iteration stopping condition is met, the training is stopped, the parameters of the current Actor and Critic networks are saved, and a deployable control agent is formed.

[0101] 604、the performance of the generated control agent is verified based on a preset test group, and if all test groups meet the preset verification indicators, the control agent is deployed;

[0102] In this embodiment, the generalization ability and stability of the control agent are verified by independent test groups to ensure that it meets the actual process requirements, thereby avoiding the risks caused by direct deployment.

[0103] Specifically, 3-5 groups of target pressures (covering a value range of 0-500, such as 100, 250, 400, and 500) are randomly selected, and each target pressure corresponds to 3 initial pressure conditions (such as target pressure 100 corresponding to initial pressures 50, 150, and 200), forming 9-15 test cases;

[0104] If the error of the stable pressure control precision is less than or equal to 0.2%, and there is no overshoot phenomenon (overshoot amount is less than or equal to 5%), no sustained oscillation phenomenon (oscillation times are less than or equal to 3), and no round termination condition is triggered, it is determined that the test group meets the verification indicators; when all test cases meet the verification indicators, the model file of the control agent is integrated into the controller of the vacuum regulating valve to complete the software deployment.

[0105] In the embodiment of the application, the real-time pressure information is acquired to construct a real-time state vector, the real-time state vector is input into the control agent, and the opening of the vacuum regulating valve is adjusted based on the output result of the control agent, which comprises:

[0106] 701, collecting real-time pressure information, the real-time pressure information comprising real-time cavity pressure;

[0107] In this embodiment, a pressure sensor is used to acquire real-time pressure information at a pressure monitoring point of the process cavity, and the real-time pressure information is close to the outlet of the regulating valve to ensure that the measured value can reflect the real pressure of the cavity.

[0108] 702, calculating a real-time pressure error based on the real-time cavity pressure and the target cavity pressure, and time-integrating the real-time pressure error to obtain a real-time pressure error integral;

[0109] 703, constructing a real-time state vector based on the real-time cavity pressure, the real-time pressure error, and the real-time pressure error integral, and inputting the real-time state vector into the control agent.

[0110] In the embodiment, the collected original pressure data is converted into a state vector recognizable by the control agent to provide a basis for real-time decision-making. The constructed real-time state vector is consistent with the state vector format in the training stage, ensuring that the control agent can be directly used without adaptation.

[0111] 704、adjusting the number of steps of the stepper motor based on the output result of the control agent, and then adjusting the opening and closing angle of the valve plate to change the gas flow area, thereby adjusting the gas outflow, and finally stabilizing the real-time cavity pressure to the target cavity pressure;

[0112] In the embodiment, the controller connected to the vacuum regulating valve converts the decision of the control agent (motor steps) into a corresponding stepper motor driving signal and transmits it to the stepper motor driver. The stepper motor rotates the corresponding number of steps according to the driving signal, drives the valve plate to rotate through the transmission mechanism, thereby changing the opening and closing angle of the valve plate. The change in the opening and closing angle of the valve plate causes the gas flow area to change, thereby causing the gas outflow to change, achieving dynamic regulation of the cavity pressure. By adjusting the gas outflow to balance the gas inflow and outflow, the cavity pressure is finally stabilized at the target cavity pressure, thereby achieving adaptive control. When the process gas inflow changes or there is external interference, the real-time cavity pressure will deviate from the target cavity pressure. At this time, the control agent perceives this deviation through the real-time state vector and quickly adjusts the motor steps to rebalance the gas outflow and inflow to maintain pressure stability.

[0113] The adaptive control method of the vacuum regulating valve in the embodiment is described above, and the adaptive control device of the vacuum regulating valve in the embodiment is described below. Please refer to Figure 2 The adaptive control device of the vacuum regulating valve in the embodiment includes one embodiment:

[0114] The acquisition module 801 is configured to acquire control demand parameters and hardware configuration parameters of the vacuum regulating valve.

[0115] The first construction module 802 is configured to construct a mathematical model of the vacuum regulating valve based on the hardware configuration parameters and the ideal gas state equation.

[0116] The definition module 803 is configured to define a state vector, and design a reward rate function and a round termination condition based on the control demand parameters.

[0117] The second construction module 804 is configured to construct a reinforcement learning framework based on a deep deterministic policy gradient algorithm.

[0118] The training module 805 is configured to use the constructed mathematical model as a simulation environment, run the reinforcement learning framework according to the defined state vector, the designed return rate function and the round termination condition to perform training, and generate and deploy a control agent when a preset iteration stop condition is met.

[0119] The adjusting module 806 is configured to acquire real-time pressure information to construct a real-time state vector, input the real-time state vector into the control agent, and adjust the opening degree of the vacuum regulating valve based on an output result of the control agent.

[0120] Based on the same idea as the method in the above embodiment, the device provided in the present application can implement the method of the above embodiment.

[0121] The above Figure 2 The adaptive control device for the vacuum regulating valve in the embodiment of the present application is described in detail from the perspective of a modular functional entity, and the adaptive control device for the vacuum regulating valve in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0122] Figure 3 FIG. 9 is a structural schematic diagram of an adaptive control device for a vacuum regulating valve provided in the embodiment of the present application. The adaptive control device 900 for the vacuum regulating valve can be quite different due to different configurations or performances, and can include one or more processors (central processing units, CPU) 910 (for example, one or more processors) and a memory 920, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 933 or data 932. The memory 920 and the storage media 930 can be temporary storage or persistent storage. The programs stored in the storage media 930 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the adaptive control device 900 for the vacuum regulating valve. Furthermore, the processor 910 can be configured to communicate with the storage media 930, execute a series of instruction operations in the storage media 930 on the adaptive control device 900 for the vacuum regulating valve, to implement the steps of the adaptive control method for the vacuum regulating valve provided in each method embodiment.

[0123] The adaptive control device 900 for the vacuum regulating valve can also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input and output interfaces 960, and / or one or more operating systems 931, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the adaptive control device 900 for the vacuum regulating valve can also include other appropriate components not shown in the figure, such as bus lines, power supplies, power supply circuits, clock circuits, arithmetic logic unit circuits, etc. Figure 3The adaptive control device structure of the vacuum regulating valve shown does not constitute a limitation on the adaptive control device of the vacuum regulating valve, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0124] The application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium, or a volatile computer readable storage medium, and the computer readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the steps of the adaptive control method of the vacuum regulating valve.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system or device, unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0126] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0127] Finally, it should be noted that: the above only describes the preferred examples of the present application, and does not limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, and for those skilled in the art, the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of adaptive control of a vacuum regulating valve, characterized in that, The method comprises the following steps: obtaining control requirement parameters and hardware configuration parameters of the vacuum regulating valve; specifically, obtaining the control requirement parameters of the vacuum regulating valve, the control requirement parameters including a pressure control range, a pressure control precision and a target cavity pressure; obtaining the hardware configuration parameters of the vacuum regulating valve, the hardware configuration parameters including a valve plate diameter, a cavity volume and a gas type; determining a molar mass and a gas constant based on the gas type; based on the hardware configuration parameters and an ideal gas state equation, a mathematical model of the vacuum regulating valve is constructed; specifically, based on the ideal gas state equation, the molar mass, the gas constant and the cavity volume, a pressure change rate formula for characterizing the relationship between the gas inflow, the gas outflow and the pressure change is derived; a calibration constant is introduced to construct a mapping relationship between the motor step number and the pressure change rate, and a step pressure correlation formula is derived; based on the valve plate diameter, the pressure change rate formula and the step pressure correlation formula, a mathematical model of the vacuum regulating valve is constructed, and the mathematical model is used to calculate the gas outflow; a state vector is defined, and a reward rate function and a round termination condition are designed based on the control requirement parameters; a reinforcement learning framework is constructed based on a deep deterministic policy gradient algorithm; the constructed mathematical model is used as a simulation environment, and the reinforcement learning framework is run to train according to the defined state vector, the designed reward rate function and the round termination condition, and a control agent is generated and deployed when a preset iteration stop condition is met; real-time pressure information is obtained to construct a real-time state vector, the real-time state vector is input into the control agent, and the opening degree of the vacuum regulating valve is adjusted based on the output result of the control agent.

2. The adaptive control method of a vacuum regulating valve according to claim 1, characterized by, The definition of the state vector and the design of the reward rate function and the round termination condition based on the control requirement parameters comprise: the state vector is defined, and the state vector includes a pressure error, an integral of the pressure error and a current cavity pressure, the pressure error being calculated based on the current cavity pressure and the target cavity pressure; the reward rate function is designed according to the pressure error, the pressure control range and the pressure control precision; the round termination condition is set according to the current cavity pressure and the pressure control precision.

3. The adaptive control method of a vacuum regulating valve according to claim 1, characterized by, The reinforcement learning framework is constructed based on the deep deterministic policy gradient algorithm, which comprises: an Actor network and a Critic network are constructed, the input end of the Actor network is configured to receive the state vector, the input end of the Critic network is configured to receive the state vector and the control action output by the Actor network, and the action value output by the Critic network is configured to provide a gradient basis for parameter optimization of the Actor network; an experience replay pool is configured, the experience replay pool is configured to store experience data groups generated in the training process; the parameter update rules of the Actor network and the Critic network are configured to complete the construction of the reinforcement learning framework, the Critic network updates the network parameters by minimizing a loss function, and the Actor network updates the network parameters based on the action value gradient output by the Critic network.

4. The adaptive control method of a vacuum regulating valve according to claim 1, characterized by, The constructed mathematical model is used as a simulation environment, and the reinforcement learning framework is run for training according to the defined state vector, the designed return rate function, and the round termination condition. When a preset iteration stop condition is met, a control agent is generated and deployed, including: The constructed mathematical model is configured as a simulation environment, and the reinforcement learning framework is run for training according to the defined state vector, the designed return rate function, and the round termination condition. In the process of each training round, the sum of reward values corresponding to each training step in the training round is calculated in real time to obtain the cumulative reward corresponding to each training round. When the number of training rounds reaches 4000 rounds or the average cumulative reward of 100 consecutive rounds is greater than or equal to 800, a control agent is generated. The performance of the generated control agent is verified based on a preset test group. If all test groups meet the preset verification indicators, the control agent is deployed.

5. The adaptive control method of a vacuum regulating valve according to claim 2, characterized by, The real-time pressure information is obtained to construct a real-time state vector, and the real-time state vector is input into the control agent. The opening degree of the vacuum regulating valve is adjusted based on the output result of the control agent, including: Real-time pressure information is collected, including real-time cavity pressure. Based on the real-time cavity pressure and the target cavity pressure, a real-time pressure error is calculated, and the real-time pressure error is time-integrated to obtain a real-time pressure error integral. Based on the real-time cavity pressure, the real-time pressure error, and the real-time pressure error integral, a real-time state vector is constructed, and the real-time state vector is input into the control agent. Based on the output result of the control agent, the number of steps of the stepper motor is adjusted, and then the valve plate opening angle is adjusted to change the gas flow area, thereby adjusting the gas outflow, and finally stabilizing the real-time cavity pressure to the target cavity pressure.

6. An adaptive control device for a vacuum regulating valve, characterized by including: The acquisition module is configured to acquire control demand parameters and hardware configuration parameters of the vacuum regulating valve. Specifically, the control demand parameters of the vacuum regulating valve are acquired, including the pressure control range, the pressure control accuracy, and the target cavity pressure. The hardware configuration parameters of the vacuum regulating valve are acquired, including the valve plate diameter, the cavity volume, and the gas type. The molar mass and the gas constant are determined based on the gas type. The first construction module is configured to construct a mathematical model of the vacuum regulating valve based on the hardware configuration parameters and the ideal gas state equation. Specifically, based on the ideal gas state equation, the molar mass, the gas constant, and the cavity volume, a pressure change rate formula is derived to represent the relationship between the gas inflow, the gas outflow, and the pressure change. A calibration constant is introduced to construct a mapping relationship between the motor step number and the pressure change rate, and a step number-pressure correlation formula is derived. Based on the valve plate diameter, the pressure change rate formula, and the step number-pressure correlation formula, a mathematical model of the vacuum regulating valve is constructed, which is used to calculate the gas outflow. The definition module is configured to define a state vector, and design a return rate function and a round termination condition based on the control demand parameters. A second construction module is configured to construct a reinforcement learning framework based on a deep deterministic policy gradient algorithm; A training module is configured to use the constructed mathematical model as a simulation environment, run the reinforcement learning framework to perform training according to the defined state vector, the designed return rate function and the round termination condition, and generate a control agent and deploy the control agent when a preset iteration stop condition is met; An adjustment module is configured to acquire real-time pressure information to construct a real-time state vector, input the real-time state vector into the control agent, and adjust the opening degree of the vacuum regulating valve based on an output result of the control agent.

7. An adaptive control device for a vacuum regulating valve, characterized by The adaptive control device of the vacuum regulating valve comprises a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory to enable the adaptive control device of the vacuum regulating valve to perform the steps of the adaptive control method of the vacuum regulating valve according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: The instructions are executed by the processor to implement the steps of the adaptive control method of the vacuum regulating valve according to any one of claims 1-5.

Images