Method for obtaining PID parameter setting intelligent agent and related device

Abstract

The invention discloses a method for obtaining a PID parameter setting agent and a related device, and relates to the technical field of automatic control. The method comprises the following steps: constructing a dynamic model training ground based on an actual production environment, priori knowledge of a target control system and a target control requirement; determining a target training hyper-parameter based on the target control demand and a preset mapping rule base; determining a target control track based on the target control demand; determining a target hyper-parameter combination based on the initial agent, the dynamic model training ground, a preset training evaluation index, a preset search strategy and a preset hyper-parameter search space; and in a dynamic model training ground, training the initial agent based on a multi-modal control target, a target control track, a target hyper-parameter combination, a target training hyper-parameter and a target reinforcement learning algorithm to obtain a PID parameter setting agent.

Description

Technical Field

[0001] This application relates to the field of automation control technology, and in particular to a method and related apparatus for obtaining PID parameter tuning agents. Background Technology

[0002] PID control technology (Proportional-Integral-Derivative control) is a core cornerstone of industrial automation. With its advantages of simple structure, convenient operation, and stable performance, PID controllers are widely used in process industry control, discrete industrial equipment control, transportation vehicles, and aerospace, playing an irreplaceable role in ensuring the safe and efficient operation of systems.

[0003] In industrial automation control systems, the control effect of a PID controller is highly dependent on the matching degree of its proportional, integral, and derivative parameters (i.e., PID control parameters). Inappropriate parameter selection can significantly degrade the system's dynamic performance, resulting in the measured value failing to track the setpoint, requiring frequent operator intervention in the control loop. In extreme cases, it can even lead to instability of the controlled object, causing loop output divergence and severely impacting the safety and efficiency of industrial production. Furthermore, the three PID parameters exhibit coupling characteristics, and the matching parameters fluctuate significantly depending on the controlled object, further increasing the technical difficulty of parameter tuning.

[0004] Therefore, how to intelligently, efficiently and accurately tune PID parameters has become one of the urgent technical problems to be solved in the field of automation control technology. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method for obtaining a PID parameter tuning agent. Through the PID parameter tuning agent in this application, PID parameters can be tuned efficiently and accurately.

[0006] The embodiments of this application disclose the following technical solutions: The first aspect of this application provides a method for obtaining a PID parameter tuning agent, including: Based on the actual production environment, prior knowledge of the target control system, and target control requirements, a dynamic model training field is constructed. Based on the target control requirements and a preset mapping rule base, target training hyperparameters are determined; the mapping rule base includes the correspondence between control requirements and training hyperparameters; the target training hyperparameters include learning rate, discount factor, and exploration rate. Based on the target control requirements, a target control trajectory is determined; the target control trajectory is a preset control trajectory for achieving the multimodal control target corresponding to the target control requirements; the target control trajectory is used as a control trajectory for post-event experience playback. Based on the initial agent, the dynamic model training field, the preset training evaluation index, the preset search strategy, and the preset hyperparameter search space, the target hyperparameter combination is determined. In the dynamic model training field, the initial agent is trained based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a PID parameter tuning agent.

[0007] In one optional implementation, the construction of a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements includes: Based on the prior knowledge of the target control system and the preset system model set, the target system model is determined; the system model set includes a liquid level loop system model, a pressure loop system model, a temperature loop system model, and a flow rate loop system model. Based on the actual production environment, a disturbance model is determined; the disturbance model is generated based on impulse disturbances and step disturbances. Based on the target control requirements, determine the control evaluation function; Based on the target system model, the disturbance model, and the control evaluation function, the dynamic model training field is constructed.

[0008] In one optional implementation, determining the control evaluation function based on the target control requirements includes: Based on the target control requirements, the target quantitative analysis function, the PID parameter compliance constraint function, and the controller output smoothness constraint function are determined; the target quantitative analysis function is one of the quantitative analysis functions corresponding to the disturbance rejection system, the quantitative analysis function corresponding to the SV tracking system, and the quantitative analysis function corresponding to the integral system. The control evaluation function is determined based on the target quantitative analysis function, the PID parameter compliance constraint function, the controller output smoothness constraint function, and a preset weight coefficient combination.

[0009] In one optional implementation, the preset training evaluation metric is a comprehensive control performance score: ; Where S is the overall control performance score, and J is the control evaluation function value under the current hyperparameter combination. The baseline J value under empirical parameters; This represents the average fluctuation of the PID parameters. To allow the maximum fluctuation; To control the precision weight, To control the smoothness weights.

[0010] In one alternative implementation, the default search strategy is a Bayesian optimized search strategy.

[0011] In one optional implementation, the initial agent includes an actor model and a critic model; both the actor model and the critic model include an input layer, a feature extraction layer, an attention layer, a fully connected layer, and an output layer; the input data of the actor model is a time series including operational variables, process variables, and setpoints; the output data of the actor model is the adjustment amount of the PID parameters; the input data of the critic model is the time series and the adjustment amount of the PID parameters; the output data of the critic model is the Q-value of reinforcement learning.

[0012] In one optional implementation, the step of training the initial agent in the dynamic model training field based on the multimodal control objective, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a PID parameter tuning agent includes: In the dynamic model training field, the initial agent is trained based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a reinforcement learning control agent; Based on the dynamic model training field and test dataset, the reinforcement learning control agent is simulated and tested to obtain evaluation results. If the evaluation results meet the evaluation control requirements, then the reinforcement learning control agent will be used as the PID parameter tuning agent. If the evaluation results do not meet the evaluation control requirements, the dynamic model training field is optimized to obtain an optimized dynamic model training field; and based on the optimized dynamic model training field, the PID parameter tuning agent is determined.

[0013] In one alternative implementation, the target reinforcement learning algorithm is one of the TD3 reinforcement learning algorithm and the PPO reinforcement learning algorithm.

[0014] A second aspect of this application provides an apparatus for obtaining a PID parameter tuning agent, comprising: The dynamic model training field construction module is used to build a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements. The target training hyperparameter determination module is used to determine the target training hyperparameters based on the target control requirements and a preset mapping rule base; the mapping rule base includes the correspondence between control requirements and training hyperparameters; the target training hyperparameters include learning rate, discount factor, and exploration rate; The target control trajectory determination module is used to determine the target control trajectory based on the target control requirements; the target control trajectory is a preset control trajectory for achieving the multimodal control target corresponding to the target control requirements; the target control trajectory is used as a control trajectory for post-event experience playback. The target hyperparameter combination determination module is used to determine the target hyperparameter combination based on the initial agent, the dynamic model training field, the preset training evaluation index, the preset search strategy, and the preset hyperparameter search space. The agent training module is used to train the initial agent in the dynamic model training field based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a PID parameter tuning agent.

[0015] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any implementation of the first aspect.

[0016] A fourth aspect of this application provides an electronic device, comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method described in any implementation of the first aspect.

[0017] Compared with the prior art, this application has the following beneficial effects: This application provides a method for obtaining a PID parameter tuning agent, comprising: constructing a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements; determining target training hyperparameters based on target control requirements and a preset mapping rule base; determining the target control trajectory based on target control requirements; determining the target hyperparameter combination based on the initial agent, the dynamic model training field, preset training evaluation indicators, preset search strategies, and preset hyperparameter search space; and training the initial agent in the dynamic model training field based on multimodal control targets, target control trajectories, target hyperparameter combinations, target training hyperparameters, and target reinforcement learning algorithms to obtain a PID parameter tuning agent.

[0018] Because the dynamic model training field in this application fully integrates the characteristics of the actual production environment with the prior knowledge of the target control system, it can accurately replicate the dynamic characteristics of the real control scenario. At the same time, it locks the optimal target hyperparameter combination through a preset mapping rule base and scientific search strategy. Combined with multimodal control objectives and reinforcement learning algorithms, it conducts targeted training on the initial agent, so that the trained PID parameter tuning agent has both the flexibility to adapt to different control requirements and the parameter tuning accuracy based on real scenario data. It effectively avoids the problems of traditional PID parameter tuning relying on human experience, adapting to a single scenario, low parameter iteration efficiency and large error. Therefore, the PID parameter tuning agent in this application can perform PID parameter tuning efficiently and accurately. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a method for obtaining PID parameter tuning agents provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of an initial intelligent agent provided in an embodiment of this application; Figure 3 A flowchart of a reinforcement learning algorithm used in training is provided as an embodiment of this application; Figure 4 This is a schematic diagram of a device for obtaining PID parameter tuning intelligence provided in an embodiment of this application. Detailed Implementation

[0021] As industrial processes become more complex and control performance requirements rise, traditional PID parameter tuning methods can no longer meet the needs of modern industry. Developing intelligent, efficient, and highly adaptable parameter tuning technologies has become an urgent development direction in the field of industrial control.

[0022] Currently, PID parameter tuning mainly involves two major technical solutions: empirical tuning methods and traditional numerical optimization tuning algorithms. All of these solutions have significant limitations and are difficult to adapt to the needs of complex industrial scenarios.

[0023] Empirical tuning methods are commonly used in engineering, including specific techniques such as the Ziegler-Nichols empirical formula method and the attenuation curve method.

[0024] The core drawback of the Ziegler-Nichols empirical formula method is that it is highly dependent on the critical response characteristics of the system. It is only applicable to systems that can reach a critical stable state. When facing complex controlled objects that are nonlinear, significantly time-varying, or whose mathematical models are unknown, the tuning effect is greatly limited and cannot guarantee the system control performance.

[0025] Among them, the attenuation curve method, as a typical representative of empirical tuning methods, involves determining the initial parameters of the PID controller based on the control loop type (such as flow loop or temperature loop) and the open-loop or closed-loop step response characteristic curve. Then, the parameters are repeatedly adjusted and the closed-loop characteristics are tested until the system performance meets the standards. This method has two major problems: first, the tuning cycle is directly related to the loop response time, which can be time-consuming in some operating conditions, making it difficult to meet the demands of high-efficiency production; second, the tuning quality highly depends on the engineering experience and understanding of the loop characteristics of technical personnel, resulting in strong subjectivity and poor consistency in tuning results among different personnel, making it impossible to establish a standardized tuning process.

[0026] Traditional numerical optimization tuning algorithms are mainly model-driven algorithms, such as the internal model method and the magnitude optimization method.

[0027] Among them, the amplitude optimization method based on model identification, proposed by Damir Vrancic in 2001, derives the transfer function of the closed-loop system after adding a PID controller by identifying the mathematical model of the controlled object, and solves the PID parameters with the goal of optimizing the closed-loop amplitude-frequency characteristics. Although this type of algorithm has a certain theoretical rigor, it still has obvious shortcomings: on the one hand, the anti-interference ability of the tuned parameters is weak, making it difficult to cope with the load disturbances and noise interference in complex industrial environments; on the other hand, the optimization objective is singular, focusing only on the optimal amplitude-frequency characteristics, and cannot be targeted to the personalized needs of specific industrial scenarios (such as fast response, high-precision steady-state control, etc.), thus limiting its applicability.

[0028] Therefore, how to intelligently, efficiently and accurately tune PID parameters has become one of the urgent technical problems to be solved in the field of automation control technology.

[0029] Based on this, this application provides a method for obtaining a PID parameter tuning agent, comprising: constructing a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements; determining target training hyperparameters based on target control requirements and a preset mapping rule base; determining the target control trajectory based on target control requirements; determining the target hyperparameter combination based on the initial agent, the dynamic model training field, preset training evaluation indicators, preset search strategies, and preset hyperparameter search space; and training the initial agent in the dynamic model training field based on the multimodal control target, target control trajectory, target hyperparameter combination, target training hyperparameters, and target reinforcement learning algorithm to obtain the PID parameter tuning agent.

[0030] Because the dynamic model training field in this application fully integrates the characteristics of the actual production environment and the prior knowledge of the target control system, it can accurately replicate the dynamic characteristics of real control scenarios. At the same time, it locks the optimal combination of target hyperparameters through a preset mapping rule base and a scientific search strategy. Combined with multimodal control targets and reinforcement learning algorithms, it conducts targeted training on the initial agent, so that the trained PID parameter tuning agent has both the flexibility to adapt to different control requirements and the parameter tuning accuracy based on real scenario data. It effectively avoids the problems of traditional PID parameter tuning relying on human experience, adapting to a single scenario, low parameter iteration efficiency and large error. Therefore, the PID parameter tuning agent in this application can perform PID parameter tuning efficiently and accurately.

[0031] To facilitate understanding of the technical solutions in this application, the technical terms used in this application will be introduced first.

[0032] A PID controller (Proportional-Integral-Differential Controller) is a classic linear closed-loop control algorithm. It processes the deviation signal of the system through proportional, integral, and derivative operations to generate corresponding control quantities to adjust the controlled object, thereby enabling the system output to quickly and stably track the set value and achieve high-precision automatic control.

[0033] Genetic Algorithm (GA) is a class of stochastic search and optimization algorithms that draw on the mechanisms of natural selection and genetic variation evolution in the biological world.

[0034] Magnitude Optimum (MO) is a classic method for tuning PID controller parameters. Its core principle is to adjust parameters such as the proportional coefficient, integral time, and derivative time of the PID controller so that the amplitude-frequency characteristics of the closed-loop control system meet the preset optimal criteria.

[0035] An open-loop system is a type of control system in which the output of the system does not affect the control action.

[0036] A closed-loop system, also known as a feedback control system, is a type of control system in which the output of the system is monitored in real time and fed back to the input to form a closed control loop, thereby correcting the control action.

[0037] A step response is the characteristic curve and process of how the output of a linear time-invariant system changes over time when the input changes abruptly from one constant value to another (i.e., the input is a step signal) under zero initial conditions.

[0038] Model identification refers to the process of selecting and determining an equivalent mathematical model that can accurately reproduce the dynamic and static characteristics of the actual system from a preset model category (such as transfer function model, state-space model, nonlinear regression model, etc.) by collecting the input and output data of the controlled system without relying on the accurate prior mathematical model of the system.

[0039] Early-stop is a regularization strategy and training termination mechanism used in machine learning and reinforcement learning to prevent model overfitting and improve training efficiency.

[0040] The integral of time multiplied by squared error (ITSE) is a type of error integral index used in control engineering to evaluate the dynamic control performance of a system. Its core is to multiply the square of the deviation between the system setpoint and the actual measured value by a time weight and then perform an integral operation to quantify the control accuracy and dynamic quality of the system throughout the entire response process.

[0041] AutoML (Automated Machine Learning) is an algorithmic framework and technical system that can automate the entire process of building machine learning models. Its core goal is to lower the application threshold of machine learning technology and reduce the cost and subjectivity of human intervention.

[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0043] Figure 1 This is a flowchart illustrating a method for obtaining PID parameter tuning agents according to an embodiment of this application. (In conjunction with...) Figure 1 As shown, the method for obtaining the PID parameter tuning agent includes: S101 constructs a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements.

[0044] A target control system refers to a combination of objects or equipment that require specific control strategies to achieve a desired operating state during industrial production processes. Target control systems possess core attributes such as prior knowledge, control objectives (i.e., target control requirements), and system characteristics.

[0045] Prior knowledge includes, but is not limited to, known information such as existing design parameters, operating rules, historical data, and industry application experience of the target control system. Target control requirements include, but are not limited to, stability, response speed, and control accuracy. System characteristics are the inherent attributes exhibited by the target control system during operation, such as dynamic response characteristics, steady-state characteristics, and anti-interference capabilities.

[0046] In one optional implementation, the construction of a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements includes the following steps: A1. Based on the prior knowledge of the target control system and the preset system model set, determine the target system model.

[0047] The system model set is constructed based on pre-defined process industry classification standards, the characteristics of controlled objects, and control scenario requirements. It encompasses a collection of system models that record the core characteristics and dynamic response patterns of various typical controlled systems in the process industry's production process. The system model set includes various system models such as level loop system models, pressure loop system models, temperature loop system models, and flow loop system models.

[0048] For example, the model that best matches the characteristics of the target control system can be selected from a pre-set set of system models based on the prior knowledge of the target control system, and then used as the target system model.

[0049] For example, a model can also be randomly selected from the system model set by default as the target system model; this selection method is suitable for general control strategy training and scenarios where there is no clear controlled object.

[0050] A2. Based on the actual production environment, determine the disturbance model.

[0051] In actual production environments, resisting external disturbances is usually one of the main control objectives. In simulation environments, external disturbances in different scenarios can be simulated by selecting disturbance models, thereby improving the parameter tuning capability of the PID tuning agent in anti-interference environments.

[0052] The disturbance model in this application mainly includes pulse and step disturbances at the input and output ends; such as pulse disturbance at the input end, step disturbance at the input end, pulse disturbance at the output end, and step disturbance at the output end, covering common interference patterns in production.

[0053] Taking the temperature control scenario of a chemical reactor as an example, the target system model first simulates the conventional operating characteristics of the reactor, such as heat conduction and material reaction heat; then, it calibrates the model parameters by combining a large amount of actual industrial field operation data to improve the accuracy of the basic simulation; through the perturbation model, it accurately reproduces the disturbances in the real scenario (such as feed flow fluctuations and random changes in the external ambient temperature); finally, it constructs a simulation environment that is highly close to the real working conditions, providing reliable scenario support for the training of the PID parameter tuning agent (reinforcement learning subject).

[0054] A3. Based on the target control requirements, determine the control evaluation function.

[0055] After determining the target system model and disturbance model, a targeted control evaluation function needs to be constructed based on the specific control requirements of the current target control system (target control requirements). Since different controlled systems have different requirements for control performance, a control evaluation function that matches the actual needs must be designed. This function simultaneously serves as both an evaluation standard for the agent's parameter tuning capability and a reward function for reinforcement learning training.

[0056] In the process of building a high-fidelity dynamic model training ground, it is necessary to closely integrate with the common control requirements in industrial scenarios. For the three core control scenarios of disturbance rejection, setpoint (SV) tracking, and integral system, a comprehensive evaluation rule system should be designed to achieve accurate quantitative evaluation of control effect and avoid dangerous working conditions through safety penalty mechanism, so as to ensure the operational safety of simulation training and subsequent practical application.

[0057] In one optional implementation, determining the control evaluation function based on the target control requirements includes: B1. Based on the target control requirements, determine the target quantitative analysis function, the PID parameter compliance constraint function, and the controller output smoothness constraint function.

[0058] The target quantitative analysis function in this application is one of the following: the quantitative analysis function corresponding to the disturbance rejection system, the quantitative analysis function corresponding to the SV tracking system, and the quantitative analysis function corresponding to the integral system.

[0059] If the target control system is a disturbance rejection system, then the core requirement of the target control system is that when external disturbances occur during the production process (such as fluctuations in raw material feed, changes in ambient temperature, etc.), the system can quickly recover to the set value and remain stable.

[0060] For the performance evaluation of this type of system, three core indicators need to be focused on: First, the cumulative deviation between the actual output value and the set value after the system is disturbed. The smaller the deviation, the better the overall control accuracy of the system. Second, the maximum deviation of the actual output value of the system from the set value after the disturbance occurs. The smaller the deviation, the stronger the stability of the system in the process of resisting disturbance. Third, the time taken for the system to recover to a stable state from being disturbed. The shorter the time, the faster the system response speed.

[0061] In practical engineering applications, the focus weight of the above three core indicators can be adjusted according to specific production needs, such as pressure control scenarios with strict requirements for fluctuation sensitivity or flow control scenarios with high requirements for response speed. This allows for differentiated emphasis on different performance dimensions and ensures that the evaluation results are highly consistent with the actual control needs on site.

[0062] For example, the expression for the quantitative analysis function corresponding to the disturbance rejection system is as follows: ; in, It is the quantitative analysis function value (evaluation function value); e(t) is the systematic error (the deviation between the set value and the actual value). It is the error integral, which reflects the cumulative deviation of the system under disturbance; The maximum error reflects the system's maximum resistance to interference fluctuations. It is the time it takes for the system to recover from the occurrence of the disturbance; , , All are weighting coefficients, which can be adjusted according to actual anti-interference requirements; t is time (independent variable).

[0063] In reinforcement learning, the reward function is: =- That is, the smaller the evaluation function value, the greater the reward, guiding the agent to learn PID parameters that enable the system to quickly stabilize after being disturbed.

[0064] If the target control system is an SV tracking system (setpoint tracking system), the core requirement of the target control system is that when the SV setpoint changes dynamically (such as the temperature rising from one value to another target value, or the flow rate being adjusted from one rate to another), the actual output of the system can accurately and quickly follow the changes in SV, and avoid excessive overshoot.

[0065] For the performance evaluation of this type of system, three core indicators need to be focused on: First, during the SV change process, the cumulative deviation between the actual output of the SV tracking system and the SV; the smaller the deviation, the higher the tracking accuracy. Second, the magnitude by which the actual output exceeds the new SV (overshoot); the smaller the magnitude, the better the stability of the tracking process, avoiding production abnormalities caused by overshoot. Third, the rise time from the start of the SV change to the first time the actual output reaches the new SV; the shorter the time, the faster the tracking response speed.

[0066] In practical applications, the focus on these three aspects can be flexibly adjusted according to the different requirements of on-site production for accuracy, stability, and speed, to adapt to different tracking scenario needs.

[0067] For example, the expression for the quantitative analysis function corresponding to the SV tracking system is as follows: ; in, It is a quantitative analysis function value (evaluation function value). The overshoot is the amount of data that exceeds the set value during the tracking process. The rise time reflects the speed of tracking; , , These are all weighting coefficients, set according to tracking accuracy and speed requirements; It is the error integral, reflecting the cumulative deviation of the system under disturbance; t is time (independent variable).

[0068] Reinforcement learning reward function =- The incentive agent learns to enable the system to output PID parameters that accurately and quickly track the set value.

[0069] If the target control system is an integral system, then the core requirement of the target control system is to eliminate the deviation (steady-state error) in the steady state of the system while avoiding equipment failure caused by integral saturation.

[0070] The performance indicators of an integral system mainly focus on two aspects: first, the deviation between the actual output and the set value when the integral system is in steady state, to ensure that the steady-state error is effectively eliminated; second, the degree to which the controller output approaches the maximum or minimum allowable output range of the device. The closer it is to the limit range, the higher the risk of integral saturation and the worse the evaluation result.

[0071] In practical applications, the evaluation weights of these two aspects can be adjusted according to the strictness of the integral system in eliminating steady-state error and the sensitivity of the equipment output limits, so as to balance control accuracy and equipment safety.

[0072] For example, the expression for the quantitative analysis function corresponding to the integral system is as follows: ; in, This is the quantitative analysis function value (evaluation function value), where u(t) is the controller output. For output limitations, It reflects how close the controller output is to the limit, and is used to avoid integral saturation; It is the error integral, which reflects the cumulative deviation of the system under disturbance; , All are weighting coefficients; t is time (independent variable).

[0073] Reinforcement learning reward function =- This guides the agent to learn PID parameters that can both eliminate steady-state error and prevent integral saturation.

[0074] After determining the control scenario (disturbance suppression system, SV tracking system, or integral system) corresponding to the target control system, the quantitative analysis function corresponding to the target control system can be determined.

[0075] Furthermore, to ensure that the PID parameter tuning results meet the requirements for safe operation of the equipment, and to avoid equipment damage caused by excessive fluctuations in the control process, two additional types of constraint rules (i.e., corresponding penalty mechanisms) need to be set: the PID parameter compliance constraint function and the controller output smoothness constraint function.

[0076] First, the PID parameter compliance constraint function.

[0077] The three core parameters of the PID controller (key parameters used to adjust the proportional, integral, and derivative actions) must meet the actual requirements of the equipment operation. On-site, reasonable parameter ranges will be preset according to the equipment type and the characteristics of the controlled object.

[0078] To ensure that the parameter tuning results meet the basic boundary conditions for safe operation of the equipment and to avoid the risk of control failure or equipment malfunction caused by parameter exceeding limits, PID parameter compliance constraint rules are set.

[0079] The rules for PID parameter compliance constraints are as follows: if any of the three parameters obtained from tuning exceeds the preset reasonable range, a corresponding penalty will be imposed on the tuning result to remind the parameter of compliance risks; if all three parameters are within the reasonable range, there will be no penalty, indicating that the parameters meet the basic requirements for safe operation of the equipment.

[0080] For example, the expression for the PID parameter compliance constraint function is as follows: ; in, It is a penalty value. When any of the PID parameters Kp (Proportional Gain), Ki (Integral Gain), or Kd (Derivative Gain) exceeds the preset reasonable range, the penalty value is 1; otherwise, the penalty value is 0.

[0081] Second, the controller outputs a smoothness constraint function.

[0082] In industrial settings, drastic fluctuations in controller output (such as valve opening and motor speed) can exacerbate equipment wear (such as wear on valve seals and current surges in motors). Therefore, it is necessary to constrain output fluctuations.

[0083] To suppress drastic fluctuations in controller output, reduce mechanical wear and electrical shocks, extend equipment lifespan, and lower maintenance costs, controller output smoothness constraint rules are set.

[0084] The smoothness constraint rule is: the greater the fluctuation amplitude and the higher the fluctuation frequency of the controller output, the stronger the penalty for the tuning result; conversely, the more stable the output, the weaker or even non-penalty the penalty. This guides the PID parameter tuning results to ensure stable equipment operation and reduce maintenance costs.

[0085] For example, the expression for the controller output smoothness constraint function is: ; in, It is a penalty term for the controller output smoothness constraint; It is the instantaneous fluctuation of the controller output signal, which is the difference between the actual output value of the controller at time t and the output value at the previous time; t is time (independent variable).

[0086] B2. Based on the target quantitative analysis function, the PID parameter compliance constraint function, the controller output smoothness constraint function, and the preset weight coefficient combination, the control evaluation function is determined.

[0087] To fully adapt to the complex needs of industrial scenarios, it is necessary to integrate three components: scenario-specific control effect evaluation, PID parameter compliance constraints, and controller output smoothness constraints, to generate a comprehensive evaluation result. During integration, the weight ratio of the three components is adjusted (e.g., increasing the weight of parameter compliance for high-risk equipment and increasing the weight of scenario control effect for precision manufacturing) to achieve a multi-dimensional balance of meeting control objectives, ensuring equipment safety, and reducing operating losses, ultimately forming a complete control evaluation function.

[0088] A4. Based on the target system model, the disturbance model, and the control evaluation function, construct the dynamic model training field.

[0089] After obtaining the target control system, disturbance system, and control evaluation function, the three can be integrated and their parameters adapted to build a closed-loop simulation verification environment and obtain a dynamic model training field.

[0090] The dynamic model training field in this application is a simulation platform for training PID parameter tuning agents, simulating the operating characteristics of controlled objects, external disturbance mechanisms, and control effect evaluation logic in real production scenarios.

[0091] S102, Based on the target control requirements and the preset mapping rule base, determine the target training hyperparameters.

[0092] Training hyperparameters are parameters used to regulate the training process of intelligent algorithms. They do not participate in the iterative updates of the model itself, but play a key role in training efficiency and final control effect, such as learning rate, discount factor, and exploration rate.

[0093] In one alternative implementation, intelligent hyperparameter matching can be achieved by establishing a mapping rule base between control requirements and hyperparameters.

[0094] For example, in SV tracking systems where response speed is extremely important, the discount factor is automatically reduced to make the agent focus more on current rewards and accelerate the learning of response strategies; for integral systems that emphasize long-term stability, the discount factor is increased to make the agent focus on long-term gains and optimize stable control strategies.

[0095] S103, Based on the target control requirements, determine the target control trajectory.

[0096] After determining the target control requirements of the target control system, the target control requirements can be precisely decomposed into multimodal control targets.

[0097] For example, in addition to the basic sub-objective of "stabilizing at the set value", the multimodal control objective can be further subdivided into sub-objectives such as fast response, low overshoot, and strong anti-interference, depending on the scenario requirements.

[0098] After determining the multi-module control objectives, the optimal control trajectory (target control trajectory) can be set for the multi-modal control objectives according to actual needs.

[0099] Taking a typical industrial scenario, a temperature SV tracking system, as an example: During the heating phase, the multimodal control objective focuses on rapid response, and the corresponding optimal control trajectory emphasizes the heating slope and response time, that is, it requires PV to quickly approach SV at a preset rate to shorten the time it takes for the system to reach the target temperature; During the heat preservation phase, the multimodal control objective focuses on low fluctuation stability, and the corresponding optimal control trajectory focuses on the fluctuation range of PV, that is, it requires the deviation of PV near SV to be maintained within a very small range to avoid large temperature fluctuations affecting process quality.

[0100] This scheme embeds multimodal control objectives and target trajectories into the reward function and state space of reinforcement learning. This guides the agent to learn PID parameter tuning strategies from multiple dimensions, enabling it to flexibly adjust parameters according to different control stages and actual scenarios, thereby improving adaptability and tuning accuracy. The target trajectory primarily replaces the target trajectory in HER (Hindsight Experience Replay) technology. Its core includes the complete change trajectory of the controller output value (MV), process variable (PV), and setpoint (SV) within the tuning period. If no specific target trajectory is set, a default target trajectory is automatically enabled. This default trajectory is designed with the core principle of eliminating the deviation between the process variable (PV) and the setpoint (SV) to ensure basic tuning effectiveness.

[0101] Simultaneously, the reinforcement learning reward function is set in conjunction with the preset control objective. This reward function can be designed based on the control evaluation function constructed in step one. Specifically, by configuring the control evaluation function... to All weight values ​​are selected to specifically adapt to the control requirements of different industrial scenarios, forming a scenario-based control evaluation function. Finally, relying on the guiding role of the optimal control trajectory and the constraint role of the scenario-based control evaluation function, a PID parameter tuning reinforcement learning training environment that is adapted to different control scenarios and has specificity is built, providing support for the efficient training of the intelligent agent.

[0102] S104. Based on the initial agent, the dynamic model training field, the preset training evaluation index, the preset search strategy, and the preset hyperparameter search space, determine the target hyperparameter combination.

[0103] In existing technologies, hyperparameter tuning is typically done using a human-experience-driven approach, training PID parameter tuning agents that integrate the TD3 (TwinDelayed Deep Deterministic Policy Gradient), PPO (Proximal Policy Optimization), and HER technologies. This approach struggles to quickly match diverse PID control scenarios such as disturbance rejection, tracking, and integral systems, and suffers from the drawbacks of relying on human experience for parameter tuning and incurring high trial-and-error costs in reinforcement learning. This step introduces Automated Machine Learning (AutoML) reinforcement learning pre-training as the core optimization step. Leveraging its automated search capabilities, it can accurately select the optimal hyperparameter combinations suitable for different PID control scenarios such as disturbance rejection, tracking, and integral systems with only a few training iterations, thereby improving agent training efficiency and algorithm performance and effectively solving the parameter tuning challenges in traditional training modes.

[0104] In one alternative implementation, the initial model is loaded and a small amount of training is performed using an automated machine learning framework to search for the optimal combination of parameters. This includes the following steps: C1, AutoML initialization and scene adaptation configuration before pre-training.

[0105] This step involves configuring the high-fidelity training field (i.e., the dynamic model training field in this application) and customizing the search space for reinforcement learning hyperparameters.

[0106] The steps for configuring a high-fidelity training ground are as follows: After loading a high-fidelity dynamic model training ground consistent with the pre-training stage, enable the corresponding system model and disturbance simulation module according to the actual control scenario (disturbance resistance / tracking / integration system).

[0107] The anti-disturbance system specifically activates typical disturbance forms such as random disturbance mutations and step disturbances, and configures disturbance amplitude, occurrence frequency and duration parameters that match the actual industrial scenario; the SV tracking system configures a dynamic change sequence of setpoints, which includes typical target trajectories such as step, ramp and sine curve, and presets trajectory switching timing and transition smoothing coefficients; the integral system enables the integral saturation simulation module, sets the upper and lower limit parameters of the controller output based on the controller hardware performance and process safety thresholds, and defines the trigger conditions for integral saturation and anti-saturation compensation strategies.

[0108] The customized definition of the hyperparameter search space for reinforcement learning is as follows: based on the characteristics of reinforcement learning algorithms (TD3 / PPO) and HER technology, and combined with the requirements of PID control scenarios, the types of hyperparameters to be optimized and the search range are clearly defined to avoid meaningless parameter traversal and improve search efficiency.

[0109] The hyperparameter search space is defined by classifying algorithm core parameters, HER parameters, general training parameters, and neural network core parameters.

[0110] Among them, the core parameters of TD3 include the learning rate of the policy network, the learning rate of the value network, the soft update coefficient, and the policy update interval; the core parameters of PPO include the learning rate of the policy network, the learning rate of the value network, the trust domain boundary, and the entropy regularization coefficient; the core parameters of HER include the number of relabeled targets and the relabeling policy; the general training parameters include the discount factor, the capacity of the empirical replay pool, and the batch sampling size; the core parameters of the neural network are divided into Transformer network parameters, CNN network parameters, and fully connected network parameters. The Transformer network parameters include the number of attention heads, the number of encoder layers, the hidden layer dimension, and the dropout rate; the CNN network parameters include the kernel size, the number of convolutional layers, the pooling type, and the output feature dimension; and the fully connected network parameters include the number of network layers, the hidden layer dimension, and the dropout rate.

[0111] The reinforcement learning agent in this application adopts a hybrid neural network model that integrates convolutional neural networks and Transformers, which has a strong ability to extract temporal features and can significantly improve the accuracy of PID parameter tuning.

[0112] Figure 2 This is a schematic diagram illustrating the structure of an initial intelligent agent provided in an embodiment of this application. The initial intelligent agent in this application includes an actor model and a critic model. The structures of the actor model and the critic model are as follows: Figure 2 As shown, each layer includes an input layer, a feature extraction layer, an attention layer, multiple fully connected layers, and an output layer. Residual connections between layers prevent gradient vanishing in deep networks, and the number of network layers can be flexibly controlled through hyperparameters.

[0113] The input data of the actor model is a time series including the operation variable, process variable, and setpoint; the output data of the actor model is the adjustment amount of the PID parameter; the input data of the critic model is the time series and the adjustment amount of the PID parameter; the output data of the critic model is the Q-value of reinforcement learning.

[0114] Combination Figure 2 As shown, the feature extraction layer consists of a convolutional layer and a max pooling layer; the temporal data input to the input layer is first fed into the convolutional layer, and local temporal features are captured through convolution operations; then the max pooling layer performs feature dimensionality reduction and retains key information.

[0115] The attention layer comprises positional encoding and multiple self-attention layers. This layer receives the feature vector output from the temporal feature extraction layer, injects temporal positional information through positional encoding, and then uses a self-attention mechanism to weighted aggregate global temporal features, accurately capturing long-term dependencies and thus uncovering global correlation patterns in the data. The fully connected layer receives the global feature vector output from the attention layer and, through non-linear mapping and feature integration, achieves feature dimension adaptation and enhanced representation capabilities.

[0116] Based on the shared backbone network, the reinforcement learning agent also includes two functional branches, which are referred to in this application as the actor model and the critic model.

[0117] C2 sets the pre-training evaluation metrics.

[0118] The core goal of AutoML (Automated Machine Learning) pre-training is to find the optimal hyperparameters with the least amount of training. Therefore, it is necessary to define quantitative evaluation metrics as the basis for selecting hyperparameter combinations. The evaluation metrics should take into account both "training efficiency" and "control performance potential".

[0119] The expressions for the core evaluation metrics in this application are as follows: ; Where S is the overall control performance score; J is the control evaluation function value under the current hyperparameter combination; The baseline J value under empirical parameters; The average fluctuation of the PID parameters. To allow the maximum fluctuation; To control the precision weight, To control the smoothness weights, in addition to considering the training convergence speed (number of iterations to reach 50% of the baseline performance) and parameter robustness (performance decay rate after small perturbations of hyperparameters) of different parameter combinations, it is also necessary to consider.

[0120] C3, select the AutoML search strategy.

[0121] By selecting a suitable AutoML framework, we can ensure that the optimal training hyperparameters in the hyperparameter search space can be efficiently searched within a small number of training iterations. In response to the core requirement of "optimization within a small number of training iterations" in this section, we abandon the traditional grid search and random search methods and give priority to the use of the efficient Bayesian optimization search strategy to optimize hyperparameters.

[0122] The specific configuration for Bayesian optimization is as follows: First, the basic model uses a Gaussian process as an alternative model to Bayesian optimization (surrogate model), which fits the mapping relationship between hyperparameters and training evaluation metrics. Second, the acquisition function is the expected improvement function, which achieves a two-way balance between exploitation of known high-quality hyperparameter ranges and exploration of unknown parameter space, ensuring the search efficiency of the global optimal hyperparameter. Third, set three convergence criteria, and terminate the hyperparameter search if any one of the criteria is met. Specifically: ① The number of hyperparameter search rounds reaches the preset upper limit (e.g., 50 rounds); ② The improvement rate of the evaluation index is less than 1% for 3 consecutive rounds of training; ③ A hyperparameter combination that meets the preset evaluation threshold (e.g., greater than or equal to 0.8) is found.

[0123] S105, in the dynamic model training field, the initial agent is trained based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters and the target reinforcement learning algorithm to obtain a PID parameter tuning agent.

[0124] The reinforcement learning algorithm system adopted in this invention integrates two core algorithms, TD3 (Twin Delayed Deep Deterministic Policy Gradient) and PPO (Proximal Policy Optimization), with HER (Hindsight Experience Replay) technology to achieve efficient training of the PID parameter tuning agent.

[0125] TD3 is a deep reinforcement learning algorithm for continuous action spaces, with its core design goal being to improve training stability and control output smoothness. This algorithm introduces a dual-critic network to suppress value overestimation bias and employs a delayed policy update mechanism to avoid training oscillations caused by excessive policy update amplitude. In the PID parameter tuning scenario of this application, the TD3 algorithm can achieve smooth adjustment of continuous parameters such as Kp, Ki, and Kd.

[0126] PPO is a reinforcement learning algorithm that balances training stability and sample utilization efficiency. Its core innovation lies in constructing a clipped objective function to limit the step size of policy updates, ensuring that the deviation between the new and old policies remains within a controllable range, thereby avoiding training collapse. This algorithm achieves stable and efficient policy improvement through multiple rounds of iterative policy optimization.

[0127] HER (Experience Augmentation) is an experience-enhancing technique for solving the sparse reward problem. For each trajectory that fails to achieve the original goal, HER resets the states traversed during the process to new goals and recalculates the reward value based on these new goals, thereby transforming the sparse original reward into a dense and effective training signal. In the PID parameter tuning scenario of this application, HER technology can significantly improve sample utilization, enabling the agent to extract effective experience from each control attempt and greatly accelerating the learning process.

[0128] This application uses AutoML to determine the optimal hyperparameter combination as a foundation, and initiates a complete training process for the TD3 and PPO algorithms, which integrate HER technology. During training, the reinforcement learning agent gradually learns high-precision PID parameter tuning capabilities through full interaction with the simulation environment and iterative optimization of the strategy. The agent's input state consists of 3-dimensional time-series data including manipulated variables (MV), process variables (PV), and setpoints (SV), encompassing historical operating data generated by the simulation environment and preset target trajectory data. The agent's output action is the adjustment amount of the PID controller's core parameters (Kp, Ki, Kd), achieving automated parameter tuning based on operating condition data.

[0129] Figure 3 A flowchart illustrating a reinforcement learning algorithm used in training, provided as an embodiment of this application. (Combined with...) Figure 3 As shown, the reinforcement learning algorithm in this application adopts a dual-algorithm parallel architecture. Figure 3 The left half of the diagram shows the TD3 (including HER) algorithm flow, and the right half shows the PPO (including HER) algorithm flow. Both follow the core logic of "initialization → data acquisition → HER remarking → algorithm iteration → policy update → termination judgment", with differences only in the data acquisition method and policy update mechanism.

[0130] It should be noted that, Figure 3 The "HER trajectory relabeling module" in the middle is the core hub connecting the original data and algorithm iteration. It relabels the target and recalculates the reward for the original experience collected by the two algorithms, transforming the sparse reward in the PID control scenario into a dense signal, thereby improving the sample utilization and learning efficiency.

[0131] It should be noted that, Figure 3 All hyperparameters in the process (such as experience pool capacity, number of relabeled targets, learning rate, etc.) are derived from the optimal combination of AutoML outputs in the preceding steps, ensuring the efficiency and stability of training.

[0132] Below, on Figure 3 The process of the TD3 (including HER) algorithm on the left side of the middle section is explained in detail.

[0133] D1, use the pre-configured high-fidelity training ground as the reinforcement learning training environment.

[0134] D2, Optimal parameter loading and network initialization.

[0135] First, load the optimal combination of hyperparameters output by AutoML, including: core parameters of the reinforcement learning algorithm (such as learning rate and trust domain boundary), Transformer and CNN network parameters (such as number of attention heads and convolution kernel size), HER parameters (number of relabeled targets k and relabeling policy), and general training parameters.

[0136] Secondly, initialize the neural network according to the parameter configuration. Specifically: for the Transformer policy network, build an encoder with a specified number of layers and configure a multi-head attention mechanism and residual connections; for the CNN value network, build a feature extraction module according to the set number of convolutional layers and kernel size, and compress the feature dimension through pooling layers; for the fully connected network, set the number of fully connected layers and the dimension of hidden layers; for the dual-target network, copy the current network parameters as the initial target network parameters.

[0137] Finally, the HER experience replay mechanism is initialized. A circular experience pool with a capacity of buffer_size is created, and the original experience region is divided into a 1:k ratio with the HER remarked experience region.

[0138] D3, Interaction and Data Acquisition.

[0139] The agent generates noisy actions based on the current policy network, interacts with the high-fidelity training field, and stores the original experience tuples (state s, action a, reward r, next state s′, original goal g) into the HER experience replay pool.

[0140] D4, HER trajectory relabeling.

[0141] Based on the optimal parameter settings, the number of relabeled targets k and the strategy are set, and k new targets g′ are generated for each trajectory. The reward r′ and the advantage value A′ are recalculated.

[0142] D5, training iteration.

[0143] Raw and relabeled data are sampled from the HER experience replay pool, and the target Q value is calculated using the discount factor γ determined by AutoML to update the critic network.

[0144] D6, Policy Update (Delay).

[0145] Update the actor network according to the optimal parameter setting strategy update interval, and softly update the target network parameters to ensure training stability.

[0146] D7, Termination condition judgment.

[0147] When the preset termination conditions are met (such as the control evaluation function meeting the standard or the number of training steps reaching the upper limit), the optimized policy network is output.

[0148] Below, on Figure 3 The process of the TD3 (including HER) algorithm on the left side of the middle section is explained in detail.

[0149] For F1, the pre-configured high-fidelity training environment is used as the reinforcement learning training environment.

[0150] F2, optimal parameter loading and network initialization.

[0151] First, load the optimal combination of hyperparameters output by AutoML, including: core parameters of the reinforcement learning algorithm (such as learning rate and trust domain boundary), Transformer and CNN network parameters (such as number of attention heads and convolution kernel size), HER parameters (number of relabeled targets k and relabeling policy), and general training parameters.

[0152] Secondly, initialize the neural network according to the parameter configuration. Specifically: for the Transformer policy network, build an encoder with a specified number of layers and configure a multi-head attention mechanism and residual connections; for the CNN value network, build a feature extraction module according to the set number of convolutional layers and kernel size, and compress the feature dimension through pooling layers; for the fully connected network, set the number of fully connected layers and the dimension of hidden layers; for the dual-target network, copy the current network parameters as the initial target network parameters.

[0153] Finally, the HER experience playback mechanism is initialized. The trajectory storage queue is initialized, and the maximum length of a single trajectory is set to 500 steps (to adapt to the dynamic response cycle of PID control).

[0154] F3, trajectory acquisition.

[0155] The agent interacts with the environment based on a policy network configured with the loaded optimal parameters, collects the complete trajectory, and records relevant data.

[0156] F4, HER trajectory relabeling.

[0157] Based on the optimal parameter settings, the number of relabeled targets k and the strategy are set, and k new targets g′ are generated for each trajectory. The reward r′ and the advantage value A′ are recalculated.

[0158] F5, strategy optimization (multiple iterations).

[0159] Using the original and HER-relabeled data, a Clipped objective function is constructed based on parameters such as the trust domain boundary and entropy regularization coefficient determined by AutoML. The total loss function is then optimized through multiple iterations to update the network.

[0160] F6, policy update judgment.

[0161] When the policy performance meets the requirements, the old policy parameters θ will be changed. old Update to the current policy network parameters θ.

[0162] F7, Termination Condition Judgment.

[0163] An Early-stop mechanism is adopted. When the control evaluation index meets the preset threshold for three consecutive rounds of training, the training is terminated and the optimized policy network is output.

[0164] S106 outputs the PID parameter tuning agent.

[0165] When the reinforcement learning training framework meets the preset termination conditions, the system automatically outputs the optimized policy network and evaluation network in ONNX format, and simultaneously saves the training parameters and model structure parameters generated during the training process for subsequent deployment and reuse.

[0166] In one alternative implementation, after obtaining the PID parameter tuning agent, the performance parameters of the PID parameter tuning agent can be verified. This process specifically includes the following steps: The first step is to test the intelligent agent simulation environment.

[0167] After the reinforcement learning agent completes training and meets the preset termination conditions, it does not directly enter the field deployment stage. Instead, it prioritizes the verification of its PID parameter tuning capabilities. Among these, agent testing based on a high-fidelity simulation environment is the core means of measuring its comprehensive performance. This stage comprehensively verifies and quantitatively evaluates the agent's control accuracy, anti-interference robustness, and scenario adaptability under different operating conditions by reproducing typical operating conditions such as disturbance resistance, dynamic setpoint tracking, and integral saturation in industrial scenarios. This provides solid performance support and decision-making basis for the subsequent migration and deployment of the agent from the simulation environment to the actual industrial field.

[0168] In one alternative implementation, the agent is tested in a simulation environment, including: G1, obtain the simulation environment.

[0169] Extract system models, disturbance models, and control evaluation functions that match the target verification scenario from the model set training field.

[0170] G2, obtains simulated test data (also known as test dataset).

[0171] The simulated test data is a standardized verification dataset covering typical operating conditions across multiple dimensions, including an input signal set, a disturbance signal set, and an evaluation test case library. The input signal set includes step signals, ramp signals, sinusoidal signals, and random signals; the disturbance signal set includes load disturbances (step / gradual), measurement noise, and parameter drift; and the evaluation test case library generates 100 standard test cases (70 training-similar test cases and 30 unseen generalized test cases).

[0172] G3, construct a test indicator system (also known as test control requirements).

[0173] The testing index system is a multi-dimensional evaluation benchmark used to comprehensively and quantitatively assess the PID parameter tuning performance of an intelligent agent. The testing index system includes: control accuracy index, robustness index, and learning efficiency index. Control accuracy indexes include: steady-state error, overshoot, and settling time; robustness indexes include: anti-interference recovery time and performance degradation rate under parameter drift; learning efficiency indexes include: number of iterations to achieve stable control and sample utilization rate.

[0174] In this application, a reinforcement learning control agent is simulated and tested based on a dynamic model training field and a test dataset to obtain evaluation results. If the test evaluation results meet the control requirements, a PID parameter agent is deployed; otherwise, the model training field is optimized.

[0175] The second step is to optimize the model training environment.

[0176] If the evaluation results do not meet the preset control index requirements, the model set training field (also known as the dynamic model training field) needs to be systematically optimized. By improving the simulation fidelity and typical working condition coverage of the training field, a simulation foundation that is more in line with the actual application scenario can be built for the retraining of the intelligent agent, ensuring that the intelligent agent ultimately meets the control requirements of the industrial site.

[0177] In one alternative implementation, obtaining the optimized dynamic model training ground includes the following steps: First, we assessed the reasons for non-compliance and analyzed their correlation with the training ground.

[0178] This step requires analyzing the correlation between the reasons for the agent's substandard performance evaluation and the model set training field, and focusing on calibrating the system's dynamic characteristic parameters.

[0179] For linear systems, system identification methods such as least squares and Kalman filtering are used to re-identify the core parameters of the transfer function, keeping the deviation rate between the simulation output and the actual system response within a low range, while expanding the range of parameter extraction for the system model.

[0180] For nonlinear systems, by supplementing the piecewise linearized model or adding nonlinear elements such as dead zones, saturation, and frictional characteristics, we can ensure that the nonlinear behavior of the system under key operating conditions is accurately reproduced, thereby improving the matching degree between the model set training field and the actual industrial scenario.

[0181] Secondly, based on the above evaluation and analysis results, further optimization work will be carried out at the scene level on the model set training field, specifically including three aspects: The system adds new scenario types where the intelligent agent's performance is poor, such as adding a composite working condition of "high-frequency small-amplitude disturbance and rapid change of set value"; it expands the system parameter drift range from ±15% to ±30% to cover extreme aging conditions of the equipment; and it adds fault scenarios such as sensor signal loss, signal jump, and signal drift.

[0182] Construct rules for progressively increasing scenario difficulty, such as gradually increasing the interference amplitude according to the training process, to ensure that the training process of the agent has a smooth and gradual nature and to avoid training failure due to sudden changes in the difficulty of the working conditions.

[0183] Establish dynamic transition rules between different scenarios to simulate the continuous changes in actual industrial site conditions and improve the training ground's fidelity to real application scenarios.

[0184] After completing the above optimizations, replace the dynamic model training field in S102 with the updated model training field (i.e., the optimized dynamic model training field), and re-execute the subsequent training and validation process.

[0185] The above approach can effectively solve the problem of substandard agent performance caused by simulation distortion in the training field or insufficient scene coverage, and build a closed-loop iterative mechanism of "testing, evaluation, optimization and retraining" to ensure that the final output agent meets the stringent requirements of actual industrial control.

[0186] The third step is to deploy the PID parameter intelligent agent.

[0187] After the training task is completed, the policy network generated by reinforcement learning training is converted into an ONNX model file using the ONNX format export capability provided by deep learning frameworks such as PyTorch and TensorFlow, providing a standardized carrier for subsequent model deployment and engineering applications.

[0188] This application provides a method for obtaining a PID parameter tuning agent, comprising: constructing a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements; determining target training hyperparameters based on target control requirements and a preset mapping rule base; determining the target control trajectory based on target control requirements; determining the target hyperparameter combination based on the initial agent, the dynamic model training field, preset training evaluation indicators, preset search strategies, and preset hyperparameter search space; and training the initial agent in the dynamic model training field based on multimodal control targets, target control trajectories, target hyperparameter combinations, target training hyperparameters, and target reinforcement learning algorithms to obtain a PID parameter tuning agent.

[0189] Because the dynamic model training field in this application fully integrates the characteristics of the actual production environment and the prior knowledge of the target control system, it can accurately replicate the dynamic characteristics of real control scenarios. At the same time, it locks the optimal combination of target hyperparameters through a preset mapping rule base and a scientific search strategy. Combined with multimodal control targets and reinforcement learning algorithms, it conducts targeted training on the initial agent, so that the trained PID parameter tuning agent has both the flexibility to adapt to different control requirements and the parameter tuning accuracy based on real scenario data. It effectively avoids the problems of traditional PID parameter tuning relying on human experience, adapting to a single scenario, low parameter iteration efficiency and large error. Therefore, the PID parameter tuning agent in this application can perform PID parameter tuning efficiently and accurately.

[0190] Based on the same inventive concept, this application also provides a PID parameter tuning device. Figure 4 This is a schematic diagram of a device for obtaining PID parameter tuning intelligence provided in an embodiment of this application. (Combined with...) Figure 4 As shown, the apparatus 400 for obtaining PID parameter tuning intelligence provided in this application includes: The dynamic model training field construction module 401 is used to construct a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements. The target training hyperparameter determination module 402 is used to determine the target training hyperparameters based on the target control requirements and a preset mapping rule base; the mapping rule base includes the correspondence between control requirements and training hyperparameters; the target training hyperparameters include learning rate, discount factor and exploration rate; The target control trajectory determination module 403 is used to determine a multimodal control target and a target control trajectory based on the system scenario corresponding to the target control system; the target control trajectory is a preset trajectory for achieving the multimodal control target; the target control trajectory is used as a preset control trajectory for post-event experience playback; The target hyperparameter combination determination module 404 is used to determine the target hyperparameter combination based on the initial agent, the dynamic model training field, the preset training evaluation index, the preset search strategy and the preset hyperparameter search space. The agent training module 405 is used to train the initial agent in the dynamic model training field based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters and the target reinforcement learning algorithm to obtain a PID parameter tuning agent.

[0191] In one alternative implementation, the dynamic model training field construction module 401 includes: The target system model determination submodule is used to determine the target system model based on the prior knowledge of the target control system and a preset system model set; the system model set includes a liquid level loop system model, a pressure loop system model, a temperature loop system model, and a flow loop system model. The disturbance model determination submodule is used to determine the disturbance model based on the actual production environment; the disturbance model is generated based on impulse disturbances and step disturbances. The control evaluation function determination submodule is used to determine the control evaluation function based on the target control requirements; The dynamic model training field construction submodule is used to construct the dynamic model training field based on the target system model, the disturbance model, and the control evaluation function.

[0192] In one alternative implementation, the control evaluation function determines the submodules, including: The first function determination unit is used to determine the target quantitative analysis function, the PID parameter compliance constraint function, and the controller output smoothness constraint function based on the target control requirements; the target quantitative analysis function is one of the quantitative analysis function corresponding to the disturbance rejection system, the quantitative analysis function corresponding to the SV tracking system, and the quantitative analysis function corresponding to the integral system; The second function determination unit is used to determine the control evaluation function based on the target quantitative analysis function, the PID parameter compliance constraint function, the controller output smoothness constraint function, and a preset weight coefficient combination.

[0193] In one alternative implementation, the agent training module 405 includes: A reinforcement learning control agent acquisition unit is used to train the initial agent in the dynamic model training field based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a reinforcement learning control agent. The evaluation result acquisition unit is used to perform simulation tests on the reinforcement learning control agent based on the dynamic model training field and test dataset, and obtain the evaluation results. The first result determination unit is used to use the reinforcement learning control agent as the PID parameter tuning agent if the evaluation result meets the evaluation control requirements. The second result determination unit is used to optimize the dynamic model training field to obtain an optimized dynamic model training field if the evaluation result does not meet the evaluation control requirements; and to determine the PID parameter tuning agent based on the optimized dynamic model training field.

[0194] Based on the intelligent agent and apparatus for obtaining PID parameter tuning provided in the foregoing embodiments, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the intelligent agent for obtaining PID parameter tuning mentioned above.

[0195] Based on the intelligent agent and device for obtaining PID parameter tuning provided in the foregoing embodiments, this application also provides an electronic device, including: A memory on which computer programs are stored; A processor is configured to execute the computer program in the memory to implement some or all of the steps in the PID parameter tuning agent provided in the foregoing embodiments.

[0196] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0197] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for obtaining a PID parameter tuning agent, characterized in that, The method includes: Based on the actual production environment, prior knowledge of the target control system, and target control requirements, a dynamic model training field is constructed. Based on the target control requirements and a preset mapping rule base, target training hyperparameters are determined; the mapping rule base includes the correspondence between control requirements and training hyperparameters; the target training hyperparameters include learning rate, discount factor, and exploration rate. Based on the target control requirements, a target control trajectory is determined; the target control trajectory is a preset control trajectory that achieves the multimodal control target corresponding to the target control requirements; the target control trajectory is used as a control trajectory for post-event experience playback. Based on the initial agent, the dynamic model training field, the preset training evaluation index, the preset search strategy, and the preset hyperparameter search space, the target hyperparameter combination is determined. In the dynamic model training field, the initial agent is trained based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a PID parameter tuning agent.

2. The method of claim 1, wherein, The dynamic model training field, constructed based on the actual production environment, prior knowledge of the target control system, and target control requirements, includes: Based on the prior knowledge of the target control system and the preset system model set, the target system model is determined; the system model set includes a liquid level loop system model, a pressure loop system model, a temperature loop system model, and a flow rate loop system model. Based on the actual production environment, a disturbance model is determined; the disturbance model is generated based on impulse disturbances and step disturbances. Based on the target control requirements, determine the control evaluation function; Based on the target system model, the disturbance model, and the control evaluation function, the dynamic model training field is constructed.

3. The method of claim 2, wherein, The determination of the control evaluation function based on the target control requirements includes: Based on the target control requirements, the target quantitative analysis function, the PID parameter compliance constraint function, and the controller output smoothness constraint function are determined; the target quantitative analysis function is one of the quantitative analysis functions corresponding to the disturbance rejection system, the quantitative analysis function corresponding to the SV tracking system, and the quantitative analysis function corresponding to the integral system. The control evaluation function is determined based on the target quantitative analysis function, the PID parameter compliance constraint function, the controller output smoothness constraint function, and a preset weight coefficient combination.

4. The method of claim 1, wherein, The preset training evaluation metric is the comprehensive control performance score: ； Where S is the overall control performance score, and J is the control evaluation function value under the current hyperparameter combination. The baseline J value under empirical parameters; The average fluctuation of the PID parameters. To allow the maximum fluctuation; To control the precision weight, To control the smoothness weights.

5. The method of claim 1, wherein, The preset search strategy is a Bayesian optimized search strategy.

6. The method of claim 1, wherein, The initial agent includes an actor model and a critic model; both the actor model and the critic model include an input layer, a feature extraction layer, an attention layer, a fully connected layer, and an output layer; the input data of the actor model is a time series including operational variables, process variables, and setpoints; the output data of the actor model is the adjustment amount of the PID parameters; the input data of the critic model is the time series and the adjustment amount of the PID parameters; the output data of the critic model is the Q-value of reinforcement learning.

7. The method of claim 1, wherein, In the dynamic model training field, based on the multimodal control objective, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm, the initial agent is trained to obtain a PID parameter tuning agent, including: In the dynamic model training field, the initial agent is trained based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a reinforcement learning control agent; Based on the dynamic model training field and test dataset, the reinforcement learning control agent is simulated and tested to obtain evaluation results. If the evaluation results meet the evaluation control requirements, then the reinforcement learning control agent will be used as the PID parameter tuning agent. If the evaluation results do not meet the evaluation control requirements, the dynamic model training field is optimized to obtain an optimized dynamic model training field; and based on the optimized dynamic model training field, the PID parameter tuning agent is determined.

8. The method of claim 7, wherein, The target reinforcement learning algorithm is one of the TD3 reinforcement learning algorithm and the PPO reinforcement learning algorithm.

9. A device for acquiring PID parameter tuning intelligence, characterized in that, The device includes: The dynamic model training field construction module is used to build a dynamic model training field based on the actual production environment, prior knowledge of the target control system, and target control requirements. The target training hyperparameter determination module is used to determine the target training hyperparameters based on the target control requirements and a preset mapping rule base; the mapping rule base includes the correspondence between control requirements and training hyperparameters; the target training hyperparameters include learning rate, discount factor, and exploration rate; The target control trajectory determination module is used to determine the target control trajectory based on the target control requirements; the target control trajectory is a preset control trajectory for achieving the multimodal control target corresponding to the target control requirements; the target control trajectory is used as a control trajectory for post-event experience playback. The target hyperparameter combination determination module is used to determine the target hyperparameter combination based on the initial agent, the dynamic model training field, the preset training evaluation index, the preset search strategy, and the preset hyperparameter search space. The agent training module is used to train the initial agent in the dynamic model training field based on the multimodal control target, the target control trajectory, the target hyperparameter combination, the target training hyperparameters, and the target reinforcement learning algorithm to obtain a PID parameter tuning agent.

10. An electronic device, comprising: include: a memory having stored thereon a computer program; a processor configured to execute the computer program in the memory to implement the steps of the method of any one of claims 1-8.

Images