Model predictive controller parameter setting method and device based on improved reinforcement learning

Abstract

The invention provides a model prediction controller parameter setting method and device based on improved reinforcement learning, and relates to the technical field of model prediction control. The method comprises the following steps: designing a state space and an action space based on an objective function of a model prediction controller; wherein the objective function is used for setting a control variable and a controlled variable of the model prediction controller; inputting the current state information into a DQN intelligent agent strategy network for action decision, and outputting optimized action parameters; the optimized action parameters are subjected to constraint processing and then applied to a model prediction controller, next state information is obtained, a reward value is calculated based on a reward function, and experience pairs are formed and stored in an experience pool; and constructing sampling probability distribution based on the time sequence difference error of the experience pair, sampling in the experience pool based on the sampling probability distribution, and optimizing the DQN intelligent agent strategy network. Manual parameter adjustment errors can be reduced, and the control precision of the controller is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model predictive control, and in particular to a model predictive controller parameter tuning method and device based on improved reinforcement learning. BACKGROUND

[0002] Model predictive control (MPC) is an advanced process control algorithm, widely used in industrial process control (such as chemical, power, oil refining, etc.) and in recent years in autonomous driving, robotics, aerospace, etc., the core idea is to predict future behavior using the dynamic model of the process, and determine the control action at the current time by solving an optimal control problem in a finite time domain. Its strong performance and ability to solve complex control problems make it an important part of modern control processes (including process industries and autonomous driving, etc.).

[0003] However, in the actual operation of the traditional MPC control link, the controller objective function weight parameters and feedback correction, reference trajectory softening coefficients that play a key role in the performance of the controller are often dependent on manual judgment, affected by the experience, preferences and subjective judgment of engineers, and often have a certain degree of manual error, which also leads to the optimization link of the controller falling into local optimum, weakening the control accuracy and generalization stability of MPC. SUMMARY

[0004] The embodiments of the present application provide a model predictive controller parameter tuning method and device based on improved reinforcement learning to solve the problem of affected MPC control accuracy and stability.

[0005] In a first aspect, the embodiments of the present application provide a model predictive controller parameter tuning method based on improved reinforcement learning, comprising: designing a state space and an action space based on a target function of a model predictive controller; wherein the target function is used to tune the control variable and the controlled variable of the model predictive controller; inputting the current state information into the DQN agent policy network for action decision, and outputting the optimized action parameters; applying the optimized action parameters to the model predictive controller after constraint processing, obtaining the next state information, calculating the reward value based on the reward function, and storing the experience pair into the experience pool; constructing a sampling probability distribution based on the time difference error of the experience pair, and sampling in the experience pool based on the sampling probability distribution to optimize the DQN agent policy network.

[0006] In a possible implementation manner, the target function is: min J || (k)- (k) +||ΔU(k)

[0007] = i = 1,2,…,p e

[0008] +h·e i = 1,2,…,p Among them, J for The objective function value at time t. (k) is the output matrix of the prediction model. (k) is the output set softening value matrix. The weighting parameters, ΔU(k), determine the degree of penalty imposed on the system's state error. To determine the weighting parameters for the system's penalty on the control input, The reference trajectory at time k, The softening coefficient is... This represents the actual output at time k. To output the set value, e The error between the predicted value and the actual value, The prediction result of the prediction model at time k for the system output at time k+1 is given. For the true value, These are the corrected multi-step predictions. is the multi-step predicted value before correction, and h is the feedback correction coefficient.

[0009] In one possible implementation, the state space includes:

[0010] in, This is the current status information. This is the current state vector, which includes control variables (determined based on actual operating conditions and process parameters). To control the current output of the object, The output of the prediction model at the current time step. This is the current feedback correction value. The current setting value, , These are the controller weight parameters from the previous time step. The tracking error at the current moment, , The rate of change of error, , , These are the parameter values ​​of the feedback correction coefficient and the softening coefficient at the previous moment, respectively; The motion space includes:

[0011]

[0012] in, Let this be the current state vector. For the current action vector, and These are the upper and lower limits of the controller weight parameters Q and R, respectively. and Feedback correction coefficients and softening coefficient The upper and lower limits.

[0013] In one possible implementation, the reward function is:

[0014]

[0015]

[0016] in, , The normalization coefficient is... Indicates a stability reward. and To stabilize the suppression parameters, , , To constrain penalties, The penalty coefficient is... For the first One control quantity, For the first The lower bound of each control variable. For each control variable, the upper bound is defined as follows: 1 represents the indicator function, 1 indicates a violation of the constraint, and 0 indicates otherwise.

[0017] In one possible implementation, the method further includes: Based on the time-series difference error, the experience pool is divided into a high-value experience area and a low-value experience area; Accordingly, the sampling probability distribution is:

[0018]

[0019] in, For sampling probability distribution, a sampling probability of a high-value experience region in the experience pool, a sampling probability of a low-value experience region in the experience replay pool, a reward value division threshold, a time difference error of an experience pair, a constant, a priority index.

[0020] In a possible implementation, the experience pool is divided into the high-value experience region and the low-value experience region based on the time difference error, and the division includes: if the time difference error of a first experience pair in the experience pool is greater than the reward value division threshold for consecutive times, the first experience pair is divided into the high-value experience region; if the time difference error of the first experience pair is less than the reward value division threshold for consecutive times, the first experience pair is divided into the low-value experience region; wherein, and are preset threshold values, and .

[0021] In a second aspect, an embodiment of the present application provides a model predictive controller parameter tuning device based on improved reinforcement learning, which includes: a variable design module, configured to design a state space and an action space based on a target function of a model predictive controller; wherein, the target function is used to tune a control variable and a controlled variable of the model predictive controller; an action optimization module, configured to input current state information into a DQN agent policy network to make an action decision, and output an optimized action parameter; a reward calculation module, configured to apply the optimized action parameter to the model predictive controller after constraint processing, obtain next state information, calculate a reward value based on a reward function, and form an experience pair to be stored in an experience pool; an experience sampling module, configured to construct a sampling probability distribution based on a time difference error of the experience pair, sample in the experience pool based on the sampling probability distribution, and optimize the DQN agent policy network.

[0022] In a third aspect, an embodiment of the present application provides an electronic device including a memory and a processor, the memory stores a computer program, and the processor implements the method in the first aspect or any possible implementation manner of the first aspect when executing the computer program.

[0023] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0024] The method and device for parameter setting of a model predictive controller based on improved reinforcement learning provided by the embodiment of the present application construct a state space by improving the DQN algorithm to observe and collect the associated variables such as the controlled input, the output, and the predicted input and output in the predictive control environment, the agent selects a policy according to the current state to construct an action space inputting the controller by the key parameters of the controller to be set, updates the objective function and solves the optimal control amount in the control period, calculates the reward function, stores the experience for updating the agent policy network to better guide the selection of the controller weight parameter in the next control period, realizes the double-cycle collaborative control process of the reinforcement learning cycle large loop + the predictive control cycle small loop, realizes the dynamic optimization and update of the key weight parameters of the controller, reduces the artificial parameter setting error, effectively improves the control precision of the controller, greatly improves the automation and standardization of the control system, and improves the running performance of the overall reinforcement learning-model predictive control (RL-MPC) framework by improving the DQN to sample and extract high-value experience with a higher priority based on the hierarchical idea in the training process, improving the model policy update rate, and strengthening the stability of the model training process. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is an implementation flowchart of the method for parameter setting of a model predictive controller based on improved reinforcement learning provided by an embodiment of the present application; Figure 2 is an implementation flowchart of the method for parameter setting of a model predictive controller based on improved reinforcement learning provided by another embodiment of the present application; Figure 3 is a structural schematic diagram of the DQN algorithm provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of the improved DQN algorithm provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of the device for parameter setting of a model predictive controller based on improved reinforcement learning provided by an embodiment of the present application; Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The application takes a multivariable, nonlinear system as an object, and invents a predictive controller parameter self-tuning method based on an improved DQN. By introducing a reinforcement learning algorithm, a reasonable action is designed according to a controller process principle and a flow-related variable analysis, a state space, a reward function, and a constraint penalty module, and the like components are designed, so that dynamic optimization update of key weight parameters and control link parameters Q, 、 of the controller is completed, a reinforcement learning-model predictive control (RL-MPC) controller parameter self-tuning framework is constructed, the control precision is improved, the self-tuning function of key parameters of the model predictive controller is realized, the manual error is greatly reduced, the control system is more automated and standardized, and meanwhile, the improved DQN with high and low value experience layered sampling mechanism can more efficiently update and train the strategy, and the running efficiency and stability of the entire parameter self-tuning framework are further improved. In the following, the embodiments of the application will be described in detail with reference to the drawings.

[0027] Referring to Figure 1 , a flowchart of implementation of a model predictive controller parameter tuning method based on improved reinforcement learning provided by an embodiment of the application is shown, and the details are as follows: Step 101: Designing a state space and an action space based on a target function of a model predictive controller; wherein the target function is used for tuning control variables and controlled variables of the model predictive controller.

[0028] In this embodiment, the specific process of the model predictive controller parameter tuning method based on improved reinforcement learning can be as shown in Figure 2 , and specifically can be divided into the following links: A analyzing a process condition under a controller environment to determine related variables; B designing a corresponding target function based on the related variables, and determining controller parameters to be optimized Q, 、 R,

[0029] C designing a state space and an action space required by a reinforcement learning algorithm based on the parameters to be tuned and the target function; D designing a reward function and a constraint penalty module based on selected characteristic variables and actual conditions; E selecting a DQN algorithm to implement a controller parameter self-tuning algorithm process, wherein the used DQN algorithm is improved to a certain extent (IDQN); a state and a reward are received and updated in a rolling manner, a strategy is updated, and an action after constraint optimization, i.e., controller weight parameters Q, 、 R, are output, and dynamic update of key parameters of the controller is completed (the new parameters will be used to update the target function, participate in the rolling optimization link to output the updated controller optimal control amount, and update the parameter of the subsequent feedback correction and reference trajectory softening link).

[0030] F uses the optimal control input to control the system and obtains feedback from the controlled variable. The obtained feedback variable is then used to correct the prediction model, updating the state and reward. Steps E and F are then repeated to gradually stabilize the controlled variable and achieve the set control objective. Based on the large-cycle step E and the small-cycle step F, the controller parameters Q and R are continuously updated. , This enables self-tuning of parameters for a reinforcement learning-based predictive control model.

[0031] In step A, determining the relevant variables includes: studying the process conditions under the current working environment of the controller, analyzing the correlation variables that have different degrees of influence on the control output under the process conditions, and determining the corresponding control variables, disturbance variables, and controlled variables. In the actual operating environment, these variables can be obtained in real time through measuring instruments such as field sensors, flow meters, and pressure gauges. If conditions permit, they can be obtained through field real-time databases and historical databases.

[0032] In step B, the objective function is designed based on the selected control variable, controlled variable, and disturbance variable. Simultaneously, the parameters to be optimized, Q and R, are determined based on the control loop. , This includes: The purpose of designing the objective function is to solve the optimization algorithm in the subsequent rolling optimization stage to obtain the optimal control quantity. Therefore, based on the selected control variable, controlled variable, and disturbance variable, the error obtained in the feedback correction stage is used. (k)- (k) (which is essentially the controlled variable) and different weight values ​​of the control input increment (control variable increment) are incorporated into the objective function.

[0033] In one possible implementation, the objective function is: min J || (k)- (k) +||ΔU(k)

[0034] = i = 1,2,…,p e

[0035] +h·e i = 1,2,…,p Among them, J for The objective function value at time t. (k) is the output matrix of the prediction model. (k) is the output set softening value matrix, ΔU(k) is the input increment, is the weight parameter for determining the punishment degree of the system on the control input, is the reference trajectory at time k, is the softening coefficient, is the actual output at time k, is the output set value, e is the error between the predicted value and the true value, is the prediction result of the prediction model at time k+1, is the true value, is the corrected multi-step prediction value, is the uncorrected multi-step prediction value, and h is the feedback correction coefficient.

[0036] In this embodiment, the objective function is designed based on the control variable, the controlled variable and the disturbance variable selected in step A: min J || (k)- (k) +||ΔU(k)

[0037] wherein (k) is the prediction model output matrix, and ΔU(k) is the input increment, (k) is the output set softening value matrix, and the softening formula is as follows: = , i = 1, 2, …, p is the reference trajectory at time k; is the softening coefficient; is the actual output at time k; is the output set value; the softening coefficient The calculation formula is: = , wherein T is the sampling period, and τ is the time constant.

[0038] In addition, the control output and the prediction output will be corrected based on the feedback correction link in the prediction control: e

[0039] +h·e , i = 1, 2, …, p wherein e is the error between the predicted value and the true value; is the prediction result of the prediction model at time k on the output of the system at time k+1; represents the true value; is the corrected multi-step predicted value; is the uncorrected multi-step predicted value; h is the feedback correction coefficient.

[0040] According to the designed objective function, the to-be-optimized parameters can be determined as the weight parameter Q that determines the punishment degree of the system on the state error, the weight parameter R that determines the punishment degree of the system on the control input, the feedback correction coefficient h, and the softening coefficient . .

[0041] In the embodiment, the state space is used to describe all possible set states of the environment with which the agent interacts. The design of the set state space needs to represent the current running state of the system, so that the agent can better select actions by processing observation information. The action space is a set of all possible actions that can be executed by the agent. In the current framework, the incremental type is selected to construct the action space, which is more consistent with the actual scene of parameter adjustment, for the reasons of ensuring learning stability and exploration efficiency. This not only reduces the range of the action space, but also makes the agent more easily learn effective adjustment strategies, and has clear physical meaning. Correspondingly, the parameter update rule is set as:

[0042] In step 102, the current state information is input into the DQN agent policy network for action decision, and the optimized action parameter is output.

[0043] In the embodiment, the state space and the action space required by the reinforcement learning algorithm designed based on the to-be-tuned parameters and the objective function in step C include: selecting the parameter values of the current state vector including the control variable , the current time output of the controlled object , the current time output of the prediction model , the current feedback correction value , the current set value , the controller weight parameter at the last time , the current time tracking error of the controlled variable , the error change rate , and the feedback correction coefficient and the softening coefficient at the last time , to constitute the state space; selecting the incremental type to construct the action space of the to-be-optimized parameters Q, R, , . In one possible implementation manner, the state space includes:

[0044] wherein, is the current state information, is the current state vector including control variables (determined based on actual working condition process conditions), is the current time output of the control object, is the current time output of the prediction model, is the current feedback correction value, is the current set value, , are the last time controller weight parameters, is the current time tracking error, , is the error change rate, , , are the last time feedback correction coefficient and softening coefficient parameter values, respectively; The action space includes:

[0045]

[0046] wherein, is the current state vector, is the current action vector, and are the upper and lower limits of the controller weight parameters Q and R, respectively, and are the upper and lower limits of the feedback correction coefficient and the softening coefficient , respectively.

[0047] The structure of the DQN algorithm is shown in Figure 3 . The action parameters corresponding to the state information at each time are calculated by the DQN agent policy network, so as to configure the objective function based on the action parameters, and then the control variables of the model predictive controller are preliminarily set, and the reward value is calculated to verify and evaluate the effect of the preliminary setting.

[0048] In step 103, the optimized action parameters are applied to the model predictive controller after constraint processing, the next state information is obtained, the reward value is calculated based on the reward function, and the experience pair is stored in the experience pool.

[0049] In this embodiment, in one possible implementation, the reward function is:

[0050]

[0051]

[0052] in, , The normalization coefficient is... Indicates a stability reward. and To stabilize the suppression parameters, , , To constrain penalties, The penalty coefficient is... For the first One control quantity, For the first The lower bound of each control variable. For each control variable, the upper bound is defined as follows: 1 represents the indicator function, 1 indicates a violation of the constraint, and 0 indicates otherwise.

[0053] In this embodiment, the reward function and constraint penalty module are designed in step D. The reward function is designed as follows:

[0054] in This represents a stability reward, used to suppress parameter fluctuations.

[0055] in and To stabilize the suppression parameter, used to control the parameters in the reward function. and The intensity of the penalty can be adjusted manually to suppress potential drastic fluctuations and enhance the stability of the control system.

[0056] in The constraint module is designed as follows:

[0057] in, , The normalization coefficient is used to balance the priorities of each objective and to penalize the deviation of nitrogen oxide concentration from the control input. This is the penalty coefficient; For the first One control quantity, For the first The lower bound of each control variable; The upper bound for each control variable; 1 represents the indicator function, 1 for a constraint violation, and 0 otherwise.

[0058] At step 104, a sampling probability distribution is constructed based on the timing difference error of the experience pair, and the DQN agent policy network is optimized based on sampling the experience pool based on the sampling probability distribution.

[0059] In the present embodiment, random sampling is taken for all experiences in the experience replay pool in the original DQN algorithm, but the experiences in the experience replay pool are not necessarily all important, and the possible key experiences are often submerged in a large number of ordinary value experiences. Uniform sampling often leads to a waste of more time in analyzing low-value experiences, so that the model is not fully utilized for high-value experiences, reducing the efficiency and stability of the model. Therefore, the experience replay pool of the DQN algorithm is improved, and a hierarchical idea is introduced. The original experience replay area is divided into a high-value experience area and a low-value experience area by a certain threshold standard, and the experiences in the high-value area and the low-value area are kept across the areas according to the training round, and the experiences in the two experience areas are dynamically updated to ensure the timeliness of the high-value experiences, and an improved DQN (IDQN) algorithm is established under the hierarchical experience value. The structure of the improved DQN algorithm (IDQN) is as shown in Figure 4 .

[0060] In reinforcement learning, the timing difference error (TD-error) is an important indicator of experience value. It represents the gap between the current Q value and the target Q value:

[0061] where r is the reward calculated in the current round; is the network policy parameter, is the target network parameter, is the discount factor.

[0062] Based on the understanding of the TD-error indicator, the greater the value of the TD-error, the greater the gap between the current and predicted and target network values, and the more experience can be learned from it, and vice versa. Based on this, a value threshold can be set to realize the hierarchical division of high and low value areas: if the current calculated value is higher than , the experience is classified as high-value experience and is classified into the high experience area, otherwise it is classified into the low experience area. Correspondingly, different sampling distribution probabilities can be set when using the experience pool for experience, to improve the utilization of high-value experiences.

[0063] In one possible implementation, the method further includes: dividing the experience pool into a high-value experience area and a low-value experience area based on the timing difference error; Correspondingly, the sampling probability distribution is:

[0064]

[0065] wherein, is the sampling probability distribution, is the sampling probability of the high-value experience region in the experience pool, is the sampling probability of the low-value experience region in the experience replay pool, is the reward value division threshold, is the temporal difference error of the experience pair, is a constant, is the priority index.

[0066] In the embodiment, the probability distribution of the hierarchical sampling region is designed as:

[0067] wherein, is the sampling probability of the high-value experience region in the experience replay pool; is the sampling probability of the low-value experience region in the experience replay pool; is the reward calculated in the current round; is the value division threshold.

[0068] Based on the hierarchical design of the high-value and low-value experience regions, when the model receives a sampling request in the actual scene, for the high-value experience region, the priority sampling based on the TD-error is adopted:

[0069] wherein, is the temporal difference error of the sample; is a small positive number (usually 1e-5), which prevents is 0 when the denominator is 0; is the priority index, (usually 0.4-0.7), which is used to control the "sharpness" of the sampling distribution.

[0070] In one possible implementation, the experience pool is divided into a high-value experience region and a low-value experience region based on the temporal difference error, including: if the temporal difference error of the first experience pair in the experience pool is greater than the reward value division threshold for consecutive times, the first experience pair is divided into the high-value experience region; if the temporal difference error of the first experience pair is less than the reward value division threshold for consecutive times, the first experience pair is divided into the low-value experience region; wherein, and are preset thresholds, and .

[0071] In this embodiment, in order to keep continuous sampling of high value areas, a flow conversion mechanism of high value experience and low value experience is given: When the experience in the experience pool is continuously TD-error satisfies:

[0072] The experience is included in the high value experience area; at the same time, if the experience in the experience is originally in the high value experience area or not, if the TD-error is continuously times satisfies:

[0073] It is sent to the low value experience area. Wherein .

[0074] Based on the high value experience priority sampling mechanism in the hierarchical experience pool, IDQN changes the distribution of experience sampling, and performs implicit weighting of the loss function based on the high and low value area, so that the actual optimization in the policy update process is the weighted loss function:

[0075] Wherein is the loss function gradient after adopting the hierarchical experience replay mechanism; respectively represent the high and low experience value area; is the importance sampling weight, used to correct the error between hierarchical sampling and original uniform sampling, and ensure that the gradient estimation is unbiased; wherein, Indicates:

[0076]

[0077] Wherein, is a predetermined sampling probability, which can be adjusted; is the total experience pool area.

[0078] The implementation process of steps E and F is as follows: System initialization: based on the historical data of the target scene (such as a cement enterprise), the state space and action space are initialized, the reinforcement learning agent is initialized, and the initial value of the to-be-determined parameter Q and R is initialized.

[0079] Loop optimization (total cycle epoch): after completing the environment initialization, based on the reinforcement learning algorithm, the state information is sent to the DQN agent policy network for action decision, and the optimized action parameters Q, R, , .

[0080] The optimized new action parameters enter the constraint penalty unit for constraint processing.

[0081] The optimized new action parameters are input into the predictive control environment, and the predictive controller starts to perform iterations to realize feedback correction, reference trajectory softening, and rolling optimization, and outputs the optimal control amount After the control link is executed, the new state variable is output and the reward .

[0082] The new state is observed, and the experience is stored in the experience pool.

[0083] The TD-error is calculated, and the experience value is evaluated.

[0084] The two experiences are stored in layers according to the high and low values and evaluation conditions, and the high and low value experience areas are sampled and the base sampling is calculated and the importance sampling weight is calculated Weighted gradient update is performed.

[0085] The policy network parameters and the value network parameters are updated, and the first step is returned to epoch+1.

[0086] In a specific embodiment, the above-mentioned controller parameter self-tuning algorithm flow is shown in Table 1:

[0087] Based on the flow, the MPC controller for ammonia water spray gun control is parameterized, including the following steps: Step A: Analyze the process conditions under the controller environment to determine the relevant variables; The agent needs to perceive and receive the state by interacting with the corresponding environment and output the action to update the target parameter, so it is necessary to analyze the relevant variables including the control variables, disturbance variables and controlled variables through the process conditions under the current environment of the controller to determine the relevant variables including the control variables, disturbance variables and controlled variables for subsequent state space construction.

[0088] Step B: Design the objective function according to the selected control variables, controlled variables and disturbance variables, and determine the controller optimization parameters Q, R, 、 according to the control link; Design the objective function min J || (k)- (k) +||ΔU(k) The parameters to be optimized are determined as follows: the weight parameter Q, which determines the penalty for the system's state error; the weight parameter R, which determines the penalty for the system's control input; and the feedback correction coefficient. and softening coefficient .

[0089] Step C: Design the state space and action space required for the reinforcement learning algorithm based on the parameters to be tuned and the objective function; Select the current state vector, including the control variables. The output of the control object at the current moment The prediction model outputs at the current time. Current time feedback correction value Current setting value Controller weight parameters at the previous time step Tracking error of the current controlled variable And the feedback coefficient and softening coefficient of the previous moment. , The parameter values ​​constitute the state space; select the parameters to be optimized: Q, R, ... , Incremental construction of action space; Step D: Design the reward function and constraint penalty module based on the selected relevant feature variables and actual working conditions; the reward function is designed as follows:

[0090] in This represents a stability reward, used to suppress parameter fluctuations.

[0091] in and To stabilize the suppression parameters.

[0092] in The constraint module is designed as follows:

[0093] in, , The normalization coefficient is used to balance the priorities of each objective and to penalize the deviation of nitrogen oxide concentration from the control input. This is the penalty coefficient; For the first One control quantity, For the first The lower bound of each control variable; The upper bound for each control variable; 1 represents the indicator function, 1 for a constraint violation, and 0 otherwise.

[0094] Step E, F: select the improved DQN algorithm to perform the controller parameter self-tuning algorithm process; roll in the updated state and reward, update the strategy and output the action after constraint optimization, that is, the controller weight parameters Q, R, 、 ; update the corresponding objective function based on the updated weight parameters in the control period, obtain the current optimal control amount by solving through the optimization algorithm, and at the same time, feedback correction and reference trajectory softening parameters in the subsequent control link in the control process; the agent calculates the current reward and stores the experience, updates the optimization strategy to guide the parameter update of the next large cycle, and continuously rolls the controller performance. Based on the above, the controller parameter update framework of the reinforcement learning large cycle + prediction control small cycle collaborative nesting is constructed.

[0095] The embodiment of the application improves the DQN algorithm to observe and collect the associated variables such as the controlled input, output, and predicted input and output in the prediction control environment to form a state space. The agent selects a strategy according to the current state to build an action space inputting the controller by the key parameters of the controller to be tuned, updates the objective function and solves the optimal control amount in the control period, calculates the reward function, stores the experience to update the agent strategy network to better guide the selection of the controller weight parameters in the next control period, realizes the double-cycle collaborative control process of the reinforcement learning cycle large cycle + prediction control cycle small cycle, realizes the dynamic optimization and update of the key weight parameters of the controller, reduces the manual parameter adjustment error, effectively improves the control precision of the controller, greatly improves the automation and standardization of the control system, and improves the running performance of the overall reinforcement learning-model prediction control (RL-MPC) framework based on the hierarchical idea to sample and extract high-value experience with a higher priority in the training process, improves the model strategy update rate, strengthens the stability of the model training process, and further improves the running performance of the overall reinforcement learning-model prediction control (RL-MPC) framework.

[0096] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0097] The following is the device embodiment of the application, and for details not described in detail, reference can be made to the corresponding method embodiments described above.

[0098] Figure 5 The structure of the improved reinforcement learning-based model prediction controller parameter tuning device provided by the embodiment of the application is shown, only the parts related to the embodiment of the application are shown for convenience of description, and the details are described as follows: As Figure 5 shown, the improved reinforcement learning-based model prediction controller parameter tuning device 5 comprises: The variable design module 51 is used to design the state space and action space based on the objective function of the model predictive controller; wherein, the objective function is used to tune the control variables and controlled variables of the model predictive controller. Action optimization module 52 is used to input the current state information into the DQN agent policy network for action decision-making and output the optimized action parameters; The reward calculation module 53 is used to constrain the optimized action parameters and apply them to the model prediction controller to obtain the next state information, calculate the reward value based on the reward function, and form experience pairs to store in the experience pool. The experience sampling module 54 is used to construct a sampling probability distribution based on the temporal difference error of the experience pair, and to sample from the experience pool based on the sampling probability distribution to optimize the DQN agent policy network.

[0099] In one possible implementation, the objective function is: min J || (k)- (k) +||ΔU(k)

[0100] = i = 1,2,…,p e

[0101] +h·e i = 1,2,…,p Among them, J for The objective function value at time t. (k) is the output matrix of the prediction model. (k) is the output set softening value matrix. The weighting parameters, ΔU(k), determine the degree of penalty imposed on the system's state error. To determine the weighting parameters for the system's penalty on the control input, The reference trajectory at time k, The softening coefficient is... This represents the actual output at time k. To output the set value, e The error between the predicted value and the actual value, The prediction result of the prediction model at time k for the system output at time k+1 is given. For the true value, These are the corrected multi-step predictions. h is a feedback correction coefficient for the pre-correction multi-step prediction value.

[0102] In a possible implementation, the state space comprises:

[0103] wherein, is the current state information, is the current state vector including the control variable (determined based on the actual working condition process condition), is the current time output of the control object, is the current time output of the prediction model, is the current feedback correction value, is the current set value, , are the controller weight parameters of the last time respectively, is the current time tracking error, , is the error change rate, , , are the parameter values of the feedback correction coefficient and the softening coefficient of the last time respectively; The action space comprises:

[0104]

[0105] wherein, is the current state vector, is the current action vector, and are the upper and lower limits of the controller weight parameters Q and R respectively, and are the upper and lower limits of the feedback correction coefficient and the softening coefficient respectively.

[0106] In a possible implementation, the reward function is:

[0107]

[0108]

[0109] wherein, , is a normalization coefficient, represents a stability reward, and are stability suppression parameters, , , is a constraint penalty term, is a penalty coefficient, is the i-th control variable, is the i-th control variable, is the lower bound of the i-th control variable, is the upper bound of each control variable, and 1 represents an indicator function, which is 1 if the constraint is violated and 0 otherwise.

[0110] In a possible implementation, the experience sampling module 54 is further configured to: divide the experience pool into a high-value experience area and a low-value experience area based on the time difference error; Correspondingly, the sampling probability distribution is:

[0111]

[0112] wherein, is the sampling probability distribution, is the sampling probability of the high-value experience area in the experience pool, is the sampling probability of the low-value experience area in the experience replay pool, is a reward value division threshold, is a time difference error of an experience pair, is a constant, is a priority index.

[0113] In a possible implementation, the experience sampling module 54 is specifically configured to: if the time difference error of a first experience pair in the experience pool is greater than the reward value division threshold for consecutive times, the first experience pair is divided into the high-value experience area; if the time difference error of the first experience pair is less than the reward value division threshold for consecutive times, the first experience pair is divided into the low-value experience area; wherein, and are preset thresholds, and .

[0114] ​The embodiment of the present application improves the DQN algorithm to observe and collect the associated variables such as the controlled input, output, and predicted input and output in the predictive control environment to form a state space, the agent selects a policy according to the current state to construct an action space inputting the controller composed of key parameters of the controller to be tuned, and updates the objective function and solves the optimal control amount in the control period; the reward function is calculated, the experience is stored for updating the agent policy network to better guide the selection of the controller weight parameter in the next control period, the double-cycle collaborative control process of the reinforcement learning cycle + the small cycle of the predictive control cycle is realized, the dynamic optimization and update of the key weight parameter of the controller are realized, the artificial parameter adjustment error is reduced, the control precision of the controller is effectively improved, the automation and standardization of the control system are greatly improved; the improved DQN extracts high-value experience based on the hierarchical idea with a higher priority in the training process, improves the model policy update rate, strengthens the stability of the model training process, and further improves the running performance of the overall reinforcement learning-model predictive control (RL-MPC) framework.

[0115] For the convenience and brevity of description, only the division of the above functional modules / units is described Figure 6 is a schematic diagram of an electronic device provided by the embodiment of the present application. As shown in Figure 6 The electronic device 6 of this embodiment includes a processor 60 and a memory 61. The memory 61 stores a computer program 62. The processor 60 implements the steps in each of the method embodiments described above when executing the computer program 62. Alternatively, the processor 60 implements the functions of each module / unit in each of the device embodiments described above when executing the computer program 62.

[0116] For example, the computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units can be a series of computer program instruction segments that can complete a specific function, which are used to describe the execution process of the computer program 62 in the electronic device 6.

[0117] The electronic device 6 can include, but is not limited to, the processor 60 and the memory 61. Those skilled in the art can understand, Figure 6 The electronic device 6 is only an example and does not constitute a limitation on the electronic device 6, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the electronic device 6 can also include an input / output device, a network access device, a bus, etc.

[0118] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0119] The memory 61 can be an internal storage unit of the electronic device 6, such as a hard disk or a memory of the electronic device 6. The memory 61 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like equipped on the electronic device 6. Further, the memory 61 can include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store the computer program 62 and other programs and data required by the electronic device 6. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0120] For example, in actual applications, the above functions can be completed by different functional modules / units according to needs. The above modules / units can be implemented in the form of hardware, in the form of software, or in the form of a combination of hardware and software.

[0121] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method in any of the above method embodiments is implemented.

[0122] The embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the method in any of the above method embodiments is implemented.

[0123] The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc.

[0124] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments. If there is no special description and logical conflict, the terms and / or descriptions of different embodiments are consistent and can be mutually referred to, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0125] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for tuning model predictive controller parameters based on improved reinforcement learning, characterized in that, include: The state space and action space are designed based on the objective function of the model predictive controller; wherein, the objective function is used to tune the control variables and controlled variables of the model predictive controller; The current state information is input into the DQN agent policy network to make action decisions, and the optimized action parameters are output. After constraining the optimized action parameters, they are applied to the model prediction controller to obtain the next state information and calculate the reward value based on the reward function. These are then combined into experience pairs and stored in the experience pool. The sampling probability distribution is constructed based on the temporal difference error of the experience pairs, and the DQN agent policy network is optimized by sampling in the experience pool based on the sampling probability distribution.

2. The method for tuning model predictive controller parameters based on improved reinforcement learning according to claim 1, characterized in that, The objective function is: my J || (k)- (k) +||ΔU(k) = ,i = 1,2,…,p e +h·e ,i = 1,2,…,p Among them, J for The objective function value at time t. (k) is the output matrix of the prediction model. (k) is the output set softening value matrix. The weighting parameters, ΔU(k), determine the degree of penalty imposed on the system's state error. To determine the weighting parameters for the system's penalty on the control input, The reference trajectory at time k, The softening coefficient is... This represents the actual output at time k. To output the set value, e The error between the predicted value and the actual value, The prediction result of the prediction model at time k for the system output at time k+1 is given. For the true value, These are the corrected multi-step predictions. is the multi-step predicted value before correction, and h is the feedback correction coefficient.

3. The method for tuning model predictive controller parameters based on improved reinforcement learning according to claim 1, characterized in that, The state space includes: in, This is the current status information. This is the current state vector, which includes control variables (determined based on actual operating conditions and process parameters). To control the current output of the object, The output of the prediction model at the current time step. This is the current feedback correction value. The current setting value, , These are the controller weight parameters from the previous time step. The tracking error at the current moment, , The rate of change of error, , , These are the parameter values ​​of the feedback correction coefficient and the softening coefficient at the previous moment, respectively; The action space includes: in, Let this be the current state vector. For the current action vector, and These are the upper and lower limits of the controller weight parameters Q and R, respectively. and Feedback correction coefficients and softening coefficient The upper and lower limits.

4. The method for tuning model predictive controller parameters based on improved reinforcement learning according to claim 1, characterized in that, The reward function is: in, , The normalization coefficient is... This indicates a stability reward. and To stabilize the suppression parameters, , , To constrain penalties, The penalty coefficient is... For the first One control quantity, For the first The lower bound of each control variable. For each control variable, the upper bound is defined as follows: 1 represents the indicator function, 1 indicates a violation of the constraint, and 0 indicates otherwise.

5. The method for tuning model predictive controller parameters based on improved reinforcement learning according to claim 1, characterized in that, The method further includes: The experience pool is divided into a high-value experience area and a low-value experience area based on the time-series difference error. Accordingly, the sampling probability distribution is: in, For sampling probability distribution, The sampling probability of high-value experience regions in the experience pool. The sampling probability for the low-value experience region in the experience replay pool. Thresholds are set for reward value. For the time difference error of the empirical pair, It is a constant. This is a priority index.

6. The method for tuning model predictive controller parameters based on improved reinforcement learning according to claim 5, characterized in that, The process of dividing the experience pool into high-value experience regions and low-value experience regions based on time-series difference error includes: If the first experience pair in the experience pool is consecutive If the temporal difference error of each time is greater than the reward value division threshold, then the first experience pair will be divided into the high-value experience area. If the first experience is continuous If the temporal difference error of each iteration is less than the reward value classification threshold, then the first experience pair is classified into the low-value experience region; wherein, and For a preset threshold, and .

7. A parameter tuning device for a model predictive controller based on improved reinforcement learning, characterized in that, include: The variable design module is used to design the state space and action space based on the objective function of the model predictive controller; wherein, the objective function is used to tune the control variables and controlled variables of the model predictive controller; The action optimization module is used to input the current state information into the DQN agent policy network for action decision-making and output the optimized action parameters. The reward calculation module is used to constrain the optimized action parameters and apply them to the model prediction controller to obtain the next state information, calculate the reward value based on the reward function, and form experience pairs to store in the experience pool. An experience sampling module is used to construct a sampling probability distribution based on the temporal difference error of experience pairs, and to sample from the experience pool based on the sampling probability distribution to optimize the DQN agent policy network.

8. The parameter tuning device for a model predictor controller based on improved reinforcement learning according to claim 7, characterized in that, The objective function is: my J || (k)- (k) +||ΔU(k) = ,i = 1,2,…,p e +h·e ,i = 1,2,…,p Among them, J for The objective function value at time t. (k) is the output matrix of the prediction model. (k) is the output set softening value matrix. The weighting parameters, ΔU(k), determine the degree of penalty imposed on the system's state error. To determine the weighting parameters for the system's penalty on the control input, The reference trajectory at time k, The softening coefficient is... This represents the actual output at time k. To output the set value, e The error between the predicted value and the actual value, The prediction result of the prediction model at time k for the system output at time k+1 is given. For the true value, These are the corrected multi-step predictions. is the multi-step predicted value before correction, and h is the feedback correction coefficient.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images