MATD3 algorithm-based controller parameter setting method

Abstract

The invention discloses a controller parameter setting method based on an MATD3 algorithm, and relates to the field of steam generator controller parameter setting, and the method comprises the steps: S1, obtaining a current controller parameter set value WLset, the controlled parameter set value WLset comprises the current controller parameter set value, the current controller parameter set value comprises a set value of a difference value between steam flow and feed water flow, and a water flow value; the controller comprises an outer ring controller, namely an auxiliary controller, and an inner ring controller, namely a main controller; s2, the current steam flow value of the steam generator is obtained, and the current steam flow value and the current controlled parameter set value WLset are input into a central controller for parameter setting to obtain target controller parameters; the central controller comprises an inner ring intelligent agent and an outer ring intelligent agent which are trained based on an MATD3 algorithm, and the inner ring intelligent agent and the outer ring intelligent agent correspond to parameter setting of the main controller and the auxiliary controller respectively.

Description

Technical Field

[0001] The present invention relates to the field of steam generator controller parameter tuning, in particular to a controller parameter tuning method based on a MATD3 algorithm. Background Art

[0002] Parameter tuning for steam generator water level controllers is a complex process, as it involves critical aspects such as nuclear power plant safety and operational efficiency. Parameter tuning requires comprehensive consideration of the specific characteristics of the steam generator, the plant's operating conditions, and safety requirements. Because steam generators operate significantly differently at high and low power levels, a single set of tuned controllers cannot meet control requirements at all power levels. Therefore, the steam generator water level control strategy currently used in most third-generation nuclear power plants employs two control modes: high-power mode and low-power mode. This control strategy enables full automatic control across the entire power operating range, but switching between high- and low-power control modes can cause short-term fluctuations or oscillations in the water level.

[0003] In order to better control the water level of the steam generator, a controller parameter tuning method based on the MATD3 algorithm is urgently needed. Summary of the Invention

[0004] To solve the above problems, this application proposes a controller parameter tuning method based on the MATD3 algorithm. The two agents are equivalent to the inner loop controller and the outer loop controller respectively. The MATD3 algorithm is used to find the parameters of the inner and outer controllers that are most suitable for the current control task during the agent training process. The specific content is as follows:

[0005] S1. Get the current controller parameter setting value WL set , the controlled parameter setting value WL set Including the current controller parameter setting value including the setting value of the difference between the steam flow rate and the feed water flow rate, the water flow rate value, the controller includes an outer loop controller, i.e., a sub-controller, and an inner loop controller, i.e., a main controller;

[0006] S2, obtain the current steam flow value of the steam generator, and compare the current steam flow value and the current controlled parameter setting value WL set Input the central controller to perform parameter tuning to obtain the target controller parameters;

[0007] The central controller includes an inner-loop agent and an outer-loop agent trained based on the MATD3 algorithm, and the inner-loop agent and the outer-loop agent correspond to the parameter setting of the main controller and the sub-controller respectively.

[0008] Preferably, the state space of the intelligent agent is the water level error of the steam generator, the integral value of the water level error, the inner loop control error and the integral value of the inner loop control error;

[0009] The state space expression is: s=(e,∫edt,e i ,∫e i dt);

[0010] Where s is the state, e is the difference between the steam generator water level and the water level set value, i is the abbreviation of integral, and t is time.

[0011] Preferably, the Actor network of the inner loop agent includes a state space, a state input layer, an ActOutLyr network layer, and an inner loop control error output layer;

[0012] The state input layer includes 2 state input neurons;

[0013] The ActOutLyr network layer consists of 1 neuron.

[0014] Preferably, the Actor network of the outer ring agent includes a state input layer, an ActOutLyr1 network layer, an ActOutLyr3 network layer, an ActOutLyr2 network layer, and an AddLyr addition layer;

[0015] The state input layer includes StateInLyr1 and StateInLyr2, and StateInLyr1 and StateInLyr2 each include 2 state input neurons;

[0016] The inputs of StateInLyr1 are the water level error and the integral of the water level error, and the inputs of StateInLyr2 are the steam flow rate and the feedwater flow rate;

[0017] The ActOutLyr1 network layer and the ActOutLyr3 network layer include one neuron, which acts as two equivalent PI controllers: the outer loop and the inner loop;

[0018] The ActOutLyr2 network layer includes one neuron, which integrates the input sequence and outputs it;

[0019] AddLyr is an additive layer that adds the input and outputs it.

[0020] Preferably, the reward function of the central controller is expressed as:

[0021]

[0022] Among them, P WL =-e 2 ;

[0023]

[0024] R WL =(3-|e|) 2 ;

[0025] Among them, r is the reward generated by the interaction between the agent and the environment, R WL is the water level error reward, P WL is the water level error penalty, is the feedwater flow penalty, e is the difference between the water level of the steam generator and the water level set value, Q fw is the water flow rate.

[0026] Preferably, the critic network structure of the MATD3 algorithm includes an input layer, a fully connected layer, a connection layer, an activation layer and an output layer;

[0027] The input layer includes two state input neurons for accepting state input and one action input neuron for accepting action input.

[0028] The fully connected layers include,Fc1: connected to the state input layer, including 32 neurons;

[0029] Fc2: connected to the action input layer, including 32 neurons;

[0030] Fc3: connected after the first ReLU activation function layer, including 32 neurons;

[0031] The connection layer connects the outputs from Fc1 and Fc2;

[0032] The activation layers include, Relu_1: ReLU activation function layer after the connection layer;

[0033] Relu_2: The second ReLU activation function layer after the fully connected layer;

[0034] The output layer includes the final Q-value output layer, which is used to output the Q value. Q is the action-value function and includes 1 neuron.

[0035] Preferably, the Actor network parameter of the inner loop agent is the proportional coefficient of the main controller and the integral coefficient

[0036] The Actor network parameter of the outer loop agent is the proportional coefficient of the sub-controller and the integral coefficient

[0037] The action space a1 of the inner loop agent is the steam flow Q sWith water flow Q fw The setting value of the difference is expressed as a1=(Q s -Q fw ) set ;

[0038] The action space a2 of the outer loop agent is the water flow Q fw , the expression is a2=Q fw .

[0039] In summary, the controller parameter tuning method based on the MATD3 algorithm presented in this paper demonstrates superior controllability and interference resistance across all operating conditions compared to conventional techniques. This MATD3-based controller parameter tuning method also offers enhanced stability, preventing significant fluctuations in the controlled parameters after disturbances, making it suitable for engineering applications.

[0040] The technical method of the present invention is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the controller parameter tuning method based on the MATD3 algorithm of the present invention;

[0042] Figure 2 Schematic diagram of the Critic network structure of the present invention;

[0043] Figure 3 This is a schematic diagram of the Actor network of the inner ring agent of the present invention;

[0044] Figure 4 This is a schematic diagram of the Actor network of the outer ring agent of the present invention;

[0045] Figure 5 The water level anti-interference test results of the traditional control strategy model and the MATD3 coordinated control model at different power levels in the present invention are as follows: Figure 5 (a) is the water level change curve of the steam generator at 10% FP power level. Figure 5 (b) is the water level change curve of the steam generator at 50% FP power level. Figure 5 (c) in FIG. 5 is when the steam generator is at 100% FP power level. DETAILED DESCRIPTION

[0046] The technical method of the present invention is further described below through the accompanying drawings and embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and values ​​described in these embodiments do not limit the scope of this application.

[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0048] Technologies, systems, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0049] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0050] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0051] like Figure 1 As shown in Figure 1, a controller parameter tuning method based on the MATD3 algorithm is shown in Figure 1. The specific contents are as follows:

[0052] S1. Get the current controller parameter setting value WL set , the controlled parameter setting value WL set Including the current controller parameter setting value including the setting value of the difference between the steam flow rate and the feed water flow rate, the water flow rate value, the controller includes an outer loop controller, i.e., a sub-controller, and an inner loop controller, i.e., a main controller;

[0053] S2, obtain the current steam flow value of the steam generator, and compare the current steam flow value and the current controlled parameter setting value WL set Input the central controller to perform parameter tuning to obtain the target controller parameters;

[0054] The central controller includes an inner-loop agent and an outer-loop agent trained based on the MATD3 algorithm. The inner-loop agent and the outer-loop agent correspond to the parameter settings of the main controller and the sub-controller, respectively. During the training process, the agent first interacts with the environment to collect state, action, and reward information. The same controller parameters are used in each episode of training. Before the start of the next episode of training, the policy network (Actor network) and value network (Critic network) of the MATD3 algorithm are updated to gradually optimize the control parameters. Experience replay and delayed update mechanisms are used to improve the stability and efficiency of training. Finally, the above process is repeated until the controller performance meets the preset requirements.

[0055] Furthermore, the state space of the central controller is the water level error of the steam generator, the integral value of the water level error, the inner loop control error and the integral value of the inner loop control error;

[0056] The state space expression is: s=(e,∫edt,ei ,∫e i dt);

[0057] Where s is the state, e is the difference between the steam generator water level and the water level set value, i is the abbreviation of integral, and t is time.

[0058] Furthermore, the Actor network of the inner-loop intelligent agent includes a state space, a state input layer, an ActOutLyr (ActorOutput Layer) network layer, and an inner-loop control error output layer. The inner-loop control error output layer represents a specific action, i.e., a change in water flow rate.

[0059] The state input layer includes 2 state input neurons;

[0060] The ActOutLyr network layer consists of 1 neuron.

[0061] Furthermore, the Actor network parameters of the inner loop agent are the proportional coefficients of the main controller. and the integral coefficient By continuously learning and optimizing these two coefficients, we can find a set of controller parameter combinations with the highest cumulative reward.

[0062] Furthermore, the Actor network of the outer ring agent includes a state input layer, an ActOutLyr1 network layer, an ActOutLyr3 network layer, an ActOutLyr2 network layer, and an AddLyr addition layer (Add Layer).

[0063] The state input layer includes StateInLyr1 and StateInLyr2, and each of StateInLyr1 and StateInLyr2 includes two state input neurons (State Input Layer, state input layer);

[0064] The inputs of StateInLyr1 are the water level error and the integral of the water level error, and the inputs of StateInLyr2 are the steam flow rate and the feedwater flow rate;

[0065] The ActOutLyr1 network layer and the ActOutLyr3 network layer include one neuron, which acts as two equivalent PI controllers: the outer loop and the inner loop;

[0066] The ActOutLyr2 network layer includes one neuron, which integrates the input sequence and outputs it;

[0067] The AddLyr layer is used to add the input and output it. The Actor network parameter of the outer loop agent is the proportional coefficient of the sub-controller and the integral coefficient

[0068] Optimize the scaling factor through continuous learning Integration coefficient Proportional coefficient and the integral coefficient Then find a set of controller parameter combinations with the highest cumulative reward.

[0069] Furthermore, the reward function of the central controller is expressed as:

[0070]

[0071] Among them, P WL =-e 2 ;

[0072]

[0073] Among them, r is the reward generated by the interaction between the agent and the environment, R WL is the water level error reward, P WL is the water level error penalty, is the feedwater flow penalty, e is the difference between the water level of the steam generator and the water level set value, Q fw is the water flow rate.

[0074] Furthermore, the critic network structure of the MATD3 algorithm includes an input layer, a fully connected layer, a connection layer, an activation layer, and an output layer;

[0075] The input layer includes two state input neurons for accepting state input and one action input neuron for accepting action input.

[0076] The fully connected layers include,Fc1: connected to the state input layer, including 32 neurons;

[0077] Fc2: connected to the action input layer, including 32 neurons;

[0078] Fc3: connected after the first ReLU activation function layer, including 32 neurons;

[0079] The connection layer connects the outputs from Fc1 and Fc2;

[0080] The activation layers include, Relu_1: ReLU activation function layer after the connection layer;

[0081] Relu_2: The second ReLU activation function layer after the fully connected layer;

[0082] The output layer includes the final Q value output layer, which is used to output the Q value and includes 1 neuron.

[0083] The action space a1 of the inner loop agent is the steam flow Q s With water flow Q fw The setting value of the difference is expressed as a1=(Q s -Q fw ) set ;

[0084] The action space a2 of the outer loop agent is the water flow Q fw , the expression is a2=Q fw .

[0085] Figure 1 The operation process is as follows:

[0086] The Owen model of the steam generator is the controlled object and generates a water level feedback signal WL.

[0087] Set the water level WL set Subtract it from the actual water level WL to obtain the error signal.

[0088] The agent receives the error information and outputs a control action a t .

[0089] Action a t Transfer to the environment, affecting the steam generator behavior.

[0090] Steam generator in action t The new water level WL and steam flow Q are generated under the action of s .

[0091] Current status i , action taken a i , the reward r i , next state s i+1 are sampled and stored in the experience pool.

[0092] The central controller samples multiple quaternion arrays N×(s i ,a i ,r i ,s i+1 ), used to train deep learning models and update the agent's strategy after learning.

[0093] The updated agent continues to interact with the system in the environment, forming a closed-loop control.

[0094] In the figure, WL set is the target water level setting value, i.e. the set water level; WL is the current steam generator water level, i.e. the actual water level; a t is the action of the agent at time step t, Q s is the steam flow rate, Q fwis the water flow rate, s i is the agent state, a i is the agent action, r i is the agent reward, s i+1 is the next state of the agent, N×(s i ,a i ,r i ,s i+1 ) is the set of experience tuples used for model training.

[0095] This application conducted a steam flow disturbance test on the traditional control strategy model and the MATD3 coordinated control model (this application) at different power levels. During the test, a steam flow step disturbance of +40kg / s was added at 10s.

[0096] MATD3 coordinated control model (this application): two intelligent agents are used to adjust the PI parameters of the main controller and the sub-controller simultaneously.

[0097] This application shows the water level change curves for the first 2000s of the low test condition and the water level change curves for the first 1000s of the medium and high test conditions, such as Figure 5 shown.

[0098] To more specifically evaluate control effectiveness, this application uses maximum deviation and recovery time as evaluation indicators to assess the dynamic performance of the water level anti-interference test. Maximum deviation refers to the maximum deviation of the controlled parameter from the set value after the system is disturbed. The permissible error range of the water level in a steam generator is typically within ±10 mm. Therefore, the recovery time specified in this application refers to the minimum time it takes for the controlled parameter to recover to within ±10 mm of the set value after the system is disturbed, and not exceed this limit during subsequent operation.

[0099] Table 1 shows the dynamic performance indicators of anti-interference test. Figure 5 As can be seen from Table 1, at low power levels, MATD3 coordinated control performs better, with smaller maximum deviation and shorter recovery time. At high power levels, MATD3 coordinated control has the best anti-interference performance, with both maximum deviation and recovery time significantly better than traditional control strategies.

[0100] It can be said that within a limited number of training rounds, the MATD3 coordinated control strategy demonstrates excellent control and interference resistance under all operating conditions. MATD3 coordinated control also offers enhanced stability, preventing significant fluctuations in controlled parameters after disturbances, making it suitable for engineering applications.

[0101] Table 1 Comparison of dynamic performance indicators of anti-interference test

[0102]

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical method of the present invention and not to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in this field should understand that they can still modify or replace the technical method of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical method to deviate from the spirit and scope of the technical method of the present invention.

Claims

1. A controller parameter tuning method based on the MATD3 algorithm, characterized in that: The following steps are involved: S1. Get the current controller parameter setting value WL set , the controlled parameter setting value WL set Including the current controller parameter setting value including the setting value of the difference between the steam flow rate and the feed water flow rate, the water flow rate value, the controller includes an outer loop controller, i.e., a sub-controller, and an inner loop controller, i.e., a main controller; S2, obtain the current steam flow value of the steam generator, and compare the current steam flow value and the current controlled parameter setting value WL set Input the central controller to perform parameter tuning to obtain the target controller parameters; The central controller includes an inner-loop agent and an outer-loop agent trained based on the MATD3 algorithm, and the inner-loop agent and the outer-loop agent correspond to the parameter setting of the main controller and the sub-controller respectively.

2. A controller parameter tuning method based on the MATD3 algorithm according to claim 1, characterized in that: The state space of the intelligent agent is the water level error of the steam generator, the integral value of the water level error, the inner loop control error and the integral value of the inner loop control error; The state space expression is: s=(e,∫edt,e i ,∫e i dt); Where s is the state, e is the difference between the steam generator water level and the water level set value, i is the abbreviation of integral, and t is time.

3. The controller parameter tuning method based on the MATD3 algorithm according to claim 1 is characterized in that: The Actor network of the inner-loop agent includes the state space, state input layer, ActOutLyr network layer, and inner-loop control error output layer; The state input layer includes 2 state input neurons; The ActOutLyr network layer consists of 1 neuron.

4. The controller parameter tuning method based on the MATD3 algorithm according to claim 2 is characterized in that: The Actor network of the outer loop agent includes the state input layer, ActOutLyr1 network layer, ActOutLyr3 network layer, ActOutLyr2 network layer, and AddLyr addition layer; The state input layer includes StateInLyr1 and StateInLyr2, and StateInLyr1 and StateInLyr2 each include 2 state input neurons; The inputs of StateInLyr1 are the water level error and the integral of the water level error, and the inputs of StateInLyr2 are the steam flow rate and the feedwater flow rate; The ActOutLyr1 network layer and the ActOutLyr3 network layer include one neuron, which acts as two equivalent PI controllers: the outer loop and the inner loop; The ActOutLyr2 network layer includes one neuron, which integrates the input sequence and outputs it; AddLyr is an additive layer that adds the input and outputs it.

5. The controller parameter tuning method based on the MATD3 algorithm according to claim 1 is characterized in that: The reward function of the central controller is expressed as: Among them, P WL =-e 2 ; R WL =(3-|e|) 2 ; Among them, r is the reward generated by the interaction between the agent and the environment, R WL is the water level error reward, P WL is the water level error penalty, is the feedwater flow penalty, e is the difference between the steam generator water level and the water level set value, Q fw is the water flow rate.

6. The controller parameter tuning method based on the MATD3 algorithm according to claim 1, characterized in that: The critic network structure of the MATD3 algorithm includes an input layer, a fully connected layer, a connection layer, an activation layer, and an output layer; The input layer includes two state input neurons for accepting state input and one action input neuron for accepting action input. The fully connected layers include,Fc1: connected to the state input layer, including 32 neurons; Fc2: connected to the action input layer, including 32 neurons; Fc3: connected after the first ReLU activation function layer, including 32 neurons; The connection layer connects the outputs from Fc1 and Fc2; The activation layers include, Relu_1: ReLU activation function layer after the connection layer; Relu_2: The second ReLU activation function layer after the fully connected layer; The output layer includes the final Q value output layer, which is used to output the Q value and includes 1 neuron.

7. The controller parameter tuning method based on the MATD3 algorithm according to claim 1, characterized in that: The Actor network parameters of the inner loop agent are the proportional coefficients of the main controller and the integral coefficient The Actor network parameter of the outer loop agent is the proportional coefficient of the sub-controller and the integral coefficient The action space a1 of the inner loop agent is the steam flow Q s With water flow Q fw The setting value of the difference is expressed as a1=(Q s -Q fw ) set ; The action space a2 of the outer loop agent is the water flow Q fw , the expression is a2=Q fw .

Images