A multi-objective control optimization method and apparatus for a nuclear steam supply system

Abstract

This invention discloses a multi-objective control optimization method and apparatus for a nuclear steam supply system. The method includes: determining the input and output parameters of an optimizer based on multiple control objectives to be optimized in the nuclear steam supply system; determining a reward function based on the multiple control objectives to be optimized, the input and output parameters of the optimizer; loading a preset deep learning network model and the reward function into the optimizer, and configuring an event triggering mechanism for the optimizer; when the optimizer is triggered to execute, controlling the optimizer to perform multiple iterations to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value; the optimizer executes control optimization operations based on the multi-objective control optimization strategy, which can optimize the performance of multiple control objectives of the nuclear steam supply system while ensuring the stability and robustness of the system's operation.

Description

Technical Field

[0001] This invention relates to the field of nuclear power plant automatic control technology, specifically to a multi-objective control optimization method and apparatus for a nuclear steam supply system. Background Technology

[0002] With the increasing global demand for energy resources and carbon emission reduction, nuclear fission energy is attracting more and more attention due to its clean and high-density characteristics. The Nuclear Steam Supply System (NSSS) is a system in nuclear power plants that uses nuclear fission energy to generate steam, primarily composed of a fission reactor steam generator. A typical NSSS transfers the heat generated by the fission reactor to the secondary loop system through the steam generator, thereby producing a steam flow to drive thermal load units for power generation or combined heat and power (CHP).

[0003] Existing nuclear steam supply system control is almost entirely achieved through traditional control methods, such as state feedback control and sliding mode control. However, due to the complex coupling between nuclear steam generators in fission reactors, human intervention by operators is often required, increasing their workload. To reduce human intervention, the control technology of nuclear steam supply systems needs to be optimized.

[0004] Current control optimization techniques for nuclear steam supply systems rely on predictions from dynamic models of the system. However, due to the nonlinear and strongly coupled nature of nuclear steam supply systems, the accuracy of model predictions is poor, making it difficult to guarantee optimization performance and system operational stability. Furthermore, this control optimization technique cannot simultaneously optimize multiple control objectives.

[0005] Therefore, there is an urgent need for a multi-objective control optimization method that can be applied to nuclear steam supply systems and does not depend on the system dynamic model. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a multi-objective control optimization method and apparatus for a nuclear steam supply system to solve the problem of poor performance of current control optimization techniques for nuclear steam supply systems.

[0007] According to a first aspect, embodiments of the present invention provide a multi-objective control optimization method for a nuclear steam supply system, the method comprising:

[0008] The input and output parameters of the optimizer are determined based on multiple control objectives to be optimized in the nuclear steam supply system; the optimizer is an optimization program used to solve multi-objective control optimization problems.

[0009] The reward function is determined based on the multiple control objectives to be optimized, the input parameters and output parameters of the optimizer;

[0010] A preset deep learning network model and the reward function are loaded into the optimizer, and an event triggering mechanism is configured for the optimizer; the event triggering mechanism is a mechanism used to trigger the optimizer to execute according to the predetermined operating state of the nuclear steam supply system;

[0011] When the optimizer is triggered to execute, the optimizer is controlled to perform multiple iterations to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value.

[0012] The optimizer is controlled to perform control optimization operations based on the multi-objective control optimization strategy to achieve optimization of multiple control objectives.

[0013] In some embodiments, after determining the reward function corresponding to the plurality of control objectives based on the input and output parameters of the optimizer, the method further includes:

[0014] The multi-objective control optimization problem is modeled as a Markov decision process, and a quintuple consisting of the input state space, action space, state transition function, discount factor, and initial state distribution is defined. The input state space is a parameter set determined based on the input parameters; the action space is a parameter set determined based on the output parameters; the state transition function is a function used to determine the state of the current time step based on the state of the previous time step and the selected action; and the discount factor is used to characterize the importance of the reward value obtained at each time step in the cumulative reward.

[0015] A deep reinforcement learning algorithm is determined based on the Markov decision process described above;

[0016] The preset deep learning network model is determined based on a deep reinforcement learning algorithm.

[0017] In some embodiments, the preset deep learning network model includes a first network and a second network; controlling the optimizer to perform multiple iterations to obtain a multi-objective control optimization strategy includes:

[0018] Initialize the network parameters of the first network and the network parameters of the second network;

[0019] The state space is initialized to obtain the current state;

[0020] For the current time step, a first processing procedure is executed; the first processing procedure includes: selecting an action in the action space based on the current state according to the policy output by the first network, executing the state transition function to obtain the state of the next time step, and obtaining the reward value of the current time step according to the reward function;

[0021] The reward value for each time step is accumulated based on the discount factor, and the second network determines whether the currently accumulated reward value is less than the preset accumulated reward function value.

[0022] If the accumulated reward value is less than the preset accumulated reward function value, the network parameters of the first network and the second network are updated based on the reward of the current time step to obtain the updated preset deep learning network model, and the first processing procedure is re-executed based on the updated preset deep learning network model in the next time step.

[0023] If the accumulated reward value is greater than or equal to the preset accumulated reward function value, the preset deep learning network model is invoked to generate the multi-objective control optimization strategy.

[0024] In some embodiments, after loading the preset deep learning network model and the reward function into the optimizer and configuring an event triggering mechanism for the optimizer, the method further includes:

[0025] At each time step, the operating status of the nuclear steam supply system is determined according to preset performance evaluation indicators;

[0026] When the operating state is in the first state, the nuclear steam supply system is controlled to issue an operating early warning reminder;

[0027] When the running state is the second state, control the start of execution of the optimizer;

[0028] If the running state is in the third state, wait to enter the next time step.

[0029] In some embodiments, the input parameters include the deviation between the actual value and the reference value corresponding to the control target, the system parameters and operating parameters of the nuclear steam supply system; the output parameter is a preset correction amount corresponding to the control target.

[0030] In some embodiments, the step of determining the reward function based on the input and output parameters of the optimizer includes:

[0031] For each control objective, obtain the difference between the input parameter corresponding to the control objective and the maximum permissible steady-state error corresponding to the control objective, and the ratio of the absolute value of the difference to the maximum permissible steady-state error;

[0032] The reward function is determined based on the ratio corresponding to each control objective.

[0033] In some embodiments, the deep reinforcement learning algorithm includes any one of the deep Q-network learning algorithm, the deep deterministic policy gradient algorithm, and the actor-critic algorithm.

[0034] According to a second aspect, embodiments of the present invention provide a multi-objective control optimization device for a nuclear steam supply system, the device comprising:

[0035] The parameter determination module is used to determine the input and output parameters of the optimizer based on multiple control objectives to be optimized in the nuclear steam supply system; the optimizer is an optimization program used to solve multi-objective control optimization problems.

[0036] The function determination module is used to determine the reward function based on multiple control objectives to be optimized, the input parameters and output parameters of the optimizer;

[0037] An optimization configuration module is used to load a preset deep learning network model and the reward function into the optimizer, and to configure an event triggering mechanism for the optimizer; the event triggering mechanism is a mechanism used to trigger the optimizer to execute according to the predetermined operating state of the nuclear steam supply system;

[0038] An iteration module is used to control the optimizer to perform multiple iterations when the optimizer is triggered to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy for obtaining a preset reward function value.

[0039] An optimization execution module is used to control the optimizer to perform control optimization operations based on the multi-objective control optimization strategy, so as to achieve optimization of multiple control objectives.

[0040] According to a third aspect, an embodiment of the present invention provides a computer device, characterized in that it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the steps of the multi-objective control optimization method for a nuclear steam supply system as described in the first aspect.

[0041] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the multi-objective control optimization method for a nuclear steam supply system as described in the first aspect.

[0042] The technical solution of this invention has the following advantages:

[0043] This invention provides a multi-objective control optimization method and apparatus for a nuclear steam supply system. The method includes: determining the input and output parameters of an optimizer based on multiple control objectives to be optimized in the nuclear steam supply system; the optimizer is an optimization program used to solve multi-objective control optimization problems; determining a reward function based on the multiple control objectives to be optimized, the input and output parameters of the optimizer; loading a preset deep learning network model and the reward function into the optimizer, and configuring an event triggering mechanism for the optimizer; the event triggering mechanism is a mechanism used to trigger the optimizer to execute according to a predetermined operating state of the nuclear steam supply system; when the optimizer is triggered to execute, controlling the optimizer to perform multiple iterations to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value; and controlling the optimizer to perform control optimization operations based on the multi-objective control optimization strategy. This invention can optimize the performance of multiple control objectives of the nuclear steam supply system while ensuring the stability and robustness of the system's operation, thereby improving the effectiveness of control optimization and increasing the operating efficiency of the nuclear steam supply system. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort:

[0045] Figure 1 This is a flowchart of a multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of a nuclear steam supply system provided in an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram illustrating the principle of multi-objective control optimization of a nuclear steam supply system provided in an embodiment of the present invention.

[0048] Figure 4 A flowchart of another multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0049] Figure 5 This is a flowchart illustrating a method for obtaining a multi-objective control optimization strategy, as provided in an embodiment of the present invention.

[0050] Figure 6 A flowchart of another multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0051] Figure 7 The figure shows an example of the effect of a multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0052] Figure 8 An example diagram illustrating the effect of another multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0053] Figure 9 This is a schematic diagram of the structure of a multi-objective control optimization device for a nuclear steam supply system provided in an embodiment of the present invention.

[0054] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Figure 1 This is a flowchart illustrating a multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention. Figure 1 As shown, the multi-objective control optimization method for the nuclear steam supply system includes steps S1-S2.

[0057] Step S1: Determine the input and output parameters of the optimizer based on the multiple control objectives to be optimized in the nuclear steam supply system.

[0058] The Nuclear Steam Supply System (NSSS) is a system in a nuclear power plant that uses nuclear fission energy to generate steam, and it mainly consists of a fission reactor nuclear steam generator.

[0059] The control objectives are indicators that need to be optimized in the nuclear steam supply system. These control objectives include, but are not limited to, thermal power, outlet steam temperature, and outlet steam pressure.

[0060] In some embodiments, the thermal power in the control target can be represented by reactor full power (RFP), and the maximum permissible error for the transient state of the control target is set to 5%, and the maximum permissible error for the steady state is set to 0.5%. The outlet steam temperature can be represented by degrees Celsius, and the maximum permissible error for the transient state of the control target is set to 2°C, and the maximum permissible error for the steady state is set to 0.2°C.

[0061] Figure 2 This is a schematic diagram of a nuclear steam supply system provided in an embodiment of the present invention. Figure 2 As shown, the operator manually adjusts the load demand, which is then input into the nuclear steam supply system as a system input. This load demand characterizes the load on the nuclear steam supply system, referring to the ratio of the actual power generation to the rated power generation. The system maps the load demand to controller reference values, such as those for the neutron flux controller, primary temperature controller, and secondary temperature controller, using a static mapping table that considers thermal-hydraulic and physical design principles. The controllers adjust actuators such as control rods, pumps, blowers, and feedwater pumps according to these reference values, ensuring that system outputs, such as thermal power and outlet steam temperature, reach the reference values, thereby meeting the load demand. To ensure the efficient and safe operation of the nuclear steam supply system, precise control of the fission reactor's thermal power and the steam generator's outlet steam temperature is required. Stable closed-loop control can be achieved at present, but due to the nonlinear and strongly coupled characteristics of the nuclear steam supply system, the reference value trajectory provided by the static mapping table can only reflect the reference steady-state value of the process variable, and the transient performance of the system control needs to be further improved.

[0062] An optimizer is an optimization program used to solve multi-objective control optimization problems.

[0063] In this embodiment of the invention, in order for the optimizer to be directly applied to the nuclear steam supply system of a nuclear power plant, the input parameters of the optimizer should be directly obtained or inferred from the nuclear steam supply system of the nuclear power plant, and include operating parameters related to the control objective.

[0064] The optimizer's input parameters include: the deviation between the actual and reference values ​​corresponding to the control target, and the system and operating parameters of the nuclear steam supply system. The optimizer's output parameters include: the preset correction amount corresponding to the control target.

[0065] In one embodiment, taking the control objectives including thermal power and outlet steam temperature as an example, the optimizer input parameters include: the deviation Δn between the actual value and the reference value of thermal power, and the deviation ΔT between the actual value and the reference value of outlet steam temperature. sThe reactor reactivity measurement ρ (system parameter), along with related flow rate G (operating parameter), temperature T (operating parameter), and pressure P (operating parameter) measurements (operating parameters) are used as input parameters to the optimizer. Furthermore, to optimize the control performance of the nuclear steam supply system, the first derivative correction of the load demand is selected as the optimizer's output. Further, considering the safety operation indicators of the nuclear power plant, the optimizer output is limited to [-10%, 10%] FP / min.

[0066] Figure 3 This is a schematic diagram illustrating the principle of multi-objective control optimization for a nuclear steam supply system, provided as an embodiment of the present invention. Figure 3 As shown, taking the control targets including thermal power and outlet steam temperature as an example, the original load demand and related operational parameters obtained from the nuclear steam supply system are input into the optimizer to obtain the first derivative revision of the load output by the optimizer. These related operational parameters include, for example, the deviation Δn between the actual and reference values ​​of thermal power and the deviation ΔT between the actual and reference values ​​of outlet steam temperature. s The reactor's reactivity measurements ρ, along with relevant flow rates G, temperature T, and pressure P, are used to determine the modified load demand. This modified load demand is then applied to the nuclear steam supply system to achieve multi-objective control optimization, such as the control processes for thermal power generation and outlet steam temperature.

[0067] Step S2: Determine the reward function based on the multiple control objectives to be optimized, the input parameters and output parameters of the optimizer.

[0068] The reward function is a function used to determine the reward value obtained by being in a certain state or taking a specific action.

[0069] In one implementation, the step of determining the reward function based on multiple control objectives to be optimized, the input parameters and output parameters of the optimizer includes: the following steps one and two.

[0070] Step 1: For each control objective, obtain the difference between the input parameter corresponding to the control objective and the maximum permissible steady-state error corresponding to the control objective, as well as the ratio of the absolute value of the difference to the maximum permissible steady-state error.

[0071] Step 2: Determine the reward function based on the ratio corresponding to each control target.

[0072] In step two, the negative of the sum of the products of each ratio and its corresponding coefficient can be used as the reward function.

[0073] Taking the control objectives of thermal power and outlet steam temperature as an example, the reward function is set as r1 as follows.

[0074] The reward function r1 = -(ε n ·β n +ε T ·β TS ),in, Used to represent thermal power control deviation and steady-state maximum permissible error ∈ n The ratio of the thermal power control deviation |Δn-∈ n | represents the input parameter corresponding to the heat power (the deviation Δn between the actual heat power value and the reference value) and the maximum permissible steady-state error corresponding to the heat power. n The absolute value of the difference; Used to indicate the deviation of outlet steam temperature control and the maximum permissible error in steady state. The proportion of the outlet steam temperature control deviation This represents the input parameter corresponding to the outlet steam temperature (the deviation ΔT between the actual and reference values ​​of the outlet steam temperature). S The maximum permissible steady-state error corresponding to thermal power ∈ n The absolute value of the difference. The coefficients corresponding to thermal power and outlet steam temperature are β. n and

[0075] Step S3: Load the preset deep learning network model and reward function into the optimizer, and configure the event triggering mechanism for the optimizer.

[0076] The preset deep learning network model is a pre-selected deep learning network model.

[0077] The event-triggered mechanism is a mechanism used to trigger the optimizer to execute based on the predetermined operating state of the nuclear steam supply system.

[0078] This invention, through the addition of a deep reinforcement learning-based optimizer to the nuclear steam supply system of a nuclear power plant, can assist operators in optimizing the control performance of the nuclear steam supply system based on the system's operating parameters, thereby reducing the operator's workload.

[0079] Furthermore, the aforementioned event-triggered mechanism can determine the optimizer's operating range. Although deep reinforcement learning-based control optimizers can obtain optimal strategies through exploration and trial and error without relying on system dynamic models, this increases the risk that the optimizer may adopt harmful strategies that negatively impact the safe operation of the nuclear power plant during training and execution. This invention introduces an event-driven mechanism to determine the optimizer's operating range, thereby ensuring that the optimizer optimizes multiple control objectives of the nuclear steam supply system under the premise of safe and stable reactor operation.

[0080] Step S4: When the optimizer is triggered to execute, control the optimizer to perform multiple iterations to obtain a multi-objective control optimization strategy.

[0081] Among them, the multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value. This preset cumulative reward function value generally refers to the maximum cumulative reward function value that can be obtained through the reward function after multiple iterations.

[0082] This invention uses a deep reinforcement learning algorithm to automatically acquire optimizers that can improve the performance of multiple control targets. This optimization process does not require human intervention and increases the intelligence level of the nuclear steam supply system.

[0083] Step S5: The control optimizer performs control optimization operations based on a multi-objective control optimization strategy to achieve optimization of multiple control objectives.

[0084] The control optimization operation includes dynamically correcting the reference values ​​of the corresponding controllers in the system based on the operating parameters of the nuclear steam supply system, so as to optimize the control performance of the nuclear steam supply system.

[0085] This invention provides a multi-objective control optimization method for a nuclear steam supply system. First, based on multiple control objectives to be optimized in the nuclear steam supply system, the input and output parameters of an optimizer are determined. The optimizer is an optimization program used to solve multi-objective control optimization problems. Then, based on the multiple control objectives to be optimized, the input and output parameters of the optimizer, a reward function is determined. A preset deep learning network model and the reward function are loaded into the optimizer, and an event triggering mechanism is configured for the optimizer. The event triggering mechanism is used to trigger the optimizer to execute according to a predetermined operating state of the nuclear steam supply system. Next, when the optimizer is triggered to execute, it is controlled to perform multiple iterations to obtain a multi-objective control optimization strategy. The multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value. Finally, the optimizer executes control optimization operations based on the multi-objective control optimization strategy to optimize the multiple control objectives. This invention can optimize the performance of multiple control objectives of a nuclear steam supply system while ensuring the stability and robustness of the system, improving the effectiveness of control optimization and increasing the operating efficiency of the nuclear steam supply system.

[0086] Figure 4 A flowchart illustrating another multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention. Figure 4 As shown, after determining the reward function corresponding to multiple control objectives based on the input and output parameters of the optimizer (step S2), the process also includes steps S6-S7.

[0087] Step S6: Model the multi-objective control optimization problem as a Markov decision process and define a quintuple consisting of the input state space, action space, state transition function, discount factor, and initial state distribution.

[0088] In this system, the input state space is a parameter set determined by the input parameters, with each input parameter corresponding to a state in the state space; the action space is a parameter set determined by the output parameters, with each output parameter corresponding to an action in the action space; the state transition function is a function used to determine the state of the current time step based on the state of the previous time step and the selected action; and the discount factor is used to characterize the importance of the reward value obtained at each time step in the cumulative reward. Therefore, the cumulative reward depends on the action chosen by the optimizer, i.e., the strategy adopted by the optimizer. In other words, the optimizer's goal is to learn a strategy that maximizes the cumulative reward from the initial state distribution.

[0089] Step S7: Determine the deep reinforcement learning algorithm based on the Markov decision process, and determine the preset deep learning network model based on the deep reinforcement learning algorithm.

[0090] Among them, the deep reinforcement learning algorithm is a reinforcement learning algorithm that can solve the Markov decision process, and the preset deep learning network model is a deep learning network model corresponding to the deep reinforcement learning algorithm.

[0091] In one implementation, the deep reinforcement learning algorithm includes any one of the deep Q-network learning algorithm, the deep deterministic policy gradient algorithm, and the actor-critic algorithm.

[0092] In one implementation, when the deep reinforcement learning algorithm is the actor-critic algorithm, the preset deep learning network model is a deep learning network model that includes an actor network and a critic network.

[0093] In one implementation, the actor-critic algorithm, using a pre-defined deep learning network model, will be used as an example for explanation. This actor-critic algorithm is an algorithmic model that uses the maximum entropy objective to formulate a random strategy.

[0094] The structure of the preset deep learning network model is as follows.

[0095] Where, π * For optimization strategy; argmax() is a function used to find the set of variable points that maximize the objective function; E represents expectation; t represents time step; S represents state, S t This represents the state at time step t; 'a' represents the action. t Indicates the action performed at time step t; ρ π The state-action distribution of the optimization strategy is represented; i takes values ​​from t to infinity and is used for counting time steps; r represents the reward function; α is the temperature coefficient. This represents the entropy of the balancing strategy.

[0096] The entropy objective can be represented by the optimal soft Q-value function, which is shown below.

[0097] Where r represents the reward function; E represents the expectation; and k takes the value [1, ∞] and is used to count the time steps.

[0098] In this optimal soft Q-value function, π * This is the optimal soft strategy.

[0099] Should Among them, V * (s t ) is the soft state value function.

[0100] Here, A represents the action space, which includes all actions; that is, A is a. tA set of.

[0101] Figure 5 This is a flowchart illustrating a method for obtaining a multi-objective control optimization strategy according to an embodiment of the present invention. In one embodiment, the preset deep learning network model includes a first network and a second network. Figure 5 As shown, the control optimizer performs multiple iterations to obtain a multi-objective control optimization strategy (step S4), including steps S41-S46.

[0102] Step S41: Initialize the network parameters of the first network and the network parameters of the second network.

[0103] In one implementation, if the preset deep learning network model is the actor-critic algorithm, the first network is the actor network and the second network is the critic network.

[0104] Step S42: Initialize the state space to obtain the current state.

[0105] In this context, the state space refers to the input state space defined when modeling a multi-objective control optimization problem as a Markov decision process. This input state space is based on a set of parameters determined by the input parameters, and each input parameter corresponds to a state in the state space.

[0106] Step S43: For the current time step, execute the first processing procedure.

[0107] The first processing procedure includes: selecting an action in the action space based on the current state according to the policy output by the first network, executing the state transition function to obtain the state of the next time step, and obtaining the reward value of the current time step according to the reward function.

[0108] The action is contained in the action space, which is a set of parameters determined based on the output parameters. Each output parameter corresponds to an action in the action space. The state transition function is a function used to determine the state of the current time step based on the state of the previous time step and the selected action. The discount factor is used to characterize the importance of the reward value obtained at each time step in the cumulative reward.

[0109] Step S44: Accumulate the reward value for each time step based on the discount factor, and the second network determines whether the currently accumulated reward value is less than the preset accumulated reward function value.

[0110] Step S45: If the accumulated reward value is less than the preset accumulated reward function value, update the network parameters of the first network and the second network respectively based on the reward of the current time step to obtain the updated preset deep learning network model, and re-execute the first processing procedure based on the updated preset deep learning network model in the next time step.

[0111] In this embodiment, during the process of updating the network parameters of the first network and the second network, the soft Q-value function parameter θ can be iteratively updated using the Bellman residual, and its gradient can be calculated using the following formula 1.

[0112]

[0113] Where ▽ represents the gradient; π φ Let Q represent the first network, and φ represent the network parameters of the first network. θ Let θ represent the second network, and θ be the network parameters of the second network. This indicates the historical network corresponding to the second network. These are historical network parameters. After initializing the network parameters of the first and second networks, the network... and network Q θ Exactly the same, in the first processing step described above, Q θ It will iterate continuously according to the following formula 2. It is based on a certain pattern (for example, every N time steps it is related to Q). θ (Same as above), as Q θ The historical network is constantly updated.

[0114] The stochastic policy can be updated by minimizing the expected KL divergence, the gradient of which can be calculated by Equation 2.

[0115]

[0116] Among these, the reparameterization technique a can be used. t =f φ (ε t ;s t This allows the policy parameter φ to be backpropagated.

[0117] Step S46: If the accumulated reward value is greater than or equal to the preset accumulated reward function value, call the preset deep learning network model to generate a multi-objective control optimization strategy.

[0118] In this embodiment of the invention, an optimizer that can improve the performance of multiple control targets is automatically obtained through a deep reinforcement learning algorithm. This optimization process does not require human intervention and increases the intelligence level of the nuclear steam supply system.

[0119] Figure 6 A flowchart illustrating another multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention. Figure 6As shown, after loading the preset deep learning network model and reward function into the optimizer and configuring the event triggering mechanism for the optimizer (step S5), the process also includes steps S8-S11.

[0120] Step S8: At each time step, determine the operating status of the nuclear steam supply system according to the preset performance evaluation indicators.

[0121] The preset performance evaluation index is used to indicate whether the nuclear steam supply system has reached the expected state. This preset performance evaluation index can be set according to the actual application process of the nuclear steam supply system; no specific limitation is made in this embodiment of the invention.

[0122] In one embodiment, the nuclear steam supply system operates in three states: a first state, a second state, and a third state. The first state is a "poor" state, indicating that the nuclear steam supply system's operation has not reached the expected state and the deviation from the expected state is greater than a preset deviation. The second state is a "neutral" state, indicating that the nuclear steam supply system's operation has not reached the expected state but the deviation from the expected state is less than a preset deviation. The third state is a "superior" state, indicating that the nuclear steam supply system's operation has reached the expected state.

[0123] Step S9: When the operating status is in the first state, control the nuclear steam supply system to issue an operating warning reminder.

[0124] In the case of the "first operating state," it indicates a significant deviation between the nuclear steam supply system's operational status and its expected state. This operational state is unstable and may pose a safety risk. Therefore, in the first operating state, control optimization cannot be performed; instead, a conservative strategy must be adopted to address the safety risk first. Consequently, the nuclear steam supply system issues an operational early warning alert to allow the operators to handle this safety risk issue.

[0125] Step S10: When the running state is the second state, the control optimizer begins execution.

[0126] In the case of the second operating state, it indicates that the operating state of the nuclear steam supply system is relatively close to the expected state, but further optimization is still needed. Furthermore, the operating state of the nuclear steam supply system is relatively stable, so a stochastic strategy can be adopted to explore and improve control performance as much as possible. Therefore, the execution of the optimizer can be controlled to achieve multi-objective control optimization.

[0127] Step S11: If the running state is in the third state, wait to enter the next time step.

[0128] In the case of the third operating state, it indicates that the nuclear steam supply system is operating at a higher level than expected, and there is no need to continue control optimization. At this time, a stabilization strategy should be adopted to maintain the current operating state and wait for the next time step.

[0129] In this embodiment of the invention, an event-driven mechanism is introduced to determine the operating range of the optimizer, thereby ensuring that the optimizer optimizes multiple control objectives of the nuclear steam supply system under the premise of safe and stable reactor operation.

[0130] Figure 7 The figure shows an example of the effect of a multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0131] like Figure 7 As shown, the horizontal axis represents the simulation time step in seconds; the vertical axis represents the relative fission nuclear power, where 1 represents full power and 0.7 represents 70% of full power. The solid line is the relative nuclear power setting curve, the dotted line at the bottom is the relative nuclear power curve before optimization, and the dotted line segment is the relative nuclear power curve after optimization. Figure 7 It can be seen that the optimized relative nuclear power curve is almost identical to the relative nuclear power setting curve under steady-state conditions, that is, the optimized relative nuclear power has reached the expected relative nuclear power setting, thus achieving optimized control.

[0132] Figure 8 An example diagram illustrating the effect of another multi-objective control optimization method for a nuclear steam supply system provided in an embodiment of the present invention.

[0133] like Figure 8 As shown, the horizontal axis represents the simulation time step in seconds; the vertical axis represents the outlet steam temperature in degrees Celsius. The dotted line represents the outlet steam temperature curve before optimization, and the dotted line segment represents the outlet steam temperature curve after optimization. Typically, 575 degrees Celsius is taken as the target value for the outlet steam temperature. As can be seen from the outlet steam temperature curves before and after optimization, the maximum deviation of the optimized outlet steam temperature from 575 degrees Celsius in the 8000-8250s time period is significantly smaller than the maximum deviation of the unoptimized outlet steam temperature from 575 degrees Celsius in the same time period. Therefore, the multi-objective control optimization method for the nuclear steam supply system provided in this embodiment of the invention can achieve control optimization of the outlet steam temperature in the nuclear steam supply system.

[0134] Figure 9 This is a schematic diagram of the structure of a multi-objective control optimization device for a nuclear steam supply system provided in an embodiment of the present invention. Figure 9 As shown, the device includes: a parameter determination module 91, a function determination module 92, an optimization configuration module 93, an iteration module 94, and an optimization execution module 95.

[0135] The parameter determination module 91 is used to determine the input and output parameters of the optimizer based on multiple control objectives to be optimized in the nuclear steam supply system. The optimizer is an optimization program used to solve multi-objective control optimization problems.

[0136] The function determination module 92 is used to determine the reward function based on multiple control objectives to be optimized, the input parameters and output parameters of the optimizer.

[0137] In one implementation, the function determination module 92 is specifically used for: for each control target, obtaining the difference between the input parameter corresponding to the control target and the maximum permissible steady-state error corresponding to the control target, and the ratio of the absolute value of the difference to the maximum permissible steady-state error; and determining the reward function based on the ratio corresponding to each control target.

[0138] In one embodiment, the device further includes a function determination module for modeling the multi-objective control optimization problem as a Markov decision process, defining a quintuple consisting of an input state space, an action space, a state transition function, a discount factor, and an initial state distribution, and determining a preset deep learning network model based on the Markov decision process. The input state space is a parameter set determined based on the input parameters; the action space is a parameter set determined based on the output parameters; the state transition function is a function used to determine the state of the current time step based on the action selected in the previous time step; and the discount factor is used to characterize the importance of the reward value obtained at each time step in the cumulative reward.

[0139] The optimization configuration module 93 is used to load the preset deep learning network model and reward function into the optimizer and configure the event triggering mechanism for the optimizer. The event triggering mechanism is a mechanism used to trigger the optimizer to execute according to the predetermined operating state of the nuclear steam supply system.

[0140] In one embodiment, the device further includes a status monitoring module and a monitoring processing module.

[0141] This status monitoring module is used to determine the operating status of the nuclear steam supply system at each time step based on preset performance evaluation indicators.

[0142] The monitoring and processing module is used to control the nuclear steam supply system to issue an operational early warning when the operating state is in the first state; it is also used to control the start of the optimizer when the operating state is in the second state; and it is also used to wait for the next time step when the operating state is in the third state.

[0143] The iteration module 94 is used to control the optimizer to perform multiple iterations when the optimizer is triggered to obtain a multi-objective control optimization strategy, wherein the multi-objective control optimization strategy is a strategy used to obtain a preset reward function value.

[0144] In one implementation, the preset deep learning network model includes a first network and a second network; the iteration module 94 is specifically used for: initializing the network parameters of the first network and the second network; initializing the state space to obtain the current state; and executing a first processing procedure for the current time step. The first processing procedure includes: selecting an action in the action space based on the policy output by the first network and the current state, executing a state transition function to obtain the state of the next time step, and obtaining the reward value of the current time step based on the reward function; accumulating the reward value of each time step based on a discount factor, and determining whether the current accumulated reward value is less than the preset accumulated reward function value; if the accumulated reward value is less than the preset accumulated reward function value, updating the network parameters of the first network and the second network based on the reward of the current time step, and re-executing the first processing procedure based on the updated preset deep learning network model in the next time step; and if the accumulated reward value is greater than or equal to the preset accumulated reward function value, calling the preset deep learning network model to generate a multi-objective control optimization strategy.

[0145] The optimization execution module 95 is used to control the optimizer to perform control optimization operations based on a multi-objective control optimization strategy in order to achieve optimization of multiple control objectives.

[0146] This invention provides a multi-objective optimization control device, wherein a parameter determination module is used to determine the input and output parameters of an optimizer based on multiple control objectives to be optimized in a nuclear steam supply system; the optimizer is an optimization program used to solve multi-objective control optimization problems; a function determination module is used to determine a reward function based on the input and output parameters of the optimizer; an optimization configuration module is used to load a preset deep learning network model and reward function into the optimizer and configure an event triggering mechanism for the optimizer; the event triggering mechanism is a mechanism used to trigger the optimizer to execute according to a predetermined operating state of the nuclear steam supply system; an iteration module is used to control the optimizer to perform multiple iterations when the optimizer is triggered to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value; and an optimization execution module is used to load the multi-objective control optimization strategy into the optimizer so that the optimizer can perform control optimization operations based on the multi-objective control optimization strategy. This device can optimize the performance of multiple control objectives of the nuclear steam supply system while ensuring the stability and robustness of the nuclear steam supply system, thereby improving the effect of control optimization and increasing the operating efficiency of the nuclear steam supply system.

[0147] Figure 10This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 10 As shown, the computer device may include a processor 1001 and a memory 1002, wherein the processor 1001 and the memory 1002 may be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0148] Processor 1001 may be a central processing unit (CPU). Processor 1001 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0149] The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-objective control optimization method for the nuclear steam supply system in the embodiments of the present invention. The processor 1001 executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory 1002, thereby realizing the multi-objective control optimization method for the nuclear steam supply system in the above method embodiments.

[0150] The memory 1002 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 1001, etc. Furthermore, the memory 1002 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include memory remotely located relative to the processor 1001, and these remote memories may be connected to the processor 1001 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0151] One or more modules are stored in memory 1002, and when executed by processor 1001, they perform actions such as... Figure 1 The multi-objective control optimization method for the nuclear steam supply system in the illustrated embodiment.

[0152] For specific details regarding the aforementioned computer equipment, please refer to the relevant documentation. Figure 1 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0154] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multi-objective control optimization method for a nuclear steam supply system, characterized in that, The method includes: The input and output parameters of the optimizer are determined based on multiple control objectives to be optimized in the nuclear steam supply system; the optimizer is an optimization program used to solve multi-objective control optimization problems. The reward function is determined based on the multiple control objectives to be optimized, the input parameters and output parameters of the optimizer; A preset deep learning network model and the reward function are loaded into the optimizer, and an event triggering mechanism is configured for the optimizer; the event triggering mechanism is a mechanism used to trigger the optimizer to execute according to the predetermined operating state of the nuclear steam supply system; When the optimizer is triggered to execute, the optimizer is controlled to perform multiple iterations to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy used to obtain a preset cumulative reward function value. The optimizer is controlled to perform control optimization operations based on the multi-objective control optimization strategy to achieve optimization of multiple control objectives.

2. The method according to claim 1, characterized in that, After determining the reward function corresponding to the multiple control objectives based on the multiple control objectives to be optimized, the input parameters and output parameters of the optimizer, the process further includes: The multi-objective control optimization problem is modeled as a Markov decision process, and a quintuple consisting of the input state space, action space, state transition function, discount factor, and initial state distribution is defined. The input state space is a parameter set determined based on the input parameters; the action space is a parameter set determined based on the output parameters; the state transition function is a function used to determine the state of the current time step based on the state of the previous time step and the selected action; and the discount factor is used to characterize the importance of the reward value obtained at each time step in the cumulative reward. A deep reinforcement learning algorithm is determined based on the Markov decision process described above; The pre-defined deep learning network model is based on the deep reinforcement learning algorithm.

3. The method according to claim 2, characterized in that, The preset deep learning network model includes a first network and a second network; controlling the optimizer to perform multiple iterations to obtain a multi-objective control optimization strategy includes: Initialize the network parameters of the first network and the network parameters of the second network; The state space is initialized to obtain the current state; For the current time step, a first processing procedure is executed; the first processing procedure includes: selecting an action in the action space based on the current state according to the policy output by the first network, executing the state transition function to obtain the state of the next time step, and obtaining the reward value of the current time step according to the reward function; The reward value for each time step is accumulated based on the discount factor, and the second network determines whether the currently accumulated reward value is less than the preset accumulated reward function value. If the accumulated reward value is less than the preset accumulated reward function value, the network parameters of the first network and the second network are updated based on the reward of the current time step to obtain the updated preset deep learning network model, and the first processing procedure is re-executed based on the updated preset deep learning network model in the next time step. If the accumulated reward value is greater than or equal to the preset accumulated reward function value, the preset deep learning network model is invoked to generate the multi-objective control optimization strategy.

4. The method according to any one of claims 1-3, characterized in that, After loading the preset deep learning network model and the reward function into the optimizer, and configuring an event triggering mechanism for the optimizer, the process further includes: At each time step, the operating status of the nuclear steam supply system is determined according to preset performance evaluation indicators; When the operating state is in the first state, the nuclear steam supply system is controlled to issue an operating early warning reminder; When the running state is the second state, control the start of execution of the optimizer; If the running state is in the third state, wait to enter the next time step.

5. The method according to claim 1, characterized in that, The input parameters include the deviation between the actual value and the reference value corresponding to the control target, the system parameters and operating parameters of the nuclear steam supply system; the output parameter is the preset correction amount corresponding to the control target.

6. The method according to any one of claims 1 or 5, characterized in that, The step of determining the reward function based on multiple control objectives to be optimized, the input parameters and output parameters of the optimizer, includes: For each control objective, obtain the difference between the input parameter corresponding to the control objective and the maximum permissible steady-state error corresponding to the control objective, and the ratio of the absolute value of the difference to the maximum permissible steady-state error; The reward function is determined based on the ratio corresponding to each control objective.

7. The method according to claim 2, characterized in that, The deep reinforcement learning algorithm includes any one of the following: deep Q-network learning algorithm, deep deterministic policy gradient algorithm, and actor-critic algorithm.

8. A multi-objective control optimization device for a nuclear steam supply system, characterized in that, The device includes: The parameter determination module is used to determine the input and output parameters of the optimizer based on multiple control objectives to be optimized in the nuclear steam supply system; the optimizer is an optimization program used to solve multi-objective control optimization problems. The function determination module is used to determine the reward function based on the input and output parameters of the optimizer; An optimization configuration module is used to load a preset deep learning network model and the reward function into the optimizer, and to configure an event triggering mechanism for the optimizer; the event triggering mechanism is a mechanism used to trigger the optimizer to execute according to the predetermined operating state of the nuclear steam supply system; An iteration module is used to control the optimizer to perform multiple iterations when the optimizer is triggered to obtain a multi-objective control optimization strategy; the multi-objective control optimization strategy is a strategy for obtaining a preset reward function value. An optimization execution module is used to load the multi-objective control optimization strategy into the optimizer, so that the optimizer can perform control optimization operations based on the multi-objective control optimization strategy to achieve optimization of multiple control objectives.

9. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the steps of the multi-objective control optimization method for a nuclear steam supply system as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-objective control optimization method for the nuclear steam supply system as described in any one of claims 1-7.

Images