A space thermionic nuclear power source control method based on DDPG deep reinforcement learning algorithm

Abstract

A method for controlling a space thermionic nuclear power source based on the DDPG deep reinforcement learning algorithm includes the following main steps: 1. Building and initializing the environment of the space thermionic nuclear power source system and determining the state space and action space of the environment; 2. Building an actor neural network and a critic neural network to fit the policy function and value function; 3. Constructing a DDPG deep reinforcement learning algorithm module; 4. Conducting an agent training process. When the reward value reaches a preset value, the trained agent parameters are saved and the training is completed; 5. Based on the trained agent, transient control conditions of the space thermionic nuclear power source are simulated to evaluate the control performance. The method of the present invention does not require pre-training data and omits the control system modeling process. It can achieve end-to-end control of the space thermionic nuclear power source and has excellent control performance.

Description

Technical Field

[0001] The present invention relates to the application of a DDPG deep reinforcement learning algorithm, and in particular to a DDPG deep reinforcement learning control method for a space thermionic nuclear power source. Background Art

[0002] Since the mid-to-late 1990s, with the rapid development of computer technology and the significant increase in computing speed, artificial intelligence research has experienced rapid growth. In 2011, traditional neural networks were modified with certain algorithms and network connections, giving rise to "deep learning" technology. In the field of reinforcement learning, deep neural networks can fit policy functions, value functions, Q functions, and other functions. They can also directly use data such as images and videos as input, improving the versatility of reinforcement learning algorithms. The DDPG algorithm (Deep Deterministic Policy Gradient algorithm) is a model-free deep reinforcement learning algorithm that can solve control problems in continuous action spaces. It can obtain input and output directly from raw environmental data.

[0003] Space nuclear power systems use a control drum to control reactor power and implement shutdown functions in the event of an accident, requiring a well-designed automatic control system. Traditional automatic control system design often requires a simplified model of the nuclear power system, followed by empirical PID parameter tuning for each control loop to find the optimal parameter combination, a labor-intensive task. Using the DDPG deep reinforcement learning algorithm, the design of the automatic control system can be simplified into an end-to-end control process from the environment to the intelligent agent, achieving excellent overall control performance. Summary of the Invention

[0004] The present invention aims to provide a method for controlling a space thermionic nuclear power source based on the DDPG deep reinforcement learning algorithm. The method involves constructing an environment for a thermionic space nuclear power source system, providing a state space and action space for the given environment, establishing an actor neural network and a critic neural network to fit the policy function and value function in the DDPG deep reinforcement learning algorithm, constructing a DDPG deep reinforcement learning algorithm module and training it. After training, control simulation is performed for the transient operating conditions of the thermionic space nuclear power source. The method can achieve end-to-end direct control of the thermionic space nuclear power source without requiring pre-training data, and exhibits excellent control performance.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Step 1: Build and initialize the environment of the space thermionic nuclear power system and determine the state space and action space of the environment;

[0007] The training process of the deep reinforcement learning algorithm is achieved through the interaction between the environment and the agent. The specific process is shown in the following formula:

[0008] (S1,A1,R1,S2)→(S2,A2,R2,S3)→…→(S t ,A t ,R t ,S t+1 )

[0009] The agent receives the observed variables S in the environment t , use DDPG deep reinforcement learning algorithm to get the action amount A t And act on the environment, the state of the environment changes and transfers to S t+1 , and returns a certain reward value R t ;

[0010] The environment includes a neutron physics model, a thermal hydraulic model, a coolant system model, and a control drum model of a space thermionic nuclear power system. These models are composed of a series of coupled nonlinear differential equations. During deep reinforcement learning training, the environment is built by simply providing the state space (output) and action space (input) of the space thermionic nuclear power system.

[0011] The environment adopts a continuous state space, including the nuclear power error value of the space thermal ion nuclear power system, the increment of the nuclear power error value, and the cumulative error value of the nuclear power. The nuclear power error value, the increment of the nuclear power error value, and the cumulative error value of the nuclear power are normalized as shown in the following formula:

[0012]

[0013] Where:

[0014] E——Relative error of nuclear power of space thermal ion nuclear power system

[0015] E t ——Relative error of nuclear power of space thermionic nuclear power system at time t

[0016] P t ——Nuclear power of the space thermionic nuclear power system at time t

[0017] P d ——Nuclear power target value of space thermionic nuclear power system

[0018] S t ——The state space of the environment at time t

[0019] The environment uses a continuous action space to control the drum at the current time step, and the control amount is normalized as shown in the following formula:

[0020] A t =U t

[0021] Where:

[0022] A t ——The action space of the environment at time t

[0023] U t ——Control quantity of the drum at time t

[0024] After the environment is built, initialize the environment and set the initial value S0 of the state space and the initial value A0 of the action space as shown in the following formula:

[0025] S0=[0.05,0,0]

[0026] A0=0

[0027] In order to evaluate the impact of the agent's actions on the environment, we consider minimizing the power error and using the smallest control amount possible, while also considering the additional rewards during the agent training process. The reward function is designed as follows:

[0028] R t =-(k e E t 2 +k u U t-1 2 )+r e +r u +r d +r s

[0029] The first two terms represent the penalty terms for error and control amount:

[0030] R t ——Reward function at time t

[0031] k e ——Error term weight coefficient

[0032] k u ——weight coefficient of control item

[0033] The last four items represent additional bonus values ​​when certain conditions are met:

[0034]

[0035] Where:

[0036] R e ——Error-based bonus

[0037] E set——Error reward threshold

[0038] R u ——Additional rewards based on control volume

[0039] U set ——Control volume reward threshold

[0040] R d ——Penalties for stopping training midway

[0041] IsDone——Training stopped midway

[0042] r s ——Rewards based on the duration of each training round

[0043] k s ——Training time bonus coefficient

[0044] T f ——The time step of each training round

[0045] T s ——Total time of each training round

[0046] Step 2: Construct the actor neural network and critic neural network to fit the policy function and value function;

[0047] The DDPG deep reinforcement learning algorithm includes a policy function π(S) and a value function Q(S,A), which are fitted using an actor neural network and a critic neural network respectively.

[0048] The input of the actor neural network is the state space of the environment, and the output is a continuously distributed one-dimensional action space. It consists of a multi-layer neural network structure. The input layer of the actor neural network is the state space of the environment, which contains 3 neurons. The hidden layer includes 3 fully connected layers, each with 128 neurons, and the activation function uses the relu function. The output layer uses the tanh activation function for normalization and multiplies it by the scale factor u. max , get [-u max ,u max ] one-dimensional continuous action space on the interval;

[0049] The critic neural network input is the state space of the environment and the action space of the actor neural network, and the output is the estimated value of the value function. It has a multi-layer network structure. The critic neural network input layer includes two parts: the state space of the environment and the output value of the actor neural network. After being processed by different hidden layers, the elements of the two parts are added together, and then passed through two fully connected layers. The final output is the estimated value of the value function.

[0050] Finally, set the hyperparameters of the actor neural network and the critic neural network respectively to complete the construction of the neural network;

[0051] Step 3: Construct the DDPG deep reinforcement learning algorithm module; use the neural network structure constructed in step 2 to establish the critic network Q(S, A; φ) and the actor network π(S; θ), where φ and θ represent the parameters of the critic network and the actor network, respectively, and S and A represent the state space and action space of the spatial thermionic nuclear power system, respectively; define the target critic network and target actor network, respectively, Q′(S, A; φ′) and π′(S; θ′); randomize the network parameters φ and θ, and set φ′=φ and θ′=θ;

[0052] The training process of the DDPG deep reinforcement learning algorithm module is as follows:

[0053] (1) Observe the initial state space of the environment and conduct the first round of training;

[0054] (2) Output action space A t =π(S t ;θ)+N t And act on the environment to get the reward value R in the action space t And the state space S of the environment at the next moment t+1 ; where N t It is a random noise process introduced artificially to increase the agent's ability to explore the environment;

[0055] (3) The current state transfer process (S t , A t , R t , S t+1 ) is stored in the experience cache space;

[0056] (4) Extracting the state transition process (S i , A i , R i , S i+1 ), a total of M groups were selected;

[0057] (5) Calculate the target function y for the target critic network i :

[0058] y i =R i +γQ′(S i+1 ,π′(S i+1 ;θ′);φ′)

[0059] Where:

[0060] Q′(Si+1 ,π′(S i+1 ;θ′);φ′)——target critic network

[0061] φ′——Target critic network parameters

[0062] π′(S i+1 ;θ′)——target actor network

[0063] θ′ — target actor network parameters

[0064] γ — discount factor

[0065] R i ——Reward value at time i

[0066] S i+1 ——The state space of the environment at time i+1

[0067] (6) Obtain the critic network loss function L in the form of mean square error:

[0068]

[0069] Where:

[0070] M——Number of sample groups

[0071] Q(S i ,π(S i ;θ);φ)——critic network

[0072] φ——critic network parameters

[0073] π(S i ;θ)——actor network

[0074] θ — actor network parameters

[0075] S i ——the state space of the environment at time i;

[0076] Update the critic network parameters φ by minimizing the loss function L;

[0077] (7) Construct the performance index J of the actor network and sample to obtain the estimated value of its policy gradient:

[0078]

[0079] Where:

[0080] M——Number of sample groups

[0081] ——The gradient of the performance indicator with respect to the actor network parameters

[0082] ——Critic network gradient to actor network

[0083] The gradient of the actor network to the actor network parameters updates the actor network parameters θ according to the policy gradient estimate.

[0084] (8) Update the target actor network and target critic network using the smoothing factor τ:

[0085] θ′=τθ+(1-τ)θ′

[0086] φ′=τφ+(1-τ)φ′

[0087] (9) After this round of training, the total reward value R of this round is obtained epi And the mean R of the total reward value avr :

[0088]

[0089] Where:

[0090] N——current number of training rounds

[0091] k — calculate the number of training rounds

[0092] (10) Use the initialization function to update the state space S of the spatial thermal ion nuclear power system and return to step (2) for the next round of training;

[0093] Step 4: Perform the agent training process, and the mean value R of the total reward value avr When the preset value is reached, the trained agent parameters are saved and the training is completed;

[0094] Step 5: Based on the trained intelligent agent, simulate the transient control conditions of the space thermionic nuclear power source to evaluate the control performance.

[0095] Compared with the prior art, the present invention has the following outstanding features:

[0096] A deep reinforcement learning (DDPG) algorithm based on deep reinforcement learning (DL-Reinforcement Programming) was used to implement automatic control of a space-based thermionic nuclear power system. By constructing an environment and a DDPG agent, the agent's parameters were updated through interaction between the two. This approach, which does not rely on complex models of thermionic nuclear power system, allows for end-to-end control of the space-based thermionic nuclear power system. The agent also performs feature extraction, directly inputting raw data and obtaining control outputs. This method, using deep reinforcement learning, builds a high-performance automatic control system, providing a research approach for the design of automatic control systems for nuclear reactors. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 Flow chart of the method of the present invention;

[0098] Figure 2 The structure of the actor neural network in the present invention;

[0099] Figure 3 is the structure of the critic neural network in the present invention;

[0100] Figure 4 This is the training process of the DDPG deep reinforcement learning algorithm module;

[0101] Figure 5 The control performance comparison between DDPG control algorithm and PID control method under nuclear power step conditions is shown. DETAILED DESCRIPTION

[0102] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0103] Step 1: Build and initialize the environment of the space thermionic nuclear power system and determine the state space and action space of the environment;

[0104] The training process of the deep reinforcement learning algorithm is achieved through the interaction between the environment and the agent. The specific process is shown in the following formula:

[0105] (S1,A1,R1,S2)→(S2,A2,R2,S3)→…→(S t A t ,R t ,S t+1 )

[0106] The agent receives the observed variables S in the environment t , use DDPG deep reinforcement learning algorithm to get the action amount A t And act on the environment, the state of the environment changes and transfers to S t+1 , and returns a certain reward value R t ;

[0107] The environment includes a neutron physics model, a thermal hydraulic model, a coolant system model, and a control drum model of a space thermionic nuclear power system. These models are composed of a series of coupled nonlinear differential equations. During deep reinforcement learning training, the environment is built by simply providing the state space (output) and action space (input) of the space thermionic nuclear power system.

[0108] The environment adopts a continuous state space, including the nuclear power error value of the space thermal ion nuclear power system, the increment of the nuclear power error value, and the cumulative error value of the nuclear power. The nuclear power error value, the increment of the nuclear power error value, and the cumulative error value of the nuclear power are normalized as shown in the following formula:

[0109]

[0110] Where:

[0111] E——Relative error of nuclear power of space thermal ion nuclear power system

[0112] E t ——Relative error of nuclear power of space thermionic nuclear power system at time t

[0113] P t ——Nuclear power of the space thermionic nuclear power system at time t

[0114] P d ——Nuclear power target value of space thermionic nuclear power system

[0115] S t ——The state space of the environment at time t

[0116] The environment uses a continuous action space to control the drum at the current time step, and the control amount is normalized as shown in the following formula:

[0117] A t =U t

[0118] Where:

[0119] A t ——The action space of the environment at time t

[0120] U t ——Control quantity of the drum at time t

[0121] After the environment is built, initialize the environment and set the initial value S0 of the state space and the initial value A0 of the action space as shown in the following formula:

[0122] S0=[0.05,0,0]

[0123] A0=0

[0124] In order to evaluate the impact of the agent's actions on the environment, we consider minimizing the power error and using the smallest control amount possible, while also considering the additional rewards during the agent training process. The reward function is designed as follows:

[0125] R t =-(k e E t 2 +k u U t-1 2 )+r e +r u +r d +r s

[0126] The first two terms represent the penalty terms for error and control amount:

[0127] R t ——Reward function at time t

[0128] k e ——Error term weight coefficient

[0129] k u ——weight coefficient of control item

[0130] The last four items represent additional bonus values ​​when certain conditions are met:

[0131]

[0132] Where:

[0133] R e ——Error-based bonus

[0134] E set ——Error reward threshold

[0135] R u ——Additional rewards based on control volume

[0136] U set ——Control volume reward threshold

[0137] R d ——Penalties for stopping training midway

[0138] IsDone——Training stopped midway

[0139] r s ——Rewards based on the duration of each training round

[0140] k s——Training time bonus coefficient

[0141] T f ——The time step of each training round

[0142] T s ——Total time of each training round

[0143] Step 2: Construct the actor neural network and critic neural network to fit the policy function and value function;

[0144] The DDPG deep reinforcement learning algorithm includes a policy function π(S) and a value function Q(S, A), which are fitted using an actor neural network (policy neural network) and a critic neural network (value neural network) respectively.

[0145] The input of the actor network is the state space of the environment, and the output is a continuously distributed one-dimensional action space. The specific neural network structure is as follows: Figure 2 As shown;

[0146] The input of the actor neural network is the state space of the environment, and the output is a continuously distributed one-dimensional action space. It consists of a multi-layer neural network structure. The input layer of the actor neural network is the state space of the environment, which contains 3 neurons. The hidden layer includes 3 fully connected layers, each with 128 neurons, and the activation function uses the relu function. The output layer uses the tanh activation function for normalization and multiplies it by the scale factor u. max , get [-u max ,u max ] one-dimensional continuous action space on the interval;

[0147] The critic neural network input is the state space of the environment and the action space of the actor neural network, and the output is the estimated value of the value function. It has a multi-layer network structure. The critic neural network input layer includes two parts: the state space of the environment and the output value of the actor neural network. After being processed by different hidden layers, the elements of the two parts are added together, and then passed through two fully connected layers. The final output is the estimated value of the value function.

[0148] like Figure 3 As shown in the figure, the critic neural network input layer includes two parts: the state space of the environment and the output value of the policy neural network. After being processed by different hidden layers, the elements of the two parts are added together, and then passed through two fully connected layers. The final output is the estimated value of the value function.

[0149] Finally, set the hyperparameters of the actor neural network and the critic neural network respectively to complete the construction of the neural network;

[0150] Step 3: Construct the DDPG deep reinforcement learning algorithm module; use the neural network structure constructed in step 2 to establish the critic network Q(S, A; φ) and the actor network π(S; θ), where φ and θ represent the parameters of the critic network and the actor network, respectively, and S and A represent the state space and action space of the spatial thermionic nuclear power system, respectively; define the target critic network and target actor network, respectively Q′(S, A; Q′) and π′(S; θ′); randomize the network parameters φ and θ, and set φ′=φ and θ′=′;

[0151] The training process of the DDPG deep reinforcement learning algorithm module is as follows:

[0152] (1) Observe the initial state space of the environment and conduct the first round of training;

[0153] (2) Output action space A t =π(S t ;θ)+N t And act on the environment to get the reward value R in the action space t And the state space S of the environment at the next moment t+1 ; where N t It is a random noise process introduced artificially to increase the agent's ability to explore the environment;

[0154] (3) The current state transfer process (S t , A t , R t , S t+1 ) is stored in the experience cache space;

[0155] (4) Extracting the state transition process (S i , A i , R i , S i+1 ), a total of M groups were selected;

[0156] (5) Calculate the target function y for the target critic network i :

[0157] y i =R i +γQ′(S i+1 ,π′(S i+1 ;θ′);φ′)

[0158] Where:

[0159] Q′(S i+1 ,π′(S i+1 ;θ′);φ′)——target critic network

[0160] φ′——Target critic network parameters

[0161] π′(S i+1 ;θ′)——target actor network

[0162] θ′ — target actor network parameters

[0163] γ — discount factor

[0164] R i ——Reward value at time i

[0165] S i+1 ——The state space of the environment at time i+1

[0166] (6) Obtain the critic network loss function L in the form of mean square error:

[0167]

[0168] Where:

[0169] M——Number of sample groups

[0170] Q(S i ,π(S i ;θ);φ)——critic network

[0171] φ——critic network parameters

[0172] π(S i ;θ)——actor network

[0173] θ — actor network parameters

[0174] S i ——the state space of the environment at time i;

[0175] Update the critic network parameters φ by minimizing the loss function L;

[0176] (7) Construct the performance index J of the actor network and sample to obtain the estimated value of its policy gradient:

[0177]

[0178] Where:

[0179] M——Number of sample groups

[0180] ——The gradient of the performance indicator with respect to the actor network parameters

[0181] ——Critic network gradient to actor network

[0182] ——Actor network gradient with respect to actor network parameters

[0183] (8) Update the target actor network and target critic network using the smoothing factor τ:

[0184] θ′=τθ+(1-τ)θ′

[0185] φ′=τφ+(1-τ)φ′

[0186] (9) After this round of training, the total reward value R of this round is obtained epi And the mean R of the total reward value avr :

[0187]

[0188] Where:

[0189] N——current number of training rounds

[0190] k — calculate the number of training rounds

[0191] (10) Use the initialization function to update the state space S of the spatial thermal ion nuclear power system and return to step (2) for the next round of training;

[0192] Step 4: Perform the agent training process, and the mean value R of the total reward value avr When the preset value is reached, the training parameters of the agent are saved and the training is completed; let the number of training rounds k = 5, consider the average reward within 5 rounds of training, and set the average reward threshold to 2000. When the threshold is reached, the training parameters are saved and the training is stopped. The training process of the agent is as follows Figure 4 shown.

[0193] Step 5: Based on the trained intelligent agent, simulate the transient control conditions of the space thermionic nuclear power source to evaluate the control performance. Figure 5 The control effect comparison between the DDPG control method and the PID control method under nuclear power step conditions is demonstrated. As can be seen from the figure, the DDPG control method of the present invention has a smaller overshoot and a shorter adjustment time, and the overall performance is superior.

Claims

1. A method for controlling a space thermal ion nuclear power source based on the DDPG deep reinforcement learning algorithm, characterized by: The following steps are involved: Step 1: Build and initialize the environment of the space thermionic nuclear power system and determine the state space and action space of the environment; The training process of the deep reinforcement learning algorithm is achieved through the interaction between the environment and the agent. The specific process is shown in the following formula: <h2 style=";text-align:left;direction:ltr">(S1, A1, R1, S2)→(S2, A2, R2, S3)→…→(S<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> A<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> R<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> ,S<h2 style=";text-align:left;direction:ltr"> t+1 <h2 style=";text-align:left;direction:ltr"> ) The agent receives the observed variables S in the environment t , use DDPG deep reinforcement learning algorithm to get the action amount A t And act on the environment, the state of the environment changes and transfers to S t+1 , and returns a certain reward value R t ; The environment includes a neutron physics model, a thermal hydraulic model, a coolant system model, and a control drum model of a space thermionic nuclear power system. These models are composed of a series of coupled nonlinear differential equations. During deep reinforcement learning training, the environment is built by simply providing the state space (output) and action space (input) of the space thermionic nuclear power system. The environment adopts a continuous state space, including the nuclear power error value of the space thermal ion nuclear power system, the increment of the nuclear power error value, and the cumulative error value of the nuclear power. The nuclear power error value, the increment of the nuclear power error value, and the cumulative error value of the nuclear power are normalized as shown in the following formula: Where: E——Relative error of nuclear power of space thermal ion nuclear power system E t ——Relative error of nuclear power of space thermionic nuclear power system at time t P t ——Nuclear power of the space thermionic nuclear power system at time t P d ——Nuclear power target value of space thermionic nuclear power system S t ——The state space of the environment at time t The environment uses a continuous action space to control the drum at the current time step, and the control amount is normalized as shown in the following formula: A t =U t Where: A t ——The action space of the environment at time t U t ——Control quantity of the drum at time t After the environment is built, initialize the environment and set the initial value S0 of the state space and the initial value A0 of the action space as shown in the following formula: S0＝[0.05，0，0] A0＝0 In order to evaluate the impact of the agent's actions on the environment, we consider minimizing the power error and using the smallest control amount possible, while also considering the additional rewards during the agent training process. The reward function is designed as follows: R t =-(k e E t 2 +k u U t-1 2 )+r e +r u +r d +r s The first two terms represent the penalty terms for error and control amount: R t ——Reward function at time t k e ——Error term weight coefficient k u ——weight coefficient of control item The last four items represent additional bonus values ​​when certain conditions are met: Where: R e ——Error-based bonus E set ——Error reward threshold R u ——Additional rewards based on control volume U set ——Control volume reward threshold R d ——Penalties for stopping training midway IsDone——Training stopped midway r s ——Rewards based on the duration of each training round k s ——Training time bonus coefficient T f ——The time step of each training round T s ——Total time of each training round Step 2: Construct the actor neural network and critic neural network to fit the policy function and value function; The DDPG deep reinforcement learning algorithm includes a policy function π(S) and a value function Q(S, A), which are fitted using an actor neural network and a critic neural network respectively. The input of the actor neural network is the state space of the environment, and the output is a continuously distributed one-dimensional action space. It consists of a multi-layer neural network structure. The input layer of the actor neural network is the state space of the environment, which contains 3 neurons. The hidden layer includes 3 fully connected layers, each with 128 neurons, and the activation function uses the relu function. The output layer uses the tanh activation function for normalization and multiplies it by the scale factor u. max , get [-u max ,u max ] one-dimensional continuous action space on the interval; The critic neural network input is the state space of the environment and the action space of the actor neural network, and the output is the estimated value of the value function. It has a multi-layer network structure. The critic neural network input layer includes two parts: the state space of the environment and the output value of the actor neural network. After being processed by different hidden layers, the elements of the two parts are added together, and then passed through two fully connected layers. The final output is the estimated value of the value function. Finally, set the hyperparameters of the actor neural network and the critic neural network respectively to complete the construction of the neural network; Step 3: Construct the DDPG deep reinforcement learning algorithm module; use the neural network structure constructed in step 2 to establish the critic network Q(S, A; φ) and the actor network π(S; θ), where φ and θ represent the parameters of the critic network and the actor network, respectively, and S and A represent the state space and action space of the spatial thermionic nuclear power system, respectively; define the target critic network and target actor network, respectively, Q′(S, A; φ′) and π′(S; θ′); randomize the network parameters φ and θ, and set φ′=φ and θ′=θ; The training process of the DDPG deep reinforcement learning algorithm module is as follows: (1) Observe the initial state space of the environment and conduct the first round of training; (2) Output action space A t =π(S t ;θ)+N t And act on the environment to get the reward value R in the action space t And the state space S of the environment at the next moment t+1 ; where N t It is a random noise process introduced artificially to increase the agent's ability to explore the environment; (3) The current state transfer process (S t , A t , R t , S t+1 ) is stored in the experience cache space; (4) Extracting the state transition process (S i , A i , R i , S i+1 ), a total of M groups were selected; (5) Calculate the target function y for the target critic network i : y i =R i +γQ′(S i+1 ,π′(S i+1 ;θ′);φ′) Where: Q′(S i+1 ,π′(S i+1 ;θ′);φ′)——target critic network φ′——Target critic network parameters π′(S i+1 ;θ′)——target actor network θ′ — target actor network parameters γ — discount factor R i ——Reward value at time i S i+1 ——The state space of the environment at time i+1 (6) Obtain the critic network loss function L in the form of mean square error: Where: M——Number of sample groups Q(S i ,π(S i ;i);f)——critic network φ——critic network parameters π(S i ;θ)——actor network θ — actor network parameters S i ——the state space of the environment at time i; Update the critic network parameters φ by minimizing the loss function L; (7) Construct the performance index J of the actor network and sample to obtain the estimated value of its policy gradient: Where: M——Number of sample groups — Gradient of performance metric with respect to actor network parameters ——Critic network gradient to actor network The gradient of the actor network to the actor network parameters updates the actor network parameters θ according to the policy gradient estimate. (8) Update the target actor network and target critic network using the smoothing factor τ: θ′=τθ+(1-τ)θ′ φ′=τφ+(1-τ)φ′ (9) After this round of training, the total reward value R of this round is obtained epi And the mean R of the total reward value avr : Where: N——current number of training rounds k — calculate the number of training rounds (10) Use the initialization function to update the state space S of the spatial thermal ion nuclear power system and return to step (2) for the next round of training; Step 4: Perform the agent training process, and the mean value R of the total reward value avr When the preset value is reached, the trained agent parameters are saved and the training is completed; Step 5: Based on the trained intelligent agent, simulate the transient control conditions of the space thermionic nuclear power source to evaluate the control performance.

Images