A tokamak plasma vertical displacement control method based on reinforcement learning

Abstract

This invention belongs to the field of nuclear fusion technology and discloses a tokamak plasma vertical displacement control method based on reinforcement learning. The technical solution includes the following steps: S1, obtaining tokamak vertical displacement data and tokamak IC coil current data as the current state vector; S2, obtaining tokamak vertical displacement data, tokamak IC coil current data, and voltage data as historical state vectors; S3, inputting the current state vector and historical state vectors into the trained vertical displacement controller, which outputs the IC coil voltage value; S4, inputting the voltage value from S3 as a control command into the control system to complete the control of the plasma vertical displacement. This invention uses reinforcement learning to train the tokamak plasma vertical displacement controller, improving the reliability of control and promoting related research on plasma control in the field of nuclear fusion.

Description

Technical Field

[0001] This invention belongs to the field of nuclear fusion technology, specifically relating to a method for vertical displacement control of tokamak plasma based on reinforcement learning. Background Technology

[0002] With the development of science and technology, controlled nuclear fusion technology is constantly improving. A tokamak device is a toroidal container that can achieve controlled nuclear fusion through magnetic confinement and vacuum adiabatic conditions. To obtain nuclear fusion energy, discharge parameters, such as density, temperature, and confinement time, must be increased to meet ignition conditions. Elongated plasma configurations help improve discharge parameters but lead to vertical displacement instabilities. The plasma disruption caused by vertical displacement instabilities is called a vertical displacement event. During a vertical displacement event, enormous electromagnetic forces and thermal shocks are generated, causing severe damage to the device.

[0003] Existing controllers are designed based on linear models, making it difficult to account for the impact of nonlinear effects on control; at the same time, existing tokamak plasma vertical displacement controllers are also designed based on linear models, making it difficult to account for the impact of nonlinear effects on control, resulting in insufficient control reliability. Summary of the Invention

[0004] The purpose of this invention is to provide a tokamak plasma vertical displacement control method based on reinforcement learning. The tokamak plasma vertical displacement controller is trained using reinforcement learning. The controller has a neural network structure, and the activation function in the neural network endows the controller with the ability to handle nonlinear effects, thereby improving the reliability of the control. This invention also promotes related research on plasma control in the field of nuclear fusion.

[0005] To achieve the above objectives, this invention proposes a tokamak plasma vertical displacement control method based on reinforcement learning, comprising the following steps:

[0006] S1. Obtain the tokamak vertical displacement data and tokamak IC coil current data as the current state vector;

[0007] S2. Obtain the tokamak vertical displacement data and the tokamak IC coil current and voltage data as a historical state vector.

[0008] S3. Input the current state vector and the historical state vector into the trained vertical displacement controller, and the vertical displacement controller outputs the voltage value of the IC coil.

[0009] S4. Input the voltage value in S3 as a control command to the control system to complete the control of the vertical displacement of the plasma.

[0010] Optionally, in S3, the vertical displacement controller inputs the current state vector and the historical state vector into the policy network to obtain the output value of the policy network: the voltage command of the IC coil.

[0011] Optionally, the specific steps for constructing the vertical displacement controller are as follows:

[0012] S31. Construct a plasma response model as an environment for reinforcement learning;

[0013] S32. Constructing intelligent agents;

[0014] S33. Construct the reward function;

[0015] S34, Training the intelligent agent.

[0016] Optionally, the plasma linear response model in S31 can be expressed in the following form through state-space equations:

[0017] x(t+1)=Ax(t)+Bu(t);

[0018] y(t) = Cx(t) + Du(t);

[0019] Where x is the current in the coil and passive structure, u is the IC coil voltage, y is the plasma vertical displacement and IC coil current, and the control frequency is 10kHz.

[0020] Optionally, the agent in S32 includes a policy network and an evaluation network.

[0021] Optionally, the policy network consists of two feature extraction networks and a fully connected neural network. The two feature extraction networks are a fully connected neural network and a long short-term memory network, which extract feature vectors of the current state vector and the historical state vector, respectively. The two extracted feature vectors are concatenated and then input into the fully connected neural network. The output is multiplied by a fixed coefficient Vicmax = 1200 to obtain the output value of the policy network.

[0022] Optionally, the method for constructing and calculating the policy network is as follows:

[0023] First, the feature extraction part consists of two parts;

[0024] Part 1: Construct an input layer consisting of 2 neurons, corresponding to two current state vectors, and then construct a fully connected layer consisting of N neurons, with each node connected to the 2 neurons in the previous layer, and the activation function is the ReLU function;

[0025] Part 2: Construct an input layer consisting of 9 neurons, corresponding to 9 historical state vectors containing three time steps, with each time step containing three data points; then construct an LSTM neural network layer with an N-dimensional output.

[0026] The next step is to construct an M-dimensional fully connected layer with the ReLU activation function, which is then connected to the K-dimensional features obtained by combining the N-dimensional features extracted by the two feature extraction networks. The calculation method is the same as the previous fully connected layer. Finally, a one-dimensional fully connected layer with the tanh activation function is constructed and connected to the M neurons of the previous layer. The role of tanh is to map the input to (-1,1) and multiply the output value by a fixed coefficient Vicmax = 1200 to obtain the output value of the policy network.

[0027] Optionally, the evaluation network consists of two feature extraction networks and a fully connected neural network. The two feature extraction networks are a fully connected neural network and a long short-term memory network, which extract feature vectors from the current state vector and the historical state vector, respectively. The two extracted feature vectors are concatenated with the IC coil voltage command at the current moment and then input into the fully connected neural network to obtain the output value of the evaluation network as the evaluation of the IC coil voltage command at the current moment.

[0028] Optionally, the specific steps for training the agent are as follows:

[0029] S34.1 The agent obtains the current state vector by observing the environment and obtains the historical state vector by reading historical information;

[0030] S34.2 The policy network obtains the voltage command through the current state vector and the historical state vector. After executing the command, the environment transitions to the next state according to the plasma response model, and at the same time, a reward is fed back to the agent.

[0031] S34.3 When the environment meets certain conditions, it is reset to the initial state, and the agent begins to interact with the environment for the next round until the training reaches the termination condition.

[0032] S34.4. The current state vector and historical state vector at the current moment, the voltage command taken at the current moment, the current state vector and historical state vector at the next moment, and the reward fed back to the agent are combined into a data unit. The agent can obtain a series of data units as a training set through continuous interaction with the environment, and finally obtain a trained policy network.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] This invention uses reinforcement learning to train a tokamak plasma vertical displacement controller. The controller is a neural network structure, and the activation function in the neural network endows the controller with the ability to handle nonlinear effects, thereby improving the reliability of the control and promoting related research on plasma control in the field of nuclear fusion. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the overall method flow of the present invention.

[0036] Figure 2 This is a schematic diagram of the strategy network structure in this invention.

[0037] Figure 3 This is a schematic diagram of the evaluation network structure in this invention.

[0038] Figure 4 This is a schematic diagram of the interaction between the intelligent agent and the environment.

[0039] Figure 5 This is a schematic diagram of the plasma vertical displacement control results.

[0040] Figure 6 This is a schematic diagram showing the change of IC coil current over time.

[0041] Figure 7 This is a schematic diagram showing the change of IC coil voltage over time.

[0042] Figure 8 This is a schematic diagram of the plasma vertical displacement control results with added noise.

[0043] Figure 9 This is a schematic diagram showing the change of IC coil current over time with added noise.

[0044] Figure 10 This is a schematic diagram showing the change of IC coil voltage over time with added noise. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 This embodiment presents a tokamak plasma vertical displacement control method based on reinforcement learning, which is implemented sequentially through the following steps:

[0047] Step 1: Obtain the tokamak vertical displacement data and tokamak IC coil current data at time t as the current state vector;

[0048] Step 2: Obtain the tokamak vertical displacement data, tokamak IC coil current data, and voltage data at times t-1, t-2, and t-3 as historical state vectors;

[0049] Step 3: Input the current state vector and the historical state vector into the trained vertical displacement controller, and the vertical displacement controller outputs the voltage value of the IC coil;

[0050] The vertical displacement controller inputs the current state vector and historical state vector into the policy network to obtain the output value of the policy network: the voltage command of the IC coil.

[0051] The specific steps for constructing a vertical displacement controller are as follows:

[0052] 1. Construct a plasma response model as the environment for reinforcement learning;

[0053] The plasma linear response model can be expressed by state-space equations as follows:

[0054] x(t+1)=Ax(t)+Bu(t);

[0055] y(t) = Cx(t) + Du(t);

[0056] Where x is the current in the coil and passive structure, u is the IC coil voltage, y is the plasma vertical displacement and IC coil current, and the control frequency is 10kHz.

[0057] 2. Construct intelligent agents;

[0058] An intelligent agent consists of a policy network and an evaluation network;

[0059] Please see Figure 2 The policy network consists of two feature extraction networks and a fully connected neural network. The two feature extraction networks are a fully connected neural network and a long short-term memory network, which extract feature vectors from the current state vector and the historical state vector, respectively. The two extracted feature vectors are concatenated and then input into the fully connected neural network. The output is multiplied by a fixed coefficient Vicmax = 1200 to obtain the output value of the policy network.

[0060] Construction and computation method of policy network: First, the feature extraction part consists of two parts.

[0061] Part 1: Construct an input layer consisting of 2 neurons, corresponding to two current state vectors. Then, construct a fully connected layer consisting of 128 neurons, with each node connected to the two neurons in the previous layer. The activation function is the ReLU function. The output of a neuron in the 128 nodes is calculated as follows: Assuming the values ​​of the two neurons in the previous layer are x1 to x2, and the weights connecting this node to the two nodes in the previous layer are w1 and w2, with a bias of b, then the output is:

[0062] The relu(x1w1+x2w2) function is defined as follows: when x is greater than 0, relu(x) = x; otherwise, relu(x) = 0. The weights are uniformly and randomly initialized in the range [-1,1], and the bias is initialized to 0.

[0063] Part Two: Construct an input layer consisting of 9 neurons, corresponding to 9 historical state vectors at time points t-3, t-2, and t-1, with three data points at each time point. Then, construct an LSTM neural network layer with a 128-dimensional output. The LSTM layer output is calculated as follows:

[0064] LSTM consists of four parts: 1. Forget gate; 2. Input gate; 3. Cell state; 4. Output gate; Assuming h0 and c0 are initialized to 0, all weights are uniformly and randomly initialized in [-1,1], and all biases are initialized to 0, where h t c represents the output of the LSTM network at time t. t The cell states at time t are represented by 128-dimensional vectors. The information from input times t-3, t-2, and t-1 is equivalent to times t = 1, 2, and 3 for the LSTM network.

[0065] Forgot Gate: Receive h t-1 and x t Output f t =sigmoid(W f [h t-1 ,x t ]+b f ),in W f b f For the weights and biases of the forget gate, [h t-1 ,x t ] indicates that h is a 128-dimensional string. t-1 and 3D x t A 131-dimensional dataset pieced together.

[0066] Input gate: i t =sigmoid(W i [h t-1 ,x t ]+b i), where W i b i The weights and biases of the input gate;

[0067] Cell state: c t =f t *c t-1 +i t *tanh(W c [h t-1 ,x t ]+b c ), where W c b c Weights and biases for cell states;

[0068]

[0069] Output gate: h t =sigmoid(W o [h t-1 ,x t ]+b o )*tanh(c t ), where W o b o The weights and biases of the output gate;

[0070] x1 is input into the LSTM. Based on the initial h0, c0, various weights and biases, h1 and c1 can be obtained. Then x2 is input to obtain h2 and c2. Finally, x3 is input to obtain the final 128-dimensional output h3.

[0071] The next step is to construct a 200-dimensional fully connected layer with the ReLU activation function. This layer is then connected to the 256-dimensional features obtained by combining the 128-dimensional features extracted by the two feature extraction networks, using the same calculation method as the previous fully connected layer. Finally, a one-dimensional fully connected layer with the tanh activation function is constructed and connected to the 200 neurons of the previous layer. The tanh function maps the input to (-1, 1), and the output value is multiplied by a fixed coefficient Vicmax = 1200 to obtain the output value of the policy network.

[0072] Please see Figure 3 The evaluation network consists of two feature extraction networks and a fully connected neural network. The two feature extraction networks are a fully connected neural network and a long short-term memory network, which extract feature vectors from the current state vector and the historical state vector, respectively. The two extracted feature vectors are concatenated with the current IC coil voltage command and then input into the fully connected neural network. The output value of the evaluation network is used as the evaluation of the current IC coil voltage command.

[0073] The only difference between the evaluation network and the policy network is that the evaluation network has an additional IC coil voltage command at the current moment as input. The calculation method, the number of neurons in each layer, and the activation function are the same as those in the policy network.

[0074] 3. Construct the reward function;

[0075] The reward function formula is as follows:

[0076]

[0077] The first objective is to bring the vertical displacement of the plasma close to zero, the second objective is to avoid overcurrent in the IC coil, and the third objective is to reduce the jitter of the vertical displacement of the plasma.

[0078] 4. Train the intelligent agent.

[0079] Please refer to section 4. The agent obtains its current state vector by observing the environment and its historical state vector by reading historical information. The policy network obtains the voltage command using the current and historical state vectors. After executing the command, the environment transitions to the next state according to the plasma response model, and a reward is given to the agent. When the environment meets certain conditions, it resets to the initial state, and the agent begins the next round of interaction with the environment until the training reaches the termination condition. A data unit is composed of six types of data: the current state vector and historical state vector, the voltage command taken at the current moment, the current and historical state vectors at the next moment, and the reward given to the agent. Through continuous interaction with the environment, the agent can obtain a series of data units as a training set to train the agent, ultimately resulting in a trained policy network.

[0080] Step 4: Input the voltage value in S3 as a control command into the control system to complete the control of the vertical displacement of the plasma; where tn represents the time n control cycles ago, and one control cycle is 100 microseconds.

[0081] Please see Figures 5-7 The vertical displacement was simulated using a trained vertical displacement controller. Considering the control delay of three control cycles in the actual experiment, the results are as follows: Figure 5 The blue line represents vertical displacement, and the yellow line represents the control target;

[0082] As can be seen, the vertical displacement was successfully controlled, the vertical displacement jitter was very small, and the IC coil current was also within the carrying range.

[0083] Please see Figures 8-10 To test the model's anti-interference capability, a segment of vertical displacement noise was extracted from the experiment and added to the observed vertical displacement values. The control results are as follows: Figure 8In the diagram, the blue line represents vertical displacement, the yellow line represents the control target, and the green line represents noise. The vertical displacement values ​​are actual values, not observed values. As can be seen, the model has strong robustness against interference.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for controlling vertical displacement of a tokamak plasma based on reinforcement learning, characterized in that, The method comprises the following steps: S1, obtaining tokamak vertical displacement data and tokamak IC coil current data as a current state vector; S2, obtaining tokamak vertical displacement data and tokamak IC coil current data and voltage data as a historical state vector; S3, inputting the current state vector and the historical state vector into the trained vertical displacement controller, and the vertical displacement controller outputs the voltage value of the IC coil; The vertical displacement controller inputs the current state vector and the historical state vector into the policy network in S3 to obtain the output value of the policy network: the voltage command of the IC coil; The construction steps of the vertical displacement controller are as follows: S31, constructing a plasma response model as an environment for reinforcement learning; S32, constructing an agent; S33, constructing a reward function; S34, training the agent; The plasma linear response model in S31 is expressed in the following form through a state space equation: ; ; wherein, x is the current on the coil and the passive structure, u is the IC coil voltage, y is the plasma vertical displacement and the IC coil current, and the control frequency is 10kHZ; The agent comprises a policy network and an evaluation network; The policy network is composed of two feature extraction networks and a fully connected neural network; the two feature extraction networks are a fully connected neural network and a long short-term memory network, respectively, and extract feature vectors of the current state vector and the historical state vector, respectively; the two extracted feature vectors are concatenated and input into the fully connected neural network, and the output is multiplied by a fixed coefficient Vicmax=1200 to obtain the output value of the policy network; The construction and calculation method of the policy network are as follows: First, the feature extraction part is composed of two parts; The first part: an input layer composed of 2 neurons is constructed, corresponding to two current state vectors, then a fully connected layer composed of N neurons is constructed, each node is connected with 2 neurons of the previous layer, and the activation function is a relu function; The second part: an input layer composed of 9 neurons is constructed, corresponding to 9 historical state vectors containing three time points, each time point containing three data; then an LSTM neural network layer with an output of N dimensions is constructed; Next, a fully connected layer with M dimensions is constructed, the activation function is relu, and the N-dimensional features extracted by the two feature extraction networks are combined to obtain K-dimensional features; the calculation method is the same as that of the previous fully connected layer; finally, a one-dimensional fully connected layer is constructed, the activation function is tanh, and the M neurons of the previous layer are connected; the function of tanh is to map the input to (-1, 1), and the output value is multiplied by a fixed coefficient Vicmax=1200 to obtain the output value of the policy network; S4, inputting the voltage value in S3 into the control system as a control command to complete the control of the plasma vertical displacement.

2. The method of claim 1, wherein the method is based on reinforcement learning. The evaluation network is composed of two feature extraction networks and a fully connected neural network; the two feature extraction networks are respectively a fully connected neural network and a long short-term memory network, which respectively extract feature vectors of the current state vector and the historical state vector; the two extracted feature vectors and the IC coil voltage command at the current moment are concatenated and input into the fully connected neural network to obtain the output value of the evaluation network as the evaluation of the IC coil voltage command at the current moment.

3. The method of claim 1, wherein the method further comprises: The specific steps of training the agent are as follows: S34.1, the agent obtains the current state vector by observing the environment and obtains the historical state vector by reading the historical information; S34.2, the policy network obtains the voltage command through the current state vector and the historical state vector, executes the command, and then the environment is transferred to the next state according to the plasma response model, while feeding back an agent reward to the agent; S34.3, when the environment meets certain conditions, it is reset to the initial state, and the agent starts the next round of interaction with the environment until the training reaches the termination condition; S34.4, the current state vector and the historical state vector at the current moment, the voltage command taken at the current moment, the current state vector and the historical state vector at the next moment, and the reward fed back to the agent are combined into a data unit, and the agent can obtain a series of data units as a training set for training the agent through continuous interaction with the environment, and finally obtain a trained policy network.

Images