Reinforcement-learning-based fusion plasma loop voltage control method

Abstract

A reinforcement-learning-based fusion plasma loop voltage control method, comprising the following steps: selecting data for model training: in the context of Tokamak fusion plasmas, using plasma parameters at the current moment as an input, and outputting vloop at the next moment; and constructing a database: collecting discharge data of an operating Tokamak apparatus and preliminary discharge data of a future apparatus, and using same as initial training data for model construction. The method can improve the precision and response speed of a Tokamak vloop control algorithm; and a reinforcement learning model has self-learning and adaptive capabilities, thereby enhancing the long-term stability and optimization effect of a vloop control method in a Tokamak system, improving the overall control effect, reducing the demand for manual intervention and operations, and reducing the operation costs and human resource input.

Description

A reinforcement learning-based method for controlling the loop voltage of fusion plasma Technical Field

[0001] This invention relates to the field of fusion plasma control technology, and in particular to a fusion plasma loop voltage control method based on reinforcement learning. Background Technology

[0002] A tokamak device is a crucial instrument for studying and realizing nuclear fusion reactions. Its working principle involves confining high-temperature plasma with a strong magnetic field, enabling nuclear fusion reactions to occur under high temperature and pressure. Control of the tokamak system is essential for achieving a stable plasma state and efficient nuclear fusion reactions. In the tokamak control system, vloop is a key parameter that directly affects plasma stability and confinement effectiveness.

[0003] The control of tokamak systems has the following drawbacks:

[0004] (1) Insufficient nonlinear response: Tokamak systems have highly nonlinear and complex dynamic characteristics. Traditional vloop control usually uses PID controllers, but this method is difficult to maintain good control performance under various operating conditions, resulting in insufficient control accuracy and stability.

[0005] (2) Parameter adjustment depends on experience: The parameters of the PID controller need to be manually adjusted based on human experience. It lacks automatic adjustment capability and is difficult to adapt to the rapid changes and complex conditions in the tokamak environment.

[0006] (3) Short-term optimization limitations: PID controllers mainly focus on error adjustment at the current moment and cannot optimize control performance over long periods, resulting in limited long-term stability and optimization effect of the system.

[0007] (4) Difficulty in handling high-dimensional states: Tokamak vloop control involves multiple high-dimensional state parameters. Traditional PID controllers have difficulty handling these complex parameters at the same time, resulting in unsatisfactory overall control performance.

[0008] In summary, this application proposes a reinforcement learning-based method for controlling the ring voltage of fusion plasma. Summary of the Invention

[0009] The purpose of this invention is to address the problem that the control effect of tokamak vloop control in nonlinear and complex dynamic environments is generally poor, and to propose a fusion plasma loop voltage control method based on reinforcement learning.

[0010] The technical solution of this invention: a method for controlling the ring voltage of fusion plasma based on reinforcement learning, comprising the following steps:

[0011] Select data for training the model: In the context of tokamak fusion plasma, the plasma parameters at the current moment are used as input, and the output is the vloop at the next moment;

[0012] Database construction: Collect discharge data of operating tokamak devices and discharge data of future devices in the early stages as initial training data for building models;

[0013] Building a neural network model: Long Short-Term Memory (LSTM) neural network is used as the training model, and the number of layers and neurons of the LSTM neural network are determined according to the specific training process;

[0014] Training of neural network models: Building and training models based on mainstream frameworks, and adjusting the network structure according to the prediction results;

[0015] Neural network model testing: Prepare new data other than the training data for model testing. Determine the effectiveness of the model by comparing the difference between the predicted results and the actual results. Perform multi-step inference verification and ensure that the predicted results in this process are consistent with the experimental data.

[0016] Constructing a reinforcement learning model: employing a proximal policy optimization algorithm;

[0017] Reinforcement learning training: The trained response model is used as the interactive environment for reinforcement learning. The exploration range is set according to the capabilities of LHW, and the optimal control command is found within the specified range.

[0018] Reinforcement learning model testing: Add some noise to the current state, use a reinforcement learning controller to give control commands based on the current state, input the control commands into the response environment, and observe the control effect;

[0019] Replace the current loop voltage controller: After meeting the control requirements, replace the traditional vloop control algorithm with the trained model, and provide the LHW power signal in real time according to the vloop control requirements by acquiring signals in real time, so as to achieve precise control of the vloop.

[0020] Optionally, the LSTM neural network is a special type of recurrent neural network (RNN) used to process sequential data and time series tasks. It forms a deep network by stacking multiple LSTM units, with the hidden state of each unit serving as the input for the next time step, in order to effectively handle long-term dependencies and capture contextual information from further back.

[0021] Optionally, the goal of the near-end policy optimization algorithm (PPO) is to maximize performance each time the policy is updated, and to ensure the stability of policy optimization by limiting the magnitude of policy updates.

[0022] Optionally, the plasma parameters include total plasma current, longitudinal coil current, PF coil current, average plasma electron string density, poloidal specific pressure, plasma internal inductance, energy storage, plasma boundary, radiation power, vloop, plasma boundary, auxiliary heating ECRH power, auxiliary heating ICRF power, and auxiliary heating LHW power.

[0023] Alternatively, the mainstream frameworks are TensorFlow or PyTorch.

[0024] Optionally, during the training of the neural network model, the error of the prediction result is less than 3%;

[0025] Optionally, the policy gradient optimization algorithm is used in the construction of the reinforcement learning model.

[0026] Optionally, the inference steps in the neural network model test are 100.

[0027] In summary, this application includes at least one of the following beneficial technical effects:

[0028] (1) Improve the accuracy and response speed of the tokamak vloop control algorithm:

[0029] By introducing deep reinforcement learning algorithms and surrogate models, this invention significantly improves the accuracy and response speed of tokamak vloop control. The reinforcement learning algorithm can adaptively adjust the control strategy under complex and nonlinear dynamic conditions, enabling the vloop to track target values ​​more accurately.

[0030] (2) Enhance the system's adaptive capability:

[0031] Reinforcement learning models possess self-learning and adaptive capabilities, enabling them to automatically optimize control strategies based on real-time environmental changes, reducing reliance on manual intervention and parameter adjustments. The system can quickly adapt to different operating conditions and unexpected situations, improving the intelligence level of the tokamak control system.

[0032] (3) Optimize long-cycle control effect:

[0033] By designing a reasonable reward function, reinforcement learning models can not only optimize control performance in the short term but also maintain good control performance over long periods. This significantly improves the long-term stability and optimization effect of the vloop control method in the tokamak system, ensuring continuous and stable nuclear fusion experimental conditions.

[0034] (4) Efficiently handles high-dimensional state parameters:

[0035] This invention leverages the powerful feature extraction capabilities of deep neural networks to effectively handle high-dimensional and complex state parameters in tokamak systems, thereby improving overall control performance. The surrogate model provides accurate vloop predictions, offering a more precise basis for the reinforcement learning controller.

[0036] (5) Reduce manual intervention and operating costs:

[0037] Because the invention mentions a reinforcement learning-based vloop control method with adaptive and automatic adjustment capabilities, it reduces the need for manual intervention and operation, thereby lowering operating costs and human resource investment. The control system can automatically optimize and adjust parameters, improving the convenience and efficiency of operation.

[0038] (6) Advancing progress in nuclear fusion research:

[0039] The control method of this invention can be extended to other control systems in the field of fusion, significantly improving the performance of tokamak control systems and providing a more stable and precise control means for nuclear fusion experiments. This is of great significance for realizing controlled nuclear fusion, promoting the progress of nuclear fusion research, and providing technical support for future energy development.

[0040] This invention introduces a reinforcement learning algorithm to replace the traditional PID controller, achieving more precise and adaptive vloop control in a tokamak system. This improves the accuracy, response speed, and long-term stability of the control system, overcoming the shortcomings of traditional controllers in nonlinear and complex dynamic environments. Attached Figure Description

[0041] Figure 1 is a schematic diagram of the interaction between reinforcement learning and neural networks;

[0042] Figure 2 is a schematic diagram of the construction steps of the reinforcement learning controller of the present invention. Detailed Implementation

[0043] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0044] This invention utilizes a neural network to construct a plasma response environment, and then obtains a vloop controller through reinforcement learning and interaction with the response environment. In experiments controlling the vloop with LHW power, the LHW drive current is affected by multiple factors, and traditional methods struggle to accurately determine the relationship between these factors. This invention uses data-driven methods and a neural network-based response model to summarize the relationship between LHW power and the vloop, and then uses reinforcement learning to explore the optimal control command for controlling the vloop under the current state. A schematic diagram of this process is shown in Figure 1, and the specific construction steps are shown in Figure 2.

[0045] I. Selecting data for training the model

[0046] In the context of tokamak fusion plasma, taking into full account the factors of LHW driving current controlling vloop, the plasma parameters at the current moment, including total plasma current, longitudinal coil current, PF coil current, average plasma electron string density, poloidal specific pressure, plasma internal inductance, energy storage, plasma boundary, radiation power, vloop, plasma boundary, auxiliary heating ECRH power, auxiliary heating ICRF power, and auxiliary heating LHW power, are taken as inputs, and the output is the vloop at the next moment.

[0047] II. Building the Database

[0048] After determining the model's input and output, data needs to be collected to build a database. Discharge data from the tokamak device during operation is stored, and existing data can be directly selected. In the early stages of the future device, some discharges will also be tested, and we can obtain this data as initial training data.

[0049] III. Constructing a Neural Network Model

[0050] Because some parameters exhibit a cumulative effect over time, a Long Short-Term Memory (LSTM) neural network is considered as the training model. An LSTM neural network is a special type of recurrent neural network (RNN) used to process sequential data and time-series tasks. LSTM networks construct deep networks by stacking multiple LSTM units, with the hidden state of each unit serving as the input for the next time step. This allows LSTM to effectively handle long-term dependencies, enabling the network to capture contextual information from further back. The number of layers and neurons in an LSTM neural network can be determined based on the specific training process.

[0051] IV. Training of Neural Network Models

[0052] This model can be built and trained using mainstream frameworks such as TensorFlow and PyTorch. Choose one framework to build the model, then feed the data into it for training. During training, the network structure needs to be adjusted based on the prediction results to ensure the prediction error is less than 3%.

[0053] V. Neural Network Model Testing

[0054] In addition to the data used for training, some data also needs to be prepared for model testing. New data is input into the model, and the difference between the predicted results and the actual results is compared to determine the model's effectiveness. This process requires multi-step inference verification; that is, the vloop obtained from the current data is used as the input for the second step of prediction, and so on, requiring approximately 100 inference steps, while ensuring that the predicted results of this process are consistent with the experimental data.

[0055] VI. Constructing a Reinforcement Learning Model

[0056] During tokamak discharge, plasma parameters generally do not change significantly; therefore, the proximal policy optimization (PPO) algorithm is considered here. The goal of PPO is to maximize performance with each policy update and ensure policy optimization stability by limiting the magnitude of policy updates. The PPO algorithm is a powerful and flexible policy optimization algorithm with good convergence, stability, and sample efficiency. It is widely used in practice and has achieved good performance in many reinforcement learning tasks. Other algorithms, such as policy gradient, can also be tried in this process.

[0057] VII. Strengthen learning and training

[0058] The pre-trained response model is used as the interaction environment for reinforcement learning, and the reinforcement learning model is trained accordingly. During training, the exploration range needs to be set based on the capabilities of the LHW (Local High-Level Weapon), and the optimal control command needs to be found within the specified range.

[0059] VIII. Reinforcement Learning Model Testing

[0060] Add some noise to the current state, and then use a reinforcement learning controller to give control commands based on the current state. Input the control commands into the response environment and observe the control effect.

[0061] 9. Replace the current loop voltage controller

[0062] After completing the tests in step eight and meeting the control requirements, the trained model can replace the traditional vloop control algorithm. By acquiring signals in real time and providing the LHW power signal according to the vloop control needs, precise vloop control can be achieved.

[0063] To verify the technical effects of the present invention, implementation verification was carried out.

[0064] Implementation Example Composition:

[0065] 1. Response Environment: The response environment employs an LSTM neural network with three hidden layers, each containing 30, 20, and 10 neurons respectively. This neural network model is based on the Python language, uses PyTorch as its framework, and is trained on a GPU. Data from the fusion discharge experiment using 1835mm discharge data points is used: 1600 discharges as the training set, 200 discharges as the validation set, and 35 discharges as the test set.

[0066] 2. Controller: The controller employs the POP reinforcement learning algorithm and is built using the Python language. In this environment, reinforcement learning control commands are input to the neural network model, which then provides the response. Training continues until a convergence condition is reached, at which point the model training stops. The convergence condition is maximizing the cumulative reward. Furthermore, a maximum number of iterations is set during training; the process stops once this maximum number of iterations is reached.

[0067] Effects:

[0068] 1. Fast Response Environment: The neural network-based response environment can provide prediction results quickly and accurately. The model makes a prediction in 1 millisecond, and the similarity of the predictions reaches 97.5%. The fast response environment ensures rapid training of reinforcement learning, reducing training time and cost.

[0069] 2. Superior control performance: The reinforcement learning-based vloop controller can better control the vloop and provides lower LHW power than traditional controllers. Good control is achieved at minimal cost.

[0070] 3. Integrated environment: Both the response environment and the control are developed based on the Python environment, making the interaction between models easier, training faster, and reducing the cost of controller design.

[0071] This implementation example demonstrates the specific structure and effects of the vloop controller of the present invention. By combining neural networks and reinforcement learning, precise control of the vloop is achieved, overcoming the shortcomings of traditional controllers and improving the control performance of the vloop.

[0072] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A method for controlling the loop voltage of fusion plasma based on reinforcement learning, characterized in that, Includes the following steps: Selecting training data for the model: Under the background of tokamak fusion plasma, the plasma parameters at the current moment are used as input, and the output is the vloop at the next moment; Building a database: Discharge data of the operating tokamak device and discharge data of the future device are collected as the initial training data for building the model; Building a neural network model: A long short-term memory neural network is used as the training model, and the number of layers and neurons of the LSTM neural network are determined according to the specific training process; Training the neural network model: The model is built and trained based on the mainstream framework, and the network structure is adjusted according to the prediction results; Testing the neural network model: New data other than the training data is prepared for model testing. The effectiveness of the model is judged by comparing the difference between the prediction results and the actual results. Multi-step inference verification is performed, and the prediction results are kept consistent with the experimental data in this process. Constructing a reinforcement learning model: employing a proximal policy optimization algorithm; Reinforcement learning training: The trained response model is used as the interactive environment for reinforcement learning. The exploration range is set according to the capabilities of LHW, and the optimal control command is found within the specified range. Reinforcement learning model testing: Add some noise to the current state, and use the reinforcement learning controller to give control commands based on the current state. Input the control commands into the response environment and observe the control effect; Replace the current loop voltage controller: After meeting the control requirements, replace the traditional Vloop control algorithm with the trained model. By acquiring signals in real time, the LHW power signal is given in real time according to the Vloop control requirements to achieve precise control of the Vloop.

2. The reinforcement learning-based fusion plasma ring voltage control method according to claim 1, characterized in that, The LSTM neural network is a special type of recurrent neural network (RNN) used to process sequential data and time series tasks. It forms a deep network by stacking multiple LSTM units, with the hidden state of each unit serving as the input for the next time step, in order to effectively handle long-term dependencies and capture contextual information from further back.

3. The reinforcement learning-based fusion plasma loop voltage control method according to claim 1, characterized in that, The goal of the Proximal Policy Optimization (PPO) algorithm is to maximize performance with each policy update and to ensure the stability of policy optimization by limiting the magnitude of policy updates.

4. The reinforcement learning-based fusion plasma ring voltage control method according to claim 1, characterized in that, The plasma parameters include total plasma current, longitudinal coil current, PF coil current, average plasma electron string density, poloidal specific pressure, plasma inductance, energy storage, plasma boundary, radiation power, vloop, plasma boundary, auxiliary heating ECRH power, auxiliary heating ICRF power, and auxiliary heating LHW power.

5. The reinforcement learning-based fusion plasma ring voltage control method according to claim 1, characterized in that, The mainstream frameworks are TensorFlow and PyTorch.

6. The reinforcement learning-based fusion plasma ring voltage control method according to claim 1, characterized in that, During the training of the neural network model, the error of the prediction result is less than 3%.

7. The reinforcement learning-based fusion plasma ring voltage control method according to claim 1, characterized in that, The policy gradient optimization algorithm is used in the construction of the reinforcement learning model.

8. The reinforcement learning-based fusion plasma ring voltage control method according to claim 1, characterized in that, In the test of the neural network model, the number of inference steps was 100.

Images