A fusion plasma ring voltage control method based on reinforcement learning

Abstract

The present invention relates to the field of fusion plasma control technology, and in particular to a fusion plasma ring voltage control method based on reinforcement learning. Its technical solution includes the following steps: selecting data for training models: in the context of tokamak fusion plasma, taking the plasma parameters at the current moment as input, and outputting the vloop at the next moment; building a database: collecting the discharge data of the running tokamak device and the discharge data of the early stage of the future device as the initial training data for building the model. The present invention improves the accuracy and response speed of the tokamak vloop control algorithm, and the reinforcement learning model has self-learning and self-adaptation capabilities, which improves the long-term stability and optimization effect of the vloop control method in the tokamak system, improves the overall control effect, reduces the need for manual intervention and operation, reduces operating costs and human resource investment, and is of great significance for realizing controlled nuclear fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of fusion plasma control, and in particular to a fusion plasma ring voltage control method based on reinforcement learning. Background Art

[0002] The Tokamak device is an important device for studying and realizing nuclear fusion reactions. Its working principle is to confine high-temperature plasma through a strong magnetic field, so that it can undergo nuclear fusion reactions under high temperature and high pressure. The control of the Tokamak system is crucial to achieving a stable plasma state and effective nuclear fusion reactions. In the Tokamak control system, vloop is a key parameter that directly affects the stability and confinement effect of the plasma.

[0003] The control of the Tokamak system has the following disadvantages:

[0004] (1) Insufficient nonlinear response: The tokamak system has highly nonlinear and complex dynamic characteristics. Traditional vloop control usually uses a PID controller, but this method is difficult to maintain good control effects under various operating conditions, resulting in insufficient control accuracy and stability.

[0005] (2) Parameter adjustment depends on experience: The parameters of the PID controller need to be manually adjusted based on human experience. It lacks automatic adjustment capabilities and is difficult to adapt to the rapid changes and complex conditions in the tokamak environment.

[0006] (3) Short-term optimization limitations: The PID controller mainly focuses on the error adjustment at the current moment and cannot optimize the long-term control performance, which limits the long-term stability and optimization effect of the system.

[0007] (4) Difficulty in processing high-dimensional states: Tokamak vloop control involves multiple high-dimensional state parameters. Traditional PID controllers find it difficult to process these complex parameters simultaneously, resulting in unsatisfactory overall control effects.

[0008] In summary, this application proposes a fusion plasma ring voltage control method based on reinforcement learning. Summary of the invention

[0009] The purpose of the present invention is to propose a fusion plasma ring voltage control method based on reinforcement learning in order to address the problem in the background technology that the control effect of tokamak vloop control in nonlinear and complex dynamic environments is generally poor.

[0010] The technical solution of the present invention is a fusion plasma ring voltage control method based on reinforcement learning, comprising the following steps:

[0011] Select the data for training the model: In the context of tokamak fusion plasma, the plasma parameters at the current moment are used as input, and the output is the vloop at the next moment;

[0012] Build a database: collect discharge data of the operating tokamak device and the early discharge data of the future device as initial training data for building the model;

[0013] Constructing a neural network model: Using a long short-term memory neural network as the training model, the number of layers and neurons of the LSTM neural network is determined according to the specific training process;

[0014] Training of neural network models: Building models based on mainstream frameworks and training them, and adjusting network structures based on prediction results;

[0015] Neural network model testing: prepare new data other than training data for model testing, judge whether the model is effective by comparing the difference between the predicted results and the actual results, perform multi-step reasoning verification, and ensure that the predicted results of the process are consistent with the experimental data;

[0016] Building a reinforcement learning model: using a proximal policy optimization algorithm;

[0017] Reinforcement learning training: Use the trained response model as the interactive environment for reinforcement learning, set the exploration range according to the capabilities of LHW, and find the optimal control command within the specified range;

[0018] Reinforcement learning model test: add a certain amount of noise to the current state, use the reinforcement learning controller to give a control command based on the current state, input the control command into the response environment, and check the control effect;

[0019] Replace the current loop voltage controller: After meeting the control requirements, the trained model will replace the traditional vloop control algorithm. By collecting signals in real time and giving LHW power signals in real time according to the requirements of vloop control, precise control of vloop can be achieved.

[0020] Optionally, the LSTM neural network is a special type of recurrent neural network (RNN) used to process sequence data and time series tasks. A deep network is formed by stacking multiple LSTM units, and the hidden state of each unit is used as the input of the next moment to effectively process long-term dependencies and capture more distant contextual information.

[0021] Optionally, the goal of the proximal policy optimization algorithm (PPO) is to maximize performance each time the policy is updated and to ensure the stability of the policy optimization by limiting the amplitude of the policy update.

[0022] Optionally, the plasma parameters include total plasma current, longitudinal field coil current, PF coil current, average density of plasma electron strings, polar pressure ratio, plasma internal inductance, energy storage, plasma boundary, radiation power, vloop, plasma boundary, auxiliary heating ECRH power, auxiliary heating ICRF power, and auxiliary heating LHW power.

[0023] Optionally, the mainstream framework is tensorflow framework or Pytorch framework.

[0024] Optionally, in the training of the neural network model, the error of the prediction result is less than 3%;

[0025] Optionally, a policy gradient optimization algorithm is used in constructing the reinforcement learning model.

[0026] Optionally, in the neural network model test, the number of inference steps is 100 steps.

[0027] In summary, the present application includes at least one of the following beneficial technical effects:

[0028] (1) Improve the accuracy and response speed of the Tokamak Vloop control algorithm:

[0029] By introducing deep reinforcement learning algorithms and agent models, the present invention significantly improves the accuracy and response speed of Tokamak vloop control. The reinforcement learning algorithm can adaptively adjust the control strategy under complex and nonlinear dynamic conditions, allowing the vloop to track the target value more accurately.

[0030] (2) Enhance the system’s adaptive capabilities:

[0031] The reinforcement learning model has the ability to self-learn and adapt, and can automatically optimize the control strategy according to real-time environmental changes, reducing the reliance on human intervention and manual adjustment of parameters. The system can quickly adapt to different operating conditions and emergencies, improving the intelligence level of the tokamak control system.

[0032] (3) Optimizing long-term control effects:

[0033] By designing a reasonable reward function, the reinforcement learning model can not only optimize the control effect in the short term, but also maintain good control performance in the long term. This significantly improves the long-term stability and optimization effect of the vloop control method in the tokamak system, ensuring continuous and stable nuclear fusion experimental conditions.

[0034] (4) Efficient processing of high-dimensional state parameters:

[0035] The present invention utilizes the powerful feature extraction capability of deep neural networks to effectively process high-dimensional and complex state parameters in the tokamak system, thus improving the overall control effect. The proxy model can provide accurate vloop predictions, providing a more accurate basis for the reinforcement learning controller.

[0036] (5) Reduce manual intervention and operating costs:

[0037] Since the invention mentions that the vloop control method based on reinforcement learning has the ability to adapt and automatically adjust, it reduces the need for manual intervention and operation, reduces operating costs and human resource investment. The control system can automatically optimize and adjust parameters, improving the convenience and efficiency of operation.

[0038] (6) Promoting the progress of nuclear fusion research:

[0039] The control method of the present invention can be extended to other control systems in the field of fusion, significantly improving the performance of the tokamak control system and providing a more stable and accurate control method for nuclear fusion experiments. This is of great significance for achieving controlled nuclear fusion, promoting the progress of nuclear fusion research, and providing technical support for future energy development.

[0040] The present invention introduces a reinforcement learning algorithm to replace the traditional PID controller to achieve more accurate and adaptive vloop control in the tokamak system, thereby improving the accuracy, response speed and long-term stability of the control system and overcoming the shortcomings of traditional controllers in nonlinear and complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the interaction between reinforcement learning and neural networks;

[0042] Figure 2 Schematic diagram of the steps for constructing the reinforcement learning controller of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.

[0044] The present invention will use a neural network to construct a plasma response environment, and then interact with the response environment through reinforcement learning to obtain a vloop controller. In the experiment of LHW power controlling vloop, the LHW driving current is affected by multiple factors, and it is difficult for traditional methods to accurately give the relationship between the two. The present invention uses data-driven and neural network-based response models to summarize the relationship between LHW power and vloop, and then uses reinforcement learning to explore the optimal control command to control vloop in the current state. The schematic diagram of the process is shown in Figure 1 The specific construction steps are as shown in Figure 2 shown.

[0045] 1. Select data for training the model

[0046] In the context of tokamak fusion plasma, the factors of LHW driving current controlling vloop are fully considered, and the plasma parameters of the current moment, namely, total plasma current, longitudinal field coil current, PF coil current, average density of plasma electron strings, poloidal pressure ratio, plasma internal inductance, energy storage, plasma boundary, radiation power, vloop, plasma boundary, auxiliary heating ECRH power, auxiliary heating ICRF power and auxiliary heating LHW power are taken as input, and the output is vloop at the next moment.

[0047] 2. Building a database

[0048] After determining the input and output of the model, we need to collect data to build a database. The discharge data of the tokamak device will be stored during operation, and we can directly select the existing data. In the early stage of the future device, we will also try some discharges, and we can also obtain these data as initial training data.

[0049] 3. Building a Neural Network Model

[0050] Since some parameters have a cumulative effect over time, the Long Short-Term Memory (LSTM) neural network is considered to be used as a training model. The LSTM neural network is a special type of recurrent neural network (RNN) used to process sequence data and time series tasks. The LSTM network forms a deep network by stacking multiple LSTM units, and the hidden state of each unit is used as the input of the next moment. In this way, LSTM can effectively handle long-term dependencies, allowing the network to capture more distant contextual information. The number of layers and neurons of the LSTM neural network can be determined according to the specific training process.

[0051] 4. Training of Neural Network Model

[0052] The model can be built and trained based on the current mainstream frameworks, such as tensorflow, Pytorch, etc. Choose one of the frameworks to build the model, and then put the data into the model for training. During the training process, the network structure needs to be adjusted according to the predicted results so that the error of the predicted results is less than 3%.

[0053] V. Neural Network Model Testing

[0054] In addition to the data used for training, some data needs to be prepared for model testing. New data is input into the model to compare the difference between the predicted results and the actual results to determine whether the model is effective. This process requires multi-step reasoning verification, that is, the vloop obtained by inputting the current data is used as the input of the second step to predict here, and so on. It takes about 100 steps of reasoning and ensures that the results predicted by this process are consistent with the experimental data.

[0055] 6. Building a reinforcement learning model

[0056] During the tokamak discharge process, the plasma parameters generally do not change much, so the proximal policy optimization algorithm (PPO) is considered here. The goal of PPO is to maximize the performance each time the policy is updated, and to ensure the stability of policy optimization by limiting the amplitude of policy updates. The PPO algorithm is a powerful and flexible policy optimization algorithm with good convergence, stability and sample efficiency. It is widely used in practice and has achieved good performance in many reinforcement learning tasks. In this process, we can also try other algorithms, such as policy gradient.

[0057] 7. Reinforcement Learning Training

[0058] The trained response model is used as the interactive environment for reinforcement learning, and the reinforcement learning model is trained. During the training process, the exploration range needs to be set according to the capabilities of LHW to find the optimal control command within the specified range.

[0059] 8. Reinforcement Learning Model Testing

[0060] Add a certain amount of noise to the current state, and then use the reinforcement learning controller to give control commands based on the current state, input the control commands into the response environment, and check the control effect.

[0061] 9. Replace the current ring voltage controller

[0062] After completing the test in step 8, the trained model can be used to replace the traditional vloop control algorithm if the control requirements are met. By collecting signals in real time and giving LHW power signals in real time according to the requirements of vloop control, accurate control of vloop can be achieved.

[0063] In order to verify the technical effect of the present invention, implementation verification is carried out.

[0064] Embodiment composition:

[0065] 1. Response environment: The response environment uses an LSTM neural network, which contains three hidden layers, with 30, 20, and 10 neurons in each hidden layer. The neural network model is based on the python language, the model framework is based on pytorch, and a GPU is deployed for model training. Based on the discharge data of 1835 guns in the fusion discharge experiment, 1600 gun data is used as the training set, 200 gun data is used as the validation set, and 35 gun data is used as the test set.

[0066] 2. Controller: The controller uses the POP reinforcement learning algorithm and is built based on the Python language. In this environment, the control command of reinforcement learning is input to the neural network model, and the neural network model gives the response result. The model training stops after continuous exploration to reach the convergence condition, and the convergence condition is to maximize the cumulative reward. In addition, during the training process, we set the maximum number of iterations, and stop if the maximum number of iterations is reached.

[0067] Effect:

[0068] 1. Fast response environment: The response environment based on neural network can quickly and accurately give prediction results. The model prediction time is 1 millisecond, and the prediction similarity reaches 97.5%. The fast response environment ensures fast training of reinforcement learning training and reduces training time and cost.

[0069] 2. Superior control effect: The vloop controller based on reinforcement learning can better control the vloop and give less LHW power than the traditional controller. Better control is achieved at the lowest cost.

[0070] 3. Integrated environment: Whether it is response environment or control, they are all developed based on Python environment. The interaction between models is easier, training is faster, and the cost of controller design is reduced.

[0071] This implementation example demonstrates the specific structure and effect of the vloop controller of the present invention. Through the combination of neural network and reinforcement learning, accurate control of vloop is achieved, which makes up for the shortcomings of traditional controllers and improves the control performance of vloop.

[0072] The above specific embodiments are only several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A fusion plasma ring voltage control method based on reinforcement learning, characterized in that: The following steps are involved: Select the data for training the model: In the context of tokamak fusion plasma, the plasma parameters at the current moment are used as input, and the output is the vloop at the next moment; Build a database: collect discharge data of the operating tokamak device and the early discharge data of the future device as initial training data for building the model; Constructing a neural network model: Using a long short-term memory neural network as the training model, the number of layers and neurons of the LSTM neural network is determined according to the specific training process; Training of neural network models: Building models based on mainstream frameworks and training them, and adjusting network structures based on prediction results; Neural network model testing: prepare new data other than training data for model testing, judge whether the model is effective by comparing the difference between the predicted results and the actual results, perform multi-step reasoning verification, and ensure that the predicted results of the process are consistent with the experimental data; Building a reinforcement learning model: using a proximal policy optimization algorithm; Reinforcement learning training: Use the trained response model as the interactive environment for reinforcement learning, set the exploration range according to the capabilities of LHW, and find the optimal control command within the specified range; Reinforcement learning model test: add a certain amount of noise to the current state, use the reinforcement learning controller to give a control command based on the current state, input the control command into the response environment, and check the control effect; Replace the current loop voltage controller: After meeting the control requirements, the trained model will replace the traditional vloop control algorithm. By collecting signals in real time and giving LHW power signals in real time according to the requirements of vloop control, precise control of vloop can be achieved.

2. The fusion plasma ring voltage control method based on reinforcement learning according to claim 1 is characterized in that: The LSTM neural network is a special type of recurrent neural network (RNN) used to process sequence data and time series tasks. A deep network is constructed by stacking multiple LSTM units. The hidden state of each unit is used as the input of the next moment to effectively process long-term dependencies and capture more distant contextual information.

3. The fusion plasma ring voltage control method based on reinforcement learning according to claim 1 is characterized in that: The goal of the Proximal Policy Optimization (PPO) algorithm is to maximize the performance each time the policy is updated and to ensure the stability of the policy optimization by limiting the amplitude of the policy update.

4. The method for controlling fusion plasma ring voltage based on reinforcement learning according to claim 1, characterized in that: The plasma parameters include total plasma current, longitudinal field coil current, PF coil current, average density of plasma electron strings, poloidal pressure ratio, plasma internal inductance, energy storage, plasma boundary, radiation power, vloop, plasma boundary, auxiliary heating ECRH power, auxiliary heating ICRF power, and auxiliary heating LHW power.

5. The method for controlling fusion plasma ring voltage based on reinforcement learning according to claim 1, characterized in that: The mainstream frameworks are tensorflow framework or Pytorch framework.

6. The method for controlling fusion plasma ring voltage based on reinforcement learning according to claim 1, characterized in that: During the training of the neural network model, the error of the prediction result is less than 3%.

7. The method for controlling fusion plasma ring voltage based on reinforcement learning according to claim 1, characterized in that: In constructing the reinforcement learning model, a policy gradient optimization algorithm is adopted.

8. The method for controlling fusion plasma ring voltage based on reinforcement learning according to claim 1, characterized in that: In the neural network model test, the number of inference steps is 100.

Images