Plasma vertical instability control method based on adversarial training

Abstract

The invention relates to the technical field of magnetic confinement nuclear fusion devices, in particular to a plasma vertical instability control method based on adversarial training, and adopts the technical scheme that the method comprises the following steps: collecting plasma displacement at several consecutive moments and current and voltage data of a vertical instability controller I C coil as observation values; the observation data are input into a plasma vertical displacement control model obtained through reinforcement learning training, and then the voltage of an I C coil is output as a control command. The opponent model is added in the adversarial reinforcement learning training process of the vertical displacement control model, and the plasma current target value in the training environment is disturbed, so that the anti-disturbance capability of the vertical instability controller is enhanced, and the overcurrent risk of the I C coil is effectively reduced.

Description

Technical Field

[0001] The invention relates to the technical field of magnetic confinement nuclear fusion devices, and in particular to a plasma vertical instability control method based on adversarial training. Background Art

[0002] The tokamak device is a toroidal container that can achieve controlled nuclear fusion through magnetic confinement and vacuum insulation. To obtain nuclear fusion energy, it is necessary to increase the discharge parameters, such as density, temperature and confinement time, to meet the ignition conditions. Elongated plasma helps to improve the discharge parameters, but it will lead to vertical displacement instability, and the loss of vertical displacement will inevitably lead to large ruptures, so we must actively control the vertical displacement. The vertical displacement is affected by many factors such as passive structure and power supply delay. It is a high-order system with complex response, limited system capacity, and complex and diverse disturbances, so the robustness of the controller is required to be high.

[0003] In the full superconducting tokamak discharge experiment, the PF coil current will be disturbed, which will cause the horizontal field equilibrium position to change. At this time, the vertical displacement controller needs a larger IC coil current to maintain the vertical displacement position, and there is a risk of overcurrent.

[0004] There are two ways to solve IC coil overcurrent in traditional vertical displacement control, but both have their own problems. The first is filtered PD control. The 14 PF coil currents I are removed by high-pass filtering. PF =[I PF1 ,I PF2 ,I PF3 ,...,I PF14 ] Overcurrent caused by slow disturbances, but the system is still unstable and there is a possibility of slow overcurrent. In addition, the control parameters need to be adjusted based on simulation or experiment, and it is difficult to obtain the optimal parameters. In order to solve the problem of slow overcurrent, the traditional control method adds speed feedback control, but the parameter range is narrow and it is difficult to adapt to the response of different configurations. If a compensator is added to adjust the dynamic response, additional jitter will be introduced.

[0005] Robust adversarial reinforcement learning can solve this problem. The logic of the agent's decision in reinforcement learning is to choose the action that can maximize the reward based on the observed value. First, we can include the size of the IC coil current in the reward setting. When the value of the IC coil current is saturated, a large negative reward is fed back to the agent, so that the agent will learn to avoid overcurrent. Secondly, the agent may be able to extract information related to the horizontal field equilibrium position from the observed value, so that the agent has the ability to adjust the position of the vertical displacement to make it closer to the equilibrium field position, thereby avoiding stabilizing the vertical displacement with a large IC current.

[0006] Robust adversarial reinforcement learning adds an opponent who is also an intelligent agent to the intelligent agent. This opponent will attack the weaknesses of the intelligent agent, and the intelligent agent needs to find the optimal strategy in the worst case. The problem we need to solve is that the intelligent agent needs to avoid overcurrent of the IC coil under the disturbance of the PF coil current. Therefore, this application proposes a plasma vertical instability control method based on adversarial training. Summary of the invention

[0007] The purpose of the present invention is to propose a plasma vertical instability control method based on adversarial training to address the problem in the background technology that a vertical displacement controller requires a larger IC coil current to maintain the vertical displacement position, which poses a risk of overcurrent.

[0008] The technical solution of the present invention is a plasma vertical instability control method based on adversarial training, comprising the following method steps:

[0009] Collect the relevant plasma displacement and IC coil current I at several consecutive moments IC and voltage V IC The data are used as observation values, and the observation values ​​are input into a trained vertical displacement controller to output control commands to control the vertical displacement of the plasma;

[0010] A plasma response model for simulating the experimental environment is constructed, which is represented by state-space equations;

[0011] Construct a reinforcement learning training environment that includes an adversary that can change the target value of the relevant coil current according to the observed value and whose reward function is opposite to the controller reward function;

[0012] Construct an agent with a policy network and a critic network containing different input data;

[0013] Construct a multi-objective reward function, perform pre-training before formal training;

[0014] Through the interaction between the agent and the environment, a good policy network is trained based on the rewards of environmental feedback and a specific data training set.

[0015] Optionally, the vertical displacement controller adopts a strategy network architecture, in which the observation values ​​include tokamak vertical displacement data at t, t-1, t-2, t-3, IC coil current data at t, t-1, t-2, t-3 and IC voltage data at t-1, t-2, t-3, and the time tn represents n control cycles relative to the current time, each control cycle is 100 microseconds, which is used to convert the observation value into a voltage command for the IC coil.

[0016] Optionally, in the reward function, a large negative reward is given to the agent when the IC coil current reaches saturation to guide the agent to avoid overcurrent.

[0017] Optionally, the agent can extract information related to the horizontal field equilibrium position from the observations and adjust the vertical displacement position to avoid the overcurrent risk caused by using a large IC current to stabilize the vertical displacement.

[0018] Optionally, an evaluation network is used to evaluate the IC coil voltage command, so that the strategy network adjusts the strategy accordingly to optimize the control parameters.

[0019] Optionally, the plasma linear response matrix in the constructed plasma response model is expressed by the state space equation x(t+1)=Ax(t)+Bu(t), y(t)=Cx(t)+Du(t), where x is the current on the coil and the passive structure, u is the IC coil voltage, y is the plasma vertical displacement and the IC coil current, and the control frequency is 10kHz.

[0020] Optionally, the opponent in the reinforcement learning training environment changes the PF coil current target value ΔPF∈[-400A,400A] based on the observed value.

[0021] Optionally, the constructed reward function includes items of reducing learning difficulty, avoiding IC coil overcurrent, and reducing plasma vertical displacement jitter. Pre-training is first performed based on the item of reducing learning difficulty, and then the coefficient of the item is set to zero for formal training.

[0022] Compared with the prior art, the present invention has at least one of the following beneficial technical effects:

[0023] The present invention enhances the anti-disturbance capability through the confrontation mechanism in the reinforcement learning training environment, and the jitter reduction term in the reward function makes the plasma motion smoother.

[0024] The negative reward mechanism for IC coil current saturation in the reward function and the intelligent agent’s adjustment of displacement position by extracting information effectively reduce the risk of overcurrent.

[0025] The present invention collects data at multiple times, constructs intelligent agents and conducts interactive training, so that the system can adapt to different environments, conditions and complex changes.

[0026] By evaluating the network-assisted strategy network adjustment strategy, multi-objective reward function training can comprehensively optimize the comprehensive performance of the controller in complex environments.

[0027] In the adversarial reinforcement learning training process of the vertical displacement control model in the present invention, an opponent model is added, and the plasma current target value in the training environment is disturbed, which enhances the anti-disturbance ability of the vertical instability controller and effectively reduces the overcurrent risk of the IC coil. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the reinforcement learning environment;

[0029] Figure 2 Schematic diagram for training an agent;

[0030] Figure 3 The control result of the P ID controller Figure 1 ;

[0031] Figure 4 The control result of RARL controller Figure 1 ;

[0032] Figure 5 The control result of the P ID controller Figure 2 ;

[0033] Figure 6 The control result of RARL controller Figure 2 . DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0035] In the following embodiments, the RARL controller is called Robust Adversarial Reinforcement Learning Controller, abbreviated as RARL controller below, and its Chinese name is Robust Adversarial Reinforcement Learning Controller.

[0036] Example

[0037] The present invention proposes a plasma vertical instability control method based on adversarial training, comprising: obtaining the tokamak vertical displacement data and the tokamak IC coil current data at the time t, t-1, t-2, and t-3, and the IC voltage data at the time t-1, t-2, and t-3 as observation values, inputting the 11-dimensional observation values ​​into the trained vertical displacement controller, and the vertical displacement controller outputs the voltage value of the IC coil, and the voltage value is input into the control system as a control command to complete the control of the vertical displacement of the plasma. Wherein the time tn represents the time before n control cycles, and one control cycle is 100 microseconds.

[0038] Among them, the plasma vertical displacement controller is a strategy network. After the observation value is input into the strategy network, the output value of the strategy network is obtained: the voltage command of the IC coil.

[0039] The policy network is obtained in the following way:

[0040] 1. Construct a plasma response model to simulate the EAST environment

[0041] The plasma linear response matrix can be expressed by the state space equation as follows:

[0042] x(t+1)=Ax(t)+Bu(t)

[0043] y(t)=Cx(t)+Du(t)

[0044] Where x is the current on the coil and the passive structure, u is the IC coil voltage, y is the plasma vertical displacement and the IC coil current, and the control frequency is 10 kHz.

[0045] 2. Build a reinforcement learning training environment

[0046] like Figure 1 As shown in the figure, the Env box is the environment that interacts with the agent, and the Agent box is the agent. Figure 1 As can be seen in the figure, the vertical displacement controller obtains the observation value from the environment and then outputs the IC coil voltage V IC The command is passed to the EAST module, which affects the system state. In addition, the adversary changes the target value of the PF coil current based on the observed value, and then the PF coil controller generates a PF coil voltage command based on this, thereby affecting the EAST system state and causing disturbances.

[0047] In the previous training, the PF target was fixed, so the PF coil current I PF The test results show that under the influence of the PF coil current, the IC coil current changes significantly, and there is a risk of overcurrent. In this embodiment, an opponent is used to change the value of PF target. The opponent is adversary, which is also a trained intelligent agent. The meaning of this value is 14 PF coil current I PF =[I PF1 ,I PF2 ,I PF3 ,...,I PF14], the range of change is no more than 400A on the original basis, that is, the opponent's output is ΔPF∈[-400A,400A]. This change does not change the system state instantly, but is subject to physical constraints. The constraint changes the target value rather than the actual value, and the actual value will approach the target value. The opponent's reward function is the opposite of the controller's reward function, which ensures that the opponent will attack the controller. The controller trained in this way can resist this attack, that is, it has the ability to resist PF current disturbances.

[0048] 3. Building an Intelligent Agent

[0049] The agent consists of a policy network and an evaluation network

[0050] The strategy network is a fully connected neural network. The input of the network is the plasma vertical displacement data and IC coil current data at time t, t-1, t-2, and t-3, as well as the IC voltage data at time t-1, t-2, and t-3. The output of the network is multiplied by 1200 to obtain the output value of the strategy network.

[0051] The input of the evaluation network is the plasma vertical displacement data, IC coil current data and IC voltage data at time t, t-1, t-2 and t-3, and one more voltage at time t than the strategy network. The output value of the evaluation network is the evaluation of the IC coil voltage command at the current moment, and the strategy network will adjust its own strategy based on the evaluation.

[0052] 4. Constructing the Reward Function

[0053] Reward function:

[0054] Among them, the first item can reduce the learning difficulty, but it affects the final performance of the controller; the second item is to avoid IC coil overcurrent; the third item is to reduce the jitter of the vertical displacement of the plasma. max , They refer to the maximum vertical displacement of the plasma, the maximum current of the IC coil, and the maximum voltage of the IC coil in the actual tokamak discharge experiment. k1, k2, and k3 are adjusted according to the training results.

[0055] First, pre-training is performed. After the pre-training is completed, k1 is set to zero and the item is removed for formal training.

[0056] 5. Training the Agent

[0057] like Figure 2As shown in the figure, the agent obtains observations by observing the environment, and the policy network obtains voltage commands through the observations. After executing the command, the environment transfers to the next state according to the plasma response model, and a reward is fed back to the agent. When the environment meets certain conditions, it is reset to the initial state, and the agent begins the next round of interaction with the environment until the training reaches the termination condition. The four types of data, namely the observation value, the voltage command taken at the current moment, the observation value at the next moment, and the reward fed back to the agent, form a data unit. The agent can obtain a series of data units as a training set for training the agent through continuous interaction with the environment, and finally obtain a trained policy network.

[0058] In order to verify the technical effect of the present invention, an experiment is conducted to verify the vertical displacement by using the trained vertical displacement controller to simulate the vertical displacement and compare it with the traditional PID controller. Considering the control delay of 3 control cycles in the actual experiment, the results are as follows: Figure 3-4 ; It can be seen that the IC coil current in the RARL controller is controlled closer to 0, effectively avoiding overcurrent.

[0059] Figure 5-6 This is the result of testing the anti-disturbance capability of the model and comparing it with PID. It can be seen that in the face of disturbances, the PID controller will make the IC current amplitude reach several thousand amperes, while the RARL controller is less than 500A, which shows that the RARL controller has strong anti-disturbance capability.

[0060] The present invention improves control stability and enhances anti-disturbance capability: by constructing a reinforcement learning training environment, an opponent is set up that changes the coil current target value based on the observed value and whose reward function is opposite to the controller reward function. This adversarial mechanism enables the controller to learn more effective control strategies in a complex interference environment, thereby improving the anti-disturbance capability of plasma vertical displacement control. For example, when the opponent changes the PF coil current target value, the controller can learn and adapt to this change and maintain the stability of the system.

[0061] At the same time, the reward function includes an item to reduce the jitter of the vertical displacement of the plasma. This allows the control method to not only focus on reaching the target position when adjusting the vertical displacement of the plasma, but also pay attention to the smoothness of the displacement process, thereby effectively reducing the jitter phenomenon in the vertical displacement process of the plasma and making the movement of the plasma more stable.

[0062] In the reward function of the present invention, a large negative reward is given to the agent when the IC coil current reaches saturation. This mechanism guides the agent to actively avoid the IC coil current from reaching saturation during the learning process, thereby effectively reducing the risk of overcurrent. The agent will avoid overcurrent by continuously adjusting the control strategy, thereby improving the safety of the system.

[0063] The intelligent agent can extract information related to the horizontal field equilibrium position from the observed value, and then adjust the vertical displacement position to avoid the risk of overcurrent caused by using large IC current to stabilize the vertical displacement. This method based on information extraction and strategy adjustment optimizes the control of the vertical displacement of the plasma from the control principle, avoiding the overcurrent problem caused by improper control.

[0064] By collecting relevant plasma displacement, coil current and voltage data at multiple times as observation values, and using these data in the training and control process of the intelligent agent, the control method can adapt to different plasma states and environmental conditions. Whether it is different displacement states, current changes or voltage fluctuations, the system can analyze and learn based on these multi-time data, thereby adjusting the control strategy to ensure the stability and reliability of the control effect.

[0065] An intelligent agent with a policy network and an evaluation network containing different input data is constructed, and a good policy network is trained through continuous interaction between the intelligent agent and the environment. This design enables the system to adapt to complex changes in the plasma system, including changes in the plasma response model, the influence of various external interferences, etc. The intelligent agent continuously learns and optimizes the control strategy during the interaction with the environment, thereby improving the adaptability of the entire control system to changes in complex systems.

[0066] It is worth noting that the evaluation network is used to evaluate the IC coil voltage command, so that the strategy network can adjust the strategy accordingly to achieve the optimization of the control parameters. The addition of the evaluation network provides a reference for the optimization of the strategy network, allowing the strategy network to adjust and optimize the network parameters based on the previous control data. ， The policy network only adjusts and optimizes parameters based on evaluation during the training process. In actual control, the parameters are fixed and will not change. This mechanism avoids the problem of difficulty in obtaining optimal parameters in traditional control methods and improves the performance of the controller.

[0067] The reward function constructed by the present invention includes multiple goals, such as reducing learning difficulty, avoiding IC coil overcurrent, and reducing plasma vertical displacement jitter. By first pre-training based on the item of reducing learning difficulty, and then setting the coefficient of this item to zero for formal training, the system can focus on different goals at different training stages, thereby comprehensively optimizing the performance of the controller. This multi-objective training method enables the controller to be optimized in multiple aspects, improving the comprehensive performance of the controller in complex environments.

[0068] The above specific embodiments are only several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A plasma vertical instability control method based on adversarial training, characterized in that: The method comprises the following steps: Collect the relevant plasma displacement and IC coil current I at several consecutive moments IC and voltage V IC The data are used as observation values, and the observation values ​​are input into a trained vertical displacement controller to output control commands to control the vertical displacement of the plasma; A plasma response model for simulating the experimental environment is constructed, which is represented by state-space equations; Construct a reinforcement learning training environment that includes an adversary that can change the target value of the relevant coil current according to the observed value and whose reward function is opposite to the controller reward function; Construct an agent with a policy network and a critic network containing different input data; Construct a multi-objective reward function, perform pre-training before formal training; Through the interaction between the agent and the environment, a good policy network is trained based on the rewards of environmental feedback and a specific data training set.

2. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: The vertical displacement controller adopts a strategy network architecture, in which the observation values ​​include the tokamak vertical displacement data at t, t-1, t-2, and t-3, the IC coil current data at t, t-1, t-2, and t-3, and the IC voltage data at t-1, t-2, and t-3. The time tn represents n control cycles before the current time, each control cycle is 100 microseconds, and is used to convert the observation value into a voltage command for the IC coil.

3. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: In the reward function, a large negative reward is given to the agent when the IC coil current reaches saturation to guide the agent to avoid overcurrent.

4. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: The intelligent agent is able to extract information related to the horizontal field equilibrium position from the observations and then adjust the vertical displacement position to avoid the overcurrent risk caused by using large IC current to stabilize the vertical displacement.

5. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: The evaluation network is used to evaluate the IC coil voltage command, so that the strategy network can adjust the strategy accordingly to achieve optimization of control parameters.

6. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: The plasma linear response matrix in the constructed plasma response model is expressed by the state space equation x(t+1)=Ax(t)+Bu(t), y(t)=Cx(t)+Du(t), where x is the current on the coil and the passive structure, u is the IC coil voltage, y is the plasma vertical displacement and the IC coil current, and the control frequency is 10kHz.

7. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: In the reinforcement learning training environment, the opponent changes the PF coil current target value ΔPF∈[-400A,400A] based on the observed value.

8. The method for controlling plasma vertical instability based on adversarial training according to claim 1, characterized in that: The constructed reward function includes items such as reducing learning difficulty, avoiding overcurrent of IC coil and reducing vertical displacement jitter of plasma. Pre-training is first performed based on the item of reducing learning difficulty, and then the coefficient of this item is set to zero for formal training.

Images