Klystron state feedback control method based on reinforcement learning

Abstract

The present invention provides a klystron state feedback control method based on reinforcement learning, including: establishing a reinforcement learning model, defining a state based on the state of the klystron, defining an action based on the amplitude and phase setting values ​​of a low-level control system, defining a reward based on the current deviation of the klystron from the ideal state and the change in the deviation after the action, using an engine to receive the amplitude and phase setting values ​​from the environment, and sending the obtained klystron state to the environment; using a virtual engine in a training mode to accelerate model training, switching to a real engine in a control mode and using a low-level control system with the above model as a feedback algorithm to feedback control the klystron state of an actual accelerator. The method of the present invention can improve the control accuracy of the low-level control system on the klystron state, make the klystron output more stable, and can greatly reduce the training time required for the feedback control algorithm to iterate to the optimal process.

Description

Technical Field

[0001] The invention relates to the field of accelerator measurement and control, in particular to a klystron state feedback control method based on reinforcement learning, and belongs to the field of new generation information technology. Background Art

[0002] As a key component of particle accelerators, microwave systems can provide energy for charged particles to accelerate them. The energy, energy dispersion, emittance and beam quality of the beam are directly affected by the stability of the microwave field. Therefore, controlling the amplitude and phase stability of the microwave field is of great significance to improving the performance of the accelerator.

[0003] A typical microwave system is mainly composed of a signal source, a low-level RF system (LLRF) and a high-power microwave system. The high-power microwave system mainly includes a solid-state amplifier, a klystron, a waveguide and a resonant cavity. After the low-level control system outputs a modulated excitation signal, it is initially amplified by the solid-state amplifier and sent to the klystron. The high-power high-frequency signal output by the klystron is then fed into the resonant cavity through the waveguide. The charged particles obtain the energy of the high-frequency signal in the resonant cavity and complete the acceleration process.

[0004] In actual environments, microwave systems are affected by various factors such as cooling water temperature and high-voltage modulator stability, and their actual output amplitude and phase will gradually deviate from the set values. In order to stabilize the amplitude and phase of the microwave system output, a low-level control system is generally used to control the amplitude and phase of the klystron output. This requires the low-level control system to use a feedback control algorithm to control the output of the klystron. The feedback control algorithm changes the amplitude and phase of the klystron output by changing the set values ​​of the amplitude and phase output of the low-level control system.

[0005] Specifically, if we want to accelerate charged particles, we need to control the amplitude and phase of the microwave field to a certain set of values. The signal output by the klystron in the high-power microwave system contains the amplitude and phase information of the microwave field. The low-level control system can down-convert the signal and demodulate it to obtain the amplitude and phase information of the microwave field. After calculating the amplitude and phase of the signal in the klystron, the low-level control system compares it with the amplitude and phase values ​​of the microwave field we want. If the two are different, the amplitude and phase of the excitation signal output by the low-level system can be changed by adjusting the setting value of the low-level control system, thereby making the amplitude and phase of the microwave field reach the setting value we want.

[0006] At present, low-level control systems mostly use PI (Proportion Integration) controller as feedback control algorithm. Although it is simple, easy to implement and has good versatility, due to the nonlinear input-output characteristics of the klystron in amplitude and phase, and the RF reference leakage and imbalance of the low-level control system during IQ (In-phase Quadrature) modulation, the phase of the klystron output signal will change when the amplitude setting value of the low-level control system changes, and the amplitude of the klystron output signal will also change when the phase setting value of the low-level control system changes, that is, there is a coupling relationship between the amplitude and phase of the actual output of the klystron, and this coupling relationship is difficult to quantitatively solve. The feedback algorithm PI controller of the traditional low-level control system is a single-variable controller, which is only applicable to the control problem of only one controlled variable or multiple controlled variables that can be operated independently of each other. When you want to control the amplitude and phase stability of the klystron output at the same time, the PI controller will not work well due to the mutual influence between the controlled quantities. In addition, improper parameter selection when using a PI controller can cause system oscillation or slow down the adjustment speed, greatly reducing the reliability and availability of the system. Selecting a suitable set of parameters requires a lot of attempts and it is impossible to determine whether the current parameters are the optimal parameters.

[0007] Therefore, it is necessary to propose a new feedback control and offline training method for low-level control systems in combination with artificial intelligence to improve the feedback control accuracy of low-level systems and significantly reduce the training time required for the feedback control algorithm to iterate to the optimal process. Summary of the invention

[0008] The object of the present invention is to provide a klystron state feedback control method based on reinforcement learning to improve the feedback control accuracy of low-level systems and significantly reduce the training time required for the feedback control algorithm to iterate to the optimal process.

[0009] In order to achieve the above object, the present invention provides a klystron state feedback control method based on reinforcement learning, comprising:

[0010] S1: Establishing a reinforcement learning model with an agent and an environment; the reinforcement learning model defines a state based at least on a state of a klystron, and defines an action based at least on the setting values ​​of the amplitude and phase of a low-level control system; defining a reward in the environment based at least on a deviation of a current state of the klystron from an ideal state of the klystron and a change in the deviation after taking an action;

[0011] S2: Setting an engine in the environment so that the engine receives the amplitude and phase setting values ​​of the low-level control system from the environment and sends the corresponding klystron state to the environment, wherein the engine includes a virtual engine and a real engine, and the real engine includes the low-level control system and the klystron in the actual accelerator;

[0012] S3: training the reinforcement learning model, and then using the trained model for feedback control; switching to the virtual engine in the training mode, and switching to the real engine in the control mode, so as to realize feedback control of the klystron state in the actual accelerator through a low-level control system based on the reinforcement learning model as a feedback control algorithm.

[0013] The action is the difference between the amplitude and phase setting values ​​of the next low-level control system output and the amplitude and phase setting values ​​of the current low-level control system; the state is the difference between the state of the klystron and the ideal state; the reward is calculated by the reward function, including the single-step reward r t As well as mission completion rewards introduced when missions are completed.

[0014] The single-step reward r t for:

[0015] ;

[0016] in, r t It is a single-step reward. r diff is a penalty item, r action is the action reward, α It is a number less than 0, indicating the weight of the penalty item;

[0017] Penalty r diff for:

[0018] ;

[0019] in, is the degree of deviation between the state of the klystron and the ideal state at step t, , [ i ] represents the first i elements, so , Respectively represent the state difference The first and second elements in k is the scaling factor;

[0020] Rewards for:

[0021] ;

[0022] in, Indicates the change in the degree of deviation after taking action. , For the t The degree of deviation between the state of the step-by-step klystron and the ideal state, For the t -1 step The degree of deviation between the state of the klystron and the ideal state, [ i ] represents the first i elements, m It represents the reward obtained if the action taken makes the klystron state closer to the ideal state compared with the previous step, n is a number less than 0, indicating the penalty if the action taken makes the klystron state further away from the ideal state than the previous step. I is the indicator function.

[0023] The step S3 specifically includes: in the training mode, the intelligent agent selects actions according to the state returned by the environment and continuously adjusts its own network parameters according to the stored data to obtain a better strategy for selecting actions, until the training is completed and switches to the control mode; in the control mode, the intelligent agent selects actions according to fixed network parameters to achieve feedback control of the klystron state in the real engine.

[0024] The real engine is a real accelerator, which includes a low-level control system and a klystron.

[0025] The virtual engine is an offline fitting model of klystron input and output. The fitting model is configured to predict the klystron state in the real accelerator under the corresponding setting values ​​after receiving the setting values ​​of the amplitude and phase of the low-level control system input by the environment.

[0026] The fitting model adopts a multi-layer perceptron network, which contains at least one hidden layer and consists of a fully connected layer, and adopts a ReLU function as an activation function.

[0027] In step S2, a fitting model of the klystron input and output is obtained as a virtual engine, which specifically includes:

[0028] S21: Construct a fitting model of the klystron input and output;

[0029] S22: setting and recording multiple sets of different amplitude and phase setting values ​​of the low-level control system in a real accelerator, and recording the amplitude and phase of the klystron output under the setting values, so as to acquire a training data set for the fitting model;

[0030] S23: Using the training data set of the fitting model to train the fitting model of the klystron input and output until the prediction accuracy can meet the requirement.

[0031] The step S22 specifically includes: firstly selecting an ideal state of the klystron, recording the setting values ​​of the amplitude and phase of the low-level control system of the klystron under the ideal state, and then determining the adjustment range of the setting values ​​of the amplitude and phase of the low-level control system with the setting value as the center value; then, within the adjustment range, setting and recording a plurality of different sets of setting values ​​of the amplitude and phase of the low-level control system, and recording the amplitude and phase output by the klystron under the setting value, so as to acquire a training data set of the fitting model.

[0032] Before the step S23, the method further includes step S230: preprocessing the training data set of the fitting model, specifically including: averaging the amplitudes and phases of multiple klystron outputs under the same amplitude and phase setting values ​​of the low-level control system as the amplitude and phase of the klystron output under the setting values; and / or filtering the amplitude of the klystron output in the training data set of the fitting model, so that the filtered data is used as the training data set used in training; and / or linearly interpolating the data in the training data set, so that the data set obtained after interpolation is used as the training data set used in training; and / or linearly normalizing the training data set, so that all input and output parameters of the training data set are mapped to the [0,1] interval through linear transformation; and / or

[0033] The step S22 also includes: under the selected sets of amplitude and phase setting values ​​of the low-level control system, collecting the amplitude of the klystron output multiple times in chronological order to obtain a time-klystron output amplitude image, and extracting a sinusoidal noise function therefrom; in the step S23, after the training is completed, adding the sinusoidal noise function to the amplitude of the trained fitting model output.

[0034] The present invention combines the latest artificial intelligence technology and applies reinforcement learning to the feedback control of the low-level control system (LLRF), so that the low-level control system can explore the strategy of controlling two interrelated variables at the same time, solves the problem that the existing technology cannot control two coupled variables well, and improves the control accuracy;

[0035] The reinforcement learning method used in the present invention can automatically learn the optimal control strategy, without spending a lot of time to find suitable parameters like the prior art, thus reducing the burden on debugging personnel;

[0036] In addition, the present invention uses a fitting model to predict the real output of the klystron, and trains the reinforcement learning model based on the fitting model, which greatly reduces the training time, reduces the normal use and debugging time of the accelerator, and improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the klystron state feedback control method based on reinforcement learning of the present invention;

[0038] Figure 2A and Figure 2B It is an algorithm structure diagram of the fitting model adopted by the klystron state feedback control method based on reinforcement learning of the present invention;

[0039] Figure 3 A training flow chart of an algorithm used for reinforcement learning in the reinforcement learning-based klystron state feedback control method of the present invention;

[0040] Figure 4 The variation of the klystron output amplitude over time under different control modes;

[0041] Figure 5 The variation of the klystron output phase over time under different control modes. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with specific examples. It should be understood that the following examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0043] The klystron state feedback control method based on reinforcement learning of the present invention is mainly based on the following principles:

[0044] Machine learning is an artificial intelligence technology that enables computers to learn from data and make predictions or decisions based on these learnings. Among them, reinforcement learning, as one of the three major modes of machine learning, is mainly composed of agents, environments, states, actions, and rewards. In reinforcement learning, the agent explores the actions on its own, and then continuously adjusts the strategy of selecting actions based on the state and rewards of environmental feedback to make the final reward as large as possible. In the present invention, the PPO (Proximal Policy Optimization) reinforcement learning algorithm is specifically adopted.

[0045] In the klystron state feedback control method based on reinforcement learning of the present invention, the agent in the reinforcement learning model needs to learn by itself the strategy of simultaneously controlling the two mutually coupled variables of the klystron output amplitude and phase according to the state of the current environment. Therefore, the specific meanings of the parameters such as environment, state, action, and reward in the present invention are as follows:

[0046] Current Status s : The difference between the current state of the klystron and the ideal state;

[0047] action a : The difference between the amplitude and phase setting values ​​of the next low-level control system output and the amplitude and phase setting values ​​of the current low-level control system;

[0048] award r : The corresponding reward calculated by the environment based on the current state of the klystron;

[0049] Next status s_ : The new state of the klystron obtained according to the setting values ​​of the amplitude and phase output by the next low-level control system, and the difference between the new state of the klystron and the ideal state.

[0050] Among them, the ideal state of the klystron is the amplitude and phase values ​​that the klystron should output, which is the ultimate goal of feedback control and is determined by the accelerator operator. For example, normal operation requires the amplitude and phase of the klystron output to be 12950 and -63.8°. The setting values ​​of the amplitude and phase output of the low-level control system refer to the amplitude and phase setting values ​​of the low-level control system selected by the intelligent agent in order to achieve the ultimate goal. If there is no interference, the amplitude of the excitation signal setting value selected by the low-level control system multiplied by a certain coefficient is the amplitude of the klystron output, and the phase setting value plus a certain number is the phase of the klystron output. However, due to the nonlinearity of the input and output characteristics of the klystron in amplitude and phase, and the RF reference leakage and imbalance of the low-level control system during IQ modulation, the amplitude and phase of the actual output of the klystron cannot be accurately known by the setting values ​​of the amplitude and phase output of the low-level control system. If there is no feedback control algorithm, the setting values ​​of the low-level control system can only be adjusted manually through field experiments to make the klystron reach the ideal output value.

[0051] In the present invention, the agent is based on the current state of the environment s (i.e. the difference between the current state of the klystron and the ideal state) Select action a (The action is the difference between the setting value of the next low-level control system and the setting value of the current low-level control system). The action is then input into the environment, and the environment converts the difference into the setting value of the next low-level control system, and inputs it into the engine to obtain the new state of the klystron. After obtaining the new state of the klystron, the difference between the new state of the klystron and the ideal state is calculated as the next state of the environment. s_ , and the reward corresponding to the current state of the klystron r , returned to the agent, the agent based on the current state of these environments s , next state s_ ,awardr ,action a Train it to choose the action a More reasonable.

[0052] Therefore, in actual control, as reinforcement learning proceeds, by adjusting the amplitude and phase setting values ​​of the selected low-level control system, the actual amplitude and phase of the klystron output will stabilize near the ideal state set by humans. For example, the ideal state includes: the amplitude and phase of the klystron output are 12950 and -63.8° respectively, then the reinforcement learning algorithm will continuously adjust the setting values ​​of the amplitude and phase of the low-level control system output to make the amplitude and phase of the actual output of the klystron sufficiently stable. Generally speaking, considering the existence of noise, our adjustment goal is to make the amplitude of the klystron output always around 12950 and the stability as good as possible. Similarly, our adjustment goal for the phase is to make the phase of the klystron output always around -63.8° and the stability as good as possible. The stability calculation method is to calculate the root mean square error (RMSE) of the klystron output amplitude and phase, and its expression is:

[0053] ;

[0054] in n is the number of samples, is the actual output of the klystron, The ideal output of the klystron.

[0055] Based on the above principle, the klystron state feedback control method based on reinforcement learning of the present invention specifically includes the following steps:

[0056] Step S1: Establish a reinforcement learning model; the reinforcement learning model defines a state based on at least the state of the klystron s , defining actions based at least on the amplitude and phase settings of the low-level control system a , and define a reward based at least on how far the current state of the klystron deviates from the ideal state of the klystron and how much the deviation changes after taking the action r ;

[0057] The state of the klystron includes the amplitude and phase of the klystron output, and the state of the klystron is measured by a low-level control system.

[0058] In the present invention, the FELiChEM accelerator device and its low-level control system are described to obtain Figure 1 The reinforcement learning-based klystron state feedback control method shown includes training and control processes.

[0059] The step S1 is implemented by the main program, wherein the step S1 may include initialization of parameters such as the selection of the ideal state of the klystron (i.e. the amplitude and phase value that the klystron is expected to output), the judgment condition for the completion of a round of training (i.e. the interval in which the current state of the klystron can be considered as completing the task), and the maximum number of steps in a round.

[0060] In step S1, in order to enable the agent to learn the general adjustment strategy under different low-level control system setting values ​​during training, a set of low-level control system amplitude and phase setting values ​​are randomly selected within a certain range in the fittable interval as initial setting values ​​when the environment is initialized. For example, at the beginning, let the program randomly select a number between 2.8-3.2 as the amplitude setting value of the low-level control system, and select a number between 22-28 as the phase setting value of the low-level control system.

[0061] The main program is used to implement the initialization of the agent, the initialization of the environment, the interaction between the agent and the environment, and the training, verification and feedback control of the agent. When the main program is running, the method of the present invention can be run in the training or control mode by specifying parameters. The rest of the implementation is in the environment (the environment will call the engine to return the status to the main program, as well as the reward corresponding to the current status and other information).

[0062] Among them, action a The state is defined based on at least the set values ​​of the amplitude and phase of the low-level control system output. s At least defined based on the state of the klystron.

[0063] Considering that the states of the accelerator klystron and the low-level control system in the real environment change slightly every day, the setting values ​​of the same low-level control system may result in slightly different klystron output values ​​on different days. At this time, if the action learned by the agent is the setting values ​​of the amplitude and phase output of the low-level control system, the model obtained by offline training on the first day may not be able to control the accelerator on the second day.

[0064] Therefore, in order to make the trained model more universal, in this embodiment, the action a The difference between the amplitude and phase setting values ​​of the next low-level control system output and the amplitude and phase setting values ​​of the current low-level control system (i.e., the change in the amplitude and phase setting values). s is the difference between the state of the klystron and the ideal state. aConverted into the required amplitude and phase setting values ​​of the next low-level control system output, input into the engine, so that the phase of the excitation signal amplitude output by the low-level control system changes, and then the amplitude and phase of the klystron output are closer to the ideal state. The conversion method is: the current amplitude and phase setting values ​​of the low-level control system (LLRF) + action value = the new amplitude and phase setting values ​​to be set for the low-level control system (LLRF).

[0065] Step S2: setting an engine in the environment so that the engine receives the amplitude and phase setting values ​​of the low-level control system from the environment and sends the state of the klystron to the environment, wherein the engine includes a virtual engine for training the reinforcement learning model and a real engine for feedback control, wherein the real engine includes the low-level control system and the klystron in the actual accelerator;

[0066] Step S3: training the reinforcement learning model, and then using the trained reinforcement learning model for feedback control; switching to the virtual engine in the training mode; switching to the real engine in the control mode, so as to realize feedback control of the klystron state in the actual accelerator through a low-level control system based on the reinforcement learning model as a feedback control algorithm.

[0067] Step S3 specifically includes: in the training mode, the agent selects actions according to the returned status and rewards and continuously adjusts its own network parameters until the training is completed and switches to the control mode; in the control mode, the agent selects actions according to fixed network parameters to achieve feedback control of the klystron state in the real engine.

[0068] That is to say, the working method of the agent in control mode is basically the same as that in training mode, the only difference is that the network parameters of the agent will not change, and the engine used is replaced by a real accelerator.

[0069] The engine can be abstracted as a function, with the input value being the setting value of the low-level control system and the return value being the current amplitude and phase of the klystron. Thus, the environment can transform a set of actions selected by the received agent into the amplitude and phase setting values ​​of the low-level control system and input them into the engine, and then process the klystron state obtained by the engine, convert it into the next state and the corresponding reward and return it to the agent for learning or judgment.

[0070] The real engine is a real accelerator, which includes a low-level control system (LLRF) and a klystron. In this embodiment, the accelerator is a FELiChEM accelerator device. The control of the low-level system in the FELiChEM device is based on the EPICS (Experimental Physics and Industrial Control System) distributed architecture, and the PV (Process Value) value in the IOC (Input Output Controller) inside the low-level control system can be obtained or changed through the CA (Channel Access) protocol (therefore, the amplitude phase setting value of the low-level control system is PV, and the state of the klystron is also PV), thereby realizing the reading and control of the device state. The klystron state obtained by the CA protocol is measured by the low-level control system, and what is changed is the setting value of the low-level control system.

[0071] Generally speaking, it takes a very large number of steps (hundreds of thousands of steps) to train an agent to converge. Assuming the repetition frequency of the accelerator is 1 Hz, the time between two adjacent steps is at least 1 second. At this time, if the online training method is used, it will take up a lot of accelerator machine time (several days), affecting normal debugging and experiments.

[0072] In the present invention, the virtual engine is an offline fitting model of the klystron input and output. The fitting model is configured to predict the klystron state (i.e., the amplitude value and phase value of the klystron output) in the real accelerator under the corresponding setting values ​​after receiving the setting values ​​of the amplitude and phase of the low-level control system input by the environment, so as to return the fitted amplitude and phase of the klystron output to the environment.

[0073] Therefore, the present invention first constructs a fitting model based on the relationship between the setting values ​​of the amplitude and phase of the low-level control system and the actual output amplitude and phase values ​​of the klystron, so that the fitting model can simulate the input and output of the real klystron, and then interacts with the fitting model through the reinforcement learning model to learn the control strategy for the real environment and realize the offline training of the intelligent agent. This allows us to not train the reinforcement learning model in a real accelerator environment (because the repetition frequency of the real accelerator FELiChEM device we use is 1Hz, and the state of the klystron can be obtained once in 1s, but the algorithm runs very fast. If online training takes several days, it will seriously affect the normal use and debugging of the accelerator. However, if the fitting model is used for offline training, the time used is greatly shortened, and the training can be completed in tens of minutes).

[0074] In the present invention, the fitting model of the klystron input and output is obtained by the following method: setting and recording multiple sets of different low-level control system amplitude and phase setting values ​​in a real accelerator, and recording the amplitude and phase of the klystron output under the setting values, as a training data set of the fitting model to train the fitting model, thereby learning the relationship between the amplitude and phase setting values ​​of the low-level control system and the actual amplitude and phase values ​​of the klystron output, and obtaining the fitting model of the klystron input and output. In addition, not all the collected data are used in the training process, only 80% or less are used for training, and the remaining unused data are used for verification; or all the data are used to detect the similarity between the input and output of the model and the original curve. The standard of this similarity is not unique, and it only needs to meet the training of the reinforcement learning model.

[0075] exist Figure 1 The accelerators on the left and right sides of the diagram are the same accelerator, except that the accelerator on the right is used to collect training data sets for fitting the model, while the accelerator on the left is the real engine of the reinforcement learning algorithm.

[0076] In step S2, a fitting model of the klystron input and output is obtained as a virtual engine, which specifically includes:

[0077] Step S21: constructing a fitting model of the klystron input and output;

[0078] In this embodiment, a multi-layer perceptron (MLP) network is used to construct the fitting model.

[0079] In order to obtain the fitting model, the multilayer perceptron network is a neural network composed of fully connected layers with at least one hidden layer. The overall structure diagram is as follows Figure 2A As shown. The neural network consists of several Figure 2B The neuron composition shown is x 1 ~ x n is the output of each neuron in the previous layer, w 1 ~ w n is the weight that needs to be obtained through training, and then the input is weighted summed, and the result is sent to an activation function to get the output of the neuron y .

[0080] The purpose of introducing the activation function is to introduce a nonlinear transformation among multiple affine transformations so that the subsequent hidden layers can process more advanced information. If there is no activation function, adding a hidden layer is just equivalent to the superposition of multiple affine transformations, and the result is still an affine transformation, that is, adding more hidden layers is only equivalent to one hidden layer.

[0081] In this embodiment, the ReLU function is used as the activation function, and its expression is:

[0082] ;

[0083] in, x is the input of the activation function, for x and 0.

[0084] Step S22: setting and recording multiple sets of different amplitude and phase setting values ​​of the low-level control system in the real accelerator, and recording the amplitude and phase of the klystron output under the setting values, so as to acquire a training data set for the fitting model.

[0085] Step S22 specifically includes: firstly selecting an ideal state of the klystron, recording the setting values ​​of the amplitude and phase of the low-level control system under the ideal state of the klystron, and then determining the adjustment range of the setting values ​​of the amplitude and phase of the low-level control system with the setting value as the center value; then, within the adjustment range, setting and recording a plurality of different sets of setting values ​​of the amplitude and phase of the low-level control system, and recording the amplitude and phase output by the klystron under the setting value, so as to acquire a training data set of the fitting model.

[0086] The ideal state of the klystron is the amplitude and phase values ​​that the klystron should output, which are set manually and are generally determined by the accelerator operator. For example, normal operation requires the amplitude and phase of the klystron output to be 12950 and -63.8° respectively.

[0087] The adjustment range of the amplitude and phase setting values ​​of the low-level control system is set so that the adjustment range of the amplitude and phase output of the klystron can cover the long-term drift of the klystron. Thus, the training data set of each accelerator is established according to the actual situation of the accelerator;

[0088] By analyzing the long-term operation data of the FELiChEM accelerator, it is found that the klystron has an amplitude variation ranging from 12850 to 13050 during the normal operation of the accelerator. After experiments, it is found that when the amplitude setting value of the low-level control system changes from 3.1 to 2.9, the klystron output amplitude can be changed to around 12950; for example, when the klystron output amplitude changes from 12950 to 12850, the low-level control system amplitude setting value only needs to be changed from 3 to 3.1 to change the klystron output amplitude from 12850 to around 12950; and when the klystron output changes from 12950 to 13050, the low-level control system amplitude setting value only needs to be changed from 3 to 2.9 to change the klystron output from 13050 to around 12950. Similarly, during the normal operation of the accelerator, the klystron has a phase change in the range of -65.8° to -61.8°. At this time, the phase setting value of the low-level control system is changed in the range of 27° to 23° to change the output phase of the klystron to near -63.8°. Taking into account the exploration needs of the intelligent agent, the present invention appropriately increases the fitting range of the fitting model, so it is determined that when collecting data for training the fitting model, the adjustment range of the amplitude setting value of the low-level control system is 2.5 to 3.5, and the adjustment range of the phase setting value is 20° to 30°, wherein the amplitude change step is 0.01, and the phase change step is 0.1; in addition, since the noise in the klystron output can be approximated as a sine function, and the period is about 3s, the klystron output corresponding to each set of setting values ​​is sampled 3 times (since the accelerator repetition frequency is 1Hz, the sampling interval is 1s, and sampling 3 times is just one period of the noise), so that the number of sample groups of the training data set is 100×100×3=30000.

[0089] Step S23: using the training data set of the fitting model to train the fitting model of the klystron input and output until the prediction accuracy can meet the requirements of the reinforcement learning model training.

[0090] The training process of the multi-layer perceptron (MLP) network is as follows: first, the training data set is input into it, the output is obtained through forward propagation, and then the mean square error (MSE) between the current output value and the predicted value of the multi-layer perceptron network is calculated by the following formula;

[0091] ;

[0092] in n is the number of samples, is the true value, The mean square error is used as the loss function of the multi-layer perceptron network to predict the value. Finally, the connection weights of each neuron are updated through the BP (Back Propagation) algorithm and the Adam (Adaptive Moment Estimation) algorithm.

[0093] Before the step S23, the method further includes a step S230: preprocessing the training data set of the fitting model to improve the accuracy of the fitting model of the klystron input and output.

[0094] In step S22, the amplitude and phase of the klystron output are collected multiple times, for example, three times, under each set of amplitude and phase setting values ​​of the low-level control system. In step S230, the training data set of the fitting model is preprocessed, including: averaging the amplitudes and phases of multiple klystron outputs under the same amplitude and phase setting values ​​of the low-level control system as the amplitude and phase of the klystron output under the setting values, so as to reduce the influence of noise on the training data set to a certain extent.

[0095] In addition, since the amplitude noise of the klystron output is large, it will seriously affect the fitting of the model, and after observing the output of the klystron for a period of time, it can be seen that the amplitude and phase noise of its output are similar to a sine function. Therefore, in step S230, the training data set of the fitting model is preprocessed to eliminate the influence of noise on the fitting, including: filtering the amplitude of the klystron output in the training data set of the fitting model, so that the filtered data is used as the training data set used in training.

[0096] The filtering may be a moving average filter. Moving average filtering is a commonly used filtering method, which is performed by sliding a window with a fixed length of a set value on a given numerical sequence and calculating the average value of the samples in the window to filter, which can effectively smooth the numerical sequence. After testing, when the size of the window of the moving mean filter is 20, a relatively smooth curve can be obtained, the independent variable of the curve is a set of low-level control system setting values ​​(for example, the first point of the independent variable is the low-level control system setting value 2.5, 20°, and the second point is the low-level control system setting value 2.5, 20.1°), and the dependent variable is the amplitude or phase of the klystron output (the low-level control system setting value-the amplitude of the klystron output is a graph, and the low-level control system setting value-the phase of the klystron output is another graph). At this time, the curve of the low-level control system setting value-the amplitude of the klystron output after filtering is in the middle position of the curve of the low-level control system setting value-the amplitude of the klystron output before filtering, so that the fitting curve can be consistent with the original curve obtained after adding the calculated sine function noise.

[0097] In addition, since the step size of each change when we collect data is small enough, the change of the klystron between two adjacent low-level control system setting values ​​can be approximated as a linear change. In order to make the model fitting more accurate, in step S230, the training data set of the fitting model is preprocessed, including: linear interpolation of the data in the training data set, so that the data set obtained after interpolation is used as the training data set used in training. The prediction accuracy requirements of the fitting model are determined according to the training situation of the reinforcement learning model. Generally speaking, the curve fitted by the fitting model is close to the original curve, relatively smooth, and can clearly reflect the trend of the original curve.

[0098] In addition, the step S22 also includes: under the selected sets of amplitude and phase setting values ​​of the low-level control system, collecting the amplitude of the klystron output multiple times in time sequence to obtain a time-image of the amplitude of the klystron output, and extracting a sinusoidal noise function therefrom; in the step S23, after the training is completed, adding a sinusoidal noise function to the amplitude of the trained fitting model output to simulate a klystron output close to the actual situation.

[0099] In this embodiment, the noise function of the amplitude output by the klystron is: 40sin(π / 3.1·times), where times is the ordinal number of the acquisition point.

[0100] Since the training process of the multilayer perceptron model is a process of continuously adjusting the connection weights, if the gap between the input and output data is large, the learned weights will be too large, which will cause the algorithm to converge more slowly or even fail to converge. Therefore, in the step S230, the training data set for the fitting model is preprocessed, and also includes: linear normalization of the training data set so that all input and output parameters of the training data set are mapped to the [0,1] interval through linear transformation.

[0101] The conversion formulas for input and output parameters in the training dataset are as follows:

[0102] ;

[0103] in x is the original data, x min and x max are the minimum and maximum values ​​in the original data, respectively. x* is the normalized data.

[0104] In the present invention, the agent in the reinforcement learning model is constructed based on the reinforcement learning algorithm, and the reinforcement learning algorithm is PPO (Proximal Policy Optimization). PPO is an on-policy algorithm, and its training goal is to maximize the final reward. The training process of the PPO reinforcement learning algorithm is as follows Figure 3 As shown. The PPO reinforcement learning algorithm consists of two networks: a policy network and a value network. Both networks are built based on a multi-layer perceptron (MLP). The input of the policy network is the current state s of the reinforcement learning model, and the output is the action that the agent wants to output. a The mean μ and variance σ , thus constructing a Gaussian distribution, namely the strategy distribution; the input of the value network is the current state s of the environment, and the output is the value of the current state of the environment V , in the same current state s Next, if the state value obtained is V The higher it is, the better the strategy.

[0105] The training process of the PPO reinforcement learning algorithm model is as follows: first, prepare an empty experience pool, and then set the current environment state s Input into the old policy network to obtain the policy distribution, and perform probability sampling based on the policy distribution to obtain an action a ,Will a Input into the environment to get rewarded r and the next status s_ , and then store these combinations [( s,a,r,s_ ),……], and then s_ Input into the old strategy network and repeat the above process until the experience pool is full.

[0106] Based on the data stored in the experience pool and the current value network, the generalized advantage estimate can be used to calculate the estimated value of the advantage function of each state. , whose expression is:

[0107] ;

[0108] in is the discount factor, which is a number less than or equal to 1. In the present invention, it is 0.95. is the current state, Indicates the state at the end of the round. V () is the state value function, and T represents the total number of time steps in a round.

[0109] Then copy the old policy network to get the new policy network, and then take out the stored ( s,a,r,s_ ) to calculate the policy network loss and the value network loss, and use this as a basis to update the policy network and the value network. The new policy network is updated several times with the same experience pool data in one update (taking the model of this application as an example, it is updated 10 times), and then the parameters of the new policy network are copied to the old policy network.

[0110] Assume that at some point t The set of training data taken out is ( s t ,a t ,r t ,s t _ ), then the current state s t Input them into the new strategy network and the old strategy network respectively, and obtain the new and old strategy distributions corresponding to the state respectively, and combine these two strategy distributions with the corresponding actions a t The probability ratio of the new and old strategy distributions can be obtained , the probability ratio of the new and old strategy distributions Crop and control its range to 1- ε With 1+ ε Between (among which ε is a hyperparameter, usually set to 0.1 or 0.2, in order to prevent the update step of the strategy from being too large, which helps to avoid a sharp drop in strategy performance. In this invention, it is set to 0.2. The clipped probability ratio is recorded as . Take the probability ratio of the new and old strategy distributions And the probability ratio after clipping With the current status s t The corresponding estimate of the advantage function The smaller value of the multiplication is used as the loss function of the current policy network, and then back-propagated to update the new policy network. t Status at all times s t The estimated value of the advantage function The state can be calculated s t Theoretical state value under V target ( s t ), the theoretical state value V target ( s t ) and the actual output of the current value network V ( st ) is used as the loss of the value network, and the value network is updated through the back propagation algorithm and the Adam algorithm to make the state value obtained by the value prediction more accurate.

[0111] Among them, the theoretical state value V target ( s t ) is derived based on the temporal difference algorithm, and its expression is:

[0112] .

[0113] In all training data ( s,a,r,s_ ) are used to update the new policy network and value network, and the training data in the same experience pool are repeatedly used to update the new policy network and value network multiple times; after a certain number of repetitions (for example, the present invention uses the same experience pool to repeat the update 10 times), the weights of the new policy network are used to update the weights of the old policy network. Finally, the experience pool is cleared and data collection is performed again until the experience pool is full. Repeat the above training steps until the policy network and value network converge.

[0114] When switching to the virtual engine in training mode to implement the offline training procedure of reinforcement learning, it is known through offline experiments that each round requires about 50 steps to adjust the output of the klystron to within the allowable error near the set value. In order to make the intelligent agent know the optimization direction clearly, the following reward function is designed. a The rewards obtained by inputting into the environment include single-step rewards r t :

[0115] ;

[0116] in, r t It is a single-step reward. r diff is a penalty item, r action is the action reward, α It is a number less than 0, indicating the weight of the penalty item.

[0117] That is, the single-step reward r t is the penalty term and action reward r action The weighted combination of the penalty term α It is adjusted according to the training conditions, and in the present invention, it is set to -0.5.

[0118] Among them, the penalty rdiff The value is obtained based on the deviation of the current state of the klystron from the ideal state of the klystron, which is used to punish the agent in the first t The state of the klystron O t (i.e. the amplitude and phase of the output) deviates from the ideal state S The state of the klystron O t and ideal state S are both vectors of length 2, with the first element representing the magnitude and the second element representing the phase. r diff The specific definition of is:

[0119] ;

[0120] in, For the t The degree of deviation between the state of the step-by-step klystron and the ideal state, , [ i ] represents the first i elements, , Respectively represent the state difference The first and second place, k It is a scaling factor used to unify the scales of the amplitude difference and the phase difference, and is set to 0.02 in the present invention.

[0121] Rewards It is generated based on the actions taken by the agent, specifically based on taking actions a The change in the new state from the ideal value compared to the previous state After each action taken by the agent, the reward Specifically expressed as:

[0122] ;

[0123] in, , For the t The degree of deviation between the state of the step-by-step klystron and the ideal state, For the t -1 step The degree of deviation between the state of the klystron and the ideal state, [ i ] represents the first i elements, m It represents the reward obtained if the action taken makes the klystron state closer to the ideal state compared with the previous step, n is a number less than 0, indicating the penalty if the action taken makes the klystron state further away from the ideal state than the previous step.I It is an indicator function. Its value is 1 when the value inside the brackets is true, otherwise it is 0.

[0124] In order to reduce the ineffective exploration of the intelligent agent and speed up the training process, we set m is 0.25, n is -1.

[0125] In addition to the single-step reward, the reward also includes a task completion reward introduced when the task is completed. If the agent makes the klystron output reach an acceptable error range near the ideal value within 50 steps, it will receive a task completion reward of 50. In order to encourage the agent to adjust as quickly as possible under different circumstances, the expression of the additional reward is 0.5×(50- s ),in, s The number of steps used in this round is used to reward the situation where the task is completed in less than 50 steps.

[0126] like Figure 4 and Figure 5 As shown, after experiments, when the klystron state feedback control method based on reinforcement learning of the present invention is used for feedback control of the klystron state, the stability of the klystron output amplitude is improved by 3.37% compared with the traditional PI controller, and the stability of the klystron output phase is improved by 27.78% compared with the traditional PI controller.

[0127] In summary, the present invention combines the latest artificial intelligence technology and applies reinforcement learning to the feedback control of the low-level control system, so that the low-level control system can explore the strategy of controlling two interrelated variables at the same time by itself, solves the problem that the prior art cannot control two coupled variables well, and improves the control accuracy;

[0128] The reinforcement learning method used in the present invention can automatically learn the optimal control strategy, without spending a lot of time to find the optimal parameters like the prior art, thus reducing the burden on the debugging personnel;

[0129] In addition, the present invention uses a fitting model to predict the real output of the klystron, and trains the reinforcement learning model based on the fitting model, which greatly reduces the training time, reduces the normal use and debugging time of the accelerator, and improves efficiency.

[0130] The above specific embodiments further illustrate the purpose, technical solutions and equipment of the present invention. It should be understood that the above description is only a specific embodiment and is not limited to the present invention. Any modification, equivalent replacement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A klystron state feedback control method based on reinforcement learning, characterized in that: include: S1: Build a reinforcement learning model with an agent and an environment; The reinforcement learning model defines a state based on at least a state of the klystron, defines an action based on at least a setting value of the amplitude and phase of the low-level control system; defines a reward in the environment based on at least a deviation of the current state of the klystron from an ideal state of the klystron and a change in the deviation after taking an action; S2: setting an engine in the environment so that the engine receives the amplitude and phase setting values ​​of the low-level control system from the environment and sends the corresponding klystron state to the environment, the engine includes a virtual engine and a real engine, and the real engine includes the low-level control system and the klystron in the actual accelerator; S3: training the reinforcement learning model and then using the trained model for feedback control; switching to the virtual engine in the training mode and switching to the real engine in the control mode to implement feedback control of the klystron state in the actual accelerator through a low-level control system based on the reinforcement learning model as a feedback control algorithm; The virtual engine is an offline fitting model of the klystron input and output. The fitting model is set to predict the klystron state in the real accelerator under the corresponding setting values ​​after receiving the setting values ​​of amplitude and phase; In S2, a fitting model of the klystron input and output is obtained as a virtual engine, including: S21: Construct a fitting model of the klystron input and output; S22: setting and recording multiple sets of different amplitude and phase setting values ​​in a real accelerator, and recording the amplitude and phase output by the klystron under the setting values, so as to acquire a training data set for the fitting model; S23: Using the training data set of the fitting model, the fitting model of the klystron input and output is trained until the prediction accuracy can meet the requirement.

2. The klystron state feedback control method based on reinforcement learning according to claim 1, characterized in that: The action is the difference between the amplitude and phase setting values ​​of the next low-level control system output and the amplitude and phase setting values ​​of the current low-level control system; the state is the difference between the state of the klystron and the ideal state; The reward is calculated by the reward function, including the single-step reward r t As well as mission completion rewards introduced when missions are completed.

3. The klystron state feedback control method based on reinforcement learning according to claim 2, characterized in that: The single-step reward r t for: ； in, r t It is a single-step reward. r diff is a penalty item, r action is the action reward, α is a number less than 0, indicating the weight of the penalty item; Penalty r diff for: ； in, is the degree of deviation between the state of the klystron and the ideal state at step t, , [ i ] represents the first i elements, so , Respectively represent the state difference The first and second elements in k is the scaling factor; Rewards for: ； in, Indicates the change in the degree of deviation after taking action. , For the t The degree of deviation between the state of the step-by-step klystron and the ideal state, For the t -1 step The degree of deviation between the state of the klystron and the ideal state, [ i ] represents the first i elements, m represents the reward obtained if the action taken makes the klystron state closer to the ideal state compared with the previous step, n is a number less than 0, indicating the penalty if the action taken makes the klystron state further away from the ideal state than the previous step. I is the indicator function.

4. The klystron state feedback control method based on reinforcement learning according to claim 1, characterized in that: The S3 specifically includes: in the training mode, the intelligent agent selects actions according to the state returned by the environment and continuously adjusts its own network parameters according to the stored data to obtain a better strategy for selecting actions, until the training is completed and switches to the control mode; in the control mode, the intelligent agent selects actions according to fixed network parameters to achieve feedback control of the klystron state in the real engine.

5. The klystron state feedback control method based on reinforcement learning according to claim 1, characterized in that: The real engine is a real accelerator, which includes a low-level control system and a klystron.

6. The klystron state feedback control method based on reinforcement learning according to claim 1, characterized in that: The fitting model adopts a multi-layer perceptron network, which contains at least one hidden layer and consists of a fully connected layer, and adopts a ReLU function as an activation function.

7. The klystron state feedback control method based on reinforcement learning according to claim 1, characterized in that: S22 specifically includes: firstly selecting an ideal state of the klystron, recording the setting values ​​of the amplitude and phase of the low-level control system under the ideal state of the klystron, and then determining the adjustment range of the setting values ​​of the amplitude and phase of the low-level control system with the setting value as the center value; then, within the adjustment range, setting and recording a plurality of different sets of setting values ​​of the amplitude and phase of the low-level control system, and recording the amplitude and phase output by the klystron under the setting value, so as to acquire a training data set for the fitting model.

8. The klystron state feedback control method based on reinforcement learning according to claim 1, characterized in that: Before S23, the method further includes S230: preprocessing the training data set of the fitting model, specifically including: averaging the amplitudes and phases of multiple klystron outputs under the same amplitude and phase setting values ​​of the low-level control system as the amplitude and phase of the klystron output under the setting values; and / or filtering the amplitude of the klystron output in the training data set of the fitting model, so that the filtered data is used as the training data set used in training; and / or linearly interpolating the data in the training data set, so that the data set obtained after interpolation is used as the training data set used in training; and / or linearly normalizing the training data set, so that all input and output parameters of the training data set are mapped to the [0,1] interval through linear transformation; and / or The S22 also includes: under the selected sets of amplitude and phase setting values ​​of the low-level control system, collecting the amplitudes of the klystron output multiple times in chronological order to obtain a time-klystron output amplitude image, and extracting a sinusoidal noise function therefrom; in the S23, after the training is completed, adding the sinusoidal noise function to the amplitude output by the trained fitting model.

Images