Beam pointing stabilization method and system based on simulation environment driven reinforcement learning

Abstract

The present invention relates to the field of optics and control technology, and discloses a light beam pointing stabilization method and system based on simulated environment driven reinforcement learning. The method divides an integrated light path into a plurality of sub-light path links according to the series relationship between upper and lower levels; records the mirror deflection angle of a fast reflection mirror group and the light beam pointing drift data before and after the swing to form a sample data set; constructs a simulation environment model based on a convolutional neural network based on the sample data set; constructs a light beam pointing correction intelligent agent using the simulation environment model as a virtual interactive environment, and uses a reinforcement learning framework for training; applies the light beam pointing correction intelligent agent to an actual light path in a real environment, and continues to perform reinforcement learning training through the actual light path in the real environment; after the training is completed, uses the light beam pointing correction intelligent agent to control the fast reflection mirror group of a light beam pointing adjustment device in the real environment; the present invention can cope with the light beam drift problem of a complex series coupled light path system in a multi-type interference environment.

Description

Technical Field

[0001] The present invention relates to the field of optics and control technology, and in particular to a light beam pointing stabilization method and system based on simulated environment driven reinforcement learning. Background Art

[0002] In laser systems, beam pointing often drifts due to various interferences, affecting laser processing and manufacturing accuracy, positioning accuracy, and laser communication efficiency. Typical interferences in beam pointing applications include: turbulent interference caused by air flow disturbances, inherent drift caused by temperature distribution and thermal deformation inside the laser cavity, changes in laser beam transmission media, elastic deformation of the optical system assembly and adjustment mechanical structure, and vibration of the pump body of auxiliary equipment (cooling pumps, molecular pumps, etc.); beam pointing misalignment mainly includes position misalignment, angle misalignment, position and angle coupling misalignment, etc. In order to reduce the adverse effects of beam pointing drift, a beam pointing control system must be added to the optical path to correct the beam pointing.

[0003] Today's beam pointing control schemes are all adjusted for a single optical path. Most of them use fast reflectors as their core components, and use the deflection of the fast reflector to adjust the beam position and pointing. The control strategy is generally deterministic mathematical methods such as PID and active disturbance rejection control. However, taking the extreme ultraviolet laser generation device as an example, there is a possibility of beam pointing drift in multiple sub-optical path links within large-scale laser equipment. Independent beam pointing adjustment devices need to be placed in multiple sub-optical path links to ensure that the beam pointing of the sub-optical path link does not produce serious deviation. The interference factors of each link are very different. The sub-optical path links are often coupled in series between the upper and lower levels. External interference varies greatly in different sub-optical path links under inconsistent physical characteristics. The traditional single beam pointing control strategy is mainly aimed at applications with a single optical path and a constant interference source, and is often unable to cope with the beam drift problem of complex series coupled optical path systems under multiple types of interference environments. Summary of the Invention

[0004] The purpose of the present invention is to solve the above problems and design a beam pointing stabilization method and system based on simulated environment driven reinforcement learning.

[0005] A first aspect of the present invention provides a method for stabilizing light beam pointing based on simulated environment driven reinforcement learning, the method comprising the following steps:

[0006] S1. The integrated optical circuit is divided into several sub-optical circuit links according to the upper and lower level series relationship, wherein each sub-optical circuit link includes an independent light beam pointing adjustment device;

[0007] S2, driving the beam pointing adjustment device and adjusting the mirror deflection angle of the fast reflection mirror group, recording the mirror deflection angle of the fast reflection mirror group and the beam pointing drift data before and after the deflection, to form a sample data set;

[0008] S3. Constructing a simulation environment model based on a convolutional neural network based on the sample data set;

[0009] S4. Using the simulated environment model as a virtual interactive environment, a beam pointing correction agent is constructed. The reward calculator, state register, action accumulator, and tensor splicer are combined and trained using a reinforcement learning framework.

[0010] S5. Apply the beam pointing correction agent obtained through pre-training reinforcement learning of the simulated environment model to the actual optical path of the real environment, and continue to conduct reinforcement learning training through the actual optical path of the real environment. After the training is completed, use the beam pointing correction agent to control the fast reflection mirror group of the beam pointing adjustment device in the real environment.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the light beam pointing adjustment device includes at least two fast reflection mirrors for adjusting the light beam pointing and two detectors for sensing the center position of the light spot offset.

[0012] Optionally, in a second implementation of the first aspect of the present invention, step S2 specifically includes the following steps:

[0013] S21. Record the spot position offset value of the near-end detector and the far-end detector in the beam pointing adjustment device at t0:

[0014]

[0015] Where, represents the spot position offset vector at time t0, and Respectively represent the spot position offset value of the near-end detector on the x-axis and y-axis, and Respectively represent the spot position offset values ​​of the remote detector on the x-axis and y-axis;

[0016] S22, keep the deflection angles of the first and second fast reflectors of the fast reflector group in the beam pointing adjustment device unchanged, record the light spot position offset values ​​of the near-end detector and the far-end detector once at intervals k, record N times in total, and Expand the row records under the matrix to form a matrix:

[0017]

[0018] S23, randomly swing the deflection axes of the first fast reflection mirror and the second fast reflection mirror, and record the light spot position offset values ​​of the near-end detector and the far-end detector:

[0019]

[0020] If the light spot has exceeded the detector target surface, that is, the detector has no light spot offset position value, then the offset value is recorded as:

[0021] D=w*Area max

[0022] Where, Area max is the maximum width of the detector target surface, and w is a penalty coefficient greater than 10;

[0023] Record the deflection angle values ​​of the first and second fast reflectors of the fast reflector group in the beam pointing adjustment device at t0:

[0024]

[0025] S24, keep the deflection angles of the first fast reflector and the second fast reflector unchanged, record the spot position offset values ​​of the near-end detector and the far-end detector once every k milliseconds, record N times in total, and t1 Expand the row records under the matrix to form a matrix:

[0026]

[0027] S25, repeat steps S23 and S24 for Q times to obtain a set of light spot position offset values ​​of the beam pointing adjustment device The set of deflection angle values ​​of the first fast reflecting mirror and the second fast reflecting mirror Based on the offset value set E and the yaw angle value set A, a dataset of the offset effect of the beam pointing adjustment device containing N*Q samples is constructed. Form a sample data set.

[0028] Optionally, in a third implementation of the first aspect of the present invention, step S3 specifically includes the following steps:

[0029] S31. Establish a convolutional neural network, where the convolutional neural network includes an input layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and an output layer;

[0030] The input layer receives the input tensor Indicates the spot position offset value at time t, the yaw angle values ​​of the first fast reflection mirror and the second fast reflection mirror;

[0031] The output layer outputs the spot position offset tensor Indicates the spot position offset value at time t+1;

[0032] S32, divide the sample data set R obtained in step S2 into a training set and a test set in proportion, and divide the x of the samples in the training set into (n) Input the convolutional neural network as an input tensor, and calculate the convolutional neural network output tensor and the corresponding sample label value y (n) The mean square error is used as the loss function to backpropagate to the previous layer of the convolutional neural network, and the weights are updated. The model is trained multiple times until the loss function value is minimized.

[0033] Optionally, in a fourth implementation of the first aspect of the present invention, step S4 specifically includes the following steps:

[0034] S41. Define random initial state The random initial state is the spot position offset value of the simulated beam pointing adjustment device at the initial time t, and the degree of spot position offset at the initial time t is recorded. Action accumulator initial value P total =[p x1 ,p y1 ,p x2 ,p y2 ]=[0,0,0,0];

[0035] S42, random initial state S t Output to the beam pointing correction agent and the status register, which stores the current state S t , the control strategy network of the beam pointing correction agent is based on S t Make decision output action This action is the fast mirror tilt angle that the beam pointing correction agent gives based on the current state to minimize the beam pointing deviation.

[0036] S43, the action accumulator receives the latest action A t , calculate the cumulative deflection angle of the fast reflector, that is:

[0037]

[0038] S44, the tensor splicer receives the historical state S from the state register t , current action A t , concatenated into a tensor And use the tensor v as the input of the simulation environment model;

[0039] S45, the simulation environment model outputs a new spot position offset value as the state of the new time t+1 according to the input tensor v The status register receives the latest status S t+1Make updates;

[0040] S46. The reward calculator calculates the reward r according to the following formula t :

[0041] r t =Diff t +Penalty t

[0042]

[0043] in, State S from time t+1 t+1 , p x1 、p y1 、p x2 、p y2 From the action accumulator P total , α is a regularization coefficient greater than 0, Diff t Used to measure the degree of beam deviation after the beam pointing adjustment device performs a rapid reflector deflection. t Used to constrain the cumulative motion swing of the fast reflector;

[0044] S47, the beam pointing correction agent accepts the reward r given by the reward calculator t If the beam points to the corrected agent convergence, then the current round of training ends, otherwise according to the new state S t+1 , continue iterative training, and iteratively update the beam pointing correction agent. One round of training lasts for a maximum of K iterations.

[0045] Optionally, in a fifth implementation of the first aspect of the present invention, the beam pointing correction agent includes 4 convolutional neural networks and 1 experience replay pool, the 4 convolutional neural networks are an action network, an evaluation network, a target action network and a target evaluation network, the action network and the target action network have the same network structure, and the evaluation network and the target evaluation network have the same network structure.

[0046] Optionally, in a sixth implementation of the first aspect of the present invention, the training process of the light beam pointing correction agent specifically includes the following steps:

[0047] Set the maximum number of training rounds to M, the number of single-round iterations to K, and the action network μ θ The network parameters θ, the evaluation network Q w The network parameters w, target action network μ θ‘ The network parameters θ′=θ, the target evaluation network Q w′ The network parameters w′=w, random noise tensor δ, reset the experience replay pool;

[0048] Obtain the random initial state S described in step S41 t = S1, execute step S42, and send the current action network μ θ , input state S t , action network μ θ Output action a t , in the output action a t Superimpose random noise, that is, let A t =a t +δ,A t As the external output action of the beam pointing correction agent;

[0049] According to steps S43-S46, the simulation environment model outputs the state S at the new time t+1. t+1 , the reward calculator gives the reward r t ;

[0050] The experience replay pool R stores a tuple (S t ,A t ,r t ,S t+1 ), randomly sample Z tuples from the experience replay pool R {(S i ,A i ,r i ,S i+1 )} i＝1...Z , for each tuple, combined with the target evaluation network Q w′ , target action network μ θ′ , calculate the target value:

[0051] y i =r i +γQ w′ (S i+1 ,μ θ′ (S i+θ1 ))

[0052] Where γ is a user-defined time series difference coefficient greater than 0, r i is the immediate reward in the sampled tuple;

[0053] Calculate iterative loss function If the iterative loss function L<∈, the beam pointing correction agent is considered to have converged and the current round of training ends, where ∈ is a convergence threshold greater than 0;

[0054] Otherwise, the iterative loss function L is used to evaluate the network Q w Loss function, update the evaluation network Q w The network parameters w are Update the action network μ θ parameter θ;

[0055] According to the action network μ θ , Evaluation Network Q w The latest network parameters θ, w, update the target action network μ θ′ , target evaluation network Q w′ The network parameters θ' and w' are, that is, θ'=τθ+(1-τ)θ', w'=τw+(1-τ)w', where τ is the proportional coefficient, 0<τ<1.

[0056] The second aspect of the present invention provides a beam pointing stabilization system based on simulated environment driven reinforcement learning, the system includes a partitioning module, a data recording module, a model building module, an intelligent agent building module and a real environment training module, wherein:

[0057] A division module is used to divide the integrated optical circuit into a number of sub-optical path links according to the upper and lower level series relationship, wherein each sub-optical path link includes an independent light beam pointing adjustment device;

[0058] A data recording module is used to drive the beam pointing adjustment device and adjust the mirror deflection angle of the fast reflector assembly, record the mirror deflection angle of the fast reflector assembly and the beam pointing drift data before and after the swing, and form a sample data set;

[0059] A model building module, configured to build a convolutional neural network-based simulation environment model based on the sample data set;

[0060] The agent building module is used to build a beam pointing correction agent using the simulated environment model as a virtual interactive environment. It combines a reward calculator, a state register, an action accumulator, and a tensor splicer, and uses a reinforcement learning framework for training.

[0061] The real-world training module is used to apply the beam pointing correction agent obtained through pre-training reinforcement learning of the simulated environment model to the actual optical path of the real environment, and to continue reinforcement learning training through the actual optical path of the real environment. After the training is completed, the beam pointing correction agent is used to control the fast reflector group of the beam pointing adjustment device in the real environment.

[0062] The third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the various steps of the beam pointing stabilization method based on simulated environment driven reinforcement learning as described in any of the above items are executed.

[0063] A fourth aspect of the present invention provides a computer-readable storage medium, which, when executed by a processor, runs the various steps in the beam pointing stabilization method based on simulated environment driven reinforcement learning as described in any of the above items.

[0064] In the technical solution provided by the present invention, a deep learning method is used to construct a beam pointing correction intelligent agent to directly control the beam pointing adjustment device. This beam pointing control solution solves the deficiency of the current beam pointing control solution in that it is oriented towards adjusting a single optical path, and can establish effective beam pointing control for a system optical path containing multiple sub-optical path links and having multiple types of interference. The effective implementation of the deep learning method requires a large amount of training on the controlled object. If real training is carried out directly under the actual beam pointing adjustment device, the overall process takes a long time, and performing large-scale and high-frequency training in a real environment is likely to cause fatigue damage to devices such as fast reflectors. This method collects an offset effect data set for the beam pointing adjustment device Based on the convolutional neural network, a simulation environment model that can reflect the actual beam pointing adjustment environment is established, and it is used as a virtual interactive environment. A reinforcement learning training method running in the virtual interactive environment is designed to train the beam pointing correction agent; in the virtual interactive environment, deep learning training can be carried out quickly, in large quantities, and without restrictions to improve the beam pointing adjustment effect of the beam pointing correction agent; based on the reinforcement learning training of the beam pointing correction agent driven by the simulation environment, the difficulty of applying the beam pointing correction agent to the real environment is greatly reduced. The beam pointing correction agent only needs to be trained for several rounds in the real environment before it can be put into actual application, and can cope with the beam drift problem of complex series coupled optical path systems under multiple types of interference environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0066] Figure 1 A flowchart of a beam pointing stabilization method based on simulated environment driven reinforcement learning provided by an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of the internal optical path system structure of an extreme ultraviolet laser generating device provided in an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of a light beam pointing adjustment device provided in an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of the reinforcement learning principle based on the simulated environment model provided by an embodiment of the present invention;

[0070] Figure 5 A diagram illustrating the principle of reinforcement learning based on a real environment provided by an embodiment of the present invention;

[0071] Figure 6A schematic diagram of the structure of a beam pointing stabilization system based on simulated environment driven reinforcement learning provided by an embodiment of the present invention;

[0072] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, apparatus, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0074] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A flowchart of a beam pointing stabilization method based on simulated environment driven reinforcement learning provided by an embodiment of the present invention, the method specifically comprising the following steps:

[0075] S1. The integrated optical circuit is divided into several sub-optical circuit links according to the upper and lower level series relationship, wherein each sub-optical circuit link includes an independent light beam pointing adjustment device;

[0076] In this embodiment, the integrated optical path system is divided into several sub-optical path links according to the upper and lower series relationship of the internal optical path. Please refer to Figure 2. Taking the extreme ultraviolet laser generating device as an example, it has multiple sub-optical path links inside, and the sub-optical path links include a beam pointing adjustment device. In this optical path system, it can be divided into three sub-optical path links, namely femtosecond laser-pulse compressor, pulse compressor-extreme ultraviolet laser, and extreme ultraviolet laser-CDI stack imaging. Each sub-optical path link has an independent beam pointing adjustment device; please refer to Figure 3. The beam pointing adjustment device includes two fast reflection mirrors for adjusting the beam pointing (the fast reflection mirror is generally a dual-axis galvanometer that can rotate around the X and Y axes), and two detectors for sensing the center position of the light spot offset (the detector is used to sense the deviation between the center position of the light spot on the current target surface and the center position of the target on the target surface. The deviation reflects the degree of the current beam pointing misalignment drift. The detector type includes a camera such as CMOS, CCD, or a position sensitive detector such as PSD, 4QD, etc.).

[0077] S2, driving the beam pointing adjustment device and adjusting the mirror deflection angle of the fast reflection mirror group, recording the mirror deflection angle of the fast reflection mirror group and the beam pointing drift data before and after the deflection, to form a sample data set;

[0078] In this embodiment, data collection is carried out link by link according to the divided sub-optical paths. Taking the beam pointing adjustment device 1 as an example, the data collection process of the femtosecond laser-pulse width compressor link is described.

[0079] S21. Record the spot position offset value of the near-end detector and the far-end detector in the beam pointing adjustment device at t0:

[0080]

[0081] Where, represents the spot position offset vector at time t0, and Respectively represent the spot position offset value of the near-end detector on the x-axis and y-axis, and Respectively represent the spot position offset values ​​of the remote detector on the x-axis and y-axis;

[0082] S22, maintaining the first fast reflection mirror of the fast reflection mirror group in the beam pointing adjustment device ( Figure 3 Middle fast reflector 1), second fast reflector ( Figure 3 2) The deflection angle of the fast reflector remains unchanged, and the spot position offset value of the near-end detector and the far-end detector is recorded once at intervals k, and recorded N times in total. Expand the row records under the matrix to form a matrix:

[0083]

[0084] S23. Randomly swing the deflection axes of the fast reflector 1 and the fast reflector 2, and record the light spot position offset values ​​of the near-end detector and the far-end detector:

[0085]

[0086] If the light spot has exceeded the detector target surface, that is, the detector has no light spot offset position value, then the offset value is recorded as:

[0087] D=w*Area max

[0088] Where, Area max is the maximum width of the detector target surface, and w is a user-defined penalty coefficient greater than 10;

[0089] Record the deflection angle values ​​of fast reflector 1 and fast reflector 2:

[0090]

[0091] S24, same as step S22, keep the deflection angle of fast reflector 1 and fast reflector 2 unchanged, record the spot position offset value of the near-end detector and the far-end detector once every k milliseconds, record N times in total, and t1 Expand the row records under the matrix to form a matrix:

[0092]

[0093] S25, repeat steps S23 and S24 for Q times to obtain a set of light spot position offset values ​​of the beam pointing adjustment device The set of yaw angle values ​​of fast reflector 1 and fast reflector 2 Based on the offset value set E and the yaw angle value set A, a dataset of the offset effect of the beam pointing adjustment device containing N*Q samples is constructed. Form a sample data set.

[0094] The sample construction rules are as follows:

[0095]

[0096] Similarly, data collection work is carried out for the beam pointing adjustment device 2 of the pulse width compressor-extreme ultraviolet laser and the beam pointing adjustment device 3 of the extreme ultraviolet laser-CDI stacked imaging link to obtain the offset effect data set R2 of the beam pointing adjustment device 2 and the offset effect data set R3 of the beam pointing adjustment device 3.

[0097] S3, build a simulation environment model based on convolutional neural network based on the sample data set;

[0098] In this embodiment, based on the offset effect data sets R1, R2, and R3 of the light beam pointing adjustment devices 1, 2, and 3, simulation environment models N1, N2, and N3 based on convolutional neural networks are respectively constructed. The simulation environment model N1 is constructed by taking the offset effect data set R1 based on the light beam pointing adjustment device 1 as an example.

[0099] S31. Establish a convolutional neural network, where the convolutional neural network includes an input layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and an output layer;

[0100] The input layer receives the input tensor Indicates the spot position offset value at time t, and the yaw angle values ​​of the fast reflector 1 and the fast reflector 2;

[0101] The output layer outputs the spot position offset tensor Indicates the spot position offset value at time t+1;

[0102] Set the training hyperparameters. According to test experience, the main hyperparameters can be set as follows: epochs = 150, batch_size = 16, lr = 0.001 (epochs is the total number of model training rounds, batch_size is the number of samples included in one round of training, lr is the learning rate). Use dropout and batch normalization in the training process to enhance the generalization performance of the network.

[0103] S32, divide the sample data set R1 obtained in step S2 into a training set and a test set in proportion, and divide the x of the samples in the training set into (n) Input the convolutional neural network as an input tensor, and calculate the convolutional neural network output tensor and the corresponding sample label value y (n) The mean square error is used as the loss function to backpropagate to the previous layer of the convolutional neural network, and the weights are updated. The model is trained multiple times until the loss function value is minimized.

[0104] The fast-reflecting mirror group can regulate the light beam multiple times, and the process of adjusting the deflection angle of the fast-reflecting mirror gradually reduces the deviation of the outgoing light beam. In the optical path of the fast-reflecting mirror group, the detector and beam splitter states are regarded as time-invariant parameters. The outgoing light beam deviation observed by the detector at time t+1 is only related to the outgoing light beam state at time t and is not affected by the past state. The pointing deviation of the initial incident light beam is already included in the outgoing light beam state at time t through a chain relationship. The simulation environment models N1, N2, and N3 based on the convolutional neural network can give the new light spot position deviation at time t+1 based on the light spot position deviation of the beam pointing adjustment device at time t and the subsequent swinging galvanometer angle.

[0105] S4. Using the simulated environment model as a virtual interactive environment, a beam pointing correction agent is constructed. The reward calculator, state register, action accumulator, and tensor splicer are combined and trained using a reinforcement learning framework.

[0106] In this embodiment, a reinforcement learning algorithm is adopted, and simulation environment models N1, N2, and N3 based on convolutional neural networks are used as virtual interactive environments for reinforcement learning to train beam pointing correction agents π1, π2, and π3 respectively. The simulation environment model N1 and the beam center of mass correction agent π1 are taken as examples.

[0107] S41. Define random initial state The random initial state is the spot position offset value of the simulated beam pointing adjustment device at the initial time t, and the degree of spot position offset at the initial time t is recorded. Action accumulator initial value P total =[p x1 ,p y1 ,p x2,p y2 ]=[0,0,0,0];

[0108] S42, random initial state S t Output to the beam pointing correction agent π1 and the status register, which stores the current state S t , the control strategy network of the beam pointing correction agent π1 is based on S t Make decision output action This action is the fast mirror tilt angle that the beam pointing correction agent π1 gives based on the current state to minimize the beam pointing deviation.

[0109] S43, the action accumulator receives the latest action A t , calculate the cumulative deflection angle of the fast reflector, that is:

[0110]

[0111] S44, the tensor splicer receives the historical state S from the state register t , current action A t , concatenated into a tensor And use the tensor v as the input of the simulation environment model N1;

[0112] S45, the simulation environment model N1 outputs a new spot position offset value as the state of the new time t+1 according to the input tensor v The status register receives the latest status S t+1 Make updates;

[0113] S46. The reward calculator calculates the reward r according to the following formula t :

[0114] r t =Diff t +Penalty t

[0115]

[0116] in, State S from time t+1 t+1 , p x1 、p y1 、p x2 、p y2 From action accumulator { total , α is a regularization coefficient greater than 0, Diff t It is used to measure the degree of beam pointing deviation after the beam pointing adjustment device performs the rapid reflector deflection. The larger the value, the smaller the beam pointing deviation and the better the control effect. tIt is used to constrain the cumulative action swing of the fast reflector. The larger its value is, the smaller the cumulative adjustment times and swing relative to the initial beam pointing deviation are, and the better the control effect is. The regularization coefficient α is used to control Penalty t+1 The importance of can be adjusted according to test experience;

[0117] S47, beam pointing correction agent π1 accepts the reward r given by the reward calculator t If the beam pointing to the correction agent π1 converges, then this round of training ends. The convergence judgment of the beam pointing to the correction agent π1 needs to be determined in combination with the loss function used in the specific network structure of the agent. Otherwise, according to the new state S t+1 , continue iterative training, continue to execute the above S42-S47 for iterative training, and iteratively update the beam pointing correction agent π1. A round of training lasts for a maximum of K iterations.

[0118] According to the above S41-S47, multiple rounds of training are repeated, with a total of M rounds of training, to further improve the generalization performance of the beam pointing correction agent π1.

[0119] The beam pointing correction agent π1 can refer to various deep learning frameworks, such as TRPO, PPO, SAC, etc. In this example, the deep deterministic policy gradient network (DDPG) is used to establish the beam pointing correction agent π1.

[0120] The beam pointing correction agent π1 consists of four convolutional neural networks, namely the action network μ θ (s), evaluation network Q w (s,a), target action network μ θ′ (s), target evaluation network Q w′ (s, a), and an experience replay pool R; action network μ θ , target action network μ θ′ With the same network structure, the tensor s accepted by the input layer is Figure 4 The state S shown t The output layer outputs a tensor of the same dimension as Figure 4 Action A shown t Tensor of the same dimension, network parameters are θ, θ'; evaluation network Q w , target evaluation network Q w′ With the same network structure, the input layer accepts tensors s and a. Tensor s is Figure 4 The state S shown tThe output layer outputs a value scalar, and the network parameters are w and w′ respectively; the experience replay pool R is a storage for storing historical training data in a single round of reinforcement learning training.

[0121] The following is a specific operation method S41-S47, S41-S47 of the beam pointing correction agent π1 in combination with the above steps:

[0122] For training rounds TrainingTimes = 1 → M:

[0123] 1) Use random numbers to define the action network μ θ The network parameters θ, the evaluation network Q w The network parameters w;

[0124] 2) Target Action Network μ θ‘ The network parameters θ′=θ, the target evaluation network Q w′ The network parameters w′=w;

[0125] 3) Define a random noise tensor δ, dimension and action network μ θ The output action is consistent;

[0126] 4) Reset the experience replay pool;

[0127] Obtain the random initial state S described in step S41 t =S1;

[0128] For a single round of iterations t = 1 → K:

[0129] 1) According to step S42, the current action network μ θ , input state S t , action network μ θ Output action a t ; In output action a t Superimpose random noise, that is, let A t =a t +δ,A t As the external output action of the beam center of mass correction agent π1;

[0130] 2) According to steps S43-S46, the simulation environment model N1 outputs the state S at the new time t+1 t+1 , the reward calculator gives the reward r t ;

[0131] 3) Experience replay pool R stores a tuple (S t ,A t ,r t ,S t+1 );

[0132] 4) Randomly sample Z tuples from the experience replay pool R {(S i ,A i ,r i ,S i+1 )} i＝1...Z , for each tuple, combined with the target evaluation network Q w′ , target action network μ θ′ (s), calculate y i =r i +γQ w′ (S i+1 ,μ θ′ (S i+1 )), γ is a user-defined time series difference coefficient greater than 0;

[0133] 5) Calculate iterative loss function If the iterative loss function L<∈, ∈ is a user-defined convergence threshold greater than 0, then the beam pointing correction agent π1 is considered to have converged, and this round of training ends; otherwise, the iterative loss function L is used to evaluate the network Q w Loss function, update the evaluation network Q w The network parameters w are Update the action network μ θ parameter θ;

[0134] 6) According to the action network μ θ , Evaluation Network Q w The latest network parameters θ, w, soft update target action network μ θ′ , target evaluation network Q w′ The network parameters θ', w', that is, θ'=τθ+(1-τ)θ', w'=τw+(1-τ)w', where τ is the proportional coefficient, 0<τ<1

[0135] End For

[0136] End For

[0137] S5. Apply the beam pointing correction agent obtained through pre-training reinforcement learning of the simulated environment model to the actual optical path of the real environment, and continue to conduct reinforcement learning training through the actual optical path of the real environment. After the training is completed, use the beam pointing correction agent to control the fast reflection mirror group of the beam pointing adjustment device in the real environment.

[0138] In this embodiment, please refer to Figure 5As shown in the figure, the beam pointing correction agents π1, π2, and π3 obtained by pre-training reinforcement learning in the simulated environment model are applied to the actual light path in the real environment. Reinforcement learning training is continued through the real environment to improve the generalization performance of the control strategy network beam pointing correction agents π1, π2, and π3 in the real environment and adapt to time-varying disturbances in the real environment.

[0139] The beam pointing correction agent π1 obtained by pre-training reinforcement learning of the simulated environment model is directly applied to the real environment; at the initial moment t, the spot offset observed by the internal detector of the beam pointing adjustment device 1 is E t That is state S t ;

[0140] S t Output to the beam pointing correction agent π1 and the status register, which stores the current state S t ; The control strategy network of the beam pointing correction agent π1 is based on S t Make decisions and take actions This action is the fast mirror tilt angle that the beam pointing correction agent π1 gives based on the current state to minimize the beam pointing deviation.

[0141] The action accumulator receives the latest action A t , calculate the cumulative deflection angle of the fast reflector, that is:

[0142]

[0143] Fast mirror driver receives action A t , drives the fast reflection mirror group of the beam pointing adjustment device 1 in the real environment femtosecond laser-pulse width compressor sub-optical path to deflect, the spot offset observed by the detector inside the beam pointing adjustment device 1 changes, and outputs the latest state S t+1 ;

[0144] The reward calculator calculates the reward r according to the following formula t :

[0145] r t =Diff t +Penalty t

[0146]

[0147] in, State S from time t+1 t+1 , p x1 、p y1 、p x2 、p y2From the action accumulator P total , α is a regularization coefficient greater than 0, Diff t It is used to measure the degree of beam pointing deviation after the beam pointing adjustment device performs the rapid reflector deflection. The larger the value, the smaller the beam pointing deviation and the better the control effect. t It is used to constrain the cumulative action swing of the fast reflector. The larger its value is, the smaller the cumulative adjustment times and swing relative to the initial beam pointing deviation are, and the better the control effect is. The regularization coefficient α is used to control Penalty t+1 The importance of can be adjusted according to test experience;

[0148] The beam pointing correction agent π1 receives the reward r from the reward calculator t , if the beam pointing to the correction agent π1 converges, then the current round of training ends. The convergence judgment of the beam pointing to the correction agent π1 needs to be determined in combination with the loss function used in the specific network structure of the agent; otherwise, according to the new state S t+1 , continue to execute the above S42-S47 for iterative training, and iteratively update the beam pointing correction agent π1. One round of training lasts for a maximum of L iterations. Since the beam pointing correction agent π1 has been pre-trained in the virtual interactive environment of the simulated environment model, the iteration process can be appropriately reduced;

[0149] Repeat multiple rounds of training according to S41-S47 above, for a total of H rounds, to further improve the generalization performance of the beam pointing correction agent π1 in the real environment. Since the beam pointing correction agent π1 has been pre-trained in the virtual interactive environment of the simulated environment model, the number of training rounds can be appropriately reduced;

[0150] After the training is completed, the beam pointing correction agent π1 is used to control the fast reflection mirror group of the beam pointing adjustment device 1 in the real environment.

[0151] In the embodiment of the present invention, a deep learning method is used to construct a beam pointing correction intelligent agent to directly control the beam pointing adjustment device. This beam pointing control scheme solves the deficiency of the current beam pointing control scheme in adjusting a single optical path, and can establish effective beam pointing control for a system optical path containing multiple sub-optical path links and multiple interference types. The effective implementation of the deep learning method requires a large amount of training on the controlled object. If real training is carried out directly under the actual beam pointing adjustment device, the overall process takes a long time, and performing large-scale and high-frequency training in a real environment is likely to cause fatigue damage to devices such as fast reflectors. This method is oriented towards beam pointing. The adjustment device collects a data set of offset effects, establishes a simulated environment model based on a convolutional neural network that can reflect the actual beam pointing adjustment environment, and uses it as a virtual interactive environment. A reinforcement learning training method running in the virtual interactive environment is designed to train the beam pointing correction agent. In the virtual interactive environment, deep learning training can be carried out quickly, in large quantities, and without restrictions to improve the beam pointing adjustment effect of the beam pointing correction agent. The reinforcement learning training of the beam pointing correction agent driven by the simulated environment can greatly reduce the difficulty of applying the beam pointing correction agent to the real environment. The beam pointing correction agent only needs to undergo several rounds of training in the real environment before it can be put into actual application.

[0152] Please refer to Figure 6 As shown, Figure 6 In some embodiments of the present invention, a beam pointing stabilization system based on simulated environment driven reinforcement learning is provided. The system includes a partitioning module, a data recording module, a model building module, an agent building module, and a real environment training module.

[0153] A division module 501 is used to divide the integrated optical circuit into a plurality of sub-optical circuit links according to the upper and lower levels of the series relationship, wherein each sub-optical circuit link includes an independent light beam pointing adjustment device;

[0154] The data recording module 502 is used to drive the beam pointing adjustment device and adjust the mirror deflection angle of the fast reflection mirror assembly, record the mirror deflection angle of the fast reflection mirror assembly and the beam pointing drift data before and after the deflection, and form a sample data set;

[0155] A model building module 503 is used to build a simulation environment model based on a convolutional neural network based on the sample data set;

[0156] An agent construction module 504 is used to construct a beam pointing correction agent using the simulated environment model as a virtual interactive environment, and is trained using a reinforcement learning framework by combining a reward calculator, a state register, an action accumulator, and a tensor splicer;

[0157] The real environment training module 505 is used to apply the beam pointing correction agent obtained through pre-training reinforcement learning of the simulated environment model to the actual optical path of the real environment, and continue to perform reinforcement learning training through the actual optical path of the real environment. After the training is completed, the beam pointing correction agent is used to control the fast reflection mirror group of the beam pointing adjustment device in the real environment.

[0158] Figure 7 : is a structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the electronic device 600. Furthermore, the processor 610 can be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the electronic device 600 to implement the method provided by the above embodiment.

[0159] The electronic device 600 may further include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating devices 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 6 The electronic device structure shown does not constitute a limitation on the computer device provided by the present invention, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0160] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the various steps of the beam pointing stabilization method based on simulation environment driven reinforcement learning provided in the above embodiments.

[0161] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0163] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A beam pointing stabilization method based on simulated environment driven reinforcement learning, characterized in that: The beam pointing stabilization method based on simulated environment driven reinforcement learning includes the following steps: S1. The integrated optical circuit is divided into several sub-optical circuit links according to the upper and lower level series relationship, wherein each sub-optical circuit link includes an independent light beam pointing adjustment device; S2, driving the beam pointing adjustment device and adjusting the mirror deflection angle of the fast reflection mirror group, recording the mirror deflection angle of the fast reflection mirror group and the beam pointing drift data before and after the deflection, to form a sample data set; S3. Constructing a simulation environment model based on a convolutional neural network based on the sample data set; S4. Using the simulated environment model as a virtual interactive environment, a beam pointing correction agent is constructed. The reward calculator, state register, action accumulator, and tensor splicer are combined and trained using a reinforcement learning framework. S5. Applying the beam pointing correction agent obtained through reinforcement learning pre-training in the simulated environment model to the actual optical path of the real environment, and continuing reinforcement learning training using the actual optical path of the real environment. After the training is completed, the beam pointing correction agent is used to control the fast reflector group of the beam pointing adjustment device in the real environment; Step S4 includes the following steps: S41. Define random initial state The random initial state is the spot position offset value of the simulated beam pointing adjustment device at the initial time t, and the degree of spot position offset at the initial time t is recorded. Action accumulator initial value P total =[p x1 ,p y1 ,p x2 ,p y2 ]=[0,0,0,0]; S42, random initial state S t Output to the beam pointing correction agent and the status register, which stores the current state S t , the control strategy network of the beam pointing correction agent is based on S t Make decision output action This action is the fast mirror tilt angle that the beam pointing correction agent gives based on the current state to minimize the beam pointing deviation. S43, the action accumulator receives the latest action A t , calculate the cumulative deflection angle of the fast reflector, that is: S44, the tensor splicer receives the historical state S from the state register t , current action A t , concatenated into a tensor And use the tensor v as the input of the simulation environment model; S45, the simulation environment model outputs a new spot position offset value as the state of the new time t+1 according to the input tensor v The status register receives the latest status S t+1 Make updates; S46. The reward calculator calculates the reward r according to the following formula t : r t =Diff t +Penalty t in, State S from time t+1 t+1 , p x1 、p y1 、p x2 、p y2 From the action accumulator P total , α is a regularization coefficient greater than 0, Diff t Used to measure the degree of beam deviation after the beam pointing adjustment device performs a rapid reflector deflection. t Used to constrain the cumulative motion swing of the fast reflector; S47, the beam pointing correction agent accepts the reward r given by the reward calculator t If the beam points to the corrected agent convergence, then the current round of training ends, otherwise according to the new state S t+1 , continue iterative training, and iteratively update the beam pointing correction agent. One round of training lasts for a maximum of K iterations.

2. The beam pointing stabilization method based on simulated environment driven reinforcement learning according to claim 1, characterized in that: The light beam pointing adjustment device comprises at least two fast reflection mirrors for adjusting the light beam pointing and two detectors for sensing the center position of the light spot offset.

3. The beam pointing stabilization method based on simulated environment driven reinforcement learning according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Record the spot position offset value of the near-end detector and the far-end detector in the beam pointing adjustment device at t0: Where, represents the spot position offset vector at time t0, and Respectively represent the spot position offset value of the near-end detector on the x-axis and y-axis, and Respectively represent the spot position offset values ​​of the remote detector on the x-axis and y-axis; S22, keep the deflection angles of the first and second fast reflectors of the fast reflector group in the beam pointing adjustment device unchanged, record the light spot position offset values ​​of the near-end detector and the far-end detector once at intervals k, record N times in total, and Expand the row records under the matrix to form a matrix: S23, randomly swing the deflection axes of the first fast reflection mirror and the second fast reflection mirror, and record the light spot position offset values ​​of the near-end detector and the far-end detector: If the light spot has exceeded the detector target surface, that is, the detector has no light spot offset position value, then the offset value is recorded as: D=w*Area max Where, Area max is the maximum width of the detector target surface, and w is a penalty coefficient greater than 10; Record the deflection angle values ​​of the first and second fast reflectors of the fast reflector group in the beam pointing adjustment device at t0: S24, keep the deflection angles of the first fast reflector and the second fast reflector unchanged, record the spot position offset values ​​of the near-end detector and the far-end detector once every k milliseconds, record N times in total, and t1 Expand the row records under the matrix to form a matrix: S25, repeat steps S23 and S24 for Q times to obtain a set of light spot position offset values ​​of the beam pointing adjustment device The set of deflection angle values ​​of the first fast reflecting mirror and the second fast reflecting mirror Based on the offset value set E and the yaw angle value set A, a dataset of the offset effect of the beam pointing adjustment device containing N*Q samples is constructed. Form a sample data set.

4. The beam pointing stabilization method based on simulated environment driven reinforcement learning according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31. Establish a convolutional neural network, where the convolutional neural network includes an input layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and an output layer; The input layer receives the input tensor Indicates the spot position offset value at time t, the yaw angle values ​​of the first fast reflection mirror and the second fast reflection mirror; The output layer outputs the spot position offset tensor Indicates the spot position offset value at time t+1; S32, divide the sample data set R obtained in step S2 into a training set and a test set in proportion, and divide the x of the samples in the training set into (n) Input the convolutional neural network as an input tensor, and calculate the convolutional neural network output tensor and the corresponding sample label value y (n) The mean square error is used as the loss function to backpropagate to the previous layer of the convolutional neural network, and the weights are updated. The model is trained multiple times until the loss function value is minimized.

5. The beam pointing stabilization method based on simulated environment driven reinforcement learning according to claim 1, characterized in that: The beam pointing correction agent includes four convolutional neural networks and one experience replay pool. The four convolutional neural networks are an action network, an evaluation network, a target action network, and a target evaluation network. The action network and the target action network have the same network structure, and the evaluation network and the target evaluation network have the same network structure.

6. The beam pointing stabilization method based on simulated environment driven reinforcement learning according to claim 5, characterized in that: The training process of the beam pointing correction agent specifically includes the following steps: Set the maximum number of training rounds to M, the number of single-round iterations to K, and the action network μ θ The network parameters θ, the evaluation network Q w The network parameters w, target action network μ θ 'Network parameters θ′=θ, target evaluation network Q w′ The network parameters w′=w, random noise tensor δ, reset the experience replay pool; Obtain the random initial state S described in step S41 t = S1, execute step S42, and send the current action network μ θ , input state S t , action network μ θ Output action a t , in the output action a t Superimpose random noise, that is, let A t =a t +δ,A t As the external output action of the beam pointing correction agent; According to steps S43-S46, the simulation environment model outputs the state S at the new time t+1. t+1 , the reward calculator gives the reward r t ; The experience replay pool R stores a tuple (S t ,A t ,r t ,S t+1 ), randomly sample Z tuples from the experience replay pool R {(S i ,A i ,r i ,S i+1 )} i＝1...Z , for each tuple, combined with the target evaluation network Q w′ , target action network μ θ′ , calculate the target value: y i =r i +γQ w′ (S i+1 ,μ θ′ (S i+1 )) Where γ is a user-defined time series difference coefficient greater than 0, r i is the immediate reward in the sampled tuple; Calculate iterative loss function If the iterative loss function L<∈, the beam pointing correction agent is considered to have converged and the current round of training ends, where ∈ is a convergence threshold greater than 0; Otherwise, the iterative loss function L is used to evaluate the network Q w Loss function, update the evaluation network Q w The network parameters w are Update the action network μ θ parameter θ; According to the action network μ θ , Evaluation Network Q w The latest network parameters θ, w, update the target action network μ θ′ , target evaluation network Q w′ The network parameters θ' and w' are, that is, θ'=τθ+(1-τ)θ', w'=τw+(1-τ)w', where τ is the proportional coefficient, 0<τ<1.

7. A system for the beam pointing stabilization method based on simulated environment driven reinforcement learning according to any one of claims 1 to 6, the system comprising a partitioning module, a data recording module, a model building module, an agent building module, and a real environment training module, wherein: A division module is used to divide the integrated optical circuit into a number of sub-optical path links according to the upper and lower level series relationship, wherein each sub-optical path link includes an independent light beam pointing adjustment device; A data recording module is used to drive the beam pointing adjustment device and adjust the mirror deflection angle of the fast reflector assembly, record the mirror deflection angle of the fast reflector assembly and the beam pointing drift data before and after the swing, and form a sample data set; A model building module, configured to build a convolutional neural network-based simulation environment model based on the sample data set; The agent building module is used to build a beam pointing correction agent using the simulated environment model as a virtual interactive environment. It combines a reward calculator, a state register, an action accumulator, and a tensor splicer, and uses a reinforcement learning framework for training. The real-world training module is used to apply the beam pointing correction agent obtained through pre-training reinforcement learning of the simulated environment model to the actual optical path of the real environment, and to continue reinforcement learning training through the actual optical path of the real environment. After the training is completed, the beam pointing correction agent is used to control the fast reflector group of the beam pointing adjustment device in the real environment.

8. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps of the beam pointing stabilization method based on simulation environment driven reinforcement learning as described in any one of claims 1 to 6 are executed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the beam pointing stabilization method based on simulated environment driven reinforcement learning as described in any one of claims 1 to 6 are executed.

Images