Control method, device and equipment of multi-loop fast reflecting mirror and medium

Abstract

The invention discloses a control method and device of a multi-loop fast reflector, equipment and a medium, relates to the technical field of optical control, and can improve the dynamic performance of a fast reflector system. According to the scheme, the method comprises the steps that the current state of the reflector is collected, wherein the current state comprises the current position, the current angle error and the last control quantity; according to the current state, a corresponding current control parameter is determined from a target control strategy table, the target control strategy table comprises a mapping relation between the state of the reflector and the control parameter, and the mapping relation is formed by inputting the state to a target adaptive strategy network after state discretization to obtain the corresponding control parameter; the target adaptive strategy network is obtained by training based on a reinforcement learning depth deterministic strategy gradient algorithm; according to the current control parameters, the current state and a preset multi-loop controller of a reflector control system, the current control quantity is calculated, the reflector is controlled based on the current control quantity, and the multi-loop controller comprises an inner loop controller and an outer loop controller.

Description

Technical Field

[0001] This application relates to the field of optical control technologies, and particularly to a control method, device, equipment and medium for a multi-loop fast steering mirror. Background Art

[0002] As the core unit of beam control, the Fast-Steering Mirror (FSM) plays an irreplaceable role in fields such as laser communication, adaptive optics, precision tracking, and image motion compensation. The tracking characteristics of the FSM system directly determine the beam aiming pointing or image motion compensation accuracy. However, the FSM is a high-order underdamped system and has significant non-linear links such as hysteresis, so there are characteristics such as difficulty in accurately establishing the model and difficulty in tuning and optimizing control parameters.

[0003] Regarding the significantly non-linear characteristics of hysteresis, although various model inverse compensation methods and advanced closed-loop control methods have been proposed, there are still difficulties in accurately establishing the model and complex calculations in the model inverse compensation method, and the advanced control method is not easy to implement. Regarding the problem of difficult tuning and optimization of control parameters, although adaptive parameter adjustment methods such as fuzzy control have achieved certain effects, the method design is complex and fuzzy rules need to be established based on experience. For high-order non-linear systems, how to design a control method to balance rapidity and tracking performance to achieve high-dynamic tracking performance, and how to perform intelligent tuning of control parameters to achieve optimal control of non-linear systems still face challenges. Summary of the Invention

[0004] This application provides a control method, device, equipment and medium for a multi-loop fast steering mirror, which can improve the dynamic performance of the fast steering mirror system control.

[0005] To achieve the above object, this application adopts the following technical solutions: In the first aspect of the embodiments of this application, a control method for a multi-loop fast steering mirror is provided. The method includes: Collect the current state of the mirror, where the current state includes: the current position of the mirror, the current angle error, and the previous control amount acting on the mirror in the previous control period, and the current angle error is the difference between the target position and the current position; Determine the corresponding current control parameter from a preset target control strategy table according to the current state. The target control strategy table includes the mapping relationships between the states of multiple mirrors and control parameters. The mapping relationships are formed by inputting the discretized states into a target adaptive strategy network to obtain the corresponding control parameters. The target adaptive strategy network is a neural network model based on the deep deterministic policy gradient algorithm of reinforcement learning; Calculate the current control quantity according to the current control parameter and the multi-loop controller of the preset control system of the mirror, and control the mirror based on the current control quantity. The multi-loop controller includes an inner-loop controller and an outer-loop controller.

[0006] As a possible implementation, the fast steering mirror system is a high-order underdamped nonlinear system. Before collecting the current state of the mirror system, the method further includes: Obtain the mathematical model of the control system of the mirror in the complex domain; Based on the mathematical model, use the proportional-integral control method to determine the outer-loop controller and configure the mirror system as a type-I system; Based on the mathematical model, use the proportional-derivative control method to determine the inner-loop controller.

[0007] As a possible implementation, the output of the outer-loop controller is:

[0008] where, is the first control parameter, is the second control parameter, is the angle error at the t-th cycle; The output of the inner-loop controller is:

[0009] where, is the third control parameter, is the fourth control parameter, is the inner-loop error.

[0010] As a possible implementation, before collecting the current state of the mirror system, the method further includes: constructing an adaptive policy network based on the deep deterministic policy gradient algorithm of reinforcement learning. The network of the adaptive policy network includes: an action network, an evaluation network, an action target network, and an evaluation target network; The state of the adaptive policy network includes: the position of the mirror, the angle error, and the control quantity; The actions of the adaptive policy network include: the first control parameter, the second control parameter, the third control parameter, and the fourth control parameter; The reward function of the adaptive policy network includes multiple indexes of the control system, and the indexes include: rise time, overshoot, steady-state error, and stability.

[0011] As a possible implementation, after constructing the adaptive policy network based on the deep deterministic policy gradient algorithm of reinforcement learning, the method further includes: Initialize the network, experience buffer pool, random noise, and state of the adaptive policy network; Execution process, which includes: Use the action network to calculate the action corresponding to the initial state; after inputting the action into the multi-loop controller, obtain the corresponding control quantity; after controlling the mirror based on the control quantity, obtain the next state; input the state into the reward function to obtain the reward value; store the sample data of the current cycle process in the experience buffer pool, where the sample data includes the current state, current action, current reward value, and next state; Repeat the execution of the process until the amount of data in the experience buffer pool is greater than the preset amount of data, then collect multiple sample data from the experience buffer pool to train the adaptive policy network. Until the number of training rounds reaches the preset threshold, obtain the target adaptive policy network; Input the discretized state into the target adaptive policy network to obtain the corresponding control parameters, and map and store the state and control parameters, the target control strategy table.

[0012] As a possible implementation, the collecting multiple sample data from the experience buffer pool to train the adaptive policy network includes: Use the sample data to calculate the gradient of the evaluation function in the action network; According to the gradient of the evaluation function, use the gradient ascent algorithm to update the network parameters of the action network; Calculate the next action corresponding to each current state in the sample data through the action target network; Use the temporal difference method to calculate the action value of the current state; According to the sample data, the next action, and the action value, calculate the gradient of the loss function of the evaluation network; According to the gradient of the loss function, update the parameters of the evaluation network.

[0013] As a possible implementation, the calculating the current control quantity according to the current control parameter and the preset multi-loop controller of the control system of the mirror, and controlling the mirror based on the current control quantity includes: Input the first control parameter, the second control parameter, and the current angle error into the outer loop controller to obtain the first control quantity; Use a tracking differentiator to calculate the speed of the mirror; According to the speed, the first control quantity, and the preset feedback coefficient, calculate the inner loop error; Input the inner loop error, the third control parameter, and the fourth control parameter into the inner loop controller to calculate the current control amount.

[0014] In a second aspect of the embodiments of the present application, a control device for a multi-loop fast steering mirror is provided. The device includes: An acquisition module, configured to collect the current state of the mirror. The current state includes the current position of the mirror, the current angle error, and the previous control amount acting on the mirror in the previous control period. The current angle error is the difference between the target position and the current position. A determination module, configured to determine the corresponding current control parameter from a preset target control strategy table according to the current state. The target control strategy table includes the mapping relationships between the states of multiple mirrors and the control parameters. The mapping relationships are formed by inputting the discretized states into a target adaptive strategy network to obtain the corresponding control parameters. The target adaptive strategy network is a neural network model based on the deep deterministic policy gradient algorithm of reinforcement learning. A processing module, configured to calculate the current control amount according to the current control parameter and a multi-loop controller of the preset control system of the mirror, and control the mirror based on the current control amount. The multi-loop controller includes an inner loop controller and an outer loop controller.

[0015] In a third aspect of the embodiments of the present application, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the control method of the multi-loop fast steering mirror in the first aspect of the embodiments of the present application is implemented.

[0016] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the control method of the multi-loop fast steering mirror in the first aspect of the embodiments of the present application is implemented.

[0017] The beneficial effects brought by the technical solutions provided by the embodiments of the present application at least include: The control method of the multi-loop fast steering mirror provided by the embodiment of the present application collects the current state of the steering mirror. The current state includes: the current position of the steering mirror, the current angle error, and the previous control amount acting on the steering mirror in the previous control period. The current angle error is the difference between the target position and the current position. According to the current state, the corresponding current control parameters are determined from a preset target control strategy table. The target control strategy table includes the mapping relationships between the states of multiple steering mirrors and the control parameters. The mapping relationships are formed by inputting the discretized states into a target adaptive policy network to obtain the corresponding control parameters. The target adaptive policy network is a neural network model based on the deep deterministic policy gradient algorithm. According to the current control parameters and a multi-loop controller of the preset control system of the steering mirror, the current control amount is calculated, and the steering mirror is controlled based on the current control amount. The multi-loop controller includes an inner-loop controller and an outer-loop controller. The control method of the multi-loop fast steering mirror provided by the present application intelligently tunes the multi-loop control parameters through a reinforcement learning algorithm to cope with the nonlinear characteristics of the system and achieve optimal tuning of the control parameters. Through simulation and experiments, this method improves the bandwidth of the FSM, shortens the adjustment time of the system, and improves the dynamic performance of the system. Description of the Drawings

[0018] Figure 1 It is a flowchart of a control method for a multi-loop fast steering mirror provided by an embodiment of the present application; Figure 2 It is a structural diagram of a control device for a multi-loop fast steering mirror provided by an embodiment of the present application; Figure 3 It is an internal structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0019] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0020] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of "plurality" is two or more.

[0021] In addition, the use of "based on" or "according to" implies openness and inclusiveness because a process, step, calculation, or other action "based on" or "according to" one or more conditions or values can, in practice, be based on additional conditions or values beyond those stated.

[0022] An embodiment of this application provides a control method for a multi-loop fast steering mirror, as Figure 1 shown. The method includes the following steps: Step 101: Acquire the current state of the steering mirror. The current state includes: the current position of the steering mirror, the current angular error, and the previous control amount applied to the steering mirror in the previous control cycle. The current angular error is the difference between the target position and the current position. Among them, the current state in step 101 is the current state of the steering mirror in the current control cycle. The steering mirror can be a two-axis fast steering mirror driven by piezoelectric ceramics. This fast steering mirror can achieve laser beam pointing adjustment or optical camera image motion compensation. When the steering mirror is a two-axis fast steering mirror driven by piezoelectric ceramics, the control amount can be a voltage drive signal.

[0023] Among them, before acquiring the current state of the steering mirror, the method further includes: receiving a target control instruction, which includes target position information.

[0024] Among them, the current position of the steering mirror can be represented by and the current angular error can be represented by The control amount can be represented by and the current state can be represented by

[0025] Step 102: Determine the corresponding current control parameter from a preset target control strategy table according to the current state. The target control strategy table includes the mapping relationships between the states of multiple steering mirrors and control parameters. The mapping relationships are formed by inputting the discretized states into a target adaptive strategy network to obtain the corresponding control parameters. The target adaptive strategy network is a neural network model based on the deep deterministic policy gradient algorithm of reinforcement learning.

[0026] Among them, the target control strategy table is obtained by discretizing the target adaptive strategy network according to the input state interval. The target adaptive strategy network is a neural network model based on the deep deterministic policy gradient algorithm of reinforcement learning. Four control parameters can be represented by The target control strategy table stores multiple - corresponding relationships.

[0027] ​Step 103: Calculate the current control quantity according to the current control parameter and the multi-loop controller of the preset control system of the mirror, and control the mirror based on the current control quantity. The multi-loop controller includes an inner-loop controller and an outer-loop controller.

[0028] The control method of the multi-loop fast steering mirror provided by the embodiment of the present application collects the current state of the mirror. The current state includes: the current position of the mirror, the current angle error, and the previous control quantity acting on the mirror in the previous control period. The current angle error is the difference between the target position and the current position; determine the corresponding current control parameter from the preset target control strategy table according to the current state. The target control strategy table is the relationship between the current state of the mirror and the control parameter. The target control strategy table is obtained by inputting multiple current states into the target adaptive strategy network to obtain the corresponding control parameter. The target adaptive strategy network is trained based on the reinforcement learning deep deterministic policy gradient algorithm; calculate the current control quantity according to the current control parameter and the multi-loop controller of the preset control system of the mirror, and control the mirror based on the current control quantity. The multi-loop controller includes an inner-loop controller and an outer-loop controller.

[0029] The control method of the multi-loop fast steering mirror provided by the present application intelligently tunes the multi-loop control parameters through the reinforcement learning algorithm to cope with the nonlinear characteristics of the system and achieve optimal tuning of the control parameters. Through simulation and experiments, this method improves the bandwidth of the FSM, shortens the adjustment time of the system, and improves the dynamic performance of the system.

[0030] Optionally, the fast steering mirror system is a high-order underdamped nonlinear system. Before obtaining the current state of the mirror system, the method further includes: Obtain the mathematical model of the control system of the mirror in the complex domain; Based on the mathematical model, use the proportional-integral control method to determine the outer-loop controller and configure the mirror system as a type-I system; Based on the mathematical model, use the proportional-derivative control method to determine the inner-loop controller.

[0031] Among them, the mathematical model of the control system of the mirror in the complex domain can also be called the transfer function of the mirror. The mathematical model or transfer function of the control system of the mirror in the complex domain can be:

[0032] Among them, is the natural frequency of the second-order link of the fast steering mirror, is the corner frequency of the first-order link of the fast steering mirror, is the characteristic of non - linear links such as hysteresis; is the damping coefficient, and < 1, is the output angle of the fast steering mirror, is the driving voltage of the fast steering mirror, is the operator in the complex domain.

[0033] It can be seen from the transfer function of the FSM that the FSM system is a third - order under - damped system with a small damping coefficient. The FSM system has a fast response, but it is easy to produce overshoot and it is difficult to maintain a balance between rapidity and no overshoot to achieve high - dynamic tracking of the coarse - stage platform error.

[0034] In order to increase the damping coefficient of this third - order system, angular velocity feedback of the FSM is introduced on the basis of the single - loop control method to form an inner velocity loop. Proportional - derivative control is used in the inner loop to cancel the inner - loop poles. In order to suppress the noise amplification effect brought by differentiation, the angular velocity of the FSM system is extracted by a tracking differentiator. To analyze the influence of the inner loop on the dynamic characteristics of the system, the hysteresis non - linear effect of the FSM system is ignored, and its non - linear characteristics are suppressed by the method of adaptive tuning of reinforcement learning parameters. At the same time, the link of the tracking differentiator is simplified to an ideal differentiator. At this time, the transfer function of the mirror control system can be deformed as:

[0035] The transfer function of the inner - loop control can be designed as: , then the transfer function of the inner loop at this time is:

[0036] Among them, when the parameters of the transfer function of the inner - loop control , when, pole cancellation can be achieved. At this time, the transfer function of the inner - loop control becomes:

[0037] At this time, the inner - loop damping becomes:

[0038] Therefore, the multi - loop control method estimates the speed, forms an inner loop with speed feedback and combines proportional - derivative control, adjusts the differential coefficient , can achieve pole cancellation, and can configure the inner loop of the FSM system as the ideal damping of a typical second - order system by configuring the feedback coefficient to improve the damping coefficient of the FSM system and optimize the dynamic performance of the system.

[0039] On the basis of the inner control loop, with the angle of the fast steering mirror For feedback, a control command is used as the input to construct an outer control loop, and the outer loop adopts proportional-integral control. To ensure the system tracking performance, the system is configured as a type-I system, and the transfer function of the outer loop control can be designed as:

[0040] According to the transfer functions of the above inner loop control and outer loop control, the output of the outer loop controller is:

[0041] Wherein, is the first control parameter, is the second control parameter, is the angle error at the t-th cycle; The output of the inner loop controller is:

[0042] Wherein, is the third control parameter, is the fourth control parameter, is the inner loop error.

[0043] After constructing the multi-loop controller of the mirror control system, to cope with the non-linear characteristics of the system and further improve the dynamic performance of the precision-level FSM system, the multi-loop control parameters are tuned by an adaptive policy network constructed by the Deep Deterministic Policy Gradient (DDPG) algorithm.

[0044] Optionally, after obtaining the multi-loop controller, the method further includes: constructing an adaptive policy network based on the Deep Deterministic Policy Gradient algorithm, the network of the adaptive policy network includes: an action network, a critic network, an action target network, and a critic target network; the state of the adaptive policy network includes: the position of the mirror, the angle error, and the control amount; the actions of the adaptive policy network include: the first control parameter, the second control parameter, the third control parameter, and the fourth control parameter; the reward function of the adaptive policy network includes multiple metrics of the control system, and the metrics include: rise time, overshoot, steady-state error, and stability.

[0045] That is to say, the adaptive policy network constructed by the DDPG algorithm includes four networks: an action actor network and a critic network , an action target actor network and a critic target network .

[0046] Obviously, the fewer the system states, the faster the training. To improve the algorithm efficiency, the states of the reinforcement learning DDPG algorithm include: the current position, the system angle error, and the control quantity, that is .

[0047] The output action a of the adaptive policy network is the 4 parameters of the multi-loop controller of the fast steering mirror system, that is .

[0048] In addition, to comprehensively evaluate the advantages and disadvantages of the policy, the concerned indicators are incorporated into the design of the reward function. The reward function designed by the adaptive policy network consists of multiple items, as shown in the formula.

[0049]

[0050] In the formula, is the rise time of the system, is the overshoot of the system, and are the control voltages output in the current control cycle and the previous control cycle, , , , and are the weight coefficients of each index respectively. The rewards and evaluate the rapidity of the system transient process through the cumulative size of the transient process error and the rise time respectively. The reward evaluates the steady-state error of the system, the reward evaluates the stability of the system, and the reward evaluates the overshoot of the system.

[0051] Optionally, after constructing the adaptive policy network based on the reinforcement learning deep deterministic policy gradient algorithm, the method further includes: Initializing the network, experience buffer pool, random noise, and state of the adaptive policy network; Execution process, the process includes: Calculating the action corresponding to the initial state by using the action network; after inputting the action into the multi-loop controller, obtaining the corresponding control quantity; after controlling the mirror based on the control quantity, obtaining the next state; inputting the state into the reward function to obtain the reward value; storing the sample data of the current cycle process into the experience buffer pool, and the sample data includes the current state, the current action, the current reward value, and the next state; Repeat the above process until the amount of data in the experience buffer pool is greater than the preset amount of data, then collect multiple sample data from the experience buffer pool to train the adaptive policy network. After the training round reaches the preset threshold, the target adaptive policy network is obtained.

[0052] Specifically, collecting multiple sample data from the experience buffer pool to train the adaptive policy network includes: Calculating the gradient of the evaluation function in the action network using the sample data; Updating the network parameters of the action network using the gradient ascent algorithm according to the gradient of the evaluation function; Calculating the next action corresponding to each current state in the sample data through the action target network; Calculating the action value of the current state using the temporal difference method; Calculating the gradient of the loss function of the evaluation network according to the sample data, the next action, and the action value; Updating the parameters of the evaluation network according to the gradient of the loss function.

[0053] During the actual execution process, the training process of the adaptive policy network can be as follows: (1) Initialize the actor network and the critic network as 4-layer fully connected networks, and initialize the network parameters and .

[0054] (2) Initialize the target actor network and the target critic network , where the target network structure and parameters , are the same as those of the source network.

[0055] (3) Initialize the experience buffer pool R, where R can be set to 100000.

[0056] (4) Set the maximum number of training rounds as , which can be set to 5000.

[0057] (5) Initialize the random noise , and generate the initial state ; (6) Set the round training period as , and the training step size as .

[0058] (7) When reaching the training sampling moment, calculate the current action by the actor network , that is, obtaining the control parameters corresponding to the current action .

[0059] (8) The control parameters obtained according to the current action , the multi-loop controller generates a control quantity with this parameter, applies this control quantity to the FSM system, and calculates the reward by combining this control parameter , state .

[0060] Specifically, the process of the multi-loop controller generating a control quantity with this parameter can be as follows: Collect the current position of the mirror , and take the difference from the target position in the control command to obtain ; Calculate the output of the outer loop controller:

[0061] Use a tracking differentiator as follows to calculate the angular velocity of the fast steering mirror ;

[0062] Among them, is the estimate of the FSM angle , is the estimate of the FSM angular velocity, is the tracking speed factor of the tracking differentiator. By reasonably configuring the tracking speed factor, while suppressing noise, the tracking differentiator can achieve the tracking of the FSM angle, can achieve the extraction of the FSM angular velocity signal.

[0063] Configure the feedback coefficient to make the damping coefficient of the FSM system be ; According to the angular velocity calculated by the tracking differentiator and the feedback coefficient, calculate the inner loop error ; Calculate the output of the inner loop controller:

[0064] Among them, the inner loop controller parameter , where is the control quantity. After acting on the fast steering mirror system with , according to , calculate the reward.

[0065] (9) When the reward is less than the minimum reward threshold, end the training for this round and jump to step (18); when it is not less than the minimum reward threshold, store the data ( ) in buffer R; (10) Randomly sample and obtain n a batch of data ( ) from the experience buffer R; (11) Use the sampled n batch of data to calculate the gradient of the evaluation function of the policy actor network:

[0066] (12) Adopt the gradient ascent algorithm to maximize the evaluation function and update the actor network;

[0067] (13) Calculate the next action of the state through the target actor network; (14) Calculate the action value of the state

[0068] (15) Use the sampled n batch of data and and to calculate the gradient of the loss function of the critic network:

[0069] (16) Use the gradient descent method to minimize the loss function of the critic network and complete the update of the critic network;

[0070] (17) Soft update the target network:

[0071]

[0072] (18) Repeat steps (6)-(17) to complete the training update for this round; (19) Repeat steps (8)-(18) until the number of training rounds reaches the maximum number of training rounds, complete the DDPG algorithm training, and obtain the target adaptive policy network.

[0073] It should be noted that after obtaining the target adaptive policy network, in order to deploy the obtained policy on an embedded platform with limited resources, the DDPG state is discretized into several intervals, and the optimal Actor network obtained by training is discretized according to the state intervals to obtain the state input and the output action control parameter mapping table.

[0074] Optionally, calculating the current control amount according to the current control parameter and the multi-loop controller of the preset control system of the mirror, and controlling the mirror based on the current control amount includes:[[]] Inputting the first control parameter, the second control parameter, and the current angle error into the first loop controller to obtain a first control amount; Using a tracking differentiator to calculate the speed of the mirror; Calculating the inner loop error according to the speed, the first control amount, and a preset feedback coefficient; Calculating the current control amount according to the inner loop error, the third control parameter, and the fourth control parameter.

[0075] In the actual execution process, the process of calculating the current control amount according to the current control parameter and the multi-loop controller of the preset control system of the mirror may be:[[]] (1) Divide the angular motion stroke of the fast steering mirror into 20 intervals, divide the angular error of the fast steering mirror into 20 intervals, and divide the control voltage into 15 intervals; (2) Using the center position of each interval as the input, calculate the output of the optimal policy actor network to obtain a discretized optimal policy netlist; (3) Initialize = 0; = 0; (4) Take the difference between the current position and the control command to obtain the angular error .

[0076] (5) Collect the current state ; (6) Calculate the outer loop control output; ; (7) Use a tracking differentiator as follows to estimate the speed of the fast steering mirror.

[0077]

[0078] (8) According to the estimated speed and the feedback coefficient, calculate the inner loop error ; (9) Calculate the inner loop control output

[0079] wherein, , when; (10) Use to act on the fast steering mirror system.

[0080] (11) Repeat steps (3)-(10) to achieve the optimal control of the multi-loop fast steering mirror based on the reinforcement learning parameter adaptation.

[0081] This application uses a tracking differentiator to extract the speed signal of the fast steering mirror, uses this speed signal as feedback and combines proportional derivative control to construct a multi-loop control, optimizes the damping coefficient of the high-order underdamped fast steering mirror system, and balances the contradiction between the rapidity and overshoot of the system; the control parameters of the multi-loop control are intelligently tuned through the reinforcement learning algorithm to adapt to the nonlinear characteristics of the system, realize the adaptive optimal tuning of the control parameters, and further improve the dynamic performance of the system. At the same time, in order to deploy this method on an embedded platform with limited resources, the state is uniformly discretized into several intervals, and the trained optimal policy network is discretized according to the input state intervals to obtain the optimal control policy netlist of the input state and control parameters, which is used in the actual system in the form of an index table to solve the problem of limited resource deployment and improve the operation efficiency of the actual system.

[0082] The embodiment of this application also provides a control device for a multi-loop fast steering mirror, as Figure 2 shown, the device includes: An acquisition module 11, configured to acquire the current state of the mirror, where the current state includes: the current position of the mirror, the current angle error, and the previous control amount acting on the mirror in the previous control period, and the current angle error is the difference between the target position and the current position; A determination module 12, configured to determine the corresponding current control parameter from a preset target control policy table according to the current state, where the target control policy table includes the mapping relationships between the states of multiple mirrors and the control parameters, and the mapping relationships are formed by inputting the discretized states into the target adaptive policy network to obtain the corresponding control parameters, and the target adaptive policy network is a neural network model based on the reinforcement learning deep deterministic policy gradient algorithm; The processing module 13 is used to calculate the current control quantity according to the current control parameters and the preset multi-loop controller of the control system of the reflector, and control the reflector based on the current control quantity, wherein the multi-loop controller includes an inner loop controller and an outer loop controller.

[0083] In one embodiment, the fast mirror system is a high-order underdamped nonlinear system, and the determining module 12 is further configured to: Obtaining a mathematical model of the control system of the reflector in a complex domain; Based on the mathematical model, an outer loop controller is determined using a proportional-integral control method, and the reflector system is configured as a Type I system; Based on the mathematical model, the inner loop controller is determined by adopting the proportional differential control method.

[0084] In one embodiment, the output of the first loop controller is:

[0085] in, is the first control parameter, is the second control parameter, is the angle error in the tth cycle; The output of the second loop controller is: in, is the third control parameter, is the fourth control parameter, is the inner loop error.

[0086] in, is the third control parameter, is the fourth control parameter, is the inner loop error.

[0087] In one embodiment, the apparatus further includes: a construction module 14, wherein the construction module 14 is configured to: An adaptive policy network is constructed based on a deep deterministic policy gradient algorithm for reinforcement learning, wherein the adaptive policy network includes an action network, an evaluation network, an action target network, and an evaluation target network; The state of the adaptive strategy network includes: the position of the reflector, the angle error and the control amount; The actions of the adaptive strategy network include: the first control parameter, the second control parameter, the third control parameter and the fourth control parameter; The reward function of the adaptive strategy network includes multiple indicators of the control system, including rise time, overshoot, steady-state error and smoothness.

[0088] In one embodiment, the device further includes: a training module 15, and the training module is configured to: Initialize the network, experience buffer pool, random noise, and state of the adaptive policy network; An execution process, and the process includes: Calculate the action corresponding to the initial state by using the action network; after inputting the action into the multi-loop controller, obtain the corresponding control quantity; after controlling the mirror based on the control quantity, obtain the next state; input the state into the reward function to obtain a reward value; store the sample data of the current cycle process in the experience buffer pool, where the sample data includes the current state, current action, current reward value, and next state; Repeat the execution of the process until the amount of data in the experience buffer pool is greater than the preset amount of data, then collect multiple sample data from the experience buffer pool to train the adaptive policy network until the training round reaches the preset threshold, and then obtain the target adaptive policy network; Input the discretized state into the target adaptive policy network to obtain the corresponding control parameters, and map and store the state and control parameters, the target control policy table.

[0089] In one embodiment, the training module 15 is further configured to: Calculate the gradient of the evaluation function in the action network by using the sample data; Update the network parameters of the action network by using the gradient ascent algorithm according to the gradient of the evaluation function; Calculate the next action corresponding to each current state in the sample data through the action target network; Calculate the action value of the current state by using the temporal difference method; Calculate the gradient of the loss function of the evaluation network according to the sample data, the next action, and the action value; Update the parameters of the evaluation network according to the gradient of the loss function.

[0090] In one embodiment, the processing module 13 is specifically configured to: Input the first control parameter, the second control parameter, and the current angle error into the outer loop controller to obtain a first control quantity; Use a tracking differentiator to calculate the speed of the mirror; Calculate the inner loop error according to the speed, the first control quantity, and a preset feedback coefficient; Input the inner loop error, the third control parameter, and the fourth control parameter into the inner loop controller to calculate the current control quantity.

[0091] The control device of the multi-loop fast steering mirror provided by the embodiment of the present application can execute the embodiment of the control method of the multi-loop fast steering mirror, and its implementation principle and technical effects are similar, so details are not described herein again.

[0092] For the specific limitations of the control device of the multi-loop fast steering mirror, reference can be made to the limitations of the control method of the multi-loop fast steering mirror in the foregoing, and details are not described herein again. Each module in the above control device of the multi-loop fast steering mirror can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0093] The execution subject of the control method of the multi-loop fast steering mirror provided by the embodiment of the present application can be an electronic device, and the electronic device can be a controller, a processor, a processing chip, a computer device, a terminal device, a server, or a server cluster. The embodiment of the present application does not make specific limitations on this.

[0094] Figure 3 It is a schematic internal structure diagram of an electronic device provided by the embodiment of the present application. As Figure 3 shown, the electronic device includes a processor and a memory connected through a system bus. Among them, the processor is used to provide computing and control capabilities. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement the steps of the control method of the multi-loop fast steering mirror provided by the above various embodiments. The internal memory provides a high-speed cache operating environment for the operating system and the computer program in the non-volatile storage medium.

[0095] Those skilled in the art can understand that Figure 3 the internal structure diagram of the electronic device shown in

[0096] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0097] In another embodiment of the present application, a computer program product is further provided. The computer program product includes computer instructions. When the computer instructions run on the control device of the multi-loop fast steering mirror, the control device of the multi-loop fast steering mirror executes each step performed by the control method of the multi-loop fast steering mirror in the method flow shown in the above method embodiment.

[0098] In the above embodiment, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that contains one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0099] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0100] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A control method for a multi-loop fast steering mirror, characterized in that The method comprises: Collecting the current state of the reflector, the current state including: the current position of the reflector, the current angle error, and the last control amount applied to the reflector in the last control cycle, the current angle error being the difference between the target position and the current position; Determining corresponding current control parameters from a preset target control strategy table according to the current state, wherein the target control strategy table includes mapping relationships between the states of the plurality of reflectors and the control parameters, wherein the mapping relationships are formed by inputting the discretized states into a target adaptive strategy network to obtain corresponding control parameters, wherein the target adaptive strategy network is a neural network model based on a reinforcement learning deep deterministic policy gradient algorithm; According to the current control parameters and a preset multi-loop controller of the control system of the reflector, a current control quantity is calculated, and the reflector is controlled based on the current control quantity. The multi-loop controller includes an inner loop controller and an outer loop controller.

2. The method according to claim 1, wherein The fast reflector system is a high-order underdamped nonlinear system. Before collecting the current state of the reflector, the method further includes: Obtaining a mathematical model of the control system of the reflector in a complex domain; Based on the mathematical model, an outer loop controller is determined using a proportional-integral control method, and the reflector system is configured as a Type I system; Based on the mathematical model, the inner loop controller is determined by adopting the proportional differential control method.

3. The method according to claim 2, wherein The output of the outer loop controller is: in, is the first control parameter, is the second control parameter, is the angle error in the tth cycle; The output of the inner loop controller is: Among them, is the third control parameter, is the fourth control parameter, is the inner loop error.

4. The method according to claim 3, wherein Before collecting the current state of the reflector system, the method further includes: An adaptive policy network is constructed based on a deep deterministic policy gradient algorithm for reinforcement learning, wherein the adaptive policy network includes an action network, an evaluation network, an action target network, and an evaluation target network; The state of the adaptive strategy network includes: the position of the reflector, the angle error and the control amount; The actions of the adaptive strategy network include: the first control parameter, the second control parameter, the third control parameter and the fourth control parameter; The reward function of the adaptive strategy network includes multiple indicators of the control system, including rise time, overshoot, steady-state error and smoothness.

5. The method according to claim 4, characterized in that After constructing the adaptive policy network based on the reinforcement learning deep deterministic policy gradient algorithm, the method further includes: Initializing the network, experience buffer pool, random noise and state of the adaptive strategy network; Execution process, the process comprising: The action network is used to calculate the action corresponding to the initial state; the action is input into the multi-loop controller to obtain the corresponding control variable; the reflector is controlled based on the control variable to obtain the next state; the state is input into the reward function to obtain the reward value; sample data of the process of the current cycle is stored in the experience buffer pool, the sample data including the current state, current action, current reward value and next state; Repeat the above process until the amount of data in the experience buffer pool is greater than the preset amount of data. Then, collect multiple sample data from the experience buffer pool to train the adaptive policy network until the number of training rounds reaches the preset threshold, and obtain the target adaptive policy network; Input the discretized state into the target adaptive policy network to obtain the corresponding control parameters, and map and store the state and control parameters to obtain the target control strategy table.

6. The method according to claim 5, wherein The step of collecting multiple sample data from the experience buffer pool to train the adaptive policy network includes: Calculate the gradient of the evaluation function in the action network using the sample data; According to the gradient of the evaluation function, use the gradient ascent algorithm to update the network parameters of the action network; Calculate the next action corresponding to each state in the sample data through the action target network; Calculate the action value of the current state using the temporal difference method; Calculate the gradient of the loss function of the evaluation network according to the sample data, the next action, and the action value; Update the parameters of the evaluation network according to the gradient of the loss function.

7. The method according to claim 4, characterized in that, The step of calculating the current control amount according to the current control parameter and the multi-loop controller of the preset control system of the mirror, and controlling the mirror based on the current control amount includes: Input the first control parameter, the second control parameter, and the current angle error into the outer loop controller to obtain the first control amount; Use a tracking differentiator to calculate the speed of the mirror; Calculate the inner loop error according to the speed, the first control amount, and a preset feedback coefficient; Input the inner loop error, the third control parameter, and the fourth control parameter into the inner loop controller to calculate the current control amount.

8. A control device for a multi-loop fast steering mirror, characterized in that, The device includes: An acquisition module, configured to acquire the current state of the mirror, where the current state includes the current position of the mirror, the current angle error, and the previous control amount applied to the mirror in the previous control cycle, and the current angle error is the difference between the target position and the current position; A determination module, configured to determine the corresponding current control parameter from a preset target control strategy table according to the current state. The target control strategy table includes the mapping relationships between the states and control parameters of multiple mirrors, and the mapping relationships are formed by inputting the discretized states into the target adaptive policy network to obtain the corresponding control parameters. The target adaptive policy network is a neural network model based on the reinforcement learning deep deterministic policy gradient algorithm; A processing module, configured to calculate the current control amount according to the current control parameter and the multi-loop controller of the preset control system of the mirror, and control the mirror based on the current control amount. The multi-loop controller includes an inner loop controller and an outer loop controller.

9. An electronic device, characterized in that, It includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the control method of the multi-loop fast steering mirror according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed by a processor, the control method of the multi-loop fast steering mirror according to any one of claims 1 to 7 is implemented.

Images