Simulated pid beam pointing stabilization control system based on reinforcement learning optimization

Abstract

The application provides a simulation PID beam pointing stability control system based on reinforcement learning optimization, applied to the technical field of beam control, through deep fusion of a reinforcement learning algorithm and an adjustable hardware simulation PID control, the simulation PID circuit retains inherent advantages of continuous time operation, sub-microsecond ultra-low delay and non-quantization error, and can meet the demand for rapid response under high-frequency complex interference; through electronic adjustable resistance-capacitance elements, the system converts traditional static control into dynamic control with self-adaptive capability, effectively overcoming technical defects of fixed parameters and poor environmental adaptability of traditional simulation PID. In cooperation with an online training mechanism of reinforcement learning based on priority experience replay and hardware safety constraints, the system not only realizes optimization of optimal hardware parameters, but also guarantees safety and robustness in complex working conditions, and has the advantages of improving beam pointing control precision and stability.

Description

Technical Field

[0001] This application relates to the field of beam control technology, and in particular to a simulated PID beam pointing stabilization control system based on reinforcement learning optimization. Background Technology

[0002] In fields such as laser communication, precision laser processing, and astronomical observation, fast-reflecting mirror beam pointing stabilization systems are crucial for ensuring system performance. Currently, most mainstream beam pointing stabilization technologies employ fixed-parameter analog PID control or rely on digital processors for digital PID control. However, the core drawback of fixed-parameter analog PID controllers is that their PID parameters are determined by fixed resistor and capacitor values ​​in the hardware circuitry, and cannot be changed after system assembly. This fixed-parameter design results in extremely poor adaptability to complex and variable environments (such as changes in dynamic characteristics caused by temperature drift, vibration spectrum drift, etc.). When external disturbance conditions change, the system is prone to overshoot, oscillation, or slow response because it cannot dynamically adjust the PID parameters according to the real-time environment, leading to a significant decrease in control accuracy or even failure to meet the requirements of high-precision applications. Existing manual debugging or parameter setting methods based on static models are time-consuming and labor-intensive, making it difficult to achieve dynamic adaptive optimization of closed-loop parameters.

[0003] Therefore, existing technologies urgently need to be improved to address the aforementioned problems. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the analog PID beam pointing stabilization control system based on reinforcement learning optimization provided in this application is applied to the field of beam control technology. It aims to solve the technical problem that the existing analog PID control parameters are fixed and cannot adapt to complex and variable interference environments, thus causing a decrease in system control accuracy.

[0005] In a first aspect, a simulated PID beam pointing stabilization control system based on reinforcement learning optimization is provided, the system comprising: The deviation detection unit is used to acquire beam pointing deviation in real time and output the corresponding analog deviation signal; The analog control unit includes a signal input terminal and a signal output terminal. The signal input terminal is connected to the deviation detection unit. The analog control unit integrates an adjustable analog PID control circuit containing resistive and capacitive elements, which is used to continuously calculate the analog deviation signal and output a control voltage. A deflection execution unit, connected to the signal output terminal, is used to drive the mirror to deflect by a corresponding deflection angle according to the control voltage, so as to correct the beam direction; The parameter optimization unit, connected to the analog control unit, is used to collect system status data in real time, input the system status data into the reinforcement learning algorithm, and output the corresponding parameter configuration instructions to the analog control unit to adjust the resistance value of the resistive element and the capacitance value of the capacitive element. The system status data includes at least the analog deviation signal, the deflection angle, the resistance value of the resistive element, and the capacitance value of the capacitive element.

[0006] Furthermore, the parameter optimization unit is used to map the parameter configuration space formed by the resistance value of the resistive element and the capacitance value of the capacitive element in the adjustable analog PID control circuit into the action space of reinforcement learning, and to map the system state data into the state space of reinforcement learning. Online training is performed in a real hardware environment. The online training process is as follows: based on the current system state data, the parameter increments of the resistance and capacitance values ​​are output through a reinforcement learning strategy. After the parameter increments are executed, the system state data at the next moment is collected and the reward signal is calculated. The network parameters of the reinforcement learning strategy are iteratively updated based on the reward signal. Based on the parameter increment, a corresponding parameter configuration instruction is generated and sent to the analog control unit via a digital communication link to adjust the resistance value of the resistive element and the capacitance value of the capacitive element.

[0007] Furthermore, when the training module performs online training, it includes: Set the initial resistance value of the resistive element and the initial capacitance value of the capacitive element; In each iteration, the current system state data is collected and input into the reinforcement learning strategy to output the parameter increment. The parameter configuration instruction is generated based on the parameter increment and sent to the analog control unit to complete the switching of resistance and capacitance values. After the system responds, the system state data at the next moment is collected and the reward signal is calculated. The current system state data, the parameter increment, the reward signal, and the system state data at the next moment are stored as a set of experience data in the experience replay pool. The network parameters of the reinforcement learning strategy are updated by sampling the experience data from the experience replay pool. After completing one round of training, the average reward value of all iterations in that round of training and the statistical value of the simulated deviation signal are calculated. If the average reward value of multiple consecutive rounds of training no longer increases and the statistical value of the beam deviation meets the preset accuracy condition, then the training is determined to be converged. After training convergence, the last updated network parameters are stored in non-volatile memory, which is used to output the corresponding parameter increments of the resistance and capacitance values ​​based on the system state data during actual operation.

[0008] Furthermore, when the training module performs online training, it also includes: In each iteration, before the reinforcement learning policy outputs the parameter increment, random noise is superimposed on the output parameter increment, and the amplitude of the random noise decreases as the training rounds increase. When updating the network parameters of the reinforcement learning strategy by sampling experience data from the experience replay pool, a priority experience replay mechanism is adopted, and the sampling weight of the experience data is determined according to the magnitude of the temporal difference error of the experience data.

[0009] Furthermore, when the training module performs online training, it also includes: In each iteration, the resistance and capacitance values ​​are limited to the minimum and maximum values ​​allowed by their respective hardware. The single-step adjustment amount of each item in the parameter increment is limited to not exceeding the preset maximum allowable single-step adjustment amount; When the simulated deviation signal exceeds the preset deviation safety limit, the resistance value and the capacitance value are restored to the initial resistance value and the initial capacitance value, and the corresponding empirical data is marked as abnormal data. When updating the network parameters of the reinforcement learning strategy, its sampling weight is reduced.

[0010] Furthermore, the resistive element is a digital potentiometer in the form of an integrated circuit, which internally consists of multiple resistors connected in series to form a voltage divider network, and controls the internal switch to select the tap to be connected through a digital communication interface, so as to realize the adjustable resistance value. The capacitor element is a programmable capacitor array, which consists of multiple capacitors that are switched by analog switches to achieve adjustable capacitance values.

[0011] Furthermore, the deviation detection unit includes: A beam position detector, which is a four-quadrant detector or a position-sensitive detector, is used to collect beam pointing deviation signals. A signal conditioning circuit, connected to the beam position detector, is used to amplify and filter the pointing deviation signal to output the analog deviation signal.

[0012] Furthermore, the deflection execution unit includes: Piezoelectric ceramic driven fast-reflecting mirror; A high-voltage drive circuit, the input of which is connected to the analog control unit and the output of which is connected to the fast-reflecting mirror, is used to convert the control voltage into a drive signal that drives the fast-reflecting mirror to generate the deflection angle.

[0013] Furthermore, the system is applied at least to ground terminals of satellite-to-ground laser communication links, precision laser processing platforms, and extreme ultraviolet laser generating devices.

[0014] Furthermore, when the system is applied to an extreme ultraviolet laser generating device, the extreme ultraviolet laser generating device has multiple series-connected sub-optical path components, each of which is equipped with the system, and each system is independently trained online.

[0015] Beneficial Effects: The analog PID beam pointing stabilization control system based on reinforcement learning optimization proposed in this application deeply integrates reinforcement learning algorithms with adjustable hardware analog PID control, constructing a closed-loop control system with real-time condition perception and autonomous evolution capabilities. The analog PID circuit retains the inherent advantages of continuous-time operation, sub-microsecond ultra-low latency, and zero quantization error, meeting the rapid response requirements under high-frequency complex interference. Through electronically adjustable RC components, the system transforms traditional static control into adaptive dynamic control, effectively overcoming the technical shortcomings of traditional analog PID with fixed parameters and poor environmental adaptability. Combined with a reinforcement learning online training mechanism based on priority experience playback and hardware safety constraints, the system not only achieves optimal hardware parameters but also ensures safety and robustness under complex operating conditions, improving the accuracy and stability of beam pointing control. Attached Figure Description

[0016] Figure 1 This is a structural diagram of a simulated PID beam pointing stabilization control system based on reinforcement learning optimization proposed in this application.

[0017] Figure 2 This paper presents a flowchart of the online training process for a reinforcement learning strategy in a simulated PID beam pointing stabilization control system based on reinforcement learning optimization, as proposed in this application.

[0018] Labeling explanations: 101, Deviation detection unit; 102, Analog control unit; 103, Deflection execution unit; 104, Parameter optimization unit. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and marked in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] In fields such as laser communication, precision laser processing, and astronomical observation, the stability of beam pointing is crucial for ensuring the performance of the entire system. However, real-world working environments often involve complex and variable interference factors such as high-frequency mechanical vibration, atmospheric turbulence, and temperature drift, which can cause minute but rapid beam deviations. Traditional digital PID control schemes, involving multiple stages such as analog-to-digital conversion, software algorithm calculation, and digital-to-analog conversion, suffer from significant timing delays, making it difficult to respond quickly and effectively to high-frequency interference. While analog PID control circuits built with fixed hardware components offer the advantage of ultra-low latency, their control parameters cannot be changed once set at the hardware level, resulting in extremely poor adaptability to complex and dynamic interference environments.

[0021] Therefore, there is an urgent need for a beam pointing stabilization control system that can maintain the ultra-low delay characteristics of analog circuits and dynamically and adaptively adjust control parameters according to real-time operating conditions.

[0022] For this, please refer to Figure 1 This application proposes a simulated PID beam pointing stabilization control system based on reinforcement learning optimization. The system includes: The deviation detection unit is used to acquire beam pointing deviation in real time and output the corresponding analog deviation signal; The analog control unit includes a signal input terminal and a signal output terminal. The signal input terminal is connected to the deviation detection unit. The analog control unit integrates an adjustable analog PID control circuit containing resistive and capacitive elements, which is used to continuously calculate the analog deviation signal and output a control voltage. The deflection actuator, connected to the signal output terminal, is used to drive the mirror to deflect by the corresponding deflection angle according to the control voltage, so as to correct the beam direction. The parameter optimization unit, connected to the analog control unit, is used to collect system status data in real time, input the system status data into the reinforcement learning algorithm, and output the corresponding parameter configuration instructions to the analog control unit to adjust the resistance value of the resistive element and the capacitance value of the capacitive element. The system status data includes at least the analog deviation signal, deflection angle, and the resistance value of the resistive element and the capacitance value of the capacitive element.

[0023] Analog PID control circuits refer to control circuits built using purely hardware electronic components such as operational amplifiers, resistors, and capacitors. They directly perform continuous-time proportional, integral, and derivative operations on analog electrical signals without the need for discretization sampling and digital calculations. More specifically, the proportional element provides an immediate response to the current deviation; the larger the deviation, the stronger the output correction. The integral element accumulates and corrects persistent small deviations to reduce long-term steady-state errors. The derivative element reflects the trend of deviation changes and applies suppression in advance when the deviation changes rapidly, thereby reducing overshoot and oscillation. For example, when a beam momentarily shifts in a certain direction due to platform vibration, the proportional element first quickly outputs a correction voltage, the integral element gradually compensates for residual errors as the deviation persists, and the derivative element applies damping in advance when the shift speed is relatively fast. The three elements work together to achieve rapid and smooth pointing recovery.

[0024] Reinforcement learning is a machine learning method whose core idea is that an agent interacts with its environment and continuously optimizes its action strategy based on reward signals from the environment, aiming to obtain the maximum long-term cumulative reward. In the application scenario of this application, the environment in reinforcement learning can be understood as a real control object composed of a deviation detection unit, a simulated control unit, a deflection execution unit, and external disturbances; the state can be understood as a representation of the current system operation, such as the magnitude of the deviation signal, the rate of deviation change, the current deflection of the mirror, and the current resistor-capacitor configuration; the action can be understood as the specific instructions for adjusting the resistive and capacitive elements; and the reward is used to evaluate whether the adjustment makes the beam more stable, the system more stable, and whether the parameter changes are too drastic. Through continuous interaction, the reinforcement learning algorithm gradually forms a parameter adjustment strategy more suitable for the current operating conditions.

[0025] The system's overall operating logic forms a dual-closed-loop structure. The inner loop is a simulated hardware closed loop, where the deviation detection unit senses the physical offset of the beam in space in real time and outputs a simulated deviation signal. The simulated control unit directly performs continuous PID calculations on this simulated signal to generate a control voltage. The deflection execution unit receives the control voltage and drives the mirror to deflect, thereby completing the physical correction of the beam's direction in a very short time. The outer loop is an intelligent parameter optimization closed loop, where the parameter optimization unit continuously monitors the operating status of the inner loop (i.e., system status data), continuously evaluates the control effect of the current hardware parameters through reinforcement learning algorithms, and dynamically outputs parameter configuration instructions to change the physical parameters of the resistive and capacitive elements in the simulated control unit in real time, thereby changing the proportional, integral, and derivative gains of the PID controller.

[0026] Furthermore, the inner loop is primarily responsible for correction, focusing on minimizing the time between the occurrence of a deviation and the actuator's action; the outer loop is primarily responsible for continuous tuning, focusing on progressively optimizing the parameter configuration of the analog PID control circuit based on the current disturbance characteristics, actuator response characteristics, and control effect. The two are not mutually exclusive but rather work in synergy: the inner loop ensures real-time control, while the outer loop guarantees adaptability. Because the parameter optimization unit does not directly replace the analog control unit in real-time control but indirectly alters the control characteristics of the analog PID control circuit by adjusting RC parameters, it retains the low-latency advantage of analog control while avoiding the problem of fixed parameters failing to accommodate various operating conditions in complex environments.

[0027] The core function of the deviation detection unit is to convert the physical offset of the light beam in space into an analog electrical signal that can be processed by the electrical system. In one possible example, a basic photodiode array combined with a simple transimpedance amplifier circuit can be used. When the position of the light beam illuminating the photodiode array changes, the photocurrent output by each diode changes accordingly. The transimpedance amplifier circuit converts these weak photocurrents into voltage signals, thereby outputting an analog deviation signal.

[0028] To describe the process more clearly and completely, a four-quadrant photodetector structure can be used as a specific implementation. After being shaped by the pre-stage optical system, the light beam illuminates four adjacent photosensitive regions. When the light spot is located at the center, the light energy received by each region is basically balanced; when the light spot shifts in a certain direction, the output current of the photosensitive region on one side of the corresponding direction increases, while the output current on the other side decreases.

[0029] The deviation detection unit can perform differential processing on the photocurrents on opposite sides to obtain analog deviation signals characterizing the X-axis offset trend and analog deviation signals characterizing the Y-axis offset trend. The transimpedance amplifier circuit converts the photocurrent into a stable voltage signal that is easy for subsequent stages to process. At the same time, by properly setting the feedback resistor and compensation capacitor, the output signal has sufficient sensitivity and stability within the target frequency band.

[0030] Furthermore, to avoid significant interference to the deviation signal caused by ambient stray light, detector dark current drift, or changes in background illumination, the deviation detection unit can also incorporate an analog filtering stage and a baseline correction stage after transimpedance amplification. The analog filtering stage suppresses high-frequency noise or low-frequency drift unrelated to target disturbances, while the baseline correction stage establishes a zero-point reference at a deviation-free or known reference position. The resulting analog deviation signal not only reflects the beam offset direction but also reliably reflects the degree of offset, providing a continuous and usable input for subsequent analog PID control.

[0031] For analog control units, the core lies in adjustable analog PID control circuits. To achieve dynamic adjustment of resistors and capacitors, in one possible example, a relay array can be used to switch between fixed discrete components with different resistance and capacitance values. By controlling the activation and deactivation of different relays, specific resistors and capacitors are connected to the feedback or input loop of the operational amplifier, thereby changing the circuit's operational parameters.

[0032] To make this implementation more feasible, the adjustable analog PID control circuit can be composed of a proportional branch, an integral branch, and a differential branch, each corresponding to a different resistor-capacitor network. The proportional branch typically changes the proportional gain by altering the combination of the input and feedback resistors; the integral branch typically uses resistors and capacitors to form an integral network to determine the cumulative correction strength for persistent deviations; and the differential branch typically uses resistors and capacitors to form a differential network to determine the sensitivity to changes in deviation trends.

[0033] In the relay array implementation, multiple sets of fixed-value resistors and capacitors can be pre-set and connected to the switchable nodes of the corresponding branches. After the parameter optimization unit issues parameter configuration commands, the drive circuit in the analog control unit controls the corresponding relays to operate, connecting the target resistor-capacitor combination to the circuit. In this way, although the analog PID control circuit still maintains pure analog continuous operation during the control process, its key parameters have reconfigurable capabilities.

[0034] Furthermore, to avoid introducing significant spikes or causing short-term instability in the control loop during relay switching, a disconnect-then-connect switching sequence can be set in the switching logic, or the switching can be performed at a time when the deviation signal is small and the system is relatively stable during parameter updates. If a dual-group parallel RC network is used, the target parameters can be connected in the backup branch first, and then the main branch can be switched through a selector switch to reduce switching disturbances.

[0035] Besides relay arrays, some implementations can also use digital potentiometers, analog switch arrays combined with resistor-capacitor networks, variable capacitor arrays, etc. to achieve resistance and capacitance parameter adjustment. However, the basic idea is still to change the actual resistance and capacitance combination in the analog PID control circuit by changing the external parameter configuration command, thereby changing the control characteristics.

[0036] The core approach of the parameter optimization unit is to use reinforcement learning algorithms to process system state data and output instructions. Specifically, the parameter optimization unit maps the parameter configuration space, consisting of the resistance values ​​of the resistors and the capacitance values ​​of the capacitors in the adjustable analog PID control circuit, to the action space of reinforcement learning, and maps the system state data to the state space of reinforcement learning.

[0037] Online training is performed in a real hardware environment. The online training process is as follows: based on the current system state data, the parameter increments of resistance and capacitance values ​​are output through the reinforcement learning strategy. After the parameter increments are executed, the system state data at the next moment is collected and the reward signal is calculated. The network parameters of the reinforcement learning strategy are iteratively updated based on the reward signal. Based on the parameter increment, corresponding parameter configuration instructions are generated and sent to the analog control unit via a digital communication link to adjust the resistance value of the resistive element and the capacitance value of the capacitive element.

[0038] Among them, the PID controller is the most widely used closed-loop control algorithm in the field of industrial control. Its continuous-time control law is expressed as follows: ,in, To control the output, This is the deviation signal (the difference between the set value and the actual value). , , These are the proportional gain, integral gain, and derivative gain, respectively. The integral variable τ represents the continuous-time independent variable from the initial time 0 to the current time t, used to analyze the deviation signal. Perform time accumulation calculation.

[0039] In analog circuits, the above calculations are directly implemented using operational amplifiers and hardware components such as resistors and capacitors, without any digital processing. A typical analog PID circuit uses an inverting input operational amplifier topology. The proportional (P) stage is implemented using an inverting amplifier. ,in For feedback resistor, The input resistance is [value]. The proportional element linearly amplifies the current deviation signal, quickly generating a corrective effect, and is a major determinant of the control system's response speed. The larger the value, the faster the system responds to deviations, but an excessively large value... This can lead to increased system overshoot or even instability.

[0040] Integration (I) stage: Implemented through an integrator circuit. ,in For input resistance, This is the integrating capacitor. The integrator accumulates the deviation signal over time, eliminating the system's steady-state error (static error). It has excellent suppression effects on long-term drift caused by the hysteresis and creep characteristics of piezoelectric ceramic actuators. Integrating time constant. This determines the strength of the integral's effect.

[0041] Differential (D) stage: Implemented by combining an RC high-pass network and an inverting amplifier. ,in It is a differential resistor. This is a differential capacitor. The differential element is sensitive to the rate of change of the deviation, can predict the trend of deviation change, and generate reverse correction in advance, effectively suppressing system overshoot and oscillation, and improving the dynamic response quality of the system.

[0042] The outstanding advantages of analog PID circuits are: real-time and continuous processing of analog electrical signals, no sampling intervals or discretization errors, signal transmission and processing are entirely determined by the circuit's own propagation delay (typically sub-microsecond level), which is far lower than the millisecond delay of digital schemes, and the dynamic range of analog signal amplitude is not limited by the number of quantization bits, theoretically enabling extremely high control resolution.

[0043] In this system, the analog PID circuit serves as the core computing unit of the fast-reflecting mirror deflection controller, undertaking the real-time closed-loop control task from deviation detection to drive execution. The specific function of the proportional (P) element is as follows: the beam deviation electrical signal output by the four-quadrant detector is transmitted through the input resistor. The signal enters the inverting amplifier. The electronically adjustable programmable feedback resistor is then adjusted. The resistance value can change the proportional gain. The size. When the beam suddenly deviates, the larger This allows the control voltage to increase rapidly, driving the piezoelectric ceramic to generate a sufficiently large deflection torque, thus achieving initial and rapid correction of the deviation. The resistance value can be dynamically adjusted within a wide range, corresponding to It is continuously adjustable over a wide dynamic range.

[0044] The specific function of the integrator (I) stage is to continuously integrate the deviation signal over time. Even under static conditions where the deviation signal is extremely small and the proportional element cannot respond effectively, the integrator can continuously accumulate and generate a gradually increasing correction signal, thereby eliminating the creep, hysteresis nonlinearity, and long-term slow drift caused by thermal deformation of the optical path structure in the piezoelectric ceramic actuator. Integrating capacitor. The capacitance value can be adjusted within a few microfarads, in conjunction with the input resistor. Adjustable integral time constant It varies over a wide range to accommodate different types of disturbances on different time scales.

[0045] The specific function of the differential (D) stage is to extract rapidly changing components in the deviation signal through an RC series high-pass network. When the beam is subjected to high-frequency vibration interference (such as the mechanical vibration of a cooling pump), the rate of change of the deviation signal increases sharply. The differential stage generates a correction signal proportional to the rate of change in advance, applying reverse drive before the deviation further expands, effectively suppressing overshoot and oscillation, and ensuring a smooth deflection process. Differential resistor and differential capacitance Together they determine the differential time constant This, in turn, determines the sensitivity of the differential action and the frequency response range.

[0046] For applications requiring extremely high real-time performance, such as fast-reflecting mirror beam pointing control, the low latency and high bandwidth advantages of analog PID control are irreplaceable by digital PID control. This is the fundamental reason why this invention insists on using analog PID control as the core control architecture.

[0047] To achieve dynamic adjustment of analog PID parameters, this invention uses electronically adjustable programmable resistors and electronically adjustable programmable capacitors to replace traditional fixed resistor-capacitor components and mechanical potentiometers: Electronically adjustable programmable resistors (digital potentiometers) are digital potentiometers in the form of integrated circuits. Internally, they consist of multiple precision resistors connected in series to form a resistor divider network, with each tap selected by a MOSFET switch. Alternatively, control words can be written via the SPI digital interface to set the resistance value to a specified value within microseconds. It offers sufficiently high resolution (multi-bit precision), and the resistance range covers the range required by the circuit design. Electronically adjustable programmable resistors have outstanding advantages such as no mechanical contacts, long lifespan, good temperature stability, and remote digital control.

[0048] Electronically adjustable programmable capacitors (programmable capacitor arrays) are implemented by switching multiple precision capacitors through a combination of analog switches. Each capacitor branch is independently controlled by an analog switch, and the on / off state of each branch is determined by configuration words written through a digital interface, thereby controlling the total equivalent capacitance value. The capacitance adjustment range covers the integral and derivative requirements of analog PID control, and the resolution meets the needs of fine-tuning.

[0049] The parameter optimization unit synchronously collects system state data at an appropriate sampling frequency to construct a multi-dimensional state vector. ,in, , These are the position deviation signals of the beam output by the four-quadrant detector in the X and Y directions, respectively.

[0050] , : These are the current deflection angles of the fast-reflecting mirror on the X and Y axes, respectively. These deflection angles can be obtained by converting the feedback voltage of the piezoelectric ceramic actuator.

[0051] This is the current configuration state vector of the electronically adjustable programmable resistive element, which includes the proportional resistor. Differential resistance The current value or numeric configuration word encoding.

[0052] This is the current configuration state vector of the electronically adjustable programmable capacitor element, which includes the integrating capacitor. Differential capacitor The current value or numeric configuration word encoding.

[0053] This state-space design fully depicts the beam pointing deviation, actuator operating point, and hardware parameter configuration of the simulated PID at the current moment.

[0054] The action space of a reinforcement learning strategy directly corresponds to the resistance and capacitance adjustment operations of electronically adjustable programmable resistors and capacitors, specifically: .

[0055] in, It is the standard mathematical notation for an action in a reinforcement learning policy, and in this application, it represents the action vector output by the reinforcement learning at the current time t.

[0056] These represent the proportional resistors. Integrating capacitor Differential resistance and differential capacitance The parameter increments. The action values ​​output by reinforcement learning are normalized and mapped, then converted into digital interface control words for the corresponding digital potentiometers and programmable capacitor arrays, and sent to the deflection execution unit via the digital communication bus.

[0057] The deflection execution unit's built-in digital interface controller parses the received instruction frames, extracts the target resistance and capacitance values ​​as configuration parameters, and then... Alternatively, the configuration word can be written to the registers of the corresponding digital potentiometer and programmable capacitor array via the SPI bus. Upon receiving the write command, the internal analog switch of the digital potentiometer completes tap switching within a very short time, updating the resistance value; similarly, the analog switch of the programmable capacitor array completes on / off switching of each capacitor branch within a very short time, updating the equivalent capacitance value. The total time for a single parameter switching is much shorter than the system response time, introducing no perceptible control delay.

[0058] After the parameters are switched, the beam detection module immediately senses the change in beam direction, and the reinforcement learning strategy calculates the reward signal based on the new system state data: ,in, This is a reward signal. The specific form of the reward function is common knowledge in the field of reinforcement learning.

[0059] Let t be the pointing deviation signal of the beam in the X direction at time t, i.e., the analog deviation signal.

[0060] Let t be the pointing deviation signal of the beam in the Y direction at time t.

[0061] The current deflection angle of the fast-reflecting mirror on the X-axis at time t.

[0062] The current deflection angle of the fast-reflecting mirror on the Y-axis at time t.

[0063] These are the action vectors output by the reinforcement learning policy at time t and the previous time t-1, respectively.

[0064] The reward function comprehensively considers the following factors: the degree of suppression of beam pointing deviation (primary target), the speed of control response (dynamic performance indicator), and the smoothness of parameter adjustment (preventing drastic jumps that could cause system oscillations). This reward signal serves as the direct basis for the reinforcement learning module to update its strategy, guiding the agent to continuously seek better hardware parameter configurations (resistance and capacitance values).

[0065] The above process repeats itself cyclically, forming a complete closed-loop optimization iteration. In each iteration, the reinforcement learning module interacts with the environment (a real physical system consisting of simulated PID hardware and fast-reflection mirror actuators) to obtain new empirical data and gradually optimize its policy network. After a sufficient number of iterations, the reinforcement learning module can autonomously learn the optimal parameter configuration strategy for the electronically adjustable resistor and capacitor values ​​under different disturbance conditions, achieving intelligent adaptive adjustment of the simulated PID parameters. Please refer to the appendix for details. Figure 2 .

[0066] The reinforcement learning training method of the present invention follows the following design principles: Adhering to the principle of algorithm independence, the training framework is designed to be compatible with a variety of mainstream reinforcement learning algorithms (such as DDPG, PPO, SAC, TD3, etc.), and can be flexibly selected and replaced according to actual application scenarios. The system solution is not bound to any specific algorithm.

[0067] The principle of direct online training in real-world environments eliminates the complex process of pre-training in virtual environments. Reinforcement learning agents are trained directly on real hardware platforms online, avoiding the additional overhead and performance loss caused by system identification, modeling, and sim-to-real transfer.

[0068] The principle of safety constraints is to set up hardware safety constraint mechanisms during training, including motion amplitude limits, parameter boundary protection, and emergency recovery from abnormal states, to ensure that online training will not damage hardware devices such as fast-reflecting mirrors.

[0069] Specifically, when the training module performs the online training process, it includes: Set the initial resistance value of the resistor element and the initial capacitance value of the capacitor element; In each iteration, the current system state data is collected and input into the reinforcement learning strategy to output parameter increments. Parameter configuration instructions are generated based on the parameter increments and sent to the analog control unit to complete the switching of resistance and capacitance values. After the system responds, the system state data at the next moment is collected and the reward signal is calculated. The current system state data, parameter increments, reward signals, and the system state data at the next moment are stored as a set of empirical data in the empirical replay pool. The empirical data is sampled from the empirical replay pool to update the network parameters of the reinforcement learning strategy. After completing one round of training, calculate the average reward value of all iterations in that round of training and the statistical value of the simulated deviation signal. If the average reward value of multiple consecutive rounds of training no longer increases and the statistical value of the beam deviation meets the preset accuracy condition, then the training is considered to have converged. After training convergence, the last updated network parameters are stored in non-volatile memory, which is used to output the corresponding resistance and capacitance values ​​as parameter increments based on system state data during actual operation.

[0070] Specifically, the online training process of the reinforcement learning strategy provided in this application on a real hardware platform is as follows: First, set the initial parameter values ​​of the electronically adjustable programmable resistor and capacitor (safe default configuration); establish a digital communication link (RS485 / ) between the reinforcement learning strategy and the fast-reflecting mirror deflection controller. / SPI), test communication is normal; initialize the policy network parameters of the reinforcement learning agent (randomly initialize or load pre-trained prior parameters); set training hyperparameters: maximum number of iterations K in a single training round, total number of training rounds M, experience replay pool capacity Nreplay, discount factor γ, etc.

[0071] In each iteration of each training round, the following interaction process is performed: Construct the current multidimensional state vector: .

[0072] The agent's policy network is based on the current state Output action: .

[0073] Actions The configuration words, mapped to digital potentiometers and programmable capacitor arrays, are sent to the controller via the digital communication bus, and the controller completes the hardware parameter switching in a very short time.

[0074] Wait for the fast-reflection mirror control system to respond and stabilize, and then acquire the multidimensional state vector at the next time step t+1. .

[0075] Then according to , , Calculate instant rewards And the tuples of empirical data Store in the experience replay pool.

[0076] Finally, a small batch of experience data is randomly sampled from the experience replay pool to update the agent's policy network and value network parameters.

[0077] In the process of iterative advancement, let Return to the previous step to construct the current multidimensional state vector and continue execution until the maximum number of iterations in a single round is reached. .

[0078] Complete one round of training ( After each iteration, the average reward value and beam deviation statistics for that round are calculated. If the average reward value no longer increases significantly over multiple consecutive rounds and the beam deviation statistics meet the system accuracy requirements, then the training is considered to have converged and can be terminated. Otherwise, start the next round of training, repeating the iterative interaction process, for a total of [number] training rounds. Second-rate.

[0079] After training converges, the optimal policy network parameters are stored in the non-volatile memory of the embedded processor.

[0080] In actual operation, the reinforcement learning module uses the solidified policy network for forward inference (without updating the network parameters) to achieve real-time optimal PID hardware parameter configuration; If a significant change in operating conditions is detected (such as a sudden change in the statistical characteristics of beam deviation), the online training process can be selectively restarted to allow the strategy to adapt to the new operating conditions.

[0081] Furthermore, the training module, during the online training process, also includes: In each iteration, before the reinforcement learning policy outputs the parameter increment, random noise is superimposed on the output parameter increment, and the amplitude of the random noise decreases as the training rounds increase. When sampling empirical data from the experience replay pool to update the network parameters of the reinforcement learning strategy, a priority experience replay mechanism is adopted, and the sampling weight of the empirical data is determined according to the magnitude of the temporal difference error of the empirical data.

[0082] During online training, the agent needs to strike a balance between maintaining system stability using the current optimal parameters and exploring unknown parameters to discover better configurations. To this end, the output actions of the policy network are superimposed with an appropriate amount of random noise (such as Ornstein-Uhlenbeck noise or Gaussian noise). The noise amplitude gradually decreases with the increase of training rounds, encouraging exploration in the initial stage and focusing on utilization in the later stage.

[0083] When conducting online training directly in a real-world environment, sample acquisition costs are high (each interaction requires waiting for a response from the physical system), thus necessitating optimization of sample utilization efficiency. Specifically, an experience replay mechanism can be employed, repeatedly utilizing historical experience data for multiple policy updates to improve data utilization.

[0084] By adopting a batch sampling approach, the strategy network is updated multiple times after each batch of experience data is collected, maximizing the information extraction efficiency of each interaction. Furthermore, the Prioritized Experience Replay technique is employed to assign higher sampling weights to experiences with abnormal rewards or large temporal difference errors, thereby accelerating the learning process.

[0085] In one specific embodiment, during the online training process of the satellite-to-ground laser communication ground terminal, in the early stage of training, the training module superimposes a Gaussian random noise with a standard deviation of 3 adjustment levels onto the calculated parameter increment, so that the system can widely test the anti-turbulence effect under different PID gains.

[0086] Here, "adjustment level" refers to the discrete parameter step unit corresponding to the switching of digital potentiometer taps or the combination switching of programmable capacitor arrays. For example, if a digital potentiometer has multiple selectable taps, a change in one tap can be defined as one adjustment level; if the programmable capacitor array switches in binary combinations, a change in the minimum effective capacitance value can be defined as one adjustment level. Using a noise standard deviation of "3 adjustment levels" means that most exploration actions remain concentrated near the current strategy, not straying too far from the controllable range, yet sufficient to cover several adjacent sets of RC combinations. During actual status data acquisition, the ground terminal can obtain the offset information of the currently received beam on the focal plane through a spot position detector, obtain platform jitter reference values ​​through attitude sensors or auxiliary stabilization modules, and combine this with the current RC configuration to form system status data.

[0087] After each training round, the standard deviation of the Gaussian noise is multiplied by 0.95 (attenuation factor). This setting aims to retain strong exploration capabilities in the early stages while gradually compressing the exploration into a finer local scope in later stages, which is beneficial for stable convergence. Simultaneously, when a sudden strong crosswind causes a drastic beam deflection, the system records this set of empirical data.

[0088] Because the current strategy failed to effectively address this sudden situation, the calculated temporal difference error was extremely large. In subsequent empirical replay sampling, this set of data containing strong crosswind disturbance characteristics was assigned a very high sampling probability. The neural network therefore performed multiple targeted weight updates on this data, quickly mastering the parameter adjustment rules for dealing with such sudden strong disturbances. To further ensure training stability, it is preferable to re-evaluate the error after each update using this type of high-priority data. If the error has significantly decreased, its priority is lowered accordingly, avoiding the system from focusing excessively on problems it has already mastered for an extended period.

[0089] Considering the physical limitations of fast-reflecting mirror piezoelectric ceramic actuators, the following safety protections are implemented during training: In each iteration, the resistance and capacitance values ​​are limited to the minimum and maximum values ​​allowed by their respective hardware. The single-step adjustment amount of each item in the limit parameter increment shall not exceed the preset maximum allowable single-step adjustment amount; When the simulated deviation signal exceeds the preset deviation safety limit, the resistance and capacitance values ​​are restored to their initial values, and the corresponding empirical data is marked as abnormal data. When updating the network parameters of the reinforcement learning strategy, its sampling weight is reduced.

[0090] Specifically, limiting the values ​​to the minimum and maximum allowed by each hardware component can be achieved by establishing a safety table for resistance and capacitance parameters within the training module. This safety table can be determined during the device design phase based on the digital potentiometer datasheet, the withstand capability of the programmable capacitor array switch, the stable operating range of the operational amplifier, and the permissible drive range of the actuator.

[0091] For example, if a digital potentiometer causes excessive branch current when the resistance is too low, or if a certain combination of capacitors causes the integral circuit to respond too slowly or even become unstable, then these regions are not included in the adjustable range.

[0092] The second layer is the rate of change limitation. By constraining the maximum allowable adjustment amount per step, it prevents control voltage jumps caused by parameter abrupt changes, thereby avoiding high-frequency shocks to the actuator. In implementation, after each strategy output parameter increment, the change amount of each parameter is compared item by item to see if it exceeds a preset threshold. If it does, it is truncated to the allowable range before being issued again. This preset threshold is preferably determined through equipment commissioning experiments: starting with a small step amount, observing whether the system has obvious overshoot, ringing, or mechanical shock, and then gradually relaxing it to a level that balances adjustment speed and safety.

[0093] The third layer is a dynamic safety circuit breaker and penalty mechanism. When the system detects that the beam deviation is about to get out of control (reaching the safety limit), it immediately strips the algorithm of its control and forces the hardware to return to the known safe initial state. More importantly, the actions that lead to loss of control are marked as abnormal and their weight is reduced (or given a huge negative reward) during experience replay, fundamentally suppressing the algorithm's tendency to try such dangerous actions again.

[0094] Specifically, the deviation signal exceeding the preset safety limit can be detected through real-time sampling by an analog comparator, a mixed-signal monitoring circuit, or a controller. To improve response speed, it is preferable to set up an independent safety comparison channel at the hardware level, directly sending the deviation signal to the comparator for comparison with a preset reference voltage; once the threshold is exceeded, the comparator outputs an interrupt signal to trigger the reset logic, without waiting for the main control software to poll and detect the anomaly.

[0095] When restoring to the initial resistance and capacitance values, the digital potentiometer and programmable capacitor array can be directly switched back to the safe configuration via a hardware preset code table or reset register, reducing software involvement and ensuring recovery in a very short time. When marking corresponding empirical data as abnormal data, it is preferable to add an abnormal flag field, an alarm type field, and a trigger time to the empirical structure so that the data can be identified during subsequent network updates.

[0096] Reducing its sampling weight does not mean completely deleting the data, as it still has significant value: it tells the system which combinations of states and actions might lead to danger. By reducing its sampling weight and combining it with a stronger penalty, the algorithm can remember the danger boundaries, but this avoids outliers dominating the training process and causing the strategy to become overly conservative.

[0097] In a specific embodiment, in the beam stabilization system of an extreme ultraviolet laser generation device, the proportional resistor of the analog control circuit is strictly limited to between 5000 ohms and 100 kiloohms, and the integrating capacitor is limited to between 0.1 microfarads and 10 microfarads. Simultaneously, it is stipulated that the resistor adjustment in each iteration must not exceed 2000 ohms. These ranges can be determined during the device integration phase: first, without integrating the reinforcement learning strategy, the effective ranges of the proportional and integrating branches are gradually scanned using traditional control methods, recording the parameter range where the control effect is stable, the operational amplifier is not saturated, and the actuator does not exceed limits. This range is then used as the absolutely safe range for online training.

[0098] In one training exploration, the algorithm attempted to increase the resistance by 10,000 ohms in a single step. This instruction was initially intercepted by the second constraint and truncated to a 2,000-ohm increase. The training module did not discard this action directly but recorded the difference between the original suggested action and the actual executed action. This allowed subsequent analysis to distinguish whether the algorithm itself was too aggressive or whether the hardware constraints were acting as a protective mechanism. If, at another moment, due to improper parameter combinations, the beam deviation signal momentarily spiked to 4.5 volts near the detector edge (the preset safety limit is 4.0 volts), a system hardware interrupt was immediately triggered, resetting the relays or switch array to the factory default of 20,000 ohms and 1 microfarad within microseconds.

[0099] The detector edge voltage is used as the safety criterion because when the light spot approaches the effective boundary of the detector, the position detection will rapidly become distorted, and further training will no longer be reliable. Therefore, prioritizing the stability of the hardware and optical path is essential. Subsequently, the training module labels the state and action pair that caused the spike in deviation as a danger zone. In subsequent network updates, the algorithm model will automatically avoid this parameter danger zone. To improve traceability, the system preferably records the state before and after the circuit breaker is triggered, the source of the circuit breaker trigger, the recovery time, and the deviation changes over several cycles after recovery. This information is used for subsequent safety boundary verification and strategy optimization.

[0100] Furthermore, the resistive element is a digital potentiometer in the form of an integrated circuit, which internally consists of multiple resistors connected in series to form a voltage divider network, and controls the internal switch to select the tap to be connected through a digital communication interface, so as to realize the adjustable resistance value; The capacitor element is a programmable capacitor array, which consists of multiple capacitors that are switched by analog switches to achieve adjustable capacitance value.

[0101] This specific implementation plan clarifies the core hardware components of the adjustable analog PID control circuit. A digital potentiometer is a mixed-signal integrated circuit manufactured using semiconductor technology. It contains an array of multiple precision miniature resistors connected in series, with CMOS solid-state switches connecting the nodes.

[0102] Through external SPI or Once the control word is written into the digital communication interface, a specific CMOS switch can be turned on instantly, changing the equivalent resistance value of the connected circuit.

[0103] Tap connection can be understood as selecting one node from multiple internal fixed resistor nodes as the output terminal, thereby obtaining different equivalent resistance values.

[0104] If the digital potentiometer is non-volatile, it can retain the most recent setting value after power failure, making it suitable for scenarios requiring parameter retention even after power loss. If the digital potentiometer is volatile, the target resistance value can be rewritten by the main controller during power-on initialization. Programmable capacitor arrays employ a similar principle, arranging multiple high-quality capacitors of different capacitance values ​​in parallel, with a low-on-resistance, low-leakage-current analog switch chip connected in series in each capacitor branch.

[0105] By controlling the on / off state of these analog switches using digital logic, binary weighted combination or linear step adjustment of the capacitance value can be achieved. Binary weighted combination means that the capacitance of each branch is configured in ascending order, and the controller can obtain various total capacitance values ​​through different switch combinations. Linear step adjustment means that the capacitance of each branch is taken into similar or equal step units, and approximately uniform change is achieved by increasing or decreasing the number of connected branches.

[0106] Furthermore, the deviation detection unit includes: A beam position detector, which can be a four-quadrant detector or a position-sensitive detector, is used to collect beam pointing deviation signals. The signal conditioning circuit, connected to the beam position detector, is used to amplify and filter the pointing deviation signal to output an analog deviation signal.

[0107] In practical implementation, beam position detectors are typically deployed at the main optical path splitting node or terminal feedback node of the optical system to receive sampled beams split into a certain proportion in real time. The dualant detector consists of four symmetrically distributed photodiodes. When the beam deviates from its geometric center, the light energy received in each quadrant changes, resulting in different output photocurrents. The position-sensitive detector, based on the transverse photoelectric effect, accurately reflects the centroid position of the beam by the proportion of current output from electrodes at the edges of the continuous photosensitive surface. Since the initial pointing deviation signal output by the detector is typically a weak photocurrent in the microampere or even nanoampere range, it cannot directly drive subsequent circuits and must undergo deep processing by a signal conditioning circuit. The signal conditioning circuit first linearly converts the weak photocurrent into a voltage signal using a low-noise transimpedance amplifier, and then uses a precision differential amplifier circuit to calculate the original voltage characterizing the direction and degree of deviation along the X and Y axes. To eliminate high-frequency noise or low-frequency baseline drift caused by ambient stray light, detector dark current drift, and power supply ripple, the signal conditioning circuit also integrates an analog low-pass or band-pass filter to smooth and correct the original voltage, ultimately outputting a clean and continuous analog deviation signal.

[0108] Furthermore, the deflection execution unit includes: Piezoelectric ceramic driven fast-reflecting mirror; The high-voltage drive circuit has its input terminal connected to the analog control unit and its output terminal connected to the fast-reflecting mirror. It is used to convert the control voltage into a drive signal that drives the fast-reflecting mirror to generate a deflection angle.

[0109] A fast-steering mirror driven by piezoelectric ceramics is a precision optical actuator that utilizes the inverse piezoelectric effect of piezoelectric materials to achieve minute-angle deflection. Compared to traditional voice coil motors or stepper motors, piezoelectric ceramics offer microsecond-level response speeds, extremely high mechanical resonant frequencies, and nanometer-level displacement resolution, making them ideal for compensating for high-frequency, minute-angle beam jitter. In actual operation, the control voltage output by the analog control unit is typically in a low-voltage range (e.g., ±5V or ±10V), which is insufficient to directly drive the piezoelectric ceramic to produce sufficient physical deformation. Therefore, a high-voltage drive circuit, acting as a bridge between low-voltage control and high-voltage execution, typically employs a high-voltage operational amplifier or discrete high-voltage transistor topology to linearly amplify the low-voltage control signal with high bandwidth and low distortion, converting it into a drive signal of tens or even hundreds of volts. This drive signal is directly applied to the piezoelectric stack inside the fast-steering mirror, causing the piezoelectric stack to undergo corresponding expansion and contraction deformation, which in turn drives the mirror surface to deflect around a flexible hinge, instantly changing the reflection path of the beam and returning it to the target position.

[0110] Furthermore, the system is applied at least to ground terminals of satellite-to-ground laser communication links, precision laser processing platforms, and extreme ultraviolet laser generating devices.

[0111] In the ground terminal of the space-to-ground laser communication link, atmospheric turbulence can cause severe and random fluctuations in the angle of arrival of the arriving beam. This system can compensate for beam jitter in real time with extremely short response time and adaptive parameter adjustment, ensuring stable coupling and uninterrupted communication link. In the precision laser processing platform, the high-speed movement of the processing head and environmental mechanical vibration can easily cause beam drift. This system can strictly lock the beam pointing accuracy at the micro-radian or even nanoradian level, ensuring the yield and consistency of micron-level processing. In the extreme ultraviolet laser generation device, facing complex strong electromagnetic interference and high-frequency vibration environment, this system can maintain precise target alignment of seed light and pump light, ensuring stable output of extreme ultraviolet light.

[0112] Furthermore, when the system is applied to an extreme ultraviolet laser generating device, the extreme ultraviolet laser generating device has multiple sub-optical path components connected in series. Each sub-optical path component is equipped with this system, and each system is trained online independently.

[0113] Extreme ultraviolet (EUV) laser generation devices typically contain multiple cascaded sub-optical paths, such as the first sub-optical path from the femtosecond laser to the pulse width compressor, the second sub-optical path from the pulse width compressor to the EUV laser, and the third sub-optical path from the EUV laser to the CDI (Continuous Laser Interference Device) stacked imaging system. Each sub-optical path suffers from independent beam pointing drift, and each sub-optical path is subject to different interference sources.

[0114] For example, the first sub-optical path is mainly affected by the thermal drift of the femtosecond laser itself, the second sub-optical path is mainly affected by the vibration of the optical components of the pulse width compressor, and the third sub-optical path is mainly affected by the airflow disturbance inside the extreme ultraviolet laser cavity. Since the frequency, amplitude, and time-varying characteristics of the interference sources are different, the optimal configuration of the analog PID hardware parameters required for each sub-optical path also varies.

[0115] Therefore, when this system is applied in an extreme ultraviolet (EUV) laser generator, a beam pointing stability control system as described in this application is independently deployed in each sub-optical path. Each system is equipped with an independent deviation detection unit, analog control unit, deflection execution unit, and parameter optimization unit, and each independently performs online reinforcement learning training. Each system independently learns and converges to the optimal PID hardware parameter configuration adapted to its own operating conditions based on the local interference characteristics of its sub-optical path, without interfering with each other. This deployment method effectively addresses the pointing stability problem caused by multi-path crosstalk in the integrated optical path system of the EUV laser generator, ensuring the beam pointing accuracy of the entire optical path system from the light source to the terminal.

[0116] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A simulated PID beam pointing stabilization control system based on reinforcement learning optimization, characterized in that, The system includes: The deviation detection unit is used to acquire beam pointing deviation in real time and output the corresponding analog deviation signal; The analog control unit includes a signal input terminal and a signal output terminal. The signal input terminal is connected to the deviation detection unit. The analog control unit integrates an adjustable analog PID control circuit containing resistive and capacitive elements, which is used to continuously calculate the analog deviation signal and output a control voltage. A deflection execution unit, connected to the signal output terminal, is used to drive the mirror to deflect by a corresponding deflection angle according to the control voltage, so as to correct the beam direction; The parameter optimization unit, connected to the analog control unit, is used to collect system status data in real time, input the system status data into the reinforcement learning algorithm, and output the corresponding parameter configuration instructions to the analog control unit to adjust the resistance value of the resistive element and the capacitance value of the capacitive element. The system status data includes at least the analog deviation signal, the deflection angle, the resistance value of the resistive element, and the capacitance value of the capacitive element.

2. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 1, characterized in that, The parameter optimization unit is used to map the parameter configuration space formed by the resistance value of the resistor element and the capacitance value of the capacitor element in the adjustable analog PID control circuit into the action space of reinforcement learning, and to map the system state data into the state space of reinforcement learning. Online training is performed in a real hardware environment. The online training process is as follows: based on the current system state data, the parameter increments of the resistance and capacitance values ​​are output through a reinforcement learning strategy. After the parameter increments are executed, the system state data at the next moment is collected and the reward signal is calculated. The network parameters of the reinforcement learning strategy are iteratively updated based on the reward signal. Based on the parameter increment, a corresponding parameter configuration instruction is generated and sent to the analog control unit via a digital communication link to adjust the resistance value of the resistive element and the capacitance value of the capacitive element.

3. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 2, characterized in that, The training module performs the online training process, including: Set the initial resistance value of the resistive element and the initial capacitance value of the capacitive element; In each iteration, the current system state data is collected and input into the reinforcement learning strategy to output the parameter increment. The parameter configuration instruction is generated based on the parameter increment and sent to the analog control unit to complete the switching of resistance and capacitance values. After the system responds, the system state data at the next moment is collected and the reward signal is calculated. The current system state data, the parameter increment, the reward signal, and the system state data at the next moment are stored as a set of experience data in the experience replay pool. The network parameters of the reinforcement learning strategy are updated by sampling the experience data from the experience replay pool. After completing one round of training, the average reward value of all iterations in that round of training and the statistical value of the simulated deviation signal are calculated. If the average reward value of multiple consecutive rounds of training no longer increases and the statistical value of the beam deviation meets the preset accuracy condition, then the training is determined to be converged. After training convergence, the last updated network parameters are stored in non-volatile memory, which is used to output the corresponding parameter increments of the resistance and capacitance values ​​based on the system state data during actual operation.

4. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 3, characterized in that, The training module, when performing online training, also includes: In each iteration, before the reinforcement learning policy outputs the parameter increment, random noise is superimposed on the output parameter increment, and the amplitude of the random noise decreases as the training rounds increase. When updating the network parameters of the reinforcement learning strategy by sampling experience data from the experience replay pool, a priority experience replay mechanism is adopted, and the sampling weight of the experience data is determined according to the magnitude of the temporal difference error of the experience data.

5. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 3, characterized in that, The training module, when performing online training, also includes: In each iteration, the resistance and capacitance values ​​are limited to the minimum and maximum values ​​allowed by their respective hardware. The single-step adjustment amount of each item in the parameter increment is limited to not exceeding the preset maximum allowable single-step adjustment amount; When the simulated deviation signal exceeds the preset deviation safety limit, the resistance value and the capacitance value are restored to the initial resistance value and the initial capacitance value, and the corresponding empirical data is marked as abnormal data. When updating the network parameters of the reinforcement learning strategy, its sampling weight is reduced.

6. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 1, characterized in that, The resistive element is a digital potentiometer in the form of an integrated circuit. It consists of multiple resistors connected in series to form a voltage divider network. The internal switch selects the tap to be connected through a digital communication interface so that the resistance value can be adjusted. The capacitor element is a programmable capacitor array, which consists of multiple capacitors that are switched by analog switches to achieve adjustable capacitance values.

7. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 1, characterized in that, The deviation detection unit includes: A beam position detector, which is a four-quadrant detector or a position-sensitive detector, is used to collect beam pointing deviation signals. A signal conditioning circuit, connected to the beam position detector, is used to amplify and filter the pointing deviation signal to output the analog deviation signal.

8. The analog PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 1, characterized in that, The deflection execution unit includes: Piezoelectric ceramic driven fast-reflecting mirror; A high-voltage drive circuit, the input of which is connected to the analog control unit and the output of which is connected to the fast-reflecting mirror, is used to convert the control voltage into a drive signal that drives the fast-reflecting mirror to generate the deflection angle.

9. A simulated PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 1, characterized in that, The system is applied at least to ground terminals of satellite-to-ground laser communication links, precision laser processing platforms, and extreme ultraviolet laser generating devices.

10. A simulated PID beam pointing stabilization control system based on reinforcement learning optimization according to claim 9, characterized in that, When the system is applied to an extreme ultraviolet laser generating device, the extreme ultraviolet laser generating device has multiple sub-optical path components connected in series, each of which is equipped with the system, and each system is independently trained online.

Images