Self-adaptive optical system parameter adjusting method and device combining offline pre-training and online reinforcement learning

Abstract

The invention discloses a self-adaptive optical system parameter adjusting method and device combining offline pre-training and online reinforcement learning, and belongs to the field of intelligent control combining self-adaptive optics and reinforcement learning. Multi-modal state modeling, off-line strategy pre-training and an online strategy enhancement mechanism guided by behavior advantages are introduced, and key control parameters in the AO system are dynamically optimized by using a reinforcement learning algorithm. According to the method, multi-modal heterogeneous data can be effectively integrated, the dynamic modeling and prediction capability of the system is improved, and stable, efficient and robust control strategy updating is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent control combining adaptive optics and reinforcement learning, and more specifically, to a method and apparatus for adjusting parameters of an adaptive optics system that combines offline pre-training and online reinforcement learning. Background Technology

[0002] Adaptive Optics (AO) systems are a class of precision optical systems used for real-time compensation of wavefront distortion, widely applied in astronomical observation, laser communication, biological imaging, and high-resolution measurement. Their core objective is to improve image sharpness or beam transmission quality under perturbation conditions (such as atmospheric turbulence). To achieve this, AO systems typically require dynamic sensing of wavefront information and real-time adjustment of actuators such as deformable mirrors via controllers to correct the wavefront.

[0003] In traditional methods, the control strategies of AO systems often employ empirical settings or fixed-parameter methods based on classical control theories (such as integral control and PID control). Key system parameters, such as integral gain, sampling frequency, exposure time, and control frequency, are frequently manually adjusted by professional engineers based on experience or prior models. This static or semi-static parameter setting approach struggles to maintain optimal system performance in highly dynamic and non-stationary environments such as atmospheric turbulence, leading to decreased image quality or insufficient system stability.

[0004] In recent years, reinforcement learning (RL) has been increasingly incorporated into the intelligent control research of AO systems due to its powerful capabilities in high-dimensional control and policy optimization. Some works have attempted to utilize RL algorithms to automatically learn control policies under different turbulent environments, achieving intelligent adjustment of system parameters. However, several key challenges remain in practical applications: 1. Difficulty in fusing multimodal heterogeneous data: Input data of AO system often includes multiple modes such as wavefront images, far-field images, slope signals, and voltage signals. Existing methods often adopt simple splicing or shallow fusion strategies, which make it difficult to capture the correlation and dynamic evolution law between different modes, resulting in insufficient state perception capability.

[0005] 2. Limited dynamic modeling capability of the system: AO system has strong nonlinearity, high dimension and rapid change dynamic characteristics. The environment modeling capability of existing reinforcement learning models is weak, making it difficult to accurately predict state transitions and performance feedback, and the response to transient changes is slow.

[0006] 3. Insufficient generalization of parameter tuning strategies: Most existing methods adopt online reinforcement learning or end-to-end policy training, which lacks a mechanism for policy initialization based on historical experience data. When faced with environmental changes or system parameter perturbations, the learning efficiency is low and the policy generalization ability is poor.

[0007] In summary, existing AO systems have significant bottlenecks in coping with dynamic disturbances and adaptively adjusting control parameters, necessitating a new parameter adjustment scheme that integrates offline pre-training and online enhancement mechanisms. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide an adaptive optics system parameter adjustment method and device that combines offline pre-training and online reinforcement learning. This method can effectively integrate multimodal heterogeneous data, improve the ability to model and predict system dynamics, and achieve stable, efficient, and robust control strategy updates.

[0009] The objective of this invention is achieved through the following solution: An adaptive optics system parameter tuning method combining offline pre-training and online reinforcement learning includes the following steps: Based on the reinforcement learning control AO system, a multimodal state modeling, offline policy pre-training, and behavior advantage-oriented online policy enhancement mechanism are introduced to dynamically optimize the control parameters in the AO system using reinforcement learning algorithms.

[0010] Furthermore, the multimodal state modeling specifically includes the following sub-steps: Image observation and structured observation data are collected from the AO system, encoded separately, and then fused into a unified state representation.

[0011] Furthermore, the offline policy pre-training specifically includes the following sub-steps: Based on historical interaction data or real expert data, a policy network and environment model are constructed to achieve preliminary control capability learning.

[0012] Furthermore, the behavior advantage-oriented online strategy enhancement mechanism specifically includes the following sub-steps: In dynamic optical environments, the policy network is enhanced and optimized based on the current design state using a behavior advantage weighted estimation method.

[0013] Furthermore, the dynamic optimization of control parameters in the AO system using reinforcement learning algorithms specifically includes the following sub-steps: The optimized policy network outputs a set of system control parameters. The action vectors output by the reinforcement learning policy network contain adjustment instructions for each key control parameter in the AO system. The hardware configuration is dynamically adjusted according to the action instructions to achieve closed-loop wavefront correction. Feedback information is collected in real time after the action is executed to provide a basis for reward calculation.

[0014] Furthermore, the image observation and structured observation data acquired in the AO system are encoded separately and fused into a unified state representation, specifically including the following sub-steps: Real observation data from the AO system, including image data and vector data, are acquired. Convolutional neural networks are used to extract features from the image data, and fully connected neural networks are used to encode the vector data. Finally, the multimodal observation results are combined into a unified state representation through feature-level concatenation. This representation is used as input for subsequent reinforcement learning.

[0015] Furthermore, the construction of a policy network and environment model based on historical interaction data or real expert data to achieve preliminary control capability learning specifically includes the following sub-steps: Organize samples of state-action-reward-state transitions from historical operation processes. Alternatively, offline datasets can be constructed by summarizing expert operational data from real systems. Offline reinforcement learning algorithms or behavior cloning strategies are used to improve the policy network. Pre-training is performed, and the corresponding value function network is trained simultaneously. This enables intelligent agents to possess basic control capabilities.

[0016] Furthermore, in the dynamic optical environment, the policy network is enhanced and optimized based on the current designed state using a behavioral advantage weighted estimation method, specifically including the following sub-steps: In the actual operation of the AO system, the reinforcement learning agent observes based on the current state. Output Action This means adjusting the corresponding control parameters and receiving system feedback rewards. and new status , will newly collected Samples are continuously added to the experience replay pool to form an online dataset. Subsequently, the policy network and value network are updated through policy optimization methods to achieve online continuous adaptive adjustment of the policy.

[0017] Furthermore, the dynamic optimization of control parameters in the AO system using reinforcement learning algorithms specifically includes the following sub-steps: A composite reward function incorporating multiple metrics is constructed, taking into account imaging quality metrics, wavefront error (WFE), and action penalty terms. The reward signal is then fed back to the reinforcement learning agent, driving the policy network to continuously optimize and update online based on offline pre-training.

[0018] An adaptive optics system parameter adjustment device combining offline pre-training and online reinforcement learning includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method described in any of the preceding claims.

[0019] The beneficial effects of this invention include: This invention, while maintaining control precision, effectively improves the responsiveness and imaging quality of the AO system to complex dynamic disturbance environments, reduces reliance on human experience, and possesses excellent intelligence, adaptability, and engineering practical value. Specifically, it has the following technological advancements: (1) Strong multimodal state fusion capability: By performing deep encoding and joint fusion of wavefront images, far-field images and structured vector data, the system achieves full perception of multi-source heterogeneous information in complex optical environments, and improves the accuracy and completeness of system state expression.

[0020] (2) Combining offline pre-training with online reinforcement: offline policy initialization is performed using historical data to give the agent basic control capabilities, reducing the sample requirements and time costs of online training; combined with behavior advantage-oriented online policy optimization, the adaptability and generalization ability of the control policy are effectively improved.

[0021] (3) Real-time dynamic adjustment of control strategy: Based on the action output of reinforcement learning, the key parameters of AO system can be adjusted in real time to form closed-loop control, effectively cope with dynamic disturbances such as atmospheric turbulence, and ensure the imaging quality and stability of the system.

[0022] (4) The composite reward function is reasonably designed: it comprehensively considers the imaging quality index and the action cost, promotes the balance between accuracy and efficiency of the control strategy, avoids over-adjustment, and improves the system's operational safety and robustness.

[0023] (5) Wide range of applications: It is not only suitable for traditional AO applications such as astronomical observation, but also for various optical systems such as laser communication and biological imaging. It has good promotion and application value and practicality. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the steps of the method in an embodiment of the present invention. Detailed Implementation

[0026] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0027] The specific implementation process of this invention is as follows: The invention specifically addresses the field of intelligent control combining Adaptive Optics (AO) and Reinforcement Learning (RL), offering a parameter adjustment scheme for AO systems that integrates offline pre-training and online reinforcement learning strategies. This scheme utilizes reinforcement learning algorithms to dynamically optimize key control parameters (including gain, sampling frequency, exposure time, and control clock) in the AO system, aiming to improve the system's adaptability and imaging performance in non-stationary environments. The proposed scheme also combines offline policy initialization and online policy update mechanisms, making it applicable to AO system control problems with strong nonlinearity and multivariate coupling characteristics.

[0028] More specifically, one embodiment proposes a parameter adjustment method for adaptive optics systems that combines offline pre-training and online reinforcement learning, suitable for the intelligent adjustment of key parameters of adaptive optics systems in dynamic and complex environments. Based on reinforcement learning control of AO systems, this method introduces multimodal state modeling, policy pre-training, and a behavior advantage-oriented online reinforcement mechanism to improve the system's responsiveness to environmental changes and control accuracy. The method includes the following steps: Step 1: Multimodal state modeling and offline policy pre-training; Multimodal state modeling, specifically, includes the following sub-steps: State observation data construction and multimodal fusion: Image observations (such as wavefront images and far-field images) and structured observations (such as wavefront slope, voltage, turbulence features, etc.) from the AO system are collected, encoded separately, and fused into a unified state representation. Specifically, real observation data from the AO system is acquired, including image data and vector data. Image data includes two-dimensional observations such as wavefront sensor images and far-field images; vector data includes multi-dimensional scalar data such as wavefront slope, corrector voltage, and system parameter settings. Convolutional neural networks are used to extract features from image observations, and fully connected neural networks are used to encode vector features. Finally, the multimodal observation results are combined through feature-level concatenation to form a unified state representation. This representation is used as input for subsequent reinforcement learning to enhance state awareness.

[0029] Offline policy pre-training, specifically, includes the following sub-steps: Offline policy construction and policy initialization training: Based on historical interaction data from other models or real expert data, a policy network and environment model are constructed to achieve initial control capability learning; specifically, state-action-reward-state transition samples from historical operation processes are compiled. Alternatively, offline datasets can be constructed by summarizing expert operational data from real systems. Offline reinforcement learning algorithms or behavior cloning strategies are used to train the policy network. Pre-training is performed, and the corresponding value function network is trained simultaneously. This enables the agent to possess basic control capabilities. This stage does not require interaction with a real environment; it relies on offline experience for initial policy learning and generalization.

[0030] Step 2, online policy enhancement: In a dynamic optical environment, the policy network is enhanced and optimized based on the current designed state using a behavioral advantage weighted estimation method; specifically, this includes the following sub-steps: Online data acquisition and dynamic policy updates: In the actual operation of the AO system, the reinforcement learning agent observes the current state... Output Action This involves adjusting the corresponding control parameters (such as integral gain, sampling frequency, exposure time, etc.) and obtaining system feedback rewards. and new status The newly collected Samples are continuously added to the experience replay pool to form an online dataset. Subsequently, the policy network and value network are updated through policy optimization methods (such as policy gradient or value iteration) to achieve online continuous adaptive adjustment of the policy.

[0031] Step 3, Action Output and System Adjustment: Output a set of system control parameters based on the optimized policy network; specifically, including: Action Execution and System Feedback Control: The action vectors output by the reinforcement learning policy network contain adjustment instructions for key control parameters in the AO system (such as deformable mirror control gain, wavefront sensor sampling frequency, exposure time, and control clock frequency). The system dynamically adjusts the hardware configuration according to the action instructions to achieve closed-loop wavefront correction. After the action is executed, the system collects feedback information in real time (including wavefront correction performance indicators such as Strell ratio and wavefront error) to provide a basis for reward calculation and ensure the accuracy and rationality of policy adjustment.

[0032] Step 4, Reward Calculation and Feedback: A composite reward function and action penalty are constructed using metrics such as Strell ratio, wavefront error, and image sharpness to guide continuous strategy optimization. Specifically, this includes the following sub-steps: Reward function design and policy optimization feedback: A composite reward function incorporating multiple metrics is constructed, comprehensively considering imaging quality metrics (such as Strellby (SR)), wavefront error (WFE), and action penalty terms. The reward signal is fed back to the reinforcement learning agent, driving the policy network to continuously optimize and update online based on offline pre-training, thereby improving policy generalization ability and overall system performance.

[0033] In other alternative embodiments, based on the above embodiments, multimodal state modeling specifically includes the following sub-steps: The state of an adaptive optics system is defined as a combination of an image observation sequence and a vector observation sequence. The image observations include the wavefront image and far-field image of the most recent K+1 time steps, and the vector observations include the wavefront slope and deformable mirror voltage of the corresponding time steps.

[0034] Image observation sequence ( , (k = 0, 1, ..., K) are input into a convolutional neural network (CNN) to extract the spatial features of each frame of the image, resulting in a sequence of latent features. These latent features are then concatenated and fused over time to form a temporal latent feature representation of the image. The vector observation sequence ( The input is processed time-by-time by a multilayer perceptron (MLP), and then concatenated to obtain the temporal latent feature representation of the vector. .

[0035] Finally, the temporal latent features of the image are represented. Temporal latent feature representation of vectors By concatenating the data, a unified state representation can be obtained. ,Right now: .

[0036] Offline policy pre-training includes the following sub-steps: Using historical datasets Pre-trained policy network ,in, This represents the system state obtained by fusing image observations and vector observations. This indicates the control action performed by the system at that time. This indicates the reward the system receives for this action. This refers to the state at the next moment after the action is performed.

[0037] During pre-training, a policy network is constructed. And it is trained with the following optimization objectives:

[0038] in, This represents the action advantage function estimated based on the current policy, used to measure the advantage of a particular action in... The quality of a state relative to the average level. The first iteration of the policy is indicated by the... Next update The system state is obtained by fusing image observations and vector observations, where 'a' represents the control action applied to the system. These are the trainable parameters of the policy network. This is a temperature coefficient used to adjust the degree of influence of the advantage function on policy updates. Indicates the current policy's control action The probability estimate is used to encourage the retention of high-advantage actions. The optimization objective is essentially to perform advantage-weighted imitation learning of the best actions from previous control data, thereby improving the initial effectiveness of the policy.

[0039] The policy network is preferably a deep neural network structure, using multimodal encoded state representation. As input, the output is the action distribution corresponding to the key control parameters of the AO system, which is used for initialization in the subsequent online training phase.

[0040] Online strategy enhancement specifically includes the following sub-steps: Collect new data during the online interaction phase And update the experience pool ; Perform a policy update based on behavioral advantage weighting using the following formula:

[0041] in, The updated parameters of the policy network represent the new policy learned through weighted imitation learning. The estimated action advantage function is used to measure the action advantage in state. Next action Compared to the average performance of the current strategy; Estimated via offline Critic network:

[0042] in, To start from the current strategy The reference action obtained from the sampling, It is a value function network with parameters. Used to evaluate in state Take action below The expected cumulative return that can be obtained; According to the policy network Output control actions This is used as a key parameter adjustment signal for the AO system at the current moment to control the operation configuration of multiple components within the system. The parameters include: the gain setting of the wavefront sensor, and the dynamic adjustment of the sampling frequency, exposure time, and clock frequency.

[0043] Specifically, policy network Receive current fusion status representation The system outputs a corresponding action vector, where each dimension of the action corresponds to a discrete selection or continuously changing value of an adjustable parameter. Based on this action vector, the system completes parameter setting and execution, forming a complete closed-loop control process.

[0044] To ensure the effectiveness and physical feasibility of the action execution, the strategy output actions must also meet certain constraints. Through the above strategy-driven action output and system dynamic adjustment, real-time optimized control of the AO system in complex dynamic environments is achieved, effectively improving the accuracy of optical correction and imaging quality.

[0045] The reward calculation and feedback process includes the following sub-steps: When designing a composite reward function, the Strelby ratio (SR), wavefront error (WFE), and action penalty are combined:

[0046] in, , These are the weighting coefficients. This is the penalty coefficient for the action; The Strell ratio is a key indicator of image quality; a higher value indicates better optical correction. This represents the wavefront error; the smaller the value, the better, as it reflects the accuracy of the system's wavefront compensation. To penalize the magnitude of actions, prevent excessive adjustments to the strategy output, and ensure system stability; Critic Network The training employs Bellman error minimization:

[0047] in, These are the optimized Critic network parameters. To perform the action The state at the next moment after the current state, r represents the state in the current state – action pair. Instant rewards based on environmental feedback; target value Calculated through the target network:

[0048] in, This is a discount factor used to balance the impact of immediate rewards and future rewards. These are the parameters of the target network, used to calculate a stable target Q value; This invention addresses the problems of current adaptive optics (AO) systems, such as reliance on manual experience for control parameter setting, lack of real-time adaptability, and low training efficiency of reinforcement learning strategies. Specifically, it utilizes historical observations to construct an offline dataset for policy initialization training, and combines a multimodal state observation fusion mechanism with an online environment interaction optimization network to improve the intelligence and dynamic adaptability of control parameter adjustment. This method has a clear overall structure, high deployment efficiency, sufficient state representation, and strong policy generalization ability, significantly enhancing the robustness and performance of AO systems in dynamic environments.

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the method for adjusting the parameters of an adaptive optics system combining offline pre-training and online reinforcement learning provided by this invention will be further described below in conjunction with other specific embodiments. It should be noted that the following description will elaborate on each step, including data acquisition, encoding fusion, model training, policy optimization, control execution, and reward design.

[0050] In the acquisition and independent encoding of multimodal observation data, in an optional embodiment, the Adaptive Optics (AO) system acquires various types of observation data in real time during operation, covering two main categories: image information and structured vector information. Specifically, the acquired data includes: wavefront sensor wavefront images: typically from the spot images formed by the microlens array, reflecting the current wavefront distortion information; far-field imaging images: reflecting the imaging quality after correction by the deformable mirror, directly demonstrating the overall correction effect of the system; and structured vector data: including wavefront slope signals (Slopes), deformable mirror driving voltages (DMVoltages), environmental parameters such as temperature and humidity fed back by the system sensors, and control parameter settings such as sampling frequency and exposure time. The acquired image data is large in size and rich in information, while the vector data has a smaller dimension but a significant time dependence.

[0051] To fully utilize various data characteristics, the following independent encoding strategy is adopted in the optional embodiments: 1) Image Encoder: A deep convolutional neural network (CNN) is used to extract spatial hierarchical features of the image. Wavefront and far-field images are encoded separately to obtain their respective latent feature vectors. , In the time dimension, temporal stacking and feature fusion are performed on the image sequences of the most recent k+1 time steps to capture dynamic change trends and form potential temporal features of the images. .

[0052] 2) Vector Encoder: For slope, voltage, and other scalar data, a multilayer perceptron (MLP) time-series encoder is used. The input is a vector sequence of the past K+1 time steps. The encoder learns temporal relevance and dynamic features through embedding layers and positional encoding, and outputs a latent representation vector. Finally, the latent features of the image are... With vector latent representation Concatenate to generate a unified state representation:

[0053] This representation integrates heterogeneous multimodal information and serves as input to the reinforcement learning policy network, effectively improving the comprehensiveness and accuracy of state perception.

[0054] In the optional embodiment, for offline policy pre-training and value function initialization, direct online policy training faces challenges such as slow convergence and low sample efficiency due to the complex environmental changes and high-dimensional control requirements of the AO system. This embodiment utilizes offline historical datasets for policy network pre-training, which can improve the generalization ability and stability of the initial policy.

[0055] Dataset Construction: Collect state-action-reward-next state quadruple data from historical runs. The data sources include historical interactions in simulated environments, expert operation records from real systems, and other reliable sources.

[0056] Advantage-weighted imitation learning: using the advantage function By weighting historical actions, the policy network is encouraged to fit control behaviors that exhibit good performance, thus optimizing the objective function.

[0057] Among them, temperature parameter Adjust the smoothness of the advantage weights to prevent overfitting of individual actions. Based on the current strategy The estimated action advantage function represents the state. Next action Relative to the performance of the average strategy, k represents the number of policy iterations. The next update refers to the version index of the current strategy, where 's' is the system state, obtained by fusing image observations and vector observations, and 'a' is the control action. These are the trainable parameters of the policy network.

[0058] Policy network design: The policy network is a multi-layer feedforward neural network, with a unified state representation as input. It outputs the motion distribution of key control parameters. The motion space supports discrete parameters, meeting the diverse control requirements of AO systems.

[0059] Value network (Critic) initialization: Simultaneously train the corresponding value function network. We estimate the expected reward of state-action pairs to provide estimation support for subsequent online augmentation. We use Bellman error minimization for target training to enhance the accuracy of the value function.

[0060] The offline pre-training phase endows the policy network with basic control capabilities, significantly reducing the exploration costs of online training.

[0061] Online data acquisition and dynamic strategy optimization: During the actual operation of the AO system, the agent's strategy is adjusted based on the current observation state. Output Action After the system executes this action, it obtains a new state. and rewards .

[0062] Data accumulation: newly generated Samples are stored in the experience pool in real time. , and offline data The data is merged to form a dynamically updated dataset.

[0063] Action advantage weighted update: Action advantage estimation based on Critic network:

[0064] s represents the current state. Q This is a Critic network (i.e., a Q-network) used to estimate the state-action value of taking action a in state s. To start from the current strategy The actions obtained from sampling These are the parameters of the Critic network; Policy updates are achieved by maximizing the log-likelihood with advantage weights:

[0065] Network parameter updates: Both the policy network and value network are optimized using gradient descent. A target network mechanism and an empirical replay pool are used to improve training stability and sample utilization.

[0066] Through online reinforcement learning, the strategy can adapt to dynamic changes in the environment in real time, enabling intelligent adjustment of control parameters.

[0067] Dynamic mapping and system closed-loop control: Action vectors output by the policy network The signals mapped to specific AO system parameter adjustment signals mainly include: wavefront sensor gain adjustment to ensure a balance between signal amplification and noise suppression; deformable mirror drive frequency and clock parameters to control the deformable mirror's response speed and correction frequency; and camera exposure time and sampling frequency to optimize imaging signal-to-noise ratio and response speed.

[0068] The action execution module is responsible for translating actions into hardware instructions, ensuring the physical feasibility of parameter adjustments and system safety. The system continuously collects feedback signals, constructs a closed-loop control process, and achieves real-time performance optimization.

[0069] Reward Function Design and Performance Evaluation: The reward function, taking into account both imaging quality and control costs, is designed as follows:

[0070] in, The Strell ratio is a key indicator of image quality; a higher value indicates better optical correction. Wavefront error, the smaller the better, reflects the accuracy of the system's wavefront compensation; Action amplitude penalty to prevent excessive adjustments in policy output and ensure system stability; weight coefficient. , and action penalty coefficient It can be adjusted according to actual needs.

[0071] This composite reward function guides the reinforcement learning agent to control the adjustment range while optimizing imaging quality, achieving a balance between efficiency and stability.

[0072] Critic Network Training and Objective Update: The value network (Critic) is used to estimate the state-action value function. The training process employs Bellman residual minimization, specifically as follows:

[0073] Where r is the immediate reward, representing the reward signal returned by the environment after taking action a in state s. The next state is the new state that the environment transitions to after the agent performs action a. The target value is:

[0074] Discount factor, which adjusts the weight of long-term rewards; These are the parameters of the target network, used to calculate stable target values ​​and prevent training oscillations caused by frequent updates to the Q network parameters; Using a dual-network structure and introducing a soft update strategy:

[0075] in, To control the update rate of the target network parameters, stabilize the training process, and avoid value explosion.

[0076] It is worth noting that the key points of system architecture design and engineering deployment are as follows: To ensure the efficiency of algorithm deployment and real-time operation in the AO system, the overall system design is as follows: Perception layer: responsible for image acquisition, data preprocessing, normalization, etc. Encoding layer: processes image and vector data separately and fuses them into a unified state representation; Strategy layer: Outputs control parameters and actions after inputting the state; Control execution layer: converts actions into physical device parameters; Evaluation layer: Calculates rewards in real time and feeds them back to the policy network optimization process; Training and update layer: maintains processes such as experience pool maintenance, strategy updates, and Critic updates.

[0077] The entire architecture is based on a modular design, which facilitates the subsequent expansion of other control parameters or the introduction of new sensor information.

[0078] The method described in the above embodiments of the present invention combines the generalization ability of offline learning with the adaptability of online learning. The offline learning stage endows the model with the basic ability to cope with dynamic changes in the real optical environment, while subsequent online reinforcement learning further improves the policy accuracy. This method can significantly improve the intelligent adjustment capability and imaging quality of the AO system in complex dynamic environments, reduce reliance on human experience, and has good practicality.

[0079] By implementing the above methods, while maintaining control accuracy, the AO system's response capability and imaging quality to complex dynamic disturbance environments are effectively improved, reducing reliance on human experience and demonstrating good intelligence, adaptability, and engineering practical value.

[0080] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0081] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0082] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. A method for adjusting parameters of an adaptive optics system combining offline pre-training and online reinforcement learning, characterized in that, Includes the following steps: Based on the reinforcement learning control AO system, a multimodal state modeling, offline policy pre-training, and behavior advantage-oriented online policy enhancement mechanism are introduced to dynamically optimize the control parameters in the AO system using reinforcement learning algorithms.

2. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 1, characterized in that, The multimodal state modeling specifically includes the following sub-steps: Image observation and structured observation data are collected from the AO system, encoded separately, and then fused into a unified state representation.

3. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 1, characterized in that, The offline strategy pre-training specifically includes the following sub-steps: Based on historical interaction data or real expert data, a policy network and environment model are constructed to achieve preliminary control capability learning.

4. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 1, characterized in that, The behavior advantage-oriented online policy enhancement mechanism specifically includes the following sub-steps: In dynamic optical environments, the policy network is enhanced and optimized based on the current design state using a behavior advantage weighted estimation method.

5. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 4, characterized in that, The step of dynamically optimizing the control parameters in the AO system using reinforcement learning algorithms specifically includes the following sub-steps: The optimized policy network outputs a set of system control parameters. The action vectors output by the reinforcement learning policy network contain adjustment instructions for each key control parameter in the AO system. The hardware configuration is dynamically adjusted according to the action instructions to achieve closed-loop wavefront correction. Feedback information is collected in real time after the action is executed to provide a basis for reward calculation.

6. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 2, characterized in that, The image observation and structured observation data acquired in the AO system are encoded separately and then fused into a unified state representation, specifically including the following sub-steps: Real observation data from the AO system, including image data and vector data, are acquired. Convolutional neural networks are used to extract features from the image data, and fully connected neural networks are used to encode the vector data. Finally, the multimodal observation results are combined into a unified state representation through feature-level concatenation. This representation is used as input for subsequent reinforcement learning.

7. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 3, characterized in that, The process of constructing a policy network and environment model based on historical interaction data or real expert data to achieve preliminary control capability learning includes the following sub-steps: Organize samples of state-action-reward-state transitions from historical operation processes. Alternatively, offline datasets can be constructed by summarizing expert operational data from real systems. Offline reinforcement learning algorithms or behavior cloning strategies are used to improve the policy network. Pre-training is performed, and the corresponding value function network is trained simultaneously. This enables intelligent agents to possess basic control capabilities.

8. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 4, characterized in that, In the dynamic optical environment, the policy network is enhanced and optimized based on the current design state using a behavioral advantage weighted estimation method, specifically including the following sub-steps: In the actual operation of the AO system, the reinforcement learning agent observes based on the current state. Output Action This means adjusting the corresponding control parameters and receiving system feedback rewards. and new status , will newly collected Samples are continuously added to the experience replay pool to form an online dataset. Subsequently, the policy network and value network are updated through policy optimization methods to achieve online continuous adaptive adjustment of the policy.

9. The adaptive optics system parameter adjustment method combining offline pre-training and online reinforcement learning according to claim 1, characterized in that, The step of dynamically optimizing the control parameters in the AO system using reinforcement learning algorithms specifically includes the following sub-steps: A composite reward function incorporating multiple metrics is constructed, taking into account imaging quality metrics, wavefront error (WFE), and action penalty terms. The reward signal is then fed back to the reinforcement learning agent, driving the policy network to continuously optimize and update online based on offline pre-training.

10. A parameter adjustment device for an adaptive optics system combining offline pre-training and online reinforcement learning, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 9.

Images