Random parallel gradient descent phase control algorithm based on reinforcement learning optimization
By combining a stochastic parallel gradient descent phase control algorithm optimized by reinforcement learning, the problems of high system complexity and difficult parameter tuning in multi-beam laser coherent synthesis are solved, achieving fast and stable phase control and simplified hardware implementation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2026-01-12
- Publication Date
- 2026-04-24
AI Technical Summary
Existing active phase control methods in coherent multi-beam laser combining suffer from high system complexity, large computational load, parameter sensitivity requiring manual adjustment, and unstable convergence speed, making it difficult to achieve efficient real-time phase control.
By combining a reinforcement learning-optimized stochastic parallel gradient descent phase control algorithm, and constructing a Q-table learning structure, the SPGD algorithm parameters are adaptively optimized to achieve automatic adjustment and real-time phase control.
It achieves fast and stable phase control, reduces dependence on environment and hardware characteristics, simplifies system structure, reduces computational complexity, and is suitable for hardware implementation on FPGA and DSP platforms.
Smart Images
Figure CN121918409A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of laser coherent synthesis, and more specifically to a stochastic parallel gradient descent phase control algorithm based on reinforcement learning optimization. Background Technology
[0002] Laser coherent combining aims to synchronize multiple laser beams by coordinating their phases, thereby obtaining a high-power and high-quality output beam. Lasers originate from stimulated emission; this process requires the generation and optical amplification of stimulated emission, and the device capable of performing this mechanism constitutes a laser. There are many types of lasers, among which fiber lasers occupy an important position in numerous applications due to their compact structure and high efficiency. With the continuous growth in demand for laser technology in industrial processing, medical treatment, and defense, higher requirements are being placed on the power, beam quality, and brightness of laser output. Although diode-pumped amplification technology is highly mature, the damage threshold and heat dissipation capacity of optical materials limit the ability of a single fiber laser to infinitely increase its output power. To obtain laser outputs of tens of kilowatts or even higher, multiple beams must be spatially combined to form a higher-energy laser beam. The prerequisite for achieving this goal is the effective control of the phase of each beam channel; therefore, phase control constitutes the core technology of multi-channel laser coherent combining.
[0003] Typically, active phase control (APC) technology corrects phase errors by detecting information from far-field beams to maximize the superposition intensity of multiple beams. Current APC methods mainly include heterodyne, dithering, stochastic parallel gradient descent (SPGD), and intelligent control methods based on deep learning or reinforcement learning. These methods can effectively correct phase errors. However, each method has its inherent limitations: heterodyne can estimate the phase difference of each beam in a single step, but requires complex optical structures, making the system overly complex and difficult to implement with a large number of beams; dithering, while requiring only a single-point detector, demands a rapidly increasing detector bandwidth with the number of beams and places high demands on the demodulation circuit speed; intelligent control methods based on deep learning or reinforcement learning largely rely on image sensors and involve significant computational costs. Although most intelligent control methods can achieve control convergence within a few steps, their computational process is complex and their update speed is slow. Traditional SPGD algorithms also suffer from the following problems: sensitivity to algorithm parameters, requiring multiple manual adjustments as the environment and hardware characteristics change, unstable convergence speed, and a rapid decrease in control bandwidth with the increase of the number of channels in large-scale arrays. Several classic improved algorithms also suffer from the problem of requiring manual adjustment of some prior parameters and overly complex parameter update calculations. Therefore, there is an urgent need for an optimization method that can automatically adjust SPGD parameters according to the environment while ensuring real-time performance and is hardware-implementable. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the existing SPGD algorithm by combining the existing algorithm with reinforcement learning methods, and to provide a stochastic parallel gradient descent phase control algorithm based on reinforcement learning optimization for real-time phase control and error correction in multi-channel laser coherent synthesis systems.
[0005] The technical solution of the present invention to solve the above-mentioned technical problems is:
[0006] A stochastic parallel gradient descent phase control method based on reinforcement learning optimization includes the following steps:
[0007] Step 1: Obtain the performance evaluation index value of the coherent synthesis system at the current iteration time, and discretize the performance evaluation index value according to the pre-set segmentation rules to map it into a unique state code of the reinforcement learning model, wherein the state code corresponds one-to-one with the performance measurement interval;
[0008] Step 2: Construct a reinforcement learning model for adaptive optimization of the parameters of the stochastic parallel gradient descent algorithm. The reinforcement learning model adopts a learning structure based on a Q-table. The dimension of the Q-table is determined by the number of unique state codes for the performance evaluation index and the number of parameter combinations for the stochastic parallel gradient descent algorithm. Each row in the Q-table corresponds to a unique state code output in Step 1, and each column corresponds to a set of parameter combinations for the stochastic parallel gradient descent algorithm. In the Q-table row corresponding to the current state, select an action according to the Q-value and a preset probability strategy, and output the parameter combination corresponding to the selected action.
[0009] Step 3: Update the control parameters of the stochastic parallel gradient descent algorithm based on the parameter combination output by the reinforcement learning model, and execute the stochastic parallel gradient descent algorithm in the coherent synthesis system to complete the phase control adjustment of the current iteration;
[0010] Step 4: Obtain the performance evaluation index value of the coherent synthesis system after executing the stochastic parallel gradient descent algorithm described in Step 3. Calculate the difference between the performance evaluation index before and after phase control, and use the difference as the reward value of the reinforcement learning model. Based on the reward value, update the Q value of the current state-action corresponding item in the Q table.
[0011] Step 5: Repeat steps 1 to 4 until the expected value of each state-action item in the Q table tends to stabilize or meets the preset convergence condition; then optimize the action selection strategy according to the distribution of the expected value in the corresponding row of each state code in the Q table.
[0012] Step 6: Repeat steps 1 to 5 until the action selection strategy meets the preset optimization conditions; select the action with the largest Q value in the Q-table row corresponding to each state code, and construct a lookup table for the optimal parameter combination of state code and stochastic parallel gradient descent algorithm, which serves as the optimal phase control strategy of the stochastic parallel gradient descent algorithm after reinforcement learning optimization.
[0013] Preferably, in step S1, the performance evaluation index adopts one or more of the following: barrel power, absolute output power, far-field peak intensity, main lobe energy ratio, Strell ratio, beam quality factor, enclosing energy, interference fringe contrast, phase consistency error, wavefront root mean square error, and sidelobe suppression ratio.
[0014] Preferably, in step S2, each entry in the Q table contains a counter, which is used to record the number of times the corresponding state-action item is accessed.
[0015] Preferably, in step S2, the preset probability strategy is an ε-greedy strategy, and the specific steps are as follows:
[0016] The action with the largest Q value in the Q-table row corresponding to the current state is selected with probability ε. Any action is randomly selected from the remaining actions in the row with equal probability 1-ε, where ε is a pre-set hyperparameter used to adjust the exploration intensity.
[0017] Preferably, in step S4, the Q value of the state-action item in the Q table is updated in the following manner:
[0018] ;
[0019] ;
[0020] Among them, s t Indicates the current state, a t This represents the action selected in the current state, r. t Indicates the current state s t Execute action a t The reward value obtained later, Q(s) t ,a t ) indicates the current state s t Execute action a t Then, the expected value of the long-term cumulative rewards that can be obtained.
[0021] Preferably, in step S5, the optimization step of the action selection strategy is as follows:
[0022] For each Q-table row corresponding to a performance metric interval, the action with the largest Q value in that row is selected as the baseline optimal action for the current performance metric interval. The baseline optimal action corresponds to a set of parameter combinations of a stochastic parallel gradient descent algorithm.
[0023] Centered on the parameter combination of the benchmark optimal action, the width of the parameter interval used in the last parameter update is reduced to 1 / M of the original, where M is a preset fixed number of parameter combinations. M new parameter combinations are generated within the reduced parameter interval as candidate actions for this performance measurement interval.
[0024] Based on the Q values of the candidate action's neighboring actions in the parameter space, the initial Q value estimate of the candidate action is obtained by interpolation.
[0025] The M groups of candidate actions and their corresponding initial Q-value estimates are used to replace the action items and Q-values corresponding to the performance metric interval in the original Q-table, thus completing the Q-table update.
[0026] Preferably, in step S6, the preset optimization condition is that the width of the parameter range for action update is less than a specified value or reaches the preset maximum number of iterations.
[0027] Compared with the prior art, the present invention has the following advantages:
[0028] 1. Compared with the existing SPGD algorithm, the adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention has a faster control convergence speed and more stable operation after convergence: The adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention can automatically find the optimal parameter combination according to different performance index ranges, so that the optimized SPGD algorithm has near-optimal performance in each performance index range; Compared with the existing SPGD algorithm, the perturbation parameter does not change throughout the control process, the present invention can usually find smaller and better perturbation parameters in the convergence range, so that the perturbation is minimized while ensuring control performance, and the operation is more stable.
[0029] 2. The adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention does not require manual intervention and can automatically explore the environment to find the optimal parameters of SPGD under different conditions: Compared with the existing SPGD algorithm, which requires manual setting and adjustment of parameters according to physical environment characteristics and hardware parameters, the adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention can automatically optimize the parameters of the SPGD algorithm under various conditions to obtain the optimal performance through interaction with the environment, without the need for external manual intervention.
[0030] 3. The adaptive stochastic parallel gradient descent parameter optimization algorithm of this invention requires minimal structural changes to the SPGD runtime, is easy to implement, and does not affect the original SPGD's running speed: After parameter optimization, this invention only needs to add a lookup table to the original SPGD algorithm to obtain the optimal parameters based on the current performance indicators, making it suitable for high-speed implementation on platforms such as FPGA and DSP. Compared to existing SPGD improvement algorithms, it eliminates the need for complex mathematical calculations that increase implementation difficulty and running time, facilitating system design and engineering applications.
[0031] 4. The adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention introduces a discretized Q-Learning agent, which can select the optimal SPGD parameter combination in real time according to the characteristics of different environments and the current beam synthesis performance index, so that the SPGD algorithm is approximately in the optimal state under various conditions.
[0032] 5. The adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention retains the advantages of the original SPGD algorithm, which is simple in structure and easy to implement on FPGA hardware. At the same time, it does not require manual parameter adjustment, image sensor, or additional complex calculations, and has strong engineering feasibility. Attached Figure Description
[0033] Figure 1 This is a flowchart illustrating the optimization process of the adaptive stochastic parallel gradient descent parameter optimization algorithm based on reinforcement learning, as described in this invention.
[0034] Figure 2 This is a flowchart of the SPGD operation proposed in this invention.
[0035] Figure 3 The beam combining effect of the coherent combining system under the optimized SPGD in an embodiment of the present invention is shown. Detailed Implementation
[0036] The present invention will be further described below with reference to specific embodiments and accompanying drawings. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
[0037] See Figure 1 The implementation of the reinforcement learning-based adaptive stochastic parallel gradient descent parameter optimization algorithm of the present invention includes the following steps:
[0038] Step 1: Obtain the performance evaluation index value of the coherent synthesis system at the current iteration time, and map the performance evaluation index value to the state of the reinforcement learning model according to the pre-set segmentation rules;
[0039] Step 2: Construct a reinforcement learning model for adaptive optimization of the parameters of the stochastic parallel gradient descent algorithm. The reinforcement learning model adopts a Q-table-based learning structure, wherein: the Q-table is constructed according to the number of segments of the system performance evaluation index and the number of parameter combinations of the stochastic parallel gradient descent algorithm. Each row of the Q-table corresponds to a segment interval of the performance evaluation index (i.e., a unique state code), and each column corresponds to a set of parameter combinations of the stochastic parallel gradient descent algorithm. In the Q-table row corresponding to the current state, an action is selected according to the Q value and a preset probability strategy, and the parameter combination corresponding to the selected action is output as the control parameters of the stochastic parallel gradient descent algorithm.
[0040] Step 3: Update the control parameters of the stochastic parallel gradient descent algorithm based on the parameter combination output by the reinforcement learning model, and execute the stochastic parallel gradient descent algorithm in the coherent synthesis system for phase control;
[0041] Step 4: Obtain the performance evaluation index value of the coherent synthesis system after executing the stochastic parallel gradient descent algorithm, and construct the reward value of the reinforcement learning model based on the performance evaluation index values before and after phase control. The reward value is the performance evaluation index J after applying the action. t Performance evaluation index J before the action is performed t-1 The difference is calculated, and the Q-value of the corresponding state-action item in the Q-table is updated based on the reward value to update the reinforcement learning policy.
[0042] Step 5: Repeat steps 1 to 4 until the expected value in the Q table stabilizes or reaches the preset condition. Then update the action based on the distribution of the expected value in each performance metric interval in the Q table.
[0043] Step 6: Repeat steps 1 to 5 until the action meets the preset conditions. Select the action with the largest Q value corresponding to each interval to form a lookup table as the policy after reinforcement learning optimization.
[0044] In step 1, the performance evaluation index can be one or more of the following: barrel power, absolute output power, far-field peak intensity, main lobe energy ratio, Strell ratio, beam quality factor, enclosing energy, interference fringe contrast, phase consistency error, wavefront root mean square error, and sidelobe suppression ratio. In this embodiment, normalized barrel power is used.
[0045] In addition, in step 2, each entry in the Q table contains a counter N, which is used to record the number of times the corresponding state-action item is accessed.
[0046] In addition, in step 2, the action selection based on the Q value according to the preset probability strategy is specifically as follows: select the action with the largest Q value in the Q table row corresponding to the current state with probability ε, and select any other action in the row with equal probability from 1 to ε, where ε is a preset hyperparameter used to adjust the exploration intensity.
[0047] Additionally, in step 4, the Q values of the state-action items in the Q table are updated as follows:
[0048] ;
[0049] ;
[0050] Among them, s t Indicates the current state, a t This indicates the action selected in the stated state, r t Indicates that in state s t and action a t The reward value obtained, Q(s) t ,a t ) represents the Q value of the corresponding state-action item in the Q table.
[0051] Specifically, in step 5, the update of the metric interval includes the following steps:
[0052] Step 501: For each interval, select the parameter combination with the largest Q value.
[0053] Step 502: Update the width of the interval based on the width of the previous parameter interval and the number of parameter intervals. , where M is the number of parameter intervals.
[0054] Step 503: Using the parameter combination with the largest Q value as the center, generate M new parameters with the currently updated interval width.
[0055] Step 504: For the new parameters, obtain an initial Q-value estimate through linear interpolation of neighboring parameter combinations to make the training process more stable.
[0056] ;
[0057] in, , for The nearest point, For new parameter combinations The estimated initial Q value, These are the normalized parameters.
[0058] In addition, there are multiple options for updating the interval width in step 502. It is only necessary to ensure that the interval width gradually shrinks during each update. In this embodiment, the interval width is changed to 1 / M of the original width, where M is the number of parameter intervals.
[0059] Furthermore, in step 5, the specific details of determining whether the estimated Q values in the Q table tend to stabilize are as follows: when the rate of change of all estimated Q values in the Q table after the update is less than a preset fixed threshold τ:
[0060] ;
[0061] This can be used to determine that the estimation tends to be stable, or simply to assume that the algorithm has reached a state of stable estimation after the number of updates reaches the preset number of rounds.
[0062] In addition, the specific details of the termination of the action when the preset conditions are met in step 6 are as follows: when the action update interval is less than the preset minimum interval width or the number of iterations reaches the preset maximum number of iterations, the optimization loop terminates.
[0063] See Figure 2 The SPGD proposed in this invention operates as follows after the strategy optimization is completed: the sensor obtains the current performance metric value, determines which interval the current metric value is in, then finds the optimal parameter combination in the current interval through a lookup table, loads the parameter combination into the traditional SPGD algorithm, and generates control actions.
[0064] See Figure 3 With the above settings, after the reinforcement learning algorithm is optimized, the convergence speed of the SPGD proposed in this invention is significantly better than that of traditional SPGD and several representative optimized variants. Furthermore, because near-optimal parameter combinations can be obtained in various parameter measurement intervals through optimization, the perturbation amplitude and stability of the SPGD proposed in this invention are significantly better than existing SPGD and its variants during the control convergence phase. At the same time, compared with various existing variable parameter SPGD algorithms, the SPGD proposed in this invention requires less modification to the SPGD algorithm execution process and incurs less additional runtime overhead, making it easier to apply on high-speed hardware such as FPGAs and DSPs, thus facilitating engineering implementation.
[0065] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A stochastic parallel gradient descent phase control method based on reinforcement learning optimization, characterized in that, Includes the following steps: Step 1: Obtain the performance evaluation index value of the coherent synthesis system at the current iteration time, and discretize the performance evaluation index value according to the pre-set segmentation rules to map it into a unique state code of the reinforcement learning model, wherein the state code corresponds one-to-one with the performance measurement interval; Step 2: Construct a reinforcement learning model for adaptive optimization of the parameters of the stochastic parallel gradient descent algorithm. The reinforcement learning model adopts a learning structure based on a Q-table. The dimension of the Q-table is determined by the number of unique state codes for the performance evaluation index and the number of parameter combinations for the stochastic parallel gradient descent algorithm. Each row in the Q-table corresponds to a unique state code output in Step 1, and each column corresponds to a set of parameter combinations for the stochastic parallel gradient descent algorithm. In the Q-table row corresponding to the current state, select an action according to the Q-value and a preset probability strategy, and output the parameter combination corresponding to the selected action. Step 3: Update the control parameters of the stochastic parallel gradient descent algorithm based on the parameter combination output by the reinforcement learning model, and execute the stochastic parallel gradient descent algorithm in the coherent synthesis system to complete the phase control adjustment of the current iteration; Step 4: Obtain the performance evaluation index value of the coherent synthesis system after executing the stochastic parallel gradient descent algorithm described in Step 3. Calculate the difference between the performance evaluation index before and after phase control, and use the difference as the reward value of the reinforcement learning model. Based on the reward value, update the Q value of the current state-action corresponding item in the Q table. Step 5: Repeat steps 1 to 4 until the expected value of each state-action item in the Q table tends to stabilize or meets the preset convergence condition; then optimize the action selection strategy according to the distribution of the expected value in the corresponding row of each state code in the Q table. Step 6: Repeat steps 1 to 5 until the action selection strategy meets the preset optimization conditions; select the action with the largest Q value in the Q-table row corresponding to each state code, and construct a lookup table for the optimal parameter combination of state code and stochastic parallel gradient descent algorithm, which serves as the optimal phase control strategy of the stochastic parallel gradient descent algorithm after reinforcement learning optimization.
2. The stochastic parallel gradient descent phase control method based on reinforcement learning optimization according to claim 1, characterized in that, In step S1, the performance evaluation index adopts one or more of the following: barrel power, absolute output power, far-field peak intensity, main lobe energy ratio, Strell ratio, beam quality factor, enclosing energy, interference fringe contrast, phase consistency error, wavefront root mean square error, and sidelobe suppression ratio.
3. The stochastic parallel gradient descent phase control method based on reinforcement learning optimization according to claim 1, characterized in that, In step S2, each entry in the Q table contains a counter, which is used to record the number of times the corresponding state-action item is accessed.
4. The stochastic parallel gradient descent phase control method based on reinforcement learning optimization according to claim 1, characterized in that, In step S2, the preset probability strategy is an ε-greedy strategy, and the specific steps are as follows: The action with the largest Q value in the Q-table row corresponding to the current state is selected with probability ε. Any action is randomly selected from the remaining actions in the row with equal probability 1-ε, where ε is a pre-set hyperparameter used to adjust the exploration intensity.
5. The stochastic parallel gradient descent phase control method based on reinforcement learning optimization according to claim 1, characterized in that, In step S4, the Q value of the state-action item in the Q table is updated as follows: ； ； Among them, s t Indicates the current state, a t This represents the action selected in the current state, r. t Indicates the current state s t Execute action a t The reward value obtained later, Q(s) t ,a t ) indicates the current state s t Execute action a t Then, the expected value of the long-term cumulative rewards that can be obtained.
6. The stochastic parallel gradient descent phase control method based on reinforcement learning optimization according to claim 1, characterized in that, In step S5, the optimization steps of the action selection strategy are as follows: For each Q-table row corresponding to a performance metric interval, the action with the largest Q value in that row is selected as the baseline optimal action for the current performance metric interval. The baseline optimal action corresponds to a set of parameter combinations of a stochastic parallel gradient descent algorithm. Centered on the parameter combination of the benchmark optimal action, the width of the parameter interval used in the last parameter update is reduced to 1 / M of the original, where M is a preset fixed number of parameter combinations. M new parameter combinations are generated within the reduced parameter interval as candidate actions for this performance measurement interval. Based on the Q values of the candidate action corresponding to the neighboring actions in the parameter space, the initial Q value estimate of the candidate action is obtained by interpolation. Replace the action items and Q values corresponding to the performance metric interval in the original Q table with the M groups of candidate actions and their corresponding initial Q value estimates, and complete the Q table update.
7. The stochastic parallel gradient descent phase control method based on reinforcement learning optimization according to claim 1, characterized in that, In step S6, the preset optimization condition is that the width of the parameter interval for action update is less than a specified value or reaches the preset maximum number of iterations.