A Reinforcement Learning-Based Adaptive Testing and Fast Convergence Method for Optical Modules

Abstract

This invention relates to the field of optical communication technology and discloses an adaptive testing and fast convergence method for optical modules based on reinforcement learning. Through an offline-trained RL agent module, this invention can dynamically generate and execute optimal test sequences for optical modules of different types or states, avoiding redundant steps in fixed processes. While ensuring test coverage, it significantly shortens test time and improves the detection probability of potential defects. Through closed-loop testing integrating the RL agent module, test actions can be adjusted in real time according to the state of individual optical modules, enabling the test process to quickly adapt to individual device differences and process fluctuations, achieving online and rapid convergence of the test strategy, and ensuring the accuracy and stability of the test. Through online iterative optimization of the RL agent module, it can continuously learn new data and changes in the production process, allowing the test strategy to continuously evolve and maintain optimal performance over the long term.

Description

Technical Field

[0001] This invention relates to the field of optical communication technology, and in particular to a method for adaptive testing and fast convergence of optical modules based on reinforcement learning. Background Technology

[0002] As a core component of optical communication systems, optical modules are primarily responsible for converting photoelectric signals, and their performance and reliability directly determine the quality of the communication network. With the development of data centers, 5G, and high-speed optical networks, optical modules are evolving towards higher speeds, smaller packages, and lower power consumption, resulting in increasingly complex internal structures and more precise manufacturing processes.

[0003] In existing large-scale production lines, the testing and debugging of optical modules is crucial to ensure the yield rate of each stage. Currently, the industry generally uses ATE systems to execute preset fixed testing procedures, which has the following problems: Due to the inherent differences in response speed, detection accuracy, and coverage of different process stages of optical modules, the test parameters and procedures of ATE systems are usually static and uniform, making it difficult to adaptively match the dynamic characteristics of different modules or process stages, resulting in low testing efficiency or missed potential defects; Relying on manual experience to adjust testing strategies and assess risks often involves delays of minutes from the occurrence of anomalies to identification and adjustment of test parameters, resulting in slow response; The lack of real-time fusion and intelligent analysis of multi-dimensional data during the production process makes it impossible to predict and warn of potential risks in advance, let alone provide dynamically optimized testing strategies, thus hindering the improvement of production efficiency. Summary of the Invention

[0004] The purpose of this invention is to address the technical problems of slow response, poor flexibility, and lack of predictability in existing fixed testing processes for optical modules, and to provide an adaptive testing and fast convergence method for optical modules based on reinforcement learning.

[0005] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A reinforcement learning-based adaptive testing and fast convergence method for optical modules includes the following sub-steps:

[0007] Based on the Markov decision process model as the rule framework and the deep Q network as the core network, an RL agent module is constructed.

[0008] By utilizing the ATE system, integrating the RL intelligent agent module and the ATE system interface, it works collaboratively with the test environment interaction module, state perception module, action execution module, and reward calculation module to jointly build an intelligent adaptive test system;

[0009] Using an intelligent adaptive testing system, an RL agent module is trained offline. The RL agent module interacts with the simulated testing environment to learn the optimal testing and debugging strategy for the target optical module.

[0010] After the RL agent module is trained offline, it is tested online with the actual optical module. The RL agent module and the actual optical module interact in the test scenario to achieve adaptive testing and fast convergence.

[0011] After the RL agent module completes online testing, it undergoes iterative optimization, continuously learning new changes in the actual optical module production process.

[0012] To address the issues of low testing efficiency, missed potential defects, poor flexibility, and lack of predictability in optical module testing, this application presents an RL (Reactive Learning) agent module based on a Markov decision process model and a deep Q-network. This module interacts with an intelligent adaptive testing system and employs a closed-loop learning process of offline training, online application, and continuous optimization to solve these technical problems. Regarding the issues of low testing efficiency and missed potential defects, the RL agent module models the testing process as a Markov decision process model. Through a deep Q-network, it learns the optimal mapping between states and actions. During the offline training phase, the RL agent module learns an efficient and high-coverage testing strategy that maximizes defect detection while avoiding unnecessary testing steps through extensive trial and error in a simulated environment. This fundamentally optimizes the testing sequence and improves detection rate and efficiency. To address the issues of poor flexibility and difficulty in adapting to dynamic production lines in testing systems, this application utilizes an RL agent module as the dynamic decision-making core during online testing. Based on real-time response data from the actual optical modules acquired by the state perception module, the module performs online fine-tuning and dynamically adjusts test stimuli, judgment thresholds, or measurement sequences through the action execution module. This achieves closed-loop control with real-time feedback, enabling rapid adaptation and convergence of testing strategies for different batches and optical modules with varying characteristics. To address the lack of predictability and inability to continuously optimize, this application continuously collects new state-action-reward data generated during online testing. This data is used for periodic incremental training of the RL agent module, allowing it to continuously learn about production line processes. This not only enables real-time adaptation but also, through learning from historical data, gradually develops the ability to predict potential testing risks or performance degradation trends.

[0013] Compared with existing technologies, the advantages of this application are as follows: Through offline-trained RL agent modules, optimal test sequences can be dynamically generated and executed for optical modules of different types or states, avoiding redundant steps in fixed processes. This significantly shortens test time while ensuring test coverage and increases the detection probability of potential defects. Through closed-loop testing integrating RL agent modules, test actions can be adjusted in real time according to the state of individual optical modules, enabling the test process to quickly adapt to individual device differences and process fluctuations, achieving rapid online convergence of the test strategy and ensuring test accuracy and stability. Through online iterative optimization of the RL agent modules, new data and changes in the production process can be continuously learned, allowing the test strategy to continuously evolve and maintain optimal performance over the long term.

[0014] Furthermore, based on the reinforcement learning-based adaptive testing and fast convergence method for optical modules, the construction of the RL agent module includes the following sub-steps:

[0015] Based on the state space and action space, a Markov decision process model is constructed by combining transition probabilities, reward functions, and discount factors.

[0016] A deep Q-network is constructed based on the main network and the target network, combined with a training mechanism.

[0017] In the aforementioned solutions, existing technologies typically employ methods based on fixed rules or static strategy libraries when constructing the decision-making core of intelligent testing systems. These methods rely on manually preset logic, making it difficult to accurately characterize and adapt to the high-dimensional, nonlinear state changes introduced by individual differences and complex couplings in optical module testing. This results in poor flexibility and low efficiency in the generated testing strategies, failing to achieve rapid convergence and continuous optimization in dynamic environments. This application addresses the technical problem of insufficient adaptive capability of decision models in dynamic testing scenarios by constructing an RL intelligent agent module with a Markov decision process model as the rule framework and a deep Q-network as the core network, integrating rule-based guidance and data-driven learning. Specifically, a Markov decision process model is constructed based on state space, action space, and reward function, injecting the test optimization goals and constraints into the model in a clear mathematical form. This provides a structured framework with clear physical meaning and business orientation for the learning process. The introduction of a deep Q-network based on a master-target network architecture and experience replay mechanism enables efficient and stable learning of complex state-action value mappings from historical interaction data with the environment, thereby dynamically supplementing and optimizing decision details that are difficult to predefine in the rule framework. This application combines a Markov decision process model as the rule framework with a deep Q-network as the core network, enabling the RL agent module to autonomously learn high-dimensional strategies through data interaction under a clear optimization direction. This improves the adaptability of the decision model to complex dynamic testing environments and the efficiency of strategy generation, ensuring that the RL agent can quickly produce efficient test sequences that fit the real-time state. Thus, it provides core, learnable decision-making capabilities for adaptive testing and rapid convergence.

[0018] Furthermore, in the reinforcement learning-based adaptive testing and fast convergence method for optical modules, the state space includes test parameter data and device response data of the optical module. The test parameter data includes bias current, TEC temperature, driving voltage, and modulation current, which reflect the test state of the optical module. The device response data includes output optical power, extinction ratio, eye diagram jitter, and threshold voltage, which reflect the performance state of the optical module.

[0019] In the aforementioned solutions, existing technologies often suffer from limitations in defining and sensing the state of optical modules during testing. This manifests as reliance on a limited number of direct test readings (e.g., only optical power) while neglecting related parameters such as driving conditions (e.g., bias current), or simply listing multi-dimensional data without characterizing its intrinsic relationship with device performance at the system level. This results in subsequent intelligent decisions (e.g., action selection) being based on incomplete information, severely restricting the accuracy and rapid convergence of adaptive testing. This application defines test parameter data (including bias current, TEC temperature, driving voltage, and modulation current) as reflecting the test conditions and excitation states applied to the optical module; and defines device response data as reflecting the key performance states exhibited by the optical module under the aforementioned excitations. This solves the technical problem of incomplete and inaccurate state characterization, which restricts the accuracy of intelligent decision-making. This application defines a two-dimensional, measurable state space, enabling the state perception module to provide the RL agent module with a comprehensive, accurate, and physically meaningful state perception. This provides a reliable information foundation for the agent to accurately understand the current test situation and evaluate the long-term value of actions, fundamentally ensuring the effectiveness of subsequent learning and decision-making. Accurate state perception is a prerequisite for adaptive decision-making, significantly improving the ability to grasp the individual characteristics and instantaneous state of the optical module, thereby supporting efficient and accurate test strategy generation and rapid convergence.

[0020] Furthermore, in the reinforcement learning-based adaptive testing and fast convergence method for optical modules, the action space adjusts the test parameter data in the state space, defining parameter adjustment instructions for the RL agent module, as follows:

[0021] ;

[0022] in, Let be the action at time t in the action space. This is the minimum adjustment step size for the bias current. This is the minimum adjustment step for the TEC temperature. This is the minimum adjustment step size for the modulation current. This is the minimum adjustment step size for the drive voltage. This is an operation to add test parameter data. This is an operation to reduce the amount of test parameter data.

[0023] In the aforementioned solutions, existing technologies, when defining the control instruction set during the optical module testing process, typically use parameter adjustment instructions based on fixed thresholds or empirical rules in a binary pass / fail manner, or allow continuous adjustments within a wide range without a definite step size. The former leads to insufficient control precision, making fine optimization impossible. The latter results in an excessively large and unstructured action space, leading to low learning efficiency and slow convergence of the agent, making it difficult to stably find the optimal test parameter configuration. This application addresses the technical problem of low learning efficiency caused by imprecise control instruction granularity and lack of constraints in the exploration space by defining a discretized, structured action space directly corresponding to the test parameters. This application transforms the complex parameter optimization problem into a series of explicit and executable fine-tuning instructions, providing a clear, operable, and physically meaningful interface for the decision output of the RL agent module. By constraining the action exploration space within a reasonable engineering range, it significantly reduces the complexity and randomness of the RL agent module's learning, enabling the RL agent module to efficiently explore the parameter space and accurately approximate the optimal operating point of the optical module through a limited, structured sequence of actions.

[0024] Furthermore, in the reinforcement learning-based optical module adaptive testing and fast convergence method, the main network includes an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0025] The main network processing steps are as follows: the standardized state vector is input into the input layer, and the input layer extracts features and outputs them to the first hidden layer;

[0026] The first hidden layer uses the ReLU function to activate features, which alleviates the gradient vanishing problem in the training of deep Q network and enhances the nonlinear fitting ability of deep Q network, enabling deep Q network to learn the complex mapping relationship between state and action. The activated features output from the first hidden layer are input into the second hidden layer.

[0027] The second hidden layer uses the ReLU function to activate features, thereby reducing the number of neurons to compress and refine features, remove redundant features, improve the generalization ability of the deep Q network, and avoid overfitting. The second hidden layer outputs the reactivated features to the output layer, and the output layer outputs the Q value of the corresponding action.

[0028] Furthermore, regarding the reinforcement learning-based adaptive testing and fast convergence method for optical modules, the preparatory work for the intelligent adaptive testing system includes:

[0029] Collect historical test data for the target optical module model, format the historical test data into historical samples, and store them in the experience playback pool. This forms an initial experience base;

[0030] Based on the testing requirements of the target optical module and industry standards, the parameter range of the Markov decision process model is set.

[0031] The weights of the main network and the target network are initialized using a random normal distribution, and the experience replay pool is initialized accordingly. .

[0032] Furthermore, the adaptive testing and fast convergence method for optical modules based on reinforcement learning includes the following sub-steps in which the RL agent module interacts with the simulated testing environment:

[0033] An initial experience base is built using historical test data, test rules are defined, and the RL agent module is initialized.

[0034] After the RL agent is initialized, the simulation test environment is reset, and the RL agent module is... The greedy strategy selects an action, executes it to change the test parameters of the optical module, and after execution, collects the new state and corresponding immediate reward, storing the experience sample in the experience replay pool. ;

[0035] The experience sample collection is performed in a loop, and the experience playback pool is updated as needed. When the number of samples reaches the sample capacity, offline network training of the RL agent module begins, and the completion of training is determined by accumulating rewards.

[0036] In the aforementioned solutions, existing technologies typically face problems such as inefficient training initialization and unstable exploration during offline training when starting and running adaptive testing systems based on reinforcement learning. This manifests as follows: historical data is not effectively constructed into an initial experience base that the agent can directly learn from, resulting in highly blind and inefficient early exploration; key model parameters (such as action range and reward function) rely on manual experience settings, which are subjective and affect the direction of policy optimization; and the lack of systematic design in network weight initialization and action selection strategies leads to slow convergence, drastic fluctuations, or getting stuck in local optima in the early stages of training. Ultimately, this results in slow startup, low policy learning efficiency, and unreliable convergence results during the offline training phase, making it difficult for the trained RL agent module to achieve the expected adaptive and fast convergence performance in actual online testing. This application addresses several key issues. First, by collecting and formatting historical data and storing it in an experience replay pool to build an initial experience base, it provides the agent with prior knowledge, accelerating early learning. Second, by setting the parameter range of the Markov decision process model according to testing requirements and industry standards, it provides precise guidance for the learning process that conforms to engineering constraints. Third, by initializing network weights using a random normal distribution, it lays a solid foundation for stable training of the deep Q-network. Fourth, by employing an ϵ-greedy strategy to select actions in environmental interactions, it achieves a dynamic balance between exploration and exploitation. Fifth, by constructing complete experience samples and storing them in the experience replay pool, it accumulates diverse training data. Sixth, by setting a sample size threshold before starting network training and using cumulative rewards as a convergence criterion, it ensures that training is started based on sufficient and deassociative data, providing an objective termination standard. This solves the technical problems of unscientific offline training initialization, inefficient exploration, and unstable training. This application constructs an efficient, stable, and goal-oriented offline training closed loop, enabling the RL agent module to quickly and stably learn high-quality and highly adaptive testing and debugging strategies in a simulated environment using limited prior knowledge. This improves the startup speed, learning efficiency, and strategy reliability of offline training, and provides a fully optimized and directly deployable intelligent decision-making core for subsequent online implementation of adaptive testing and rapid convergence of the optical module.

[0037] Furthermore, based on the reinforcement learning-based adaptive testing and fast convergence method for optical modules, the offline network training of the RL agent module includes the following sub-steps:

[0038] Judgment Experience Replay Pool If the number of samples in the pool has not reached the sample capacity, the empirical sample collection will be repeated. If the sample capacity has been reached, the samples will be retrieved from the empirical replay pool. N empirical samples are randomly selected from the data to form an offline training batch;

[0039] For each empirical sample in the offline training batch, the target Q-value is calculated using the target network, with the following formula:

[0040] ;

[0041] in, For the target Q value, As a discount factor, For target network exist Time-space state Next for all actions Q-value output, For target network Predicted Time-space state The maximum Q value;

[0042] The mean squared error loss is calculated by using the predicted Q-value and the target Q-value, as follows:

[0043] ;

[0044] For mean square error loss, This refers to the offline training batch size. For the first The target Q value for each sample For the first The predicted Q value for each sample;

[0045] The Adam optimizer is used to minimize the loss function through backpropagation, the weight parameters of the deep Q network are updated, and the weight parameters of the deep Q network are synchronized to the target network.

[0046] The variance of the test convergence step is calculated using the following formula:

[0047] ;

[0048] in, for The variance of the test convergence steps for a series of consecutive offline training batches. This represents the number of offline training batches used for continuous evaluation. For the first In each offline training batch, the number of iterations required for the RL agent to bring the performance of the optical module in the simulated test environment to a qualified standard (i.e., the number of loops of "state perception-action execution-reward feedback" in that batch). for The average number of test convergence steps for a consecutive offline training batch;

[0049] when Accumulated rewards for consecutive offline training batches Furthermore, when the variance of the test convergence step is less than 5%, it is considered that the optimal test and debugging strategy for the target optical module has been adapted, the offline training of the RL agent is stopped, and the optimal weight parameters of the deep Q network are saved.

[0050] In the aforementioned schemes, existing technologies typically face problems of low training efficiency and insufficient stability when performing offline network training for deep reinforcement learning. This results in lengthy offline training phases, large fluctuations in policy performance improvement curves, and difficulty in guaranteeing the quality and reliability of the ultimately trained policy. This application sets a sample size threshold and randomly samples from experience replays to form training batches, breaking the temporal correlation between experience samples and making the sampled data closer to independent and identically distributed, thus improving training stability. A target network with an update cycle different from the main network is used to provide a stable target estimate, solving the training instability problem caused by the rapid changes in target estimate with the main network. A mean squared error loss is constructed, and the Adam optimizer is used to update network parameters. The error between the predicted Q-value and the target Q-value is calculated, and the Adam optimizer minimizes this loss through backpropagation. The Adam optimizer can achieve faster and more robust convergence in complex parameter spaces. This application constructs an efficient and stable offline learning mechanism for deep Q-networks. The introduction of random experience sampling and the target network significantly improves the stability of the training process. The application of the Adam optimizer accelerates the convergence speed of network weight optimization, enabling the RL agent module to efficiently and reliably learn accurate value function estimates from historical and simulated interaction data. This lays a solid model foundation for generating high-performance adaptive testing strategies and ensures the achievement of the goal of rapid convergence in the subsequent online testing phase.

[0051] Furthermore, the online testing of the actual optical module based on reinforcement learning-based adaptive testing and fast convergence method includes the following sub-steps:

[0052] The ATE system interface module connects the RL agent to the automated test equipment, adapts the communication protocol, initializes the main network according to the optimal parameters of the deep Q network, and provides a target performance set as a benchmark for testing and debugging.

[0053] The state perception module collects the initial state of the current optical module, standardizes the initial state data and inputs it into the RL agent module. The RL agent module calculates the Q value of all actions in the initial state through the main network, selects the action with the largest Q value as the optimal action, and outputs parameter adjustment instructions to the action execution module.

[0054] The action execution module converts the output parameter adjustment instructions into control signals that the automatic test equipment can recognize, driving the automatic test equipment to perform parameter adjustment operations;

[0055] After the parameters are adjusted, the state perception module collects the new state of the optical module and the corresponding device response data again, and the reward calculation module calculates the instant reward based on the new state and the target performance set.

[0056] The repeated execution state sensing module collects the optical module's status and outputs parameter adjustment instructions to the action execution module. The action execution module drives the automatic testing equipment to perform parameter adjustment operations until the optical module's performance meets the qualification standard.

[0057] In the aforementioned solutions, the core deficiency of existing technologies in automating online testing of optical modules lies in the static and open-loop nature of their decision-making logic. The testing system can only execute operations according to a preset, linear script, and cannot make real-time decisions and optimize paths based on the dynamic response of the optical module under test. When the optical module deviates from the expected behavior due to manufacturing tolerances or test environment disturbances, the system lacks the ability to adjust its strategy online and can only rely on completing all preset steps or performing inefficient repetitive scans to ensure coverage. This results in unpredictable testing time, low debugging efficiency, and difficulty in achieving rapid and accurate convergence of test parameters to the optimal operating point. This application achieves hardware and software integration and communication adaptation through the ATE system interface module, initializes the network with the optimal strategy obtained from offline training, and loads the target performance set as a benchmark to lay the foundation for online adaptive testing. The system uses a closed loop of "perception-decision-execution" where the state perception module collects data in real time, the RL agent module calculates and outputs the action with the maximum Q value based on the main network, and the action execution module drives the ATE to perform adjustments. After each adjustment, the reward calculation module generates an instant reward based on the difference between the new state and the target performance set. This feedback signal drives the loop to continue until the performance target is met. The high-dimensional policy model obtained from offline training is transformed into a serialized control command with millisecond-level response to the actual optical module. This enables the RL agent module to mimic expert experience and dynamically plan the shortest path debugging action sequence based on the unique real-time feedback of each module. This significantly reduces the number of iterations and testing time required to achieve qualified performance, realizes fast and adaptive convergence of the online testing process, directly improves the throughput and intelligence level of production line testing, and solves the technical problem that static test scripts cannot be dynamically optimized, resulting in low online debugging efficiency and slow convergence speed.

[0058] Furthermore, based on the reinforcement learning-based adaptive testing and fast convergence method for optical modules, the iterative optimization of the RL agent module includes the following sub-steps:

[0059] Newly generated valid experience samples during online testing will be continuously stored in the experience replay pool. Expand training data;

[0060] Set a regular iteration cycle and utilize the experience replay pool. Incremental training with accumulated effective experience samples to update the weight parameters of the deep Q network;

[0061] Save the updated weight parameters of the deep Q network. The generated weight parameters will be used for online testing of the optical module the next day.

[0062] In the aforementioned solutions, the decision-making model of existing adaptive testing systems becomes fixed after deployment, failing to absorb newly generated data and knowledge from online production to cope with dynamic changes in the production line. This leads to performance degradation and loss of adaptive advantages after long-term operation. This application addresses the technical problem of performance degradation due to environmental changes during long-term system operation by establishing an iterative optimization mechanism for the RL agent module. Specifically, new experience samples generated by continuous online testing are stored in an experience replay pool to provide a data foundation for iteration. A fixed period is set, and the deep Q-network is incrementally trained using samples in the pool to update its strategy. The updated network parameters are saved and deployed for subsequent production testing. This application uses the RL agent module for iterative optimization, endowing the RL agent module with continuous online learning and self-optimization capabilities, enabling its strategy to evolve synchronously with the production process, thereby ensuring continuous, stable, high-precision, and high-efficiency testing. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of a reinforcement learning-based adaptive testing and fast convergence method for optical modules.

[0065] Figure 2 This is a schematic diagram of the structure of the RL agent module.

[0066] Figure 3 This is a schematic diagram of the main network structure.

[0067] Figure 4 This is a schematic diagram of the structure of an intelligent adaptive testing system.

[0068] Figure 5 This is a flowchart for online testing of optical modules. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0070] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements or an indirect connection via other elements.

[0071] This invention is achieved through the following technical solutions, such as... Figure 1 As shown, the reinforcement learning-based adaptive testing and fast convergence method for optical modules includes the following steps:

[0072] like Figure 2 As shown, S1: Based on the Markov Decision Process (MDP) model as the rule framework and the Deep Q Network (DQN) as the core network, an RL agent module is constructed.

[0073] Specifically, S11: Based on the state space and action space, a Markov decision process model is constructed by combining transition probabilities, reward functions, and discount factors. The formula is as follows:

[0074] ;

[0075] in, For Markov decision process models, For state space, For the action space, For the transition probability, For the reward function, This is the discount factor.

[0076] More specifically, the state space includes test parameter data and device response data of the optical module. The test parameter data includes bias current, TEC temperature, drive voltage, and modulation current, which reflect the test status of the optical module. The device response data includes output optical power, extinction ratio, eye diagram jitter, and threshold voltage, which reflect the performance status of the optical module.

[0077] The state space reflects the test and performance states of the optical module at time t, providing a decision-making basis for the RL agent module. The formula is:

[0078] ;

[0079] in, Let t be the spatial state. The bias current at time t (unit: mA) TEC temperature at time t (unit: ), The driving voltage at time t (unit: V). Let t be the modulation current (unit: mA). The output optical power at time t (unit: dBm) The extinction ratio at time t (unit: dB). Eye diagram jitter at time t (unit: ps) Let t be the threshold voltage (in V). For transpose;

[0080] The action space adjusts the test parameter data in the state space, defining parameter adjustment instructions for the RL agent module, with the following formula:

[0081] ;

[0082] in, Let be the action at time t in the action space. This is the minimum adjustment step size for the bias current. This is the minimum adjustment step for the TEC temperature. This is the minimum adjustment step size for the modulation current. This is the minimum adjustment step size for the drive voltage. This is an operation to add test parameter data. This is an operation to reduce test parameter data;

[0083] In this embodiment, even minor changes in test parameter data can alter the performance state of the optical module. To ensure adjustment accuracy, a minimum adjustment step size is set according to industry standards for optical module testing. This ensures that testing requirements are met while avoiding slow convergence due to excessively small step sizes. , , , Thus, the RL agent module can adjust the test parameter data by selecting different actions.

[0084] More specifically, the transition probability reflects the transition of the optical module to the action module after the action at time t in the action space. The probability of the spatial state at time step is given by the formula:

[0085] ;

[0086] in, For optical module transfer to The probability of the spatial state at any given time. for Temporal and spatial states;

[0087] The transition probability is determined by the device characteristics of the optical module, the accuracy of the automatic testing equipment, and the noise of the testing environment. Individual differences in the device characteristics of the optical module lead to different performance responses under the same parameter adjustment. The accuracy error of the automatic testing equipment affects the actual effect of parameter adjustment, and the noise of the testing environment (such as temperature fluctuations and electromagnetic interference) interferes with the acquisition results of device response data.

[0088] In this embodiment, the transition probability specifically refers to comprehensively collecting state transition data of the same type of optical module under various spatial states and action combinations (i.e., the new spatial state data of the optical module after performing different parameter adjustment actions under different initial spatial states). Based on this type of state transition data, a statistical probability distribution model is constructed. The model quantifies the state transition rules corresponding to different state-action combinations, and finally determines the specific value of the transition probability in each scenario.

[0089] More specifically, the reward function includes immediate reward and termination reward;

[0090] The reward function quantifies state transitions, enabling the RL agent module to determine whether the current action contributes to convergence towards the target performance.

[0091] The combination of target performance characteristics constitutes a target performance set, as shown in the formula:

[0092] ;

[0093] in, For the target performance set, The target output optical power of the optical module, To maximize the allowable eye diagram jitter, This represents the acceptable range for the threshold voltage. , This is the minimum boundary value of the threshold voltage. This represents the maximum boundary value of the threshold voltage.

[0094] The instant reward is obtained based on the deviation between the spatial state at time t and the target performance, as shown in the formula:

[0095] ;

[0096] in, The instantaneous reward at time t. , , , , , The weighting coefficients for the reward function are set according to the priority of optical module testing (for example, when output optical power is the core indicator, the weighting coefficient can be increased). The optimal combination of weight coefficients is determined through offline training and optimization. , This is the weighting coefficient corresponding to the optical power deviation. , The weighting coefficient corresponding to the extinction ratio deviation. The weighting coefficient corresponding to the jitter deviation. The weighting coefficient corresponding to the threshold voltage deviation. Optical power deviation reflects the difference between the actual output optical power and the target value; The extinction ratio deviation reflects the difference between the actual extinction ratio and the target value. This is the jitter deviation. A positive deviation occurs when the actual jitter exceeds the maximum allowable value, and the deviation is 0 when it does not exceed the maximum allowable value. This represents the threshold voltage deviation. A positive deviation occurs when the actual threshold voltage exceeds the acceptable range, and the deviation is 0 when it is within the range.

[0097] The termination reward incentive RL agent module quickly converges to a qualified state when , , , When all performance deviations of the optical module meet the qualification standards, the termination reward is triggered, and the formula is:

[0098] ;

[0099] in, This is the allowable deviation of optical power. The allowable deviation for the extinction ratio, The preset maximum reward, To terminate the reward;

[0100] In the embodiments, , To effectively incentivize the RL agent module to bring the optical module to a qualified state as quickly as possible, The reward is far greater than the immediate reward.

[0101] More specifically, the discount factor weighs the immediate reward and the future reward, enabling the RL agent module to consider both the immediate benefit of the current action and the cumulative benefit of future actions, thus avoiding short-term decision-making. The formula is as follows:

[0102] ;

[0103] To accumulate rewards, represent the weighted sum of all rewards from the initial step 0 to the test termination step T;

[0104] In the embodiments, , The reward decays exponentially with each step, reflecting the discount effect of future rewards.

[0105] Existing technologies, when constructing decision models to guide testing and debugging, typically employ empirical strategies based on fixed rules or simplified heuristic models. These methods often struggle to provide a precise mathematical description and systematic optimization of the dynamic process of optical module testing. The state definitions are often one-sided, focusing only on a few output performance indicators while neglecting the key test parameters driving these performance metrics. This results in the RL agent module being unable to fully perceive the complete working state of the optical module. Furthermore, the direction of parameter adjustment lacks a refined structural definition, causing the automatic debugging process to either have excessively large step sizes that prevent precise convergence or be inefficient due to an unordered search space. The goal orientation is vague, lacking an evaluation mechanism that can quantify the immediate and long-term benefits of each debugging action, leading to a blind, lengthy debugging process that is prone to getting trapped in local optima. This application precisely characterizes the complete operating condition of the optical module at time t by defining a state space that integrates test parameter data and device response data, providing a comprehensive and quantitative basis for decision-making. By designing an action space discretized with minimum adjustment step size, the complex continuous parameter optimization problem is transformed into a series of explicit and executable fine-tuning instructions, providing a structured and refined operation interface for the agent and effectively constraining the search space. By constructing a reward function that includes immediate and termination rewards and performing quantitative calculations based on the deviation from the target performance set, the high-level goal of rapidly converging to a qualified state is transformed into a numerical signal that the agent can directly optimize and pursue, achieving strong goal orientation in the debugging process. By introducing a discount factor to define cumulative rewards, the RL agent module is guided to balance immediate and long-term benefits when making action decisions, avoiding short-sighted behavior.

[0106] Specifically, S12: Based on the main network and the target network, and combined with the training mechanism, a deep Q-network (DQN) is constructed.

[0107] More specifically, such as Figure 3 As shown, the main network includes an input layer, a first hidden layer, a second hidden layer, and an output layer;

[0108] The input layer includes 8 neurons, corresponding to the dimension of the spatial state. The first hidden layer includes 64 neurons, using the ReLU function as the activation function. The second hidden layer includes 32 neurons, using the ReLU function as the activation function. The output layer includes 8 neurons, corresponding to the actions in the action space.

[0109] The processing steps of the main network are as follows:

[0110] The standardized state vector is input to the input layer. After feature extraction, the input layer outputs to the first hidden layer. The first hidden layer uses the ReLU function to activate the features, alleviating the gradient vanishing problem in deep Q-network training and enhancing the non-linear fitting ability of the deep Q-network, enabling it to learn the complex mapping relationship between states and actions. The activated features output from the first hidden layer are input to the second hidden layer, which uses the ReLU function to activate the features. By reducing the number of neurons, it achieves feature compression and refinement, removes redundant features, improves the generalization ability of the deep Q-network, and avoids overfitting. The reactivated features output from the second hidden layer are then sent to the output layer, which outputs the Q-value corresponding to the action. ;

[0111] in, To predict the Q-value, i.e., the spatial state at time t. Action at time t in the next execution action space Expected cumulative rewards;

[0112] More specifically, the target network has the same hierarchical structure as the main network, and the target network synchronizes the parameters of the deep Q network to calculate the target Q value.

[0113] More specifically, the training mechanism of the deep Q-network includes the Adam optimizer and the experience replay pool;

[0114] The Adam optimizer updates the weight parameters of the deep Q-network, and the empirical replay pool. Used to store experience samples generated from interactions between RL agents and the test environment. During the training process, batches of samples are randomly sampled from the replay pool for network training.

[0115] like Figure 4 As shown, S2: Utilizing the ATE system, the RL intelligent agent module and the ATE system interface are integrated to work in collaboration with the test environment interaction module, state perception module, action execution module, and reward calculation module to jointly build an intelligent adaptive test system;

[0116] The intelligent adaptive testing system includes an RL agent module, an ATE system interface, a test environment interaction module, a state perception module, an action execution module, and a reward calculation module.

[0117] Specifically, the RL intelligent agent module is used to realize intelligent decision-making in the testing and debugging process of optical modules;

[0118] The test environment interaction module serves as a bridge between the RL agent module and the actual test scenario, collecting status data in real time during the test process and issuing action commands generated by the RL agent module.

[0119] The state awareness module is used to acquire test parameter data and device response data of the optical module through automated testing equipment and convert them into state information in a unified format.

[0120] The action execution module is used to drive the automated testing equipment to perform parameter adjustment operations through the parameter adjustment instructions output by the RL intelligent agent module;

[0121] The reward calculation module is used to calculate the reward value in real time based on the device response data obtained by the state perception module and the preset target performance, and feed it back to the RL agent module to guide the RL agent module to learn towards the optimal decision direction.

[0122] The ATE system interface module is used to connect automated testing equipment to achieve protocol compatibility and data interaction.

[0123] The ATE system, RL agent module, and automated test equipment are each deployed with a precise PTP slave clock. The PTP master clock deployed on the control server of the ATE system ensures that the decision-making time of the RL agent module, the action execution time of the automated test equipment, and the status acquisition time are kept synchronized.

[0124] The ATE system interface module supports three mainstream interface types: GPIB, USB 3.0, and Ethernet.

[0125] S3: Utilize an intelligent adaptive testing system to train the RL agent module offline. The RL agent module interacts with the simulated testing environment to learn the optimal testing and debugging strategy for the target optical module.

[0126] Specifically, S31: Construct an initial experience base using historical test data, define test rules, and initialize the RL agent module;

[0127] More specifically, S311: Collect historical test data of the target optical module model, format the historical test data into historical samples, and store them in the experience playback pool. This forms an initial experience base;

[0128] The historical test data includes bias current, TEC temperature, drive voltage, modulation current, output optical power, extinction ratio, eye diagram jitter, and threshold voltage.

[0129] S312: Based on the testing requirements of the target optical module and industry standards, set the parameter range of the Markov decision process model;

[0130] The parameter range of the Markov decision process model is specifically as follows:

[0131] Transition probability Discount factor Bias current TEC temperature driving voltage Modulation current Output optical power Extinction ratio Eye image jitter Threshold voltage Weight coefficients of the reward function .

[0132] S313: Initialize the weights of the main network and the target network using a random normal distribution, and initialize the experience replay pool. .

[0133] It is important to note that initializing the weights of the main network and the target network ensures that the training of the RL agent module starts from a controlled, random starting point, and that the experience replay pool is initialized. S31 ensures that the experience pool starts working from a known state containing the initial experience base, rather than containing residual data from the last run. In other words, it ensures that both the initial experience base and the new interactive experiences that the agent will generate are stored simultaneously. This is the preparation that S31 makes for the RL intelligence module.

[0134] S32: After the RL agent is initialized, the simulation test environment is reset, and the RL agent module... Greedy strategy selects an action and executes the selected action. Change the test parameters of the optical module, and collect the new status after the operation is completed. With corresponding instant rewards , to use empirical samples Store in the experience replay pool ;

[0135] More specifically, S321: Reset the optical module's test parameters to the initial test state. , The greedy strategy selects the action with the highest Q value based on the depth Q network;

[0136] The core of the greedy strategy described above is to achieve a balance between trying new actions and selecting the known optimal action, specifically by using probability... Randomly select actions (exploring unknown paths to avoid the RL agent getting trapped in local optima), with probability. Choose the action with the highest current Q value (use the known optimal path to accelerate convergence);

[0137] In the embodiment, the initial test state for , , , , The initial value is 0.9, which decreases linearly to 0.1 with the number of training steps at a rate of 0.0001 / step. The high exploration probability in the initial stage helps the RL agent explore more state space, while the low exploration probability in the later stage helps stabilize the optimal policy.

[0138] S322: The action execution module drives the simulation test equipment to execute the selected action. By changing the test parameters of the optical module, the state perception module collects data after the action is executed. Time-space state and the corresponding device response data;

[0139] S323: The reward calculation module is based on... Time-space state With target performance set Calculate instant rewards and empirical samples Store in the experience replay pool This completes one experience collection session.

[0140] It should be noted that after the S32 is prepared for the RL agent module, it realizes environmental interaction and experience collection, and generates the first experience sample.

[0141] S33: Loop through experience sample collection, when the experience playback pool... When the number of samples reaches the sample capacity, offline network training of the RL agent module begins, accumulating rewards. Determine whether the training has been completed;

[0142] The sample size includes experience samples and historical samples from the initial experience base;

[0143] More specifically, S331: Determine the experience replay pool Check if the number of samples in the pool has reached the sample capacity. If not, repeat step S32. If the sample capacity has been reached, retrieve samples from the experience replay pool. N empirical samples are randomly selected from the data to form an offline training batch;

[0144] S332: For each empirical sample in the offline training batch, calculate the target Q-value using the target network, with the following formula:

[0145] ;

[0146] in, For the target Q value, As a discount factor, For target network Spatial state at time t+1 Next for all actions Q-value output, For target network Predicted spatial state at time t+1 The maximum Q value;

[0147] S333: Calculate the mean squared error loss by using the predicted Q-value and the target Q-value, using the following formula:

[0148] ;

[0149] For mean square error loss, This refers to the offline training batch size. For the first The target Q value for each sample For the first The predicted Q value for each sample;

[0150] S334: Use the Adam optimizer to minimize the loss function through backpropagation, update the weight parameters of the deep Q network, and synchronize the weight parameters of the deep Q network to the target network;

[0151] In the embodiment, the learning rate of the Adam optimizer The sample size of the experience replay pool was set to 1000, the offline training batch size was 32, and the parameters of the target network were synchronized every 100 steps.

[0152] S335: Calculate the variance of the test convergence step using the following formula:

[0153] ;

[0154] in, for The variance of the test convergence steps for a series of consecutive offline training batches. This represents the number of offline training batches used for continuous evaluation. For the first In each offline training batch, the number of iterations required for the RL agent to bring the performance of the optical module in the simulated test environment to a qualified standard (i.e., the number of loops of state perception-action execution-reward feedback in that batch). for The average number of test convergence steps for a consecutive offline training batch;

[0155] S336: When Accumulated rewards for consecutive offline training batches Furthermore, when the variance of the test convergence step is less than 5%, it is considered that the optimal test and debugging strategy for the target optical module has been adapted, the offline training of the RL agent is stopped, and the optimal weight parameters of the deep Q network are saved.

[0156] In the embodiments, .

[0157] like Figure 5 As shown, S4: After the RL agent module is trained offline, the actual optical module is tested online. The RL agent module and the actual optical module test scenario interact to achieve adaptive testing and fast convergence.

[0158] S41: Connects the RL agent to the automated testing equipment via the ATE system interface module, adapts the communication protocol, initializes the main network based on the optimal parameters of the deep Q-network (online training eliminates the need for a target network), and provides the target performance set. Provides a benchmark for testing and debugging;

[0159] The automatic testing equipment includes a programmable power supply, a temperature controller, an optical power meter, an eye diagram meter, and an interface;

[0160] The programmable power supply is used to provide bias current and drive voltage, the temperature controller is used to control TEC temperature, the optical power meter is used to measure output optical power, the eye diagram meter is used to measure extinction ratio and eye diagram jitter, and the interface type supports standardized interfaces such as GPIB, USB3.0, and Ethernet.

[0161] In the embodiments, The settings are based on the current specifications of the optical module being tested.

[0162] S42: The state awareness module acquires the initial state of the current optical module. The initial state data is standardized and then input into the RL agent module. The RL agent module calculates the Q-value of all actions in the initial state through the main network and selects the action with the largest Q-value as the optimal action. Output parameter adjustment instructions to the action execution module;

[0163] The initial state This includes the initial bias current, initial TEC temperature, initial drive voltage, initial modulation current, initial output optical power, initial extinction ratio, initial eye diagram jitter, and initial threshold voltage of the current optical module.

[0164] S43: The action execution module converts the output parameter adjustment command into a control signal that can be recognized by the automatic test equipment, driving the automatic test equipment to perform parameter adjustment operations (e.g., increasing the bias current by 0.1mA).

[0165] S44: After the parameter adjustment is completed, the state sensing module collects the new state of the optical module again. Based on the corresponding device response data, the reward calculation module calculates the reward according to the new state. With target performance set Calculate instant rewards (Online training rewards are primarily used to provide feedback on the effectiveness of the current action; no experience samples need to be stored.)

[0166] S45: Repeat steps S42-S44, continuously adjusting the test parameters until the performance of the optical module meets the qualification standard.

[0167] Specifically, during the repeated execution of S42-S44, the spatial state at time t is monitored in real time by the state awareness module. If all performance deviations of the optical module meet the qualification standards, the final test results (optimal test parameter data, device response data, and number of convergence steps) will be output and synchronized to the ATE system.

[0168] S5: After the online testing of the RL agent module is completed, the RL agent module will undergo iterative optimization and continuously learn new changes in the optical module production process;

[0169] S51: Newly generated valid experience samples during online testing Continuously depositing experience into the replay pool Expand training data;

[0170] The effective empirical samples are those that bring the performance of optical modules closer to the qualified standard;

[0171] S52: Set a periodic iteration cycle and utilize the experience replay pool. Incremental training with accumulated effective experience samples to update the weight parameters of the deep Q network;

[0172] In this embodiment, the periodic iteration cycle is set to the end of each production day. The incremental training and offline training processes are the same, but the number of training steps can be appropriately reduced (e.g., 500 training steps per iteration) to avoid overtraining that could cause policy fluctuations.

[0173] S53: Save the updated weight parameters of the deep Q network. The generated weight parameters are used for online testing of the optical module the next day.

[0174] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reinforcement learning-based adaptive testing and fast convergence method for optical modules, characterized in that, Includes the following sub-steps: Based on the Markov decision process model as the rule framework and the deep Q network as the core network, an RL agent module is constructed. By utilizing the ATE system, integrating the RL intelligent agent module and the ATE system interface, it works collaboratively with the test environment interaction module, state perception module, action execution module, and reward calculation module to jointly build an intelligent adaptive test system; Using an intelligent adaptive testing system, an RL agent module is trained offline. The RL agent module interacts with the simulated testing environment to learn the optimal testing and debugging strategy for the target optical module. After the RL agent module is trained offline, it is tested online with the actual optical module. The RL agent module and the actual optical module interact in the test scenario to achieve adaptive testing and fast convergence. After the online testing of the RL agent module is completed, the RL agent module undergoes iterative optimization and continuously learns new changes in the actual optical module production process; The construction of the RL agent module includes the following sub-steps: Based on the state space and action space, a Markov decision process model is constructed by combining transition probabilities, reward functions, and discount factors. Based on the main network and the target network, and combined with the training mechanism, a deep Q-network is constructed; The state space includes test parameter data and device response data of the optical module. The test parameter data includes bias current, TEC temperature, drive voltage, and modulation current, and the test parameter data reflects the test status of the optical module. The device response data includes output optical power, extinction ratio, eye diagram jitter, and threshold voltage, and the device response data reflects the performance status of the optical module. The action space adjusts the test parameter data in the state space, defining parameter adjustment instructions for the RL agent module, with the following formula: ； in, Let be the action at time t in the action space. This is the minimum adjustment step size for the bias current. This is the minimum adjustment step for the TEC temperature. This is the minimum adjustment step size for the modulation current. This is the minimum adjustment step size for the drive voltage. This is an operation to add test parameter data. This is an operation to reduce test parameter data; , , , ; The reward function includes immediate reward and termination reward; The reward function quantifies state transitions, enabling the RL agent module to determine whether the current action contributes to convergence towards the target performance. The combination of target performance characteristics constitutes a target performance set, as shown in the formula: ； in, For the target performance set, The target output optical power of the optical module, To maximize the allowable eye diagram jitter, This represents the acceptable range for the threshold voltage. , This is the minimum boundary value of the threshold voltage. This represents the maximum boundary value of the threshold voltage. The instant reward is obtained based on the deviation between the spatial state at time t and the target performance, as shown in the formula: ； in, The instantaneous reward at time t. , , , , , The weighting coefficients for the reward function are set according to the priority of the optical module test. , This is the weighting coefficient corresponding to the optical power deviation. , This is the weighting coefficient corresponding to the extinction ratio deviation. The weighting coefficient corresponding to the jitter deviation. The weighting coefficients corresponding to the threshold voltage deviation are: This refers to optical power deviation; For extinction ratio deviation, This is the jitter deviation. A positive deviation occurs when the actual jitter exceeds the maximum allowable value, and the deviation is 0 when it does not exceed the maximum allowable value. This is the threshold voltage deviation. A positive deviation occurs when the actual threshold voltage exceeds the acceptable range, and the deviation is 0 when it is within the range. The termination reward incentive RL agent module quickly converges to a qualified state when , , , When all performance deviations of the optical module meet the qualification standards, the termination reward is triggered, and the formula is: ； in, This is the allowable deviation of optical power. The allowable deviation for the extinction ratio, The preset maximum reward, To terminate the reward; The ATE system interface module is used to connect automated testing equipment to achieve protocol compatibility and data interaction. The ATE system, RL agent module, and automated test equipment are each deployed with a precise PTP slave clock. The PTP master clock deployed on the control server of the ATE system ensures that the decision-making time of the RL agent module, the action execution time of the automated test equipment, and the status acquisition time are kept synchronized. The ATE system interface module supports three mainstream interface types: GPIB, USB 3.0, and Ethernet.

2. The method for adaptive testing and fast convergence of optical modules based on reinforcement learning according to claim 1, characterized in that, The main network includes an input layer, a first hidden layer, a second hidden layer, and an output layer; The main network processing steps are as follows: the standardized state vector is input into the input layer, and the input layer extracts features and outputs them to the first hidden layer; The first hidden layer uses the ReLU function to activate features, which alleviates the gradient vanishing problem in the training of deep Q network and enhances the nonlinear fitting ability of deep Q network, enabling deep Q network to learn the complex mapping relationship between state and action. The activated features output from the first hidden layer are input into the second hidden layer. The second hidden layer uses the ReLU function to activate features, thereby reducing the number of neurons to compress and refine features, remove redundant features, improve the generalization ability of the deep Q network, and avoid overfitting. The second hidden layer outputs the reactivated features to the output layer, and the output layer outputs the Q value of the corresponding action.

3. The method for adaptive testing and fast convergence of optical modules based on reinforcement learning according to claim 1, characterized in that, The preparation work for the intelligent adaptive testing system includes: Collect historical test data for the target optical module model, format the historical test data into historical samples, and store them in the experience playback pool. This forms an initial experience base; Based on the testing requirements of the target optical module and industry standards, the parameter range of the Markov decision process model is set. The weights of the main network and the target network are initialized using a random normal distribution, and the experience replay pool is initialized accordingly. .

4. The method for adaptive testing and fast convergence of optical modules based on reinforcement learning according to claim 1, characterized in that, The interaction between the RL agent module and the simulation test environment includes the following sub-steps: An initial experience base is built using historical test data, test rules are defined, and the RL agent module is initialized. After the RL agent is initialized, the simulation test environment is reset, and the RL agent module is... The greedy strategy selects an action, executes it to change the test parameters of the optical module, and after execution, collects the new state and corresponding immediate reward, storing the experience sample in the experience replay pool. ; The experience sample collection is performed in a loop, and the experience playback pool is updated as needed. When the number of samples reaches the sample capacity, offline network training of the RL agent module begins, and the completion of training is determined by accumulating rewards.

5. The method for adaptive testing and fast convergence of optical modules based on reinforcement learning according to claim 4, characterized in that, The offline network training of the RL agent module includes the following sub-steps: Judgment Experience Replay Pool If the number of samples in the pool has not reached the sample capacity, the empirical sample collection will be repeated. If the sample capacity has been reached, the samples will be retrieved from the empirical replay pool. N empirical samples are randomly selected from the data to form an offline training batch; For each empirical sample in the offline training batch, the target Q-value is calculated using the target network, with the following formula: ； in, For the target Q value, As a discount factor, For target network exist Time-space state Next for all actions Q-value output, For target network Predicted Time-space state The maximum Q value; The mean squared error loss is calculated by using the predicted Q-value and the target Q-value, as follows: ； For mean square error loss, This refers to the offline training batch size. For the first The target Q value for each sample For the first The predicted Q value for each sample; The Adam optimizer is used to minimize the loss function through backpropagation, the weight parameters of the deep Q network are updated, and the weight parameters of the deep Q network are synchronized to the target network. The variance of the test convergence step is calculated using the following formula: ； in, for The variance of the test convergence steps for a series of consecutive offline training batches. This represents the number of offline training batches used for continuous evaluation. For the first In each offline training batch, the number of iterations required for the RL agent to bring the performance of the optical module in the simulated test environment to a passing standard. for The average number of test convergence steps for a consecutive offline training batch; when Accumulated rewards for consecutive offline training batches Furthermore, when the variance of the test convergence step is less than 5%, it is considered that the optimal test and debugging strategy for the target optical module has been adapted, the offline training of the RL agent is stopped, and the optimal weight parameters of the deep Q network are saved.

6. The method for adaptive testing and fast convergence of optical modules based on reinforcement learning according to claim 1, characterized in that, The online testing of the actual optical module includes the following sub-steps: The ATE system interface module connects the RL agent to the automated test equipment, adapts the communication protocol, initializes the main network according to the optimal parameters of the deep Q network, and provides a target performance set as a benchmark for testing and debugging. The state perception module collects the initial state of the current optical module, standardizes the initial state data and inputs it into the RL agent module. The RL agent module calculates the Q value of all actions in the initial state through the main network, selects the action with the largest Q value as the optimal action, and outputs parameter adjustment instructions to the action execution module. The action execution module converts the output parameter adjustment instructions into control signals that the automatic test equipment can recognize, driving the automatic test equipment to perform parameter adjustment operations; After the parameters are adjusted, the state perception module collects the new state of the optical module and the corresponding device response data again, and the reward calculation module calculates the instant reward based on the new state and the target performance set. The repeated execution state sensing module collects the optical module's status and outputs parameter adjustment instructions to the action execution module. The action execution module drives the automatic testing equipment to perform parameter adjustment operations until the optical module's performance meets the qualification standard.

7. The method for adaptive testing and fast convergence of optical modules based on reinforcement learning according to claim 1, characterized in that, The iterative optimization of the RL agent module includes the following sub-steps: Newly generated valid experience samples during online testing will be continuously stored in the experience replay pool. Expand training data; Set a regular iteration cycle and utilize the experience replay pool. Incremental training with accumulated effective experience samples to update the weight parameters of the deep Q network; Save the updated weight parameters of the deep Q network. The generated weight parameters will be used for online testing of the optical module the next day.

Images