A reinforcement learning-based dynamic baseline and threshold adaptive adjustment system and method for Flash ADC

Abstract

This invention discloses a reinforcement learning-based dynamic baseline and threshold adaptive adjustment system for a Flash ADC, comprising a signal acquisition module, a state feature extraction module, a reinforcement learning agent, an execution control module, a reward function calculation module, and a learning update module. This invention also provides a method for operating the reinforcement learning-based dynamic baseline and threshold adaptive adjustment system for a Flash ADC. This scheme, through interaction between the reinforcement learning agent and the ADC system, autonomously makes decisions based on real-time state to adjust the baseline and threshold voltages, dynamically optimizing multiple performance indicators such as signal-to-noise ratio, baseline stability, and detection efficiency. This solves the problems of insufficient adaptability and performance imbalance of traditional methods in complex time-varying environments, significantly improving measurement accuracy and reliability.

Description

Technical Field

[0001] This invention relates to the field of high-speed data acquisition and signal processing technology, and in particular to a Flash ADC dynamic baseline and threshold adaptive adjustment system and method based on reinforcement learning. Background Technology

[0002] High-speed Flash analog-to-digital converters (ADCs), especially high-sampling-rate, high-resolution Flash ADCs such as 250MHz / 500MHz and 12-bit / 14-bit, are core front-end components of modern precision measurement systems and are widely used in nuclear physics experiments, particle detector readout, medical imaging, and high-end instrumentation. In these applications, the Flash ADC is responsible for converting the fast, weak pulse signals generated by the detector into digital quantities, and its conversion accuracy and stability directly determine the measurement performance and data reliability of the entire system.

[0003] However, in actual operation, especially in harsh environments such as long-term, continuous operation in nuclear physics experiments, the performance of Flash ADCs is easily affected by environmental factors or radiation detector factors such as temperature drift, power supply noise, component aging, and pulse stacking, leading to baseline drift and inaccurate detection thresholds. In nuclear physics pulse signal processing, accurate baseline maintenance and threshold setting are extremely important.

[0004] Traditional baseline adjustment methods often employ fixed-rule digital signal processing, such as digital error correction DACs combined with operational amplifiers, or simple feedback control, such as PID controllers. These methods have significant limitations when dealing with complex, time-varying nonlinear interferences that occur during the long-term operation of nuclear physics experimental facilities. These limitations include: poor adaptability (unable to effectively learn and adapt to complex patterns of environmental changes, and limited optimization capabilities); insufficient flexibility (thresholds are usually fixed or can only be manually set according to simple rules, unable to be dynamically optimized based on real-time signal characteristics, and difficult to achieve optimal discrimination in low signal-to-noise ratio environments); and performance bottlenecks (rule-based algorithms struggle to achieve a global, dynamic optimal balance among multiple conflicting objectives such as signal quality, baseline stability, and detection efficiency).

[0005] Therefore, existing technologies are insufficient to meet the ever-increasing demands for data quality in high-precision nuclear physics experiments. There is an urgent need in this field for an adaptive adjustment scheme capable of autonomously learning and adapting to dynamic environmental changes, possessing intelligent decision-making capabilities, and thus achieving synergistic optimization among multiple performance indicators to ensure the performance and data quality of high-speed data acquisition systems during long-term operation. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a reinforcement learning-based dynamic baseline and threshold adaptive adjustment system and method for Flash ADCs, which can achieve real-time, accurate, and adaptive adjustment of baseline voltage and threshold voltage, effectively suppress baseline drift, and dynamically optimize threshold settings.

[0007] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0008] A reinforcement learning-based dynamic baseline and threshold adaptive adjustment system for a Flash ADC includes: a signal acquisition module, including the Flash ADC, for acquiring and outputting raw digital signals and environmental sensor data; a state feature extraction module, for calculating and generating a state vector representing the current state of the system in real time based on the raw digital signals and environmental sensor data; a reinforcement learning agent, for receiving the state vector and outputting a corresponding action vector based on an internally pre-trained policy network, the action vector including voltage adjustment amounts for the baseline adjustment circuit and voltage adjustment amounts for the threshold adjustment circuit; an execution control module, for converting the action vector into analog control voltages and applying them to the baseline adjustment circuit and the threshold adjustment circuit respectively to adjust and form a new baseline voltage and detection threshold for the Flash ADC; a reward function calculation module, where the system operates under the new Flash ADC baseline voltage and detection threshold, the state feature extraction module provides the new state vector, and the reward function calculation module calculates an immediate reward that comprehensively considers signal quality, baseline stability, and detection efficiency; and a learning update module, for storing experience data of system operation and periodically updating the internal policy of the reinforcement learning agent based on experience data including the immediate reward.

[0009] Furthermore, the state vector includes at least one or more of the following:

[0010] The mean μ_t of the current signal window, the standard deviation σ_t of the current signal window, the peak statistic p_t of the signal, the historical baseline value sequence b_{t-1},...,b_{tk}, the ambient temperature reading T_t, and the power supply noise estimate N_t.

[0011] Furthermore, the reward function R calculated by the reward function calculation module is a weighted combination of signal-to-noise ratio (SNR), baseline stability (L_baseline), detection efficiency (E_detection), and motion amplitude penalty.

[0012] Furthermore, the reward function R(s, a, s') = α·SNR(s') + β·L_baseline(s') + γ·E_detection(s') - η·||a||2

[0013] Where s is the current state, i.e., the real-time characteristics of the system, a is the action, and s' is the new state;

[0014] SNR(s') represents the adjusted signal-to-noise ratio, SNR(s) = 10·log10(σ_signal² / σ_noise²);

[0015] L_baseline(s') represents the baseline stability penalty term, L_baseline(s') = -λ_b·|b_t - b_{target}|;

[0016] E_detection(s') represents the detection efficiency term, E_detection(s') = TPR - FPR, where TPR is the true positive rate and FPR is the false positive rate;

[0017] ||a||2 is the L2 norm of vector a, representing the penalty term for the magnitude of the action;

[0018] b_t is the baseline voltage value at the current time t; b_{target} is the expected baseline voltage value; λ_b is the weighting coefficient, which is a real number greater than zero;

[0019] α, β, γ, η are weight coefficients, α+β+γ=1.

[0020] Furthermore, the reinforcement learning agent employs either the Proximal Policy Optimization (PPO) algorithm or the Deep Deterministic Policy Gradient (DDPG) algorithm.

[0021] Furthermore, the learning update module uses a proximal policy optimization algorithm to update the reinforcement learning agent, and its policy update objective function is:

[0022] J(θ) = E[min( r_t(θ)·A_t, clip(r_t(θ), 1-ε, 1+ε)·A_t )]

[0023] Where r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) is the probability ratio of the new and old strategies;

[0024] A_t is the dominance function, calculated using the generalized dominance estimation (GAE):

[0025] A_t = Σ (γλ)^l· δ_{t+l}

[0026] l represents how many steps forward from the current time t; l=0 means looking at the first step, l=1 means looking at the second step; δ_{t+l} represents the "positive impact" of the l-th step in the future; (γλ)^l represents the composite weighting factor, which decays exponentially with the step size l; λ is a parameter between 0 and 1.

[0027] The time-series difference error is calculated as δ_t = r_t + γV(s_{t+1}) - V(s_t), where V(s_t) is the expected return and r_t + γV(s_{t+1}) is the actual return. When δ_t>0, it represents a positive impact of the decision; when δ_t<0, it represents a negative impact of the decision; and when δ_t=0, it represents an impact that meets the expected outcome.

[0028] Furthermore, the state feature extraction module is implemented in hardware logic on the FPGA and is used to perform real-time feature calculation on the FlashADC data stream.

[0029] This invention also provides a method for operating a Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning, comprising the following steps:

[0030] S1, acquires the status information of the Flash ADC system in real time and generates a status vector;

[0031] S2, input the state vector into the pre-trained reinforcement learning agent;

[0032] S3, through the reinforcement learning policy network of the reinforcement learning agent, the optimal action vector is inferred based on the state vector, and the action vector includes the adjustment amount of the baseline voltage and the threshold voltage;

[0033] S4, execute the action vector, and adjust the baseline adjustment circuit and threshold adjustment circuit of the Flash ADC through the DAC control interface;

[0034] S5, observe the adjusted new state vector and calculate the immediate reward;

[0035] S6. Using the empirical data composed of the state vector, action vector, immediate reward, and new state vector, the strategy of the reinforcement learning agent is updated and optimized.

[0036] Steps S1-S6 form an adaptive closed-loop control cycle.

[0037] This invention offers the following advantages: It models the baseline adjustment problem as a Markov decision process, enabling an agent to interact with the environment and autonomously learn the optimal control strategy. This achieves real-time, accurate, and adaptive adjustment of the baseline and threshold voltages, effectively suppressing baseline drift and dynamically optimizing threshold settings to maximize the capture of weak signals and reduce the risk of misjudgment. This solution requires no prior model, can actively learn and adapt to complex time-varying environments, and simultaneously optimizes the signal-to-noise ratio, baseline stability, and detection efficiency through a carefully designed reward function. Attached Figure Description

[0038] Figure 1 This is a system architecture block diagram according to an embodiment of the present invention.

[0039] Figure 2 The diagram below illustrates the implementation of the state feature extraction module in an FPGA, taking 250MHz as an example. Detailed Implementation

[0040] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] A reinforcement learning-based dynamic baseline and threshold adaptive adjustment system for Flash ADC, such as... Figure 1 As shown, it includes:

[0042] The signal acquisition module, including the Flash ADC, is used to acquire and output raw digital signals and environmental sensor data;

[0043] The state feature extraction module is used to calculate and generate a state vector representing the current state of the system in real time based on the original digital signal and environmental sensor data.

[0044] A reinforcement learning agent is used to receive the state vector and output a corresponding action vector based on an internally pre-trained policy network. The action vector includes a voltage adjustment amount for the baseline adjustment circuit and a voltage adjustment amount for the threshold adjustment circuit.

[0045] An execution control module is used to convert the action vector into an analog control voltage and apply it to the baseline adjustment circuit and the threshold adjustment circuit respectively, so as to adjust the baseline voltage and detection threshold for forming a new Flash ADC;

[0046] The reward function calculation module calculates an instantaneous reward that comprehensively considers signal quality, baseline stability, and detection efficiency, while the system operates under the new Flash ADC baseline voltage and detection threshold. The state feature extraction module provides a new state vector.

[0047] The learning update module is used to store experience data from system operation and to periodically update the internal policy of the reinforcement learning agent based on the experience data, including the immediate rewards.

[0048] State space definition:

[0049] The state vector is defined as:

[0050] s_t = [μ_t, σ_t, p_t, b_{t-1}, b_{t-2}, ..., b_{tk}, T_t, N_t]

[0051] in,

[0052] μ_t is the mean of the current signal window;

[0053] σ_t is the standard deviation of the current signal window;

[0054] p_t represents peak statistics, such as the number of times the threshold is crossed or the peak amplitude.

[0055] b_{ti} is a sequence of historical baseline values ​​(i=1 to k), used to characterize the trend;

[0056] T_t represents the ambient temperature sensor reading;

[0057] N_t is the estimated value of power supply noise.

[0058] Action space definition:

[0059] Action vectors are defined as:

[0060] a_t = [ΔV_{baseline}, ΔV_{threshold}] ∈ℝ²

[0061] Where ΔV is the voltage adjustment applied to the corresponding DAC, ΔV_{baseline} is the adjustment to the baseline voltage, and ΔV_{threshold} is the adjustment to the threshold. Their absolute values ​​are limited by hardware: |ΔV| ≤ V_max.

[0062] V_max is the maximum value of the reference voltage, and ℝ² is the two-dimensional real number space set.

[0063] Reward function design:

[0064] The reward function comprehensively considers signal quality, stability, and detection efficiency, and its core form is:

[0065] R(s, a, s') =α·SNR(s') +β·L_baseline(s') +γ·E_detection(s') -η·||a||2

[0066] Where s is the current state vector, i.e. the real-time feature of the system, a is the action vector, and s' is the new state vector;

[0067] SNR(s') represents the adjusted signal-to-noise ratio, SNR(s) = 10·log10(σ_signal² / σ_noise²);

[0068] L_baseline(s') represents the baseline stability penalty term, L_baseline(s') = -λ_b·|b_t - b_{target}|;

[0069] E_detection(s') represents the detection efficiency term, E_detection(s') = TPR - FPR, where TPR is the true positive rate and FPR is the false positive rate;

[0070] ||a||2 is the L2 norm of vector a, also known as the Euclidean norm, which represents a penalty term for the magnitude of the action and is used to encourage smooth adjustments;

[0071] b_t is the baseline voltage value at the current time t; b_{target} is the expected baseline voltage value; λ_b is the weighting coefficient, which is a real number greater than zero.

[0072] α, β, γ, η are weight coefficients, and α+β+γ=1.

[0073] η is used as a weighting coefficient to balance the response speed and stability of the control system.

[0074] When η=0, the AI ​​adopts a very aggressive control strategy; the larger η is, the more conservative the control strategy becomes. Therefore, η needs to be of a moderate value and needs to be adjusted through experiments to achieve the best balance between "response speed" and "stability".

[0075] The reinforcement learning agent employs either the Proximal Policy Optimization (PPO) algorithm or the Deep Deterministic Policy Gradient (DDPG) algorithm. In this scheme, the Proximal Policy Optimization (PPO) algorithm is preferred, and its objective function is:

[0076] J(θ) = E[min( r_t(θ)·A_t, clip(r_t(θ), 1-ε, 1+ε)·A_t )]

[0077] in,

[0078] The Clip() function restricts r_t(θ) to the range [1-ε, 1+ε], ensuring smooth policy updates. This clipping function is a key mechanism for stable training in the PPO algorithm, where ε is the clipping parameter, such as 0.1 or 0.2. The introduction of the Clip() function ensures the safety of this scheme, preventing the AI ​​from issuing extreme commands that could cause the ADC operating point to go out of control and interrupt data acquisition, thus ensuring stable system operation. This design avoids drastic fluctuations in control commands, thereby improving system robustness.

[0079] The Min() function is used to ensure the stability of this scheme. This ensures that the learning process does not oscillate drastically, and the AI ​​controller can steadily converge to an optimal policy.

[0080] r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t), which is the probability ratio of the new and old strategies;

[0081] A_t is the dominance function, calculated using the generalized dominance estimation (GAE):

[0082] A_t = Σ (γλ)^l· δ_{t+l}

[0083] l represents how many steps forward from the current time t; l=0 means looking at the first step, l=1 means looking at the second step, and so on; δ_{t+l} represents the "positive impact" of the l-th step in the future; (γλ)^l represents the composite weighting factor, which decays exponentially with the step size l. In this formula, γ is the discount factor, representing the uncertainty of future rewards, and λ is the GAE decay parameter, representing the trade-off between bias and variance in advantage estimation.

[0084] The temporal difference error is δ_t = r_t + γV(s_{t+1}) - V(s_t), which measures the difference between the actual return and the expected return. Here, V(s_t) is the expected return, and r_t + γV(s_{t+1}) is the actual return, representing the total return re-estimated after entering the next state following the decision. s_t is the current state vector, and s_{t+1} is the new state vector.

[0085] When δ_t>0, it represents a positive impact of the decision; when δ_t<0, it represents a negative impact of the decision; and when δ_t=0, it represents an impact that meets expectations.

[0086] λ is a parameter between 0 and 1, and the commonly used value λ=0.95 is selected in this scheme.

[0087] The system architecture and executable scheme in this solution are as follows:

[0088] FPGA-side state feature extraction module:

[0089] Implement high-speed state feature extraction hardware logic on an FPGA, such as Verilog / VHDL, to calculate μ_t, σ_t, p_t, etc. in real time and package them into a state vector s_t for the agent to read. Figure 2 The diagram below illustrates the implementation of the state feature extraction module in an FPGA, taking 250MHz as an example.

[0090] Software-side RL agent:

[0091] An intelligent agent consists of a policy network (Actor) and a value network (Critic).

[0092] Real-time control loop:

[0093] The control process includes: state reading, policy reasoning, action execution (DAC adjustment), reward calculation, experience storage, and periodic policy updates, forming a complete self-learning closed loop.

[0094] The training and deployment strategy for this solution is as follows:

[0095] A phased training scheme is adopted:

[0096] 1. Simulation pre-training: Train the initial policy in a simulation environment to ensure safety.

[0097] 2. Online fine-tuning: Conducting security exploration and fine-tuning on a real system using conservative strategies.

[0098] 3. Continuous learning: After system deployment, strategies can be updated regularly with new data to adapt to long-term drift.

[0099] This solution also provides a method for operating a Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning, including the following steps:

[0100] S1, acquires the status information of the Flash ADC system in real time and generates a status vector;

[0101] S2, input the state vector into the pre-trained reinforcement learning agent;

[0102] S3, through the reinforcement learning policy network of the reinforcement learning agent, the optimal action vector is inferred based on the state vector, and the action vector includes the adjustment amount of the baseline voltage and the threshold voltage;

[0103] S4, execute the action vector, and adjust the baseline adjustment circuit and threshold adjustment circuit of the Flash ADC through the DAC control interface;

[0104] S5, observe the adjusted new state vector and calculate the immediate reward;

[0105] S6. Using the empirical data composed of the state vector, action vector, immediate reward, and new state vector, the strategy of the reinforcement learning agent is updated and optimized.

[0106] Steps S1-S6 form an adaptive closed-loop control cycle.

[0107] This solution has the following beneficial effects:

[0108] 1. Fully adaptive: No prior model required; can actively learn and adapt to complex time-varying environments.

[0109] 2. Multi-objective optimization: Through a carefully designed reward function, signal-to-noise ratio, baseline stability, and detection efficiency are optimized simultaneously.

[0110] 3. High stability: The PPO algorithm ensures smooth policy updates, avoids drastic fluctuations in control instructions, and guarantees system security.

[0111] 4. High performance: Especially suitable for high-speed and high-precision Flash ADC systems of 250MHz and above, with fast response speed and high adjustment accuracy.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning, characterized in that, include: The signal acquisition module, including the Flash ADC, is used to acquire and output raw digital signals and environmental sensor data; The state feature extraction module is used to calculate and generate a state vector representing the current state of the system in real time based on the original digital signal and environmental sensor data. A reinforcement learning agent is used to receive the state vector and output a corresponding action vector based on an internally pre-trained policy network. The action vector includes a voltage adjustment amount for the baseline adjustment circuit and a voltage adjustment amount for the threshold adjustment circuit. An execution control module is used to convert the action vector into an analog control voltage and apply it to the baseline adjustment circuit and the threshold adjustment circuit respectively, so as to adjust the baseline voltage and detection threshold for forming a new Flash ADC; The reward function calculation module calculates an instantaneous reward that comprehensively considers signal quality, baseline stability, and detection efficiency, while the system operates under the new Flash ADC baseline voltage and detection threshold. The state feature extraction module provides a new state vector. The learning update module is used to store experience data from system operation and to periodically update the internal policy of the reinforcement learning agent based on the experience data, including the immediate rewards.

2. The Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 1, characterized in that, The state vector includes at least one or more of the following: The mean μ_t of the current signal window, the standard deviation σ_t of the current signal window, the peak statistic p_t of the signal, the historical baseline value sequence b_{t-1},...,b_{tk}, the ambient temperature reading T_t, and the power supply noise estimate N_t.

3. The Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 1, characterized in that, The reward function R calculated by the reward function calculation module is a weighted combination of signal-to-noise ratio (SNR), baseline stability (L_baseline), detection efficiency (E_detection), and motion amplitude penalty.

4. The Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 3, characterized in that, The reward function R(s, a, s') = α·SNR(s') + β·L_baseline(s') + γ·E_detection(s') - η·||a||2 Where s is the current state, i.e., the real-time characteristics of the system, a is the action, and s' is the new state; SNR(s') represents the adjusted signal-to-noise ratio, SNR(s) = 10·log10(σ_signal² / σ_noise²); L_baseline(s') represents the baseline stability penalty term, L_baseline(s') = -λ_b·|b_t - b_{target}|; E_detection(s') represents the detection efficiency term, E_detection(s') = TPR - FPR, where TPR is the true positive rate and FPR is the false positive rate; ||a||2 is the L2 norm of vector a, representing the penalty term for the magnitude of the action; b_t is the baseline voltage value at the current time t; b_{target} is the expected baseline voltage value; λ_b is the weighting coefficient, which is a real number greater than zero; α, β, γ, η are weight coefficients, α+β+γ=1.

5. The Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning agent employs either the Proximal Policy Optimization (PPO) algorithm or the Deep Deterministic Policy Gradient (DDPG) algorithm.

6. The Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 1, characterized in that, The learning update module uses a proximal policy optimization algorithm to update the reinforcement learning agent, and its policy update objective function is: J(θ) = E[min( r_t(θ)·A_t, clip(r_t(θ), 1-ε, 1+ε)·A_t )] Where r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) is the probability ratio of the new and old strategies; A_t is the dominance function, calculated using the generalized dominance estimation (GAE): A_t = Σ (γλ)^l· δ_{t+l} l represents how many steps forward from the current time t; l=0 means looking at the first step, l=1 means looking at the second step; δ_{t+l} represents the positive impact of the l-th step in the future; (γλ)^l represents the composite weighting factor, which decays exponentially with the step size l; λ is a parameter between 0 and 1. The time-series difference error is calculated as δ_t = r_t + γV(s_{t+1}) - V(s_t), where V(s_t) is the expected return and r_t + γV(s_{t+1}) is the actual return. When δ_t>0, it represents a positive impact of the decision; when δ_t<0, it represents a negative impact of the decision; and when δ_t=0, it represents an impact that meets the expected outcome.

7. The Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 1, characterized in that, The state feature extraction module is implemented in hardware logic on the FPGA and is used to perform real-time feature calculation on the FlashADC data stream.

8. A method for operating a Flash ADC dynamic baseline and threshold adaptive adjustment system based on reinforcement learning according to claim 1, characterized in that, Includes the following steps: S1, acquires the status information of the Flash ADC system in real time and generates a status vector; S2, input the state vector into the pre-trained reinforcement learning agent; S3, through the reinforcement learning policy network of the reinforcement learning agent, the optimal action vector is inferred based on the state vector, and the action vector includes the adjustment amount of the baseline voltage and the threshold voltage; S4, execute the action vector, and adjust the baseline adjustment circuit and threshold adjustment circuit of the Flash ADC through the DAC control interface; S5, observe the adjusted new state vector and calculate the immediate reward; S6. Using the empirical data composed of the state vector, action vector, immediate reward, and new state vector, the strategy of the reinforcement learning agent is updated and optimized. Steps S1-S6 form an adaptive closed-loop control cycle.

Images