Electronic Load Gate Voltage Regulation Method Based on Improved DQN Algorithm

Abstract

This invention proposes a method for regulating the gate voltage of an electronic load based on an improved DQN algorithm. This method employs a DQN reinforcement learning architecture to acquire high-frequency waveform data of the electronic load's load current. Through Hilbert transform, it obtains the instantaneous amplitude signal, instantaneous phase signal, and instantaneous frequency signal as the environmental state s. t ; to change the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor. t As action a t Establish a reward function r with the goal of achieving optimal control performance. t The agent in environmental state s t Perform action a t Obtain new environmental state s t+1 , will (s t a t r t s t+1 The data is stored in the experience playback cache pool; network parameters are updated by prioritizing experience playback, thereby achieving precise control of the electronic load current.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of power electronic control technology and artificial intelligence, and specifically relates to a method for regulating the gate voltage of an electronic load based on an improved DQN algorithm. Background Technology

[0002] Modern power supplies strive for smaller size and higher output power, posing challenges to the dynamic response, accuracy, and power handling capabilities of test equipment. The core function of electronic loads is to accurately simulate various load conditions for testing power supply equipment. Modern high-performance electronic loads generally use power MOS transistors as their power dissipation components. These devices control the current flowing through the device channel by controlling their gate voltage. The control system of the electronic load achieves the set load mode by rapidly adjusting the gate voltage of the power transistor. Traditional methods achieve electronic load gate voltage regulation by switching resistor networks with fixed resistors or relays. The disadvantages are low regulation accuracy, inability to achieve continuous and smooth control, all power consumption is converted into heat, requiring a complex heat dissipation system, large size, and susceptibility to temperature drift and power supply ripple, leading to current control errors. By using a reinforcement learning architecture for electronic load gate voltage regulation, precise control of the electronic load's load current can be achieved. Summary of the Invention

[0003] The purpose of this invention is to provide a method for regulating the gate voltage of an electronic load based on an improved DQN algorithm, so as to achieve precise control of the load current of the electronic load.

[0004] To solve the above-mentioned technical problems, the present invention provides a method for achieving precise control of the load current of electronic loads, including:

[0005] High-frequency waveform data of the electronic load current are collected, and instantaneous amplitude, instantaneous phase, and instantaneous frequency signals are obtained through Hilbert transform as environmental state s. t ;

[0006] The environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor. t As action a t ;

[0007] The reward function for overshoot is designed as follows: ,in, It is the base value for overshoot reward. It controls the intensity of the overshoot penalty. y max It is the maximum value of the output response, y s This is the steady-state value. The smaller the actual overshoot σ, the better the reward function for the overshoot. The larger the value, the greater the reward function for the steady-state error. ,in, It is the steady-state error reward base value. It is the steady-state error penalty intensity. y t This is the network response output value; if the steady-state error e s The smaller the value, the greater the reward function of the steady-state error. The larger the value; the reward function for adjusting the time is designed as follows: ,in, It is the base value for adjusting time rewards. It controls the intensity of punishment for deviations in time. , It is the steady-state error band width, if the settling time t s The shorter the time, the better the reward function adjusts. The larger the value, the more likely a reward function is to be established, with the objectives of minimizing overshoot, minimizing steady-state error, and minimizing settling time. In this process, a weighted summation method is used to balance the weights of the three indicators. , and And satisfy ;

[0008] The agent in environmental state s t Perform action a t Obtain new environmental state s t+1 , will the environmental state s t Action a t Rewards r t and new environmental states t+1 Constitutes a tuple (s) t a t r t s t+1 ) are stored in the experience replay cache pool, and samples are taken from the experience replay cache pool during training (s i a i r i s i+1 );

[0009] Sample batches from the experience replay cache pool according to priority. ,in, This refers to the batch size of the sampling; the TD error for each sampling experience is calculated. , As a discount factor, It is the target network. It is the main network; according to priority Sampling experience introduces importance sampling weight w1 to correct bias Where p is the sampling priority, To prevent zero values, N is the capacity of the experience replay buffer pool, and β is the weight decay coefficient. The loss function combines the TD error and the importance sampling weights to calculate the gradient and update the main network parameters θ through backpropagation. Every C steps, the main network parameters θ are copied to the target network to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

[0010] Optionally, the high-frequency waveform data of the electronic load current is collected and subjected to Hilbert transform to obtain instantaneous amplitude signal, instantaneous phase signal, and instantaneous frequency signal as environmental state s. t ,include:

[0011] High-frequency waveform data of the electronic load current were collected, and the data was denoised and then subjected to Hilbert transform to obtain the analytical signal. Where x(t) is the current signal, j is the imaginary unit, and H is the Hilbert transform; the instantaneous amplitude signal is obtained by calculating the modulus of the analytic signal. The instantaneous phase signal is obtained by unwinding the phase of the analytic signal. The instantaneous frequency signal is obtained by differentiating the instantaneous phase signal. The obtained instantaneous amplitude signal Instantaneous phase signal and instantaneous frequency signals As environmental state s t .

[0012] Optionally, the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor. t As action a t ,include:

[0013] The environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the GRU1 is used to extract the instantaneous amplitude signal. Extracting instantaneous amplitude features from the instantaneous phase signal using GRU2. Instantaneous phase features are extracted from the instantaneous frequency signal using GRU3. Instantaneous frequency features are extracted, and correlation weights between different features are calculated using a self-attention mechanism. These attention weights are then used to weight and fuse the three features to obtain multimodal fused features. These multimodal fused features are input into the DeepSeek distillation model, and the Softmax function is used for classification to output the gate voltage regulation of the electronic load MOS transistor, ΔP1, ΔP2, ..., ΔP. n The gate voltage adjustment ΔP of the electronic load MOSFET t As action a t .

[0014] This invention proposes a method for regulating the gate voltage of an electronic load based on an improved DQN algorithm. This method employs a DQN reinforcement learning architecture to acquire high-frequency waveform data of the electronic load's load current. Through Hilbert transform, it obtains the instantaneous amplitude signal, instantaneous phase signal, and instantaneous frequency signal as the environmental state s. t ; to change the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor as action a. t Establish a reward function r with the goal of achieving optimal control performance. t The agent in environmental state s t Perform action a t Obtain new environmental state s t+1 , will (s t a t r t s t+1 The data is stored in the experience playback cache pool; network parameters are updated by prioritizing experience playback, thereby achieving precise control of the electronic load current. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic flowchart of an electronic load gate voltage adjustment method provided in an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the multimodal DeepSeek distillation model framework for the electronic load gate voltage regulation method provided in this embodiment of the invention;

[0018] Figure 3 A schematic diagram of the DQN reinforcement learning architecture for the electronic load gate voltage regulation method provided in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, Figure 1 The flowchart of the electronic load gate voltage adjustment method provided in the embodiment of the present invention includes five parts.

[0021] S1: Acquire high-frequency waveform data of the electronic load current, and obtain the instantaneous amplitude signal, instantaneous phase signal, and instantaneous frequency signal through Hilbert transform as the environmental state s. t .

[0022] It should be noted that the traditional Fourier transform is inadequate in the face of non-stationarity of signals, only providing global frequency information. This invention uses Hilbert transform to track instantaneous changes and effectively analyze the changes of local frequency components of the signal over time. In strong interference scenarios, it can effectively extract the impact features that are masked by noise and strong background signals. The method specifically includes four steps.

[0023] S11: Denoise the high-frequency waveform data of the electronic load current, and perform Hilbert transform on the denoised signal to obtain the analytical signal. , where x(t) is the current signal, j is the imaginary unit, and H is the Hilbert transform; the instantaneous amplitude signal A(t) is obtained by calculating the modulus of the analytic signal, and its functional expression is shown in equation (1);

[0024]

[0025] S12: Calculate the phase angle based on the analytic signal z(t), unwind the original phase value within the interval [-π, π], and if the difference between two adjacent phase points is greater than π, a jump is considered to have occurred, and subsequent phases are compensated by adding an integer multiple of 2π to obtain a continuous instantaneous phase signal. Its functional expression is shown in equation (2);

[0026]

[0027] S13: Based on the dewinding phase, the continuous instantaneous phase signal is obtained by differentiation calculation to obtain the instantaneous frequency signal f(t), and its functional expression is shown in equation (3);

[0028]

[0029] S14: The instantaneous amplitude signal, instantaneous phase signal, and instantaneous frequency signal obtained by applying Hilbert transform to the high-frequency waveform data are used as the environmental state s. t .

[0030] Based on the foregoing discussion, in an optional embodiment of the present invention, the specific details include:

[0031] In an optional embodiment of S1, the unwinding phase setting jump threshold is π radians, and the compensation step size is 2π.

[0032] S2: Set the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor. t As action a t .

[0033] It should be noted that this invention employs a multimodal DeepSeek distillation model agent because it has a particularly strong cross-modal fusion capability. It uses a self-attention mechanism to automatically adjust the weights of different modes, which can efficiently process non-stationary signals. This is very important for time-varying signals such as instantaneous frequencies, and it can reduce the amount of computation while maintaining high performance.

[0034] like Figure 2 As shown, the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor as action a. t The process specifically includes three steps.

[0035] S21: Change the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the instantaneous amplitude feature h is extracted from the instantaneous amplitude signal A(t) using GRU1. A Using GRU2 to obtain instantaneous phase signals Extracting instantaneous phase features h ϕ The instantaneous frequency feature h is extracted from the instantaneous frequency signal f(t) using GRU3. f .

[0036] S22: Combine the three eigenvectors into a matrix L=[h A ;h ϕ ;h f ] T Through a learnable weight matrix W Q W K W VThe input features are mapped to a query, key, and value space, and the attention weight W is obtained by calculating the dependencies between pairs of features through a self-attention mechanism. F As shown in equation (4), the calculated attention weight W is used. F The multimodal fusion features are obtained by weighted summation of the value matrix V;

[0037]

[0038] in, It is the representation vector of the current spatial position of the feature vector. It is the transpose of all spatial location feature vectors. It is the attention dimension.

[0039] S23: Input the multimodal fusion features into the DeepSeek distillation model, and use the Softmax function to classify and output the electronic load MOS transistor gate voltage regulation ΔP1, ΔP2, ..., ΔP n The gate voltage adjustment ΔP of the electronic load MOSFET t As action a t .

[0040] Based on the foregoing discussion, in an optional embodiment of the present invention, the specific details include:

[0041] In an optional embodiment of S2, the batch size of the multimodal DeepSeek distillation model agent is 64, and the epoch is set to 100.

[0042] S3: Establish a reward function r that aims to minimize overshoot, minimize steady-state error, and minimize settling time. t .

[0043] It should be noted that the target reward function design of this invention accurately quantifies the core performance target of electronic load gate voltage regulation, and achieves strong incentives for near-target performance through a sub-reward function in the form of exponential decay. The priority of different targets is flexibly set using a weighted summation method. The smoothness and dense reward characteristics of its target reward function are very suitable for efficient search of optimal control parameters when combined with the DQN algorithm. The target reward function design specifically includes four steps.

[0044] S31: Overshoot refers to the maximum deviation of the output response from the steady-state value. If the overshoot is too large, it will cause network instability and damage to electronic equipment. The overshoot should be as small as possible. The reward function for the overshoot is designed as shown in equation (5).

[0045]

[0046] in, It is the base value for overshoot reward. It controls the intensity of the overshoot penalty. y max It is the maximum value of the output response, y s This is the steady-state value. The smaller the actual overshoot σ, the better the reward function for the overshoot. The larger.

[0047] S32: Steady-state error is the deviation between the output response of the control system and the expected target value when it reaches steady state. If the steady-state error is too large, the expected target cannot be achieved. Therefore, it needs to be minimized. The reward function for steady-state error is designed as shown in equation (6).

[0048]

[0049] in, It is the steady-state error reward base value. It is the steady-state error penalty intensity. y t This is the network response output value; if the steady-state error e s The smaller the value, the greater the reward function of the steady-state error. The larger.

[0050] S33: The settling time is the time required to transition from the initial state to the steady state within a specified range. If the settling time is too long, it will lead to low network efficiency. Therefore, the settling time should be as short as possible. The reward function for the settling time is designed as shown in equation (7).

[0051]

[0052] in, It is the base value for adjusting time rewards. It controls the intensity of punishment for deviations in time. , It is the steady-state error band width, if the settling time t s The shorter the time, the larger the reward function for adjusting the time.

[0053] S34: Taking into account the three key indicators of overshoot, steady-state error and settling time, a reward function is established with the objectives of minimizing overshoot, minimizing steady-state error and minimizing settling time as shown in Equation (8);

[0054]

[0055] The weighted summation method is used to balance the weights of the three indicators. , and And satisfy .

[0056] Based on the foregoing discussion, in an optional embodiment of the present invention, the specific details include:

[0057] In an optional embodiment of S3, the overshoot determination is set with a small hysteresis region, and the steady-state determination is set with a sufficient delay window. The value is 3%.

[0058] S4: The agent is in environmental state s t Perform action a t Obtain new environmental state s t+1 , will (s t a t r t s t+1 Stored in the experience replay cache pool.

[0059] It should be noted that traditional Q-Learning is prone to parameter oscillations due to local data correlations. The agent in this invention generates tuples (s) during continuous decision-making. t a t r t s t+1 It has strong temporal dependencies, and a single experience in the experience replay cache pool can be reused to alleviate the data sparsity problem. The process specifically includes two steps.

[0060] S41: The agent is in environmental state s t Execute the action of regulating the gate voltage of the electronic load MOSFET a t Obtain new environmental state s t+1 The obtained environmental state s t Action a t Rewards r t and new environmental states t+1 Constitutes a tuple (s) t a t r t s t+1 ).

[0061] S42: The tuple (s) t a t r t s t+1 ) are stored in the experience replay cache pool, and samples are taken from the experience replay cache pool during training (s i a i r i s i+1 ).

[0062] Based on the foregoing discussion, in an optional embodiment of the present invention, the specific details include:

[0063] In an optional embodiment of S4, the capacity parameter of the experience replay cache pool is set to 10000, and the experience replay cache pool is constructed using a circular queue structure.

[0064] S5: Prioritizes experience playback to optimize and update network parameters, thereby achieving precise control of electronic load current.

[0065] It should be noted that by adopting priority experience replay, the probability of sampling experience from the experience replay cache pool can be changed, which can provide experience with greater learning benefits and higher sampling priority, thereby significantly improving the efficiency and performance of deep reinforcement learning, especially in terms of network parameter update and optimization. This process specifically includes three steps.

[0066] S51: Sample from the experience replay cache pool according to priority, and extract sampling batches. ,in, It refers to the sampling batch size.

[0067] S52: Calculate the TD error for each sampling experience as shown in equation (9), according to priority. Sampling experience, among which To prevent zero values, importance sampling weights w are introduced. i The correction deviation is shown in equation (10);

[0068]

[0069]

[0070] in, As a discount factor, It is the target network. is the main network, p is the sampling priority, N is the capacity of the experience replay buffer pool, and β is the weight decay coefficient.

[0071] S53: The loss function combines the TD error and importance sampling weights to calculate the gradient and update the main network parameters θ through backpropagation. Every C steps, the main network parameters θ are copied to the target network to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

[0072] Based on the foregoing discussion, in an optional embodiment of the present invention, the specific details include:

[0073] In an optional embodiment of S5, β is set to an initial value of 0.4 and linearly increased to 1.0 to gradually offset the priority sampling bias, and the main network parameters θ are copied to the target network every 1000 steps.

[0074] like Figure 3As shown, a DQN reinforcement learning architecture is used to collect high-frequency waveform data of the electronic load current. The instantaneous amplitude signal, instantaneous phase signal, and instantaneous frequency signal are obtained through Hilbert transform as the environmental state s. t , will the environmental state s t The input is fed into the multimodal DeepSeek distillation model agent (the main network of the DQN reinforcement learning architecture), and the main network operates in environment state s. t Obtain action a t and reward function r t Obtain new environmental states s from the target network t+1 , will the environmental state s t Action a t Rewards r t and new environmental states t+1 Constitutes a tuple (s) t a t r t s t+1 ) are stored in the experience replay cache pool, and during training, samples are taken from the experience replay cache pool according to priority (s i a i r i s i+1 The TD error of each sampling experience is calculated. Based on the priority sampling experience, the importance sampling weight is introduced to correct the deviation. The loss function combines the TD error and the importance sampling weight to calculate the gradient and update the main network parameter θ through backpropagation. Every C steps, the main network parameter θ is copied to the target network to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

[0075] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0076] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for regulating the gate voltage of an electronic load based on an improved DQN algorithm, characterized in that, include: A DQN reinforcement learning architecture is employed to collect high-frequency waveform data of the electronic load current. The instantaneous amplitude, phase, and frequency signals are obtained through Hilbert transform and used as the environmental state s. t ; The environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor. t As action a t ; The reward function for overshoot is designed as follows: ,in, It is the base value for overshoot reward. It controls the intensity of the overshoot penalty. y max It is the maximum value of the output response, y s This is the steady-state value. The smaller the actual overshoot σ, the better the reward function for the overshoot. The larger the value, the greater the reward function for the steady-state error. ,in, It is the steady-state error reward base value. It is the steady-state error penalty intensity. y t This is the network response output value; if the steady-state error e s The smaller the value, the greater the reward function of the steady-state error. The larger the value; the reward function for adjusting the time is designed as follows: ,in, It is the base value for adjusting time rewards. It controls the intensity of punishment for deviations in time. , It is the steady-state error band width, if the actual settling time t s The shorter the time, the better the reward function adjusts. The larger the value, the more likely a reward function is to be established, with the objectives of minimizing overshoot, minimizing steady-state error, and minimizing settling time. In this process, a weighted summation method is used to balance the weights of the three indicators. , and And satisfy ; The agent in environmental state s t Perform action a t Obtain new environmental state s t+1 , will the environmental state s t Action a t Rewards r t and new environmental states t+1 Constitutes a tuple (s) t a t r t s t+1 ) are stored in the experience replay cache pool, and samples are taken from the experience replay cache pool during training (s i a i r i s i+1 ); Sample batches from the experience replay cache pool according to priority. ,in, This refers to the batch size of the sampling; the TD error for each sampling experience is calculated. , As a discount factor, It is the target network. It is the main network; according to priority Sampling experience introduces importance sampling weight w1 to correct bias Where p is the sampling priority, To prevent zero values, N is the buffer capacity and β is the weight decay coefficient. The loss function combines the TD error and the importance sampling weight to calculate the gradient and update the main network parameter θ through backpropagation. Every C steps, the main network parameter θ is copied to the target network to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

2. The electronic load gate voltage adjustment method as described in claim 1, characterized in that, High-frequency waveform data of the electronic load current are collected, and instantaneous amplitude, instantaneous phase, and instantaneous frequency signals are obtained through Hilbert transform as environmental state s. t ,include: A DQN reinforcement learning architecture is used to collect high-frequency waveform data of the electronic load current. After denoising the data, a Hilbert transform is performed to obtain the analytical signal. Where x(t) is the current signal, j is the imaginary unit, and H is the Hilbert transform; the instantaneous amplitude signal is obtained by calculating the modulus of the analytic signal. The instantaneous phase signal is obtained by unwinding the phase of the analytic signal. The instantaneous frequency signal is obtained by differentiating the instantaneous phase signal. The obtained instantaneous amplitude signal Instantaneous phase signal and instantaneous frequency signals As environmental state s t .

3. The electronic load gate voltage adjustment method as described in claim 1, characterized in that, The environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the agent outputs the gate voltage regulation ΔP of the electronic load MOS transistor. t As action a t ,include: The environmental state s t The input is fed into the multimodal DeepSeek distillation model agent, and the GRU1 is used to extract the instantaneous amplitude signal. Extracting instantaneous amplitude features from the instantaneous phase signal using GRU2. Instantaneous phase features are extracted from the instantaneous frequency signal using GRU3. Instantaneous frequency features are extracted, and correlation weights between different features are calculated using a self-attention mechanism. These attention weights are then used to weight and fuse the three features to obtain multimodal fused features. These multimodal fused features are input into the DeepSeek distillation model, and the Softmax function is used for classification to output the gate voltage regulation of the electronic load MOS transistor, ΔP1, ΔP2, ..., ΔP. n The gate voltage adjustment ΔP of the electronic load MOSFET t As action a t .

Images