Electronic load grid voltage adjusting method based on improved DQN algorithm

Abstract

The invention provides an electronic load grid voltage adjusting method based on an improved DQN algorithm, and the method comprises the steps: employing a DQN reinforcement learning architecture, collecting the high-frequency waveform data of the load pulling current of an electronic load, obtaining an instantaneous amplitude signal, an instantaneous phase signal and an instantaneous frequency signal through Hilbert transformation, and enabling the instantaneous amplitude signal, the instantaneous phase signal and the instantaneous frequency signal to serve as an environment state st; the environment state st is input into a multi-mode DeepSeek distillation model intelligent agent, and the intelligent agent outputs an electronic load MOS tube grid voltage regulating variable delta Pt as an action at; establishing a reward function rt with optimal control performance as a target; the intelligent agent executes the action at in the environment state st to obtain a new environment state st + 1, and (st, at, rt, st + 1) is stored in an experience playback buffer pool; and the network parameters are replayed, optimized and updated by adopting priority experience, so that the accurate control of the load pulling current of the electronic load is realized.

Description

Technical Field

[0001] The present invention belongs to the intersection of power electronic control technology and artificial intelligence, and specifically relates to an electronic load gate voltage regulation method based on an improved DQN algorithm. Background Art

[0002] Modern power supplies pursue smaller size and higher output power, which poses challenges to the dynamic response, accuracy and power handling capabilities of test equipment. The core function of electronic loads is to accurately simulate various load conditions for testing power supply equipment. Modern high-performance electronic loads generally use power MOS transistors as their power dissipation elements. These devices control the current flowing through the device channel by controlling their gate voltage. The control system of the electronic load quickly adjusts the gate voltage of the power tube to achieve the set load mode. The traditional method uses a fixed resistor or relay switching resistor network to achieve electronic load gate voltage regulation. The disadvantage is low regulation accuracy and inability to achieve continuous and smooth control. All power consumption is converted into heat, requiring a complex heat dissipation system, large size, and susceptible to temperature drift and power supply ripple, resulting in current control errors. Through the electronic load gate voltage regulation of the reinforcement learning architecture, precise control of the electronic load load current can be achieved. Summary of the Invention

[0003] The purpose of the present invention is to provide an electronic load gate voltage regulation method based on an improved DQN algorithm to achieve precise control of the electronic load pull current.

[0004] To solve the above technical problems, the present invention provides a method for achieving precise control of the load current of an electronic load, including: Collect the high-frequency waveform data of the electronic load current, and obtain the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal as the environmental state s through Hilbert transform t ; The environmental state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment value ΔP t As an action t ; The reward function for overshoot is designed as ,in, is the overshoot reward base value, is the intensity of the overshoot penalty control, ,y max is the maximum output response, y s is the steady-state value. If the actual overshoot σ is smaller, the reward function of the overshoot The larger the value, the reward function of the steady-state error is designed as ,in, is the steady-state error reward base value, is the steady-state error penalty intensity, ,y t is the network response output value, if the steady-state error e s The smaller is, the reward function of the steady-state error The larger the value, the reward function for adjusting the time is designed to be ,in, It is to adjust the base value of time reward. is to control the time deviation penalty intensity, , is the width of the steady-state error band. If the adjustment time t s The shorter the time, the reward function of the adjustment time The larger the value, the smaller the reward function is; establish a reward function with the goal of minimizing overshoot, minimizing steady-state error, and minimizing adjustment time. , among which, the weighted summation method is used to balance the weights of the three indicators 、 and and satisfy ; The agent is in the environment state s t Execute action a t Get the new environment state s t+1 , the environmental state s t 、Action a t , reward r t and the new environment state s t+1 Constructing a tuple (s t , a t , r t , s t+1 ) is stored in the experience replay cache pool, and sampling is performed from the experience replay cache pool during training (s i , a i , r i , s i+1 ); Sample batches from the experience replay buffer pool by priority ,in, is the sampling batch size; calculate the TD error of each sampling experience , is the discount factor, is the target network, Is the main network; according to priority Sampling experience introduces importance sampling weight w1 to correct deviation , where p is the sampling priority, is to prevent zero values, N is the capacity of the experience replay buffer pool, and β is the weight decay coefficient; the loss function combines the TD error and the importance sampling weight to calculate the gradient through back propagation to update the main network parameter θ. The main network parameter θ is copied to the target network every C steps to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

[0005] Optionally, the high-frequency waveform data of the electronic load current is collected, and the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal are obtained by Hilbert transform as the environmental state s t ,include: Collect high-frequency waveform data of the electronic load current, remove noise from the data, and perform Hilbert transform to obtain the analytical signal , where x(t) is the current signal, j is the imaginary unit, and H is the Hilbert transform; the modulus of the analytical signal is calculated to obtain the instantaneous amplitude signal , based on the analytical signal, the instantaneous phase signal is obtained by unwrapping the phase , the instantaneous phase signal is derived to obtain the instantaneous frequency signal , the instantaneous amplitude signal obtained , instantaneous phase signal And the instantaneous frequency signal As the environmental state s t .

[0006] Optionally, the environmental state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment value ΔP t As an action t ,include: The environmental state s t Input into the multimodal DeepSeek distillation model agent, using GRU1 from the instantaneous amplitude signal Extract the instantaneous amplitude feature and use GRU2 to extract the instantaneous phase signal Extract the instantaneous phase feature and use GRU3 to extract the instantaneous frequency signal Extract instantaneous frequency features, calculate the correlation weights between different features through the self-attention mechanism, use the attention weights to perform weighted fusion of the three features to obtain multimodal fusion features, input the multimodal fusion features into the DeepSeek distillation model, and classify them through the Softmax function to output the electronic load MOS tube gate voltage adjustment values ​​ΔP1, ΔP2, ..., ΔP n , adjust the gate voltage of the electronic load MOS tube by ΔP t As an action t .

[0007] The present invention proposes an electronic load gate voltage regulation method based on an improved DQN algorithm. The method adopts a DQN reinforcement learning architecture to collect high-frequency waveform data of the electronic load current, and obtains the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal as the environmental state s through Hilbert transform. t ; Set the environmental state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment amount ΔP as action a t ; Establish a reward function r with the goal of optimizing control performance t ; The agent is in the environment state s t Execute action a t Get the new environment state s t+1 , will (s t , a t , r t , s t+1 ) are stored in the experience replay buffer pool; priority experience replay is used to optimize and update network parameters, thereby achieving precise control of the electronic load current. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 A schematic flow chart of a method for regulating the gate voltage of an electronic load provided in an embodiment of the present invention; Figure 2 A schematic diagram of a multi-modal DeepSeek distillation model framework for an electronic load gate voltage adjustment method according to an embodiment of the present invention; Figure 3 Schematic diagram of the DQN reinforcement learning architecture of the electronic load gate voltage adjustment method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0010] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0011] like Figure 1 As shown, Figure 1 The flowchart of the electronic load gate voltage adjustment method provided by the embodiment of the present invention specifically includes 5 contents.

[0012] S1: Collect high-frequency waveform data of the electronic load current, and obtain the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal as the environmental state s through Hilbert transform t .

[0013] It should be noted that the traditional Fourier transform is powerless in the face of signal non-stationarity and only provides global frequency information. The present invention uses the Hilbert transform to track instantaneous changes and effectively analyze the changes in the local frequency components of the signal over time. In strong interference scenarios, it can effectively extract impact features masked by noise and strong background signals. The method specifically includes four steps.

[0014] S11: De-noise the high-frequency waveform data of the electronic load current, perform Hilbert transform on the de-noised signal, and obtain the analytical signal , where x(t) is the current signal, j is the imaginary unit, and H is the Hilbert transform. The instantaneous amplitude signal A(t) is obtained by calculating the modulus of the analytical signal, and its function expression is shown in Equation (1).

[0015] S12: Calculate the phase angle based on the analytical signal z(t), and limit the original phase value to the interval [-π, π] to unwrap the phase. If the difference between two adjacent phase points is greater than π, it is considered that a jump has occurred, and the subsequent phase is compensated by adding an integer multiple of 2π to obtain a continuous instantaneous phase signal. , its function expression is shown in formula (2);

[0016] S13: Based on the unwrapped phase, the continuous instantaneous phase signal is derived and the instantaneous frequency signal f(t) is obtained. Its function expression is shown in formula (3);

[0017] S14: The instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal obtained by Hilbert transforming the high-frequency waveform data are used as the environmental state s t .

[0018] Based on the above discussion, in an optional embodiment of the present invention, the above specifically includes: In an optional embodiment of S1, the unwrapping phase setting jump threshold is π radians, and the compensation step is 2π.

[0019] S2: Change the environment state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment value ΔP t As an action t .

[0020] It should be noted that the present invention adopts the multimodal DeepSeek distillation model agent because it has a particularly strong cross-modal fusion capability. It uses the self-attention mechanism to automatically adjust the weights of different modalities and can efficiently process non-stationary signals, which is very important for time-varying signals such as instantaneous frequency. It can reduce the amount of computation while maintaining high performance.

[0021] like Figure 2 As shown, the environmental state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment amount ΔP as action a t , the process specifically includes 3 steps.

[0022] S21: The environmental state s t Input into the multimodal DeepSeek distillation model agent, and use GRU1 to extract the instantaneous amplitude feature h from the instantaneous amplitude signal A(t) A , using GRU2 from the instantaneous phase signal Extract the instantaneous phase feature h ϕ , GRU3 is used to extract the instantaneous frequency feature h from the instantaneous frequency signal f(t) f .

[0023] S22: Combine the three eigenvectors into a matrix L = [h A ;h ϕ ;h f ] T , through the learnable weight matrix W Q 、W K 、W V , map the input features to the query, key, and value space, and calculate the dependency between the two features through the self-attention mechanism to obtain the attention weight W F As shown in formula (4), the attention weight W is calculated using F Perform weighted summation on the value matrix V to obtain multimodal fusion features;

[0024] in, is the representation vector of the current spatial position of the eigenvector, is the transpose of the eigenvectors of all spatial positions, It is the attention dimension.

[0025] S23: Input the multimodal fusion features into the DeepSeek distillation model, and classify and output the electronic load MOS tube gate voltage adjustment values ​​ΔP1, ΔP2, ..., ΔP through the Softmax function. n , adjust the gate voltage of the electronic load MOS tube by ΔP t As an action t .

[0026] Based on the above discussion, in an optional embodiment of the present invention, the above specifically includes: In an optional embodiment of S2, the batch size of the multimodal DeepSeek distillation model agent is 64 and the epoch is set to 100.

[0027] S3: Establish a reward function r with the goal of minimizing overshoot, minimizing steady-state error, and minimizing adjustment time t .

[0028] It should be noted that the target reward function design of the present invention accurately quantifies the core performance target of the electronic load gate voltage regulation, achieves strong incentives for close-to-target performance through a sub-reward function in the form of exponential decay, and flexibly sets the priorities of different targets using a weighted summation method. The smoothness and dense reward characteristics of its target reward function are very suitable for combining with the DQN algorithm to efficiently search for optimal control parameters. The target reward function design specifically includes four steps.

[0029] S31: Overshoot refers to the maximum deviation of the output response from the steady-state value. If the overshoot is too large, the network will be unstable, which will cause damage to the electronic equipment. The overshoot should be as small as possible. The reward function for the overshoot is designed as shown in formula (5);

[0030] in, is the overshoot reward base value, is the intensity of the overshoot penalty control, ,y max is the maximum output response, y s is the steady-state value. If the actual overshoot σ is smaller, the reward function of the overshoot The bigger.

[0031] S32: Steady-state error is the deviation between the output response of the control system and the desired target value when it reaches steady state. If the steady-state error is too large, the expected target cannot be achieved. Therefore, it needs to be minimized. The reward function design of steady-state error is shown in formula (6);

[0032] in, is the steady-state error reward base value, is the steady-state error penalty intensity, ,y t is the network response output value, if the steady-state error e s The smaller is, the reward function of the steady-state error The bigger.

[0033] S33: The adjustment time is the time required to transition from the initial state to the steady-state value within the specified range. If the adjustment time is too long, the network efficiency will be low. Therefore, the adjustment time should be as short as possible. The reward function design of the adjustment time is shown in formula (7);

[0034] in, It is to adjust the base value of time reward. is to control the time deviation penalty intensity, , is the width of the steady-state error band. If the adjustment time t s The shorter it is, the greater the reward function of the adjustment time.

[0035] S34: Taking into account the three key indicators of overshoot, steady-state error and adjustment time, a reward function with the goals of minimizing overshoot, minimizing steady-state error and minimizing adjustment time is established as shown in formula (8);

[0036] Among them, the weighted summation method is used to balance the weights of the three indicators. 、 and and satisfy .

[0037] Based on the above discussion, in an optional embodiment of the present invention, the above specifically includes: In an optional embodiment of S3, a small hysteresis zone is set for overshoot determination, and a sufficient delay window is set for steady-state determination. The value is 3%.

[0038] S4: The agent is in the environment state s t Execute action a t Get the new environment state s t+1 , will (s t , a t , r t , s t+1 ) is stored in the experience replay cache pool.

[0039] It should be noted that traditional Q-Learning is prone to parameter oscillation due to local data correlation. The agent in this invention generates tuples (s t , a t , r t , s t+1 ) has strong temporal dependence, and the single experience in the experience replay cache pool can be reused to alleviate the data sparsity problem. The process specifically includes two steps.

[0040] S41: The agent is in the environment state s t Execute electronic load MOS tube gate voltage adjustment action a t Get the new environment state s t+1 , the obtained environmental state s t 、Action a t , reward r t and the new environment state s t+1 Constructing a tuple (s t , a t , r t , s t+1 ).

[0041] S42: The tuple (s t , a t , r t , s t+1 ) is stored in the experience replay cache pool, and sampling is performed from the experience replay cache pool during training (s i , a i , r i , s i+1 ).

[0042] Based on the above discussion, in an optional embodiment of the present invention, the above specifically includes: In an optional embodiment of S4, the experience replay buffer pool capacity parameter is set to 10,000, and the experience replay buffer pool is constructed using a ring queue structure.

[0043] S5: Use priority experience playback to optimize and update network parameters, thereby achieving precise control of the electronic load current.

[0044] It should be noted that the use of prioritized experience replay can provide experience with greater learning benefits and higher sampling priority by changing the probability of sampling experience from the experience replay cache pool, thereby significantly improving the efficiency and performance of deep reinforcement learning, especially in the update and optimization of network parameters. The process specifically includes three steps.

[0045] S51: Sample from the experience replay buffer pool according to priority and extract sampling batches ,in, is the sampling batch size.

[0046] S52: Calculate the TD error of each sampling experience as shown in formula (9), according to the priority Sampling experience, including To prevent zero values, the importance sampling weight w is introduced i The corrected deviation is shown in formula (10);

[0047] in, is the discount factor, is the target network, is the main network, p is the sampling priority, N is the capacity of the experience replay buffer pool, and β is the weight decay coefficient.

[0048] S53: The loss function combines the TD error and the importance sampling weight to calculate the gradient through back propagation to update the main network parameter θ. The main network parameter θ is copied to the target network every C steps to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

[0049] Based on the above discussion, in an optional embodiment of the present invention, the above specifically includes: In an optional embodiment of S5, β is set to an initial value of 0.4 and linearly increased to 1.0 to gradually offset the priority sampling deviation, and the main network parameter θ is copied to the target network every 1000 steps.

[0050] like Figure 3 As shown in the figure, the DQN reinforcement learning architecture is used to collect the high-frequency waveform data of the electronic load current, and the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal are obtained as the environmental state s through Hilbert transform. t , the environmental state s t Input is fed into the multimodal DeepSeek distillation model agent (the main network of the DQN reinforcement learning architecture), which is in the environment state s t Get action a t and the reward function r t , obtain the new environment state s from the target network t+1 , the environmental state s t 、Action a t , reward r t and the new environment state s t+1 Constructing a tuple (s t , a t , r t , s t+1) is stored in the experience replay cache pool, and is sampled from the experience replay cache pool according to priority during training (s i , a i , r i , s i+1 ), calculate the TD error of each sampling experience, introduce the importance sampling weight according to the priority sampling experience to correct the deviation, and the loss function combines the TD error and the importance sampling weight to calculate the gradient through back propagation to update the main network parameter θ. The main network parameter θ is copied to the target network every C steps to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

[0051] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0052] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An electronic load gate voltage adjustment method based on an improved DQN algorithm is characterized in that: include: The DQN reinforcement learning architecture is used to collect high-frequency waveform data of the electronic load current, and the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal are obtained through Hilbert transform as the environmental state s t ; The environmental state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment value ΔP t As an action t ; The reward function for overshoot is designed as ,in, is the overshoot reward base value, is the intensity of the overshoot penalty control, ,y max is the maximum output response, y s is the steady-state value. If the actual overshoot σ is smaller, the reward function of the overshoot The larger the value, the reward function of the steady-state error is designed as ,in, is the steady-state error reward base value, is the steady-state error penalty intensity, ,y t is the network response output value, if the steady-state error e s The smaller is, the reward function of the steady-state error The larger the value, the reward function for adjusting the time is designed to be ,in, It is to adjust the base value of time reward. is to control the time deviation penalty intensity, , Is the width of the steady-state error band. If the actual adjustment time t s The shorter the time, the reward function of the adjustment time The larger the value, the smaller the reward function is; establish a reward function with the goal of minimizing overshoot, minimizing steady-state error, and minimizing adjustment time. , among which, the weighted summation method is used to balance the weights of the three indicators 、 and and satisfy ; The agent is in the environment state s t Execute action a t Get the new environment state s t+1 , the environmental state s t 、Action a t , reward r t and the new environment state s t+1 Constructing a tuple (s t , a t , r t , s t+1 ) is stored in the experience replay cache pool, and sampling is performed from the experience replay cache pool during training (s i , a i , r i , s i+1 ); Sample batches from the experience replay buffer pool by priority ,in, is the sampling batch size; calculate the TD error of each sampling experience , is the discount factor, is the target network, Is the main network; according to priority Sampling experience introduces importance sampling weight w1 to correct deviation , where p is the sampling priority, is to prevent zero value, N is the buffer capacity, and β is the weight attenuation coefficient; the loss function combines the TD error and the importance sampling weight to calculate the gradient through back propagation to update the main network parameter θ, and copies the main network parameter θ to the target network every C steps to ensure the stability of the target value, thereby achieving precise control of the electronic load current.

2. The electronic load gate voltage adjustment method according to claim 1, wherein: Collect the high-frequency waveform data of the electronic load current, and obtain the instantaneous amplitude signal, instantaneous phase signal and instantaneous frequency signal as the environmental state s through Hilbert transform t ,include: The DQN reinforcement learning architecture is used to collect high-frequency waveform data of the electronic load current, and the Hilbert transform is performed after denoising the data to obtain the analytical signal. , where x(t) is the current signal, j is the imaginary unit, and H is the Hilbert transform; the modulus of the analytical signal is calculated to obtain the instantaneous amplitude signal , based on the analytical signal, the instantaneous phase signal is obtained by unwrapping the phase , the instantaneous phase signal is derived to obtain the instantaneous frequency signal , the instantaneous amplitude signal obtained , instantaneous phase signal And the instantaneous frequency signal As the environmental state s t .

3. The electronic load gate voltage adjustment method according to claim 1, wherein: The environmental state s t Input into the multimodal DeepSeek distillation model agent, the agent outputs the electronic load MOS tube gate voltage adjustment value ΔP t As an action t ,include: The environmental state s t Input into the multimodal DeepSeek distillation model agent, using GRU1 from the instantaneous amplitude signal Extract the instantaneous amplitude feature and use GRU2 to extract the instantaneous phase signal Extract the instantaneous phase feature and use GRU3 to extract the instantaneous frequency signal Extract instantaneous frequency features, calculate the correlation weights between different features through the self-attention mechanism, use the attention weights to perform weighted fusion of the three features to obtain multimodal fusion features, input the multimodal fusion features into the DeepSeek distillation model, and classify them through the Softmax function to output the electronic load MOS tube gate voltage adjustment values ​​ΔP1, ΔP2, ..., ΔP n , adjust the gate voltage of the electronic load MOS tube by ΔP t As an action t .

Images