Electronic load pull load current adjusting method based on deep reinforcement learning

Abstract

The invention provides an electronic load pull load current adjusting method based on deep reinforcement learning. The method comprises the following steps of: simulating an adjusting process of load pulling current of an electronic load into a Markov decision process based on a reinforcement learning architecture, taking a high-frequency time sequence signal of the load pulling current as an environment state, and solving an optimal MOS (Metal Oxide Semiconductor) tube grid voltage adjusting quantity sequence by an intelligent agent; the intelligent agent selects a multi-scale jump separable convolutional neural network model; after the MCU adjusting quantity predicted by the intelligent agent is applied to the electronic load, an adjusting quantity reward value is established according to the difference value between the actual current and the target current, and a training data set is established through multi-step adjustment of the intelligent agent; and based on the influence degree on the load pulling current, applying a weight to the reward value sequence to establish a reward target function, and based on a gradient descent algorithm, optimizing agent parameters by taking the maximum reward target function value as a target. According to the deep reinforcement learning method, the training samples do not need to be labeled, manual participation is reduced, and the learning efficiency is improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent control based on deep learning, and specifically relates to a method for regulating the load current of an electronic load based on deep reinforcement learning. Background Art

[0002] The basic working principle of electronic DC load is to control the working current of MOS tube flowing from drain to source by controlling the gate voltage of MOS tube. The working current is called load current. The larger the load current is, the larger the load is under the same test voltage. The size of simulated load is controlled by adjusting the size of load current. Controlling the load current by adjusting the gate voltage of MOS tube is an iterative adjustment process. The most appropriate adjustment amount sequence needs to be calculated by the control algorithm according to the feedback current to achieve the load current adjustment requirements of short adjustment time, small overshoot and small steady-state error.

[0003] At present, electronic DC load current regulation is mainly achieved through PID algorithm or fuzzy PID control algorithm, which requires manual setting of three parameters: integral, differential and proportional. It takes a long time to design an algorithm to ensure the best regulation process and the control performance is difficult to guarantee. With the development of deep neural networks, machines can solve the above problems by self-learning based on historical operation data, and reinforcement learning with deep networks as intelligent agents can perfectly solve the problem of load current regulation. Summary of the invention

[0004] The purpose of this application is to design a control algorithm that can autonomously learn based on historical data, achieve fast and accurate adjustment of load current, reduce the design time of the control algorithm, and improve its control accuracy.

[0005] In order to solve the above technical problems, the present application proposes a method for regulating the load current of an electronic load based on deep reinforcement learning, comprising: The regulation process of the electronic load current is simulated as a Markov decision process. The high-frequency timing signal of the load current is used as the environmental state, and the intelligent agent solves the optimal MOS tube gate voltage regulation sequence. Agent selection of multi-scale jump separable convolutional neural network model with reinforcement learning architecture; The adjustment reward value is established based on the difference between the actual current and the target current, and the training data set is established through multi-step adjustment of the intelligent agent; Based on the influence of the load current, the reward value sequence is weighted to establish a reward objective function, and the agent parameters are optimized based on the gradient descent algorithm with the maximum value of the reward objective function as the goal; The trained intelligent agent is used as the controller and the environmental state is used as input to give a discrete sequence of the optimal adjustment amount of the MOS tube gate voltage, thereby achieving fast and high-precision adjustment of the load current.

[0006] Optionally, the process of adjusting the load current of the electronic load is simulated as a Markov decision process, and the high-frequency timing signal of the load current is used as the environmental state, and the intelligent agent solves the optimal MOS tube gate voltage adjustment amount sequence, including: The current and target load current are used as the initial state respectively. and the final state , taking the high-frequency sampling timing signal of the load current as the observed quantity of the environment state, and taking the observed quantity as input, the intelligent agent gives a discrete sequence of the MOS tube gate voltage adjustment quantity that controls the load current size , forming a state transition set of load current under discrete sequence control , thus simulating the load current regulation process into a Markov decision process.

[0007] Optional, reinforcement learning architecture agent selection multi-scale jump separable convolutional neural network model, including: The intelligent agent selects a neural network model, converts the high-frequency current sampling timing signal into a grayscale image and then enters the multi-core convolution layer. The multi-core convolution selects four different sizes of convolution kernels: 1*1, 1*1 convolution followed by 3*3 convolution, 1*1 convolution followed by 5*5 convolution, and 1*1 convolution followed by 7*7 convolution. The second layer of the model first uses 1*1 point-by-point convolution to adjust the number of features, then selects depthwise separable convolution to extract information, and then uses 1*1 point-by-point convolution to fuse the information to obtain feature information. Jump connections are used between the input and feature information to achieve information exchange. The feature information enters a channel attention module, and the weight of each feature information vector in the second layer is optimized.

[0008] Optionally, an adjustment reward value is established based on the difference between the actual current and the target current, and a training data set is established through multi-step adjustment of the agent, including: To achieve the load current from arrive The state transition that the agent needs to complete The MOS tube gate voltage is adjusted to form a discrete sequence of adjustment quantities. , the agent passes After the cycle adjustment, the A discrete sequence, each adjustment value is applied to the circuit, and the load current feedback value is , calculate the target current The difference between the feedback current value If after adjustment If it is close to the target value, Multiply by the ratio + , otherwise multiply by the ratio - , get the reward value for each adjustment amount , all reward values ​​of a regulation sequence After adding weights, the reward value is accumulated , The adjustment sequence is obtained Reward value ， The sequence of reward values ​​and adjustment amounts forms the training dataset for reinforcement learning.

[0009] Optionally, a reward objective function is established by applying weights to the reward value sequence based on the degree of influence on the load current, and the agent parameters are optimized based on the gradient descent algorithm with the maximum value of the reward objective function as the goal, including: A dual-loop structure intelligent agent model parameter optimization method is proposed, the outer loop is The current state, the inner loop is A sequence of adjustment quantities needs to be sampled in each state A set of training sample sequences, When optimizing model parameters in this state, there is no need to consider The reward value before time , while considering t The reward value after time right The influence of model parameters gradually weakens under this state, and the reward value sequence is adjusted according to the degree of influence on the model parameters. Add weights to get the reinforcement learning reward objective function , Represents the weight of the impact of the operation amount on the reward value, Represents the time when the state occurs, status, is a constant variable, express Take the current state, Representatives when entering After the state, only the current state When , the maximum reward value weight is 1, and the reward weight of the subsequent state gradually decreases; then use Calculated in The reward gradient value under the state, where Representative samples, total Samples, Indicates Samples in Adjustment amount under status The probability of occurrence, based on the parameter update formula renew Model parameters under the state, Represents the average value of the rewards to prevent all rewards from being positive.

[0010] Optionally, the trained agent is used as a controller, and the environmental state is used as input to provide a discrete sequence of optimal adjustment amounts of the MOS tube gate voltage, so as to achieve fast and high-precision adjustment of the load current, including: Will The high-frequency sampling timing signal of the load current in the state is input into the trained intelligent model, and the model outputs the optimal MOS tube gate voltage adjustment value , after applying this adjustment amount, the current enters After continuous iteration, the model outputs the voltage regulation sequence with the shortest regulation time, the least overshoot and the smallest steady-state error. .

[0011] The present application proposes a method for regulating the load current of an electronic load based on deep reinforcement learning. Based on the reinforcement learning architecture, the regulation process of the load current of the electronic load is simulated into a Markov decision process, and the high-frequency timing signal of the load current is used as the environmental state. The intelligent agent solves the optimal sequence of MOS tube gate voltage regulation quantities; the intelligent agent selects a multi-scale jump separable convolutional neural network model; after the MCU regulation quantity predicted by the intelligent agent is applied to the electronic load, the regulation quantity reward value is established according to the difference between the actual current and the target current, and a training data set is established through multi-step regulation of the intelligent agent; a reward objective function is established by applying weights to the reward value sequence based on the degree of influence on the load current, and the intelligent agent parameters are optimized based on the gradient descent algorithm with the maximum value of the reward objective function as the goal. The deep reinforcement learning method proposed in the present application does not require labeling of training samples, reduces manual participation, and improves learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A schematic diagram of a process for optimizing and adjusting the load current of an electronic load based on a deep reinforcement learning method provided in an embodiment of the present application; Figure 2 An electronic load current pulling circuit provided by an embodiment of the present invention; Figure 3 A deep reinforcement learning model architecture provided for an embodiment of the present invention; Figure 4 A multi-scale jump separable convolutional neural network model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0015] like Figure 1 Show, Figure 1 A flow chart of a method for regulating electronic load current based on deep reinforcement learning is provided in an embodiment of the present application, which specifically includes five contents.

[0016] S11: The regulation process of the electronic load current is simulated as a Markov decision process, and the reinforcement learning architecture is used to realize current regulation. The high-frequency timing signal of the load current is used as the environmental state, and the intelligent agent solves the optimal MOS tube gate voltage regulation sequence.

[0017] It should be noted that the reinforcement learning model architecture can be used to describe the Markov decision process. Adjusting the gate voltage of the electronic load MOS tube will change the load current. The electronic load is Figure 2 The process of adjusting the load current to the target value is a current state transfer process driven by the adjustment amount. The adjustment amount is calculated by the agent according to the environmental state. This process can be described as a Markov decision process. The electronic load reinforcement learning current regulation architecture proposed in this application is as follows: Figure 3 , mainly includes the following steps.

[0018] Step 11: Environmental perception, using Hall-effect current transformer to collect data Figure 2 The drain current of BG3 in the middle forms a time series, outliers are removed and missing values ​​are interpolated for the time series signal, and the sliding slice forms the environmental observation after filtering and denoising;

[0019] Step 12: State definition, if the load current is changed from Adjust to , then Defined as initial state ,Will Defined as the final state ,state It is defined as the load current after the first adjustment of the gate voltage of MOS tube BG3. Similarly, the state set of the adjustment process is: ;

[0020] Step 13: Agent, for the Markov decision process, the decision model gives an adjustment strategy for the current state according to the environmental observation. The decision model is called an agent in reinforcement learning. This application uses a deep learning model based on the high-frequency timing signal of the load current after filtering and denoising to give the gate adjustment voltage of the MOS tube BG3 under different states;

[0021] Step 14: Adjust the amount from the initial state Initially, the first adjustment of the gate voltage of MOS tube BG3 is defined as , whose value is given by the agent based on the environmental observation, can be described as "+0.1v, +0.2v, -0.2v, -0.15v, ..., +0.2v", and the last adjustment of the MOS tube is , and so on, the set of adjustment quantities in the adjustment process is ;

[0022] Based on the above discussion, in an optional embodiment of the present application, the sampling frequency of the Hall current transformer is 10k, and the sliding window size is set = 8000, sliding step = 500, the agent chooses a deep convolutional neural network.

[0023] S12: Reinforcement learning agent selection of multi-scale jump separable convolutional neural network models.

[0024] It should be noted that the electronic DC load is an instrument. The intelligent agent model established in this application needs to be embedded in the electronic load controller ARM chip. The traditional deep neural network has a large amount of calculation and consumes a lot of memory. This application proposes a method such as Figure 4 As shown in the figure, the deep network model based on the combination of 1*1 convolution and depth-wise separable convolution strategy greatly reduces the amount of calculation. At the same time, multi-core and multi-scale convolution is used to extract the input information to ensure the integrity and availability of the extracted information. The model building process mainly includes the following key steps.

[0025] Step 21: To facilitate two-dimensional convolution, the collected long sequence signal of the load current is connected end to end and divided into segments, the first segment of the time sequence signal is used as the first row of the two-dimensional grayscale image, the second segment of the time sequence signal is used as the second row of the grayscale image, and so on, to establish the two-dimensional grayscale image data of the load current;

[0026] Step 22: The embedded intelligent body proposed in this application runs in the ARM chip of the electronic load, and the computational amount of the convolution process needs to be compressed, but the ability of the model to extract features cannot be reduced. For this reason, a new model architecture of 1*1 convolution followed by multi-scale convolution is proposed: ① Use a 1*1 convolution kernel to convolve the input grayscale image to form the first feature block, ② Use a 1*1 convolution kernel to convolve the input grayscale image, and then perform 3*3, 5*5 and 7*7 convolutions on the feature map after 1*1 convolution to form the second feature block, ③ Do not convolve the original grayscale image, and directly perform 2*2 maximum pooling processing to form the third feature block, and finally splice the three feature blocks to form the first layer of convolution feature map;

[0027] Step 23: Take the first convolution feature map as the input of the second convolution layer. First, use the conventional 1*1 point-by-point convolution kernel to fuse the input feature map. Then, in order to reduce the amount of convolution calculation, use the depth-separable convolution method. Each convolution kernel only convolves the feature map of the current channel, and does not fuse multiple feature maps. There are as many output feature maps as there are input feature maps after convolution. Then, use the conventional 1*1 point-by-point convolution again to fuse and convolve the multiple feature maps to form the output feature map.

[0028] Step 24: A jump connection is used between the convolutional feature map of the first layer and the output feature map of the second layer to achieve information exchange, eliminate useless information and reduce parameters to be optimized. The output feature layer after the jump connection enters the channel attention mechanism module, optimizes the weight allocation of the feature map of each channel, selects important information, and further reduces the amount of calculation of the next layer.

[0029] Based on the above discussion, in an optional embodiment of the present application, the first feature block of the first convolution layer selects 32 channels, and the second feature blocks 3*3, 5*5 and 7*7 convolutions each have 64 channels.

[0030] S13: Establish an adjustment reward value based on the difference between the actual current and the target current, and establish a training data set through multi-step adjustment of the intelligent agent.

[0031] It should be noted that the reinforcement learning training data set has no manual labels. The model optimizes and updates parameters based on the cumulative reward value of the adjustment sequence in each state during the state transfer process, and finally obtains the voltage adjustment sequence that guarantees the highest reward. Therefore, the design of the reward function is a key step in reinforcement learning. In the state migration process, each state must be used as the starting state for a new state migration until the final state. The training data set must be resampled in each state, and the model parameters are updated in this state based on the newly sampled data set. The reward function design and training data sampling specifically include the following steps.

[0032] Step 31: In the initial state , the environmental observation is input into the intelligent agent, and the intelligent agent gives the MOS tube gate voltage adjustment value based on the existing model parameters ,Will After being applied to the electronic load, the load current feedback value is ， Calculate the target current The difference between the feedback current If after adjustment If it is close to the target value, Multiply by the ratio + , otherwise multiply by the ratio - , get the reward value of the first adjustment amount , enter the state ;

[0033] Step 32: From State Start by repeating step 1 until the final target state , this process is called a training sample sampling, and the state sequence set of the first sampling is obtained , adjustment quantity sequence set , the reward sequence set after each adjustment , accumulate all rewards of the first sampling Get cumulative rewards ;

[0034] Step 33: Repeat steps 31-32 times, can be sampled A set of training sample sequences: ① State Sequence Set , , ... , , ② A set of adjustment sequence, , , ... , , ③ Cumulative reward sequence set ;

[0035] Step 34: The above steps 31-33 are from arrive In the sampling process, in the state transition process based on MOS tube gate voltage regulation, Each state of needs to execute steps 1-3, that is, from state , ,..., Each state transition process must be sampled The sequence sample set as described in step 3 has different numbers of sample sequences in different states.

[0036] Based on the above discussion, in an optional embodiment of the present application, the number of sampling samples in each state is a hyperparameter that needs to be optimized based on real data. , proportional value It is also a hyperparameter that needs to be statistically optimized according to the actual adjustment process.

[0037] S14: A reward objective function is established by applying weights to the reward value sequence based on the degree of influence on the load current, and the agent parameters are optimized based on the gradient descent algorithm with the goal of maximizing the reward objective function value.

[0038] It should be noted that the initial parameters of the agent model are randomly given. Therefore, the MOS tube voltage adjustment output after the input state observation cannot guarantee the maximum reward value. Only by repeatedly sampling to obtain a large number of training samples and continuously optimizing the agent model parameters based on the objective function of maximum reward using the gradient descent algorithm can the goal of maximizing the cumulative reward value of the state migration process be achieved. The agent parameter optimization process includes the following steps:

[0039] Step 41: Design a double loop reward accumulation structure, the outer loop is The current state of the inner loop is A sequence of adjustment quantities needs to be sampled in each state A set of training sample sequences;

[0040] Step 42: Design a reinforcement learning reward function, When optimizing model parameters in this state, there is no need to consider t The reward value before time , while considering t The reward value after time right The influence of model parameters in this state gradually weakens, and the reinforcement learning reward objective function is obtained: (1) t Represents the time when the state occurs, status, Represents the current state, Represents the weight of the impact of the operation amount on the reward value;

[0041] Step 43: Use the gradient descent calculation formula: (2) Calculated in The reward gradient value under the state is based on the parameter update formula: (3) renew The model parameters under the state, where i Representative samples, total Samples, Prevent all rewards from being positive. It is an impact factor and represents the step size of each optimization.

[0042] Based on the above discussion, in an optional embodiment of the present application, the gradient descent algorithm selects the stochastic gradient descent method. Take the average of historical rewards and the decay factor Take 0.6. The impact factor needs to be optimized based on the specific data used.

[0043] S15: The trained intelligent agent is used as a controller, and the environmental state is used as input to provide a discrete sequence of optimal adjustment values ​​for the gate voltage of the MOS tube, thereby achieving fast and high-precision adjustment of the load current.

[0044] It should be noted that the trained agent gives a MOS tube gate voltage adjustment sequence based on the current state of the electronic load. After applying this sequence to the electronic load circuit, the target value tracking control of the current can be achieved quickly and accurately. The specific process includes:

[0045] The collected The initial state load current mean and high-frequency waveform data are used as observations to input the trained intelligent model. The model outputs a MOS tube gate voltage adjustment value. After adjustment, the load current changes and enters state, and collect the observations in the new state again and return them to the intelligent agent. This process is repeated until the load current reaches the error range of the target value, and the adjustment process is ended. The operation sequence formed by the adjustment process can ensure the shortest adjustment time, the highest accuracy and the smallest overshoot.

[0046] Based on the above discussion, in an optional embodiment of the present application, the static error of the target value is required to be within ±5%, and the adjustment time is based on Initial state to Final state calculation.

[0047] The present application uses specific examples to illustrate the principles and implementation methods of the present invention, and the description of the above embodiments is only used to help understand the method and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for regulating electronic load current based on deep reinforcement learning, characterized in that: include: The regulation process of the electronic load current is simulated as a Markov decision process. The high-frequency timing signal of the load current is used as the environmental state, and the intelligent agent solves the optimal MOS tube gate voltage regulation sequence. Agent selection of multi-scale jump separable convolutional neural network model with reinforcement learning architecture; The adjustment reward value is established based on the difference between the actual current and the target current, and the training data set is established through multi-step adjustment of the intelligent agent; Based on the influence of the load current, the reward value sequence is weighted to establish a reward objective function, and the agent parameters are optimized based on the gradient descent algorithm with the maximum value of the reward objective function as the goal; The trained intelligent agent is used as the controller and the environmental state is used as input to give a discrete sequence of the optimal adjustment amount of the MOS tube gate voltage, thereby achieving fast and high-precision adjustment of the load current.

2. The electronic load current regulation method based on deep reinforcement learning according to claim 1, characterized in that: The regulation process of the electronic load current is simulated as a Markov decision process. The high-frequency timing signal of the load current is used as the environmental state, and the intelligent agent solves the optimal MOS tube gate voltage regulation sequence, including: The current and target load current are used as the initial state respectively. and the final state , taking the high-frequency sampling timing signal of the load current as the observed quantity of the environment state, and taking the observed quantity as input, the intelligent agent gives a discrete sequence of the MOS tube gate voltage adjustment quantity that controls the load current size , forming a state transition set of load current under discrete sequence control , thus simulating the load current regulation process into a Markov decision process.

3. The electronic load current regulation method based on deep reinforcement learning according to claim 1, characterized in that: The agent of the reinforcement learning architecture selects a multi-scale jump separable convolutional neural network model, including: The intelligent agent selects a neural network model, converts the high-frequency current sampling timing signal into a grayscale image and then enters the multi-core convolution layer. The multi-core convolution selects four different sizes of convolution kernels: 1*1, 1*1 convolution followed by 3*3 convolution, 1*1 convolution followed by 5*5 convolution, and 1*1 convolution followed by 7*7 convolution. The second layer of the model first uses 1*1 point-by-point convolution to adjust the number of features, then selects depthwise separable convolution to extract information, and then uses 1*1 point-by-point convolution to fuse the information to obtain feature information. Jump connections are used between the input and feature information to achieve information exchange. The feature information enters a channel attention module, and the weight of each feature information vector in the second layer is optimized.

4. The electronic load current regulation method based on deep reinforcement learning according to claim 1, characterized in that: The adjustment reward value is established based on the difference between the actual current and the target current, and the training data set is established through multi-step adjustment of the intelligent agent, including: To achieve the load current from arrive The state transition that the agent needs to complete The MOS tube gate voltage is adjusted to form a discrete sequence of adjustment quantities. , the agent passes After the cycle adjustment, the A discrete sequence, each adjustment value is applied to the circuit, and the load current feedback value is , calculate the target current The difference between the feedback current If after adjustment If it is close to the target value, Multiply by the ratio + , otherwise multiply by the ratio - , get the reward value for each adjustment amount , all reward values ​​of a sequence of adjustment quantities After adding weights, the reward value is accumulated , The adjustment sequence is obtained Reward value ， The sequence of reward values ​​and adjustment amounts forms the training dataset for reinforcement learning.

5. The electronic load current regulation method based on deep reinforcement learning according to claim 1, characterized in that: Based on the influence of the load current, a reward objective function is established by weighting the reward value sequence. Based on the gradient descent algorithm, the agent parameters are optimized with the maximum value of the reward objective function as the goal, including: A dual-loop structure intelligent agent model parameter optimization method is proposed, the outer loop is The current state, the inner loop is A sequence of adjustment quantities needs to be sampled in each state A set of training sample sequences, When optimizing model parameters in this state, there is no need to consider The reward value before time , while considering t The reward value after time right The influence of model parameters gradually weakens under this state, and the reward value sequence is adjusted according to the degree of influence on the model parameters. Add weights to get the reinforcement learning reward objective function , Represents the weight of the impact of the operation amount on the reward value, Represents the time when the state occurs, status, is a constant variable, express Take the current state, Representatives when entering After the state, only the current state When , the maximum reward value weight is 1, and the reward weight of the subsequent state gradually decreases; then use Calculated in The reward gradient value under the state, where Representative samples, total Samples, Indicates Samples in Adjustment amount under status The probability of occurrence, based on the parameter update formula renew Model parameters under the state, Represents the average value of the rewards to prevent all rewards from being positive.

6. The electronic load current regulation method based on deep reinforcement learning according to claim 1, characterized in that: The trained agent is used as the controller, and the environmental state is used as input to give the optimal discrete sequence of the gate voltage of the MOS tube, so as to achieve fast and high-precision adjustment of the load current, including: Will The high-frequency sampling timing signal of the load current in the state is input into the trained intelligent model, and the model outputs the optimal MOS tube gate voltage adjustment value , after applying this adjustment amount, the current enters After continuous iteration, the model outputs the voltage regulation sequence with the shortest regulation time, the least overshoot and the smallest steady-state error. .

Images