Model-free micro-grid control method and system based on transfer learning and medium

Abstract

The invention relates to a model-free micro-grid control method and system based on transfer learning and a medium, and the method comprises the following steps: dividing a DC micro-grid into a plurality of converters, and constructing a converter simulation model; realizing a deep reinforcement learning DDQN algorithm in combination with the converter simulation model, constructing a reward function, an action space and a state space, and performing normalization processing on the state space; parameters of the deep reinforcement learning DDQN algorithm are adjusted; training the converter by using the deep reinforcement learning DDQN algorithm and a reward function, and reconstructing a mapping function; and inputting an action space in the mapping function, outputting a duty ratio to PWM (Pulse Width Modulation), and adjusting the output voltage and the output power of the converter by dynamically adjusting the duty ratio, thereby realizing model-free control on the direct-current micro-grid. Compared with the prior art, the transient performance, the steady-state performance and the generalization performance of the controller are improved.

Description

Technical Field

[0001] The present invention relates to the field of direct current microgrids, and in particular to a model-free microgrid control method, system and medium based on transfer learning. Background Art

[0002] DC microgrids have more advantages in energy distribution and power quality control because they do not have the problems of reactive power and frequency in AC microgrids. At the same time, DC microgrids have the characteristics of simple control, low system cost, high reliability, and high efficiency, which makes them perform well in a variety of application scenarios. For example, in terms of renewable energy access and energy storage system integration, DC microgrids can more efficiently handle the power output from distributed energy sources such as photovoltaic power generation and wind power generation, reducing losses in the energy conversion process, thereby improving overall energy utilization efficiency. In addition, DC microgrids can also simplify the structural design of power electronic equipment and reduce equipment manufacturing and maintenance costs. In order to take into account both the transient and steady-state performance of DC microgrids, existing methods can be divided into the following categories:

[0003] 1. Use a model-based control method, such as the FCS-MPC-based control method of a photovoltaic-storage DC microgrid system in China's patent CN117424284B. Based on the finite control set model predictive control, the switch control prediction model of the distributed photovoltaic power generation unit and the switch control prediction model of the distributed energy storage unit are respectively constructed, and predictive control is performed based on the constructed models; based on the different operation modes of the photovoltaic-storage DC microgrid system, the switch control prediction model of the distributed photovoltaic power generation unit and the switch control prediction model of the distributed energy storage unit are used for predictive control, and the switch states of the distributed photovoltaic power generation unit and the distributed energy storage unit are changed respectively, so that the power balance of the photovoltaic-storage DC microgrid system is achieved, and the output power of multiple distributed photovoltaic power generation units and distributed energy storage units in the photovoltaic-storage DC microgrid is automatically distributed according to the pre-set settings without communication, thereby improving the utilization rate of each unit in the photovoltaic-storage DC microgrid, and having excellent dynamic response and strong robustness. This type of method describes the control strategy of the system dynamic behavior by accurately modeling the power electronic system, and has the advantages of high precision, fast response, strong adaptability and robustness. However, it has disadvantages such as dependence on precise models and high computational complexity.

[0004] 2. Model-free control methods, such as Chinese patent CN118263842B, which discloses a DC microgrid coordinated control method based on multi-agent system and deep reinforcement learning, takes voltage stability and current equalization of isolated DC microgrids based on distributed control architecture as the goal, introduces the management system and communication mechanism of the multi-agent system, and adopts the deep deterministic policy gradient algorithm (DDPG) to control the DC microgrid to achieve the target task. This method improves the limitations of traditional DC microgrid droop control, can solve the problem that the load distribution of isolated microgrids cannot simultaneously meet the requirements of high-precision power balancing and voltage regulation, and greatly improves the stability of DC microgrids. This type of method does not need to model the system, but directly learns and optimizes in the actual control process, which greatly improves the practicality and robustness of DC microgrid control, and also greatly reduces the design space that needs to be explored in the field of power electronic controller design, mainly including deep reinforcement learning. However, its performance depends to a large extent on the diversity and coverage of training data, and there are still problems such as data scarcity and insufficient sample complexity. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a model-free microgrid control method, system and medium based on transfer learning, so as to improve the transient performance, steady-state performance and generalization performance of the deep reinforcement learning controller.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A model-free microgrid control method based on transfer learning, characterized in that it includes the following steps:

[0008] Divide the DC microgrid into multiple converters and build a converter simulation model;

[0009] Implementing a deep reinforcement learning DDQN algorithm in combination with the converter simulation model, constructing a reward function, an action space, and a state space, and normalizing the state space;

[0010] Adjust the parameters of the deep reinforcement learning DDQN algorithm;

[0011] Use the deep reinforcement learning DDQN algorithm and the reward function to train the transformer and reconstruct the mapping function;

[0012] The action space is input into the mapping function, the duty cycle is output into PWM, and the output voltage and output power of the converter are adjusted by dynamically adjusting the duty cycle to realize model-free control of the DC microgrid.

[0013] Furthermore, the deep reinforcement learning DDQN algorithm parameters include the number of training rounds, learning rate, experience buffer length, smoothing factor, discount factor, reward function reward item and penalty item and reward function sub-goal.

[0014] Furthermore, the reward function is as follows:

[0015]

[0016] Where, e(t) is the tracking error between the output voltage and the bus reference voltage, α 1～4 Represent the reward items of the corresponding subtasks, τ 1～5 They represent the critical range of the decomposed subtasks, and β represents the penalty term of each subtask.

[0017] Furthermore, the bus reference voltage is obtained by droop control, and the calculation formula is as follows:

[0018] V droop =V outref -k di P i

[0019] Where V droop is the bus reference voltage, V outref To specify the reference voltage, P i is the output power of the ith converter, k di is the droop coefficient of the i-th converter;

[0020] The calculation formula of the tracking error is as follows:

[0021] e(t)=V droop -v0(t)

[0022] Wherein, v0(t) is the output voltage of the converter.

[0023] Furthermore, the state space is as follows:

[0024]

[0025] In the formula, S t is the state space, is the delayed signal of the output voltage, v′0(t) is the derivative of the output voltage with respect to time, and e delay (t) is the delayed signal of the tracking error, and e′(t) is the time derivative of the tracking error.

[0026] Furthermore, the state space is normalized, and the expression is as follows:

[0027]

[0028] In the formula, S′ i is the state output after normalization, S i is the unnormalized state of the input, S min is the minimum value of the current input state space, S max is the maximum value of the current input state space.

[0029] Furthermore, the training steps of training the transformer using the deep reinforcement learning DDQN algorithm are as follows:

[0030] The DC microgrid is considered as a combination of n converters, which are then divided into n converters;

[0031] If it is the first time to use deep reinforcement learning training, directly train the first converter, and then use the voltage and current double closed-loop PI to control the remaining converters; after the training is completed, extract the weights and biases of the first agent and reconstruct them into a mapping function;

[0032] If the jth converter is to be trained, the first j-1 mapping functions are first applied to the corresponding converter, and then the deep reinforcement learning DDQN algorithm is used to train converter j. For the remaining converters, the voltage and current dual closed-loop PI control is still used;

[0033] The above steps are repeated until all PI controllers are replaced by the mapping function.

[0034] Furthermore, the mapping function is as follows:

[0035]

[0036] In the formula, w is the weight vector, b is the bias term, x is the input feature vector, a is the weighted sum, y is the final output of the neuron, and max(0,a) is the ReLU activation function.

[0037] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model-free microgrid control method based on transfer learning can be implemented.

[0038] According to another aspect of the present invention, a model-free microgrid intelligent control system based on transfer learning is provided, comprising:

[0039] A simulation model building module, used to divide the DC microgrid into multiple converters and build a converter simulation model;

[0040] A deep reinforcement learning controller design module, used to implement a deep reinforcement learning DDQN algorithm in combination with the converter simulation model, construct a reward function, an action space and a state space, and normalize the state space;

[0041] An algorithm parameter adjustment module, used to adjust the parameters of the deep reinforcement learning DDQN algorithm;

[0042] A transformer training module, used to train the transformer using the deep reinforcement learning DDQN algorithm and a reward function to reconstruct a mapping function;

[0043] The model-free control module is used to input the action space in the mapping function, output the duty cycle to the PWM, and adjust the output voltage and output power of the converter by dynamically adjusting the duty cycle to achieve model-free control of the DC microgrid.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention divides the DC microgrid into multiple converters, constructs a converter simulation model and implements the deep reinforcement learning DDQN algorithm, adjusts the algorithm parameters and normalizes the state space, and proposes a domain-adaptive interactive training method based on transfer learning. Other controllers are used to gradually guide the learning of the deep reinforcement learning controller, thereby improving its generalization performance under different working conditions.

[0046] 2. The present invention applies model-free deep reinforcement learning technology to feed back the corresponding duty cycle signal output from the action space to the PWM generator, and the PWM generator generates a control signal based on this to adjust the output voltage and output power of the converter. The proposed deep reinforcement learning controller can adaptively adjust the DC bus voltage and achieve precise power distribution, thereby significantly improving its transient and steady-state performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of the model-free microgrid intelligent control method based on transfer learning proposed in the present invention;

[0048] Figure 2 A block diagram of deep reinforcement learning;

[0049] Figure 3 Schematic diagram of the internal structure of the DDQN algorithm;

[0050] Figure 4 This is a flowchart of domain adaptation;

[0051] Figure 5The changes of voltage and current when the constant power load changes, where (5a) is the voltage change and (5b) is the current change;

[0052] Figure 6 is the change of voltage and current when the droop coefficient and constant power load change, where (6a) is the change of voltage and (6b) is the change of current. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0054] Abbreviations involved:

[0055] Double Deep Q-Network: Double Deep Q-Network, DDQN

[0056] Deep Q-Network, DQN

[0057] Pulse Width Modulation: Pulse Width Modulation, PWM

[0058] Deep Neural Network: DNN

[0059] Example 1

[0060] This embodiment provides a model-free microgrid intelligent control method based on transfer learning. Figure 1 As shown, the following steps are included:

[0061] S1. Divide the DC microgrid into multiple converters and build a converter simulation model;

[0062] This embodiment takes the DC-DC Buck converter as an example and constructs a simulation model in MATLAB / Simulink. The overall block diagram is shown in FIG. Figure 2 As shown, the parameter specifications of the DC-DC Buck converter are shown in Table 1.

[0063] Table 1 Buck converter circuit parameters

[0064]

[0065] S2. Implementing a deep reinforcement learning DDQN algorithm in combination with the converter simulation model, constructing a reward function, an action space, and a state space, and normalizing the state space;

[0066] DDQN is an enhancement of the traditional DQN algorithm, which aims to alleviate the problem of overestimation of Q value. The internal structure diagram of DDQN algorithm is as follows: Figure 3 As shown in Figure 1, it mainly reduces this problem by separating action selection and Q value estimation. The main steps of the algorithm are as follows

[0067] Use the current Q network to select actions,

[0068]

[0069] In the formula, θ is the parameter of the current Q network, s′ is the next state, and a represents the action taken in this state;

[0070] Use the target Q network to evaluate the value of the above actions,

[0071] Q target =r+γQ(s′,a′;θ - )

[0072] In the formula, Q target is the target Q value, θ - is the parameter of the target Q network, γ is the discount factor, and r is the immediate reward;

[0073] Update the current Q network and update the parameters of the current Q network by minimizing the following loss function,

[0074]

[0075] Where L(θ) is the loss function, represents the expected value, and Q(s,a;θ) is the output of the current Q network.

[0076] Update the target Q network: Every certain number of steps, copy the parameters θ of the current Q network to the target Q network θ - , maintain a certain delayed update strategy.

[0077] The design of the reward function is mainly determined by the tracking error e(t) between the current output voltage and the bus reference voltage. At the same time, in order to help the agent learn more effectively, the task is decomposed into multiple subtasks. The reward function is defined as follows:

[0078]

[0079] In the formula, R t is the reward value after executing the action, e(t) is the tracking error between the output voltage and the bus reference voltage, τ 1～5 Represent the critical range of the decomposed subtasks, α 1～4 They represent the reward items of the corresponding subtasks, and β represents the penalty item of each subtask.

[0080] The input of the deep reinforcement learning state space must first consider the output voltage v0(t) of the Buck converter. Since a parallel system is considered here, only one output voltage needs to be considered. Secondly, the bus reference voltage is obtained by droop control, and the calculation method is as follows:

[0081] V droop =V outref -k di P i

[0082] Where V droop Represents the reference voltage given by the droop control, V outref is the specified reference voltage, k di represents the droop coefficient of the i-th converter, P i represents the output power of the ith converter.

[0083] Since the bus reference voltage is constantly changing, the tracking error e(t) = V droop -v0(t) is taken into account. In addition, considering the power change of the Buck converter caused by the change of the droop coefficient, the output power and droop coefficient of each converter are also regarded as inputs of the state space. At the same time, considering the continuous dynamic charging and discharging of the capacitor and inductor of the converter, the delayed signals and derivative values ​​of v0(t) and e(t) are used as supplementary features to consider factors such as system response speed and trend. By combining these states, a more comprehensive and informative representation of the system state can be provided, enabling the agent to more sensitively perceive dynamic changes in the environment. The state space is defined as follows:

[0084]

[0085] Where, v0(t), v ′ 0(t) respectively represent the output voltage, the output voltage at the previous moment, and the derivative of the output voltage with respect to time, e(t), e delay (t),e ′ (t) represent the tracking error, the tracking error at the previous moment, and the derivative of the tracking error with respect to time, respectively. i ,k di represent the output power of the i-th converter and the droop coefficient of the i-th converter respectively.

[0086] In addition, in order to keep the input state dimensions as consistent as possible, they are normalized:

[0087]

[0088] In the formula, S i ′is the state output after normalization, S i is the unnormalized state of the input, S min and S max are the maximum and minimum values ​​of the current input state space respectively.

[0089] In the field of power electronics, the duty cycle of the Buck converter plays a vital role. The duty cycle refers to the proportion of the time that the switch tube (such as MOSFET) is on to the entire switching cycle. Dynamically adjusting the duty cycle can adjust the output voltage and output power, thereby affecting the output power and efficiency of the Buck converter.

[0090] Specifically, the size of the duty cycle directly affects the average value of the output voltage. By increasing the duty cycle, the output voltage can be reduced; reducing the duty cycle will increase the output voltage. In addition, the choice of duty cycle also affects the efficiency of the Buck converter. The appropriate duty cycle can find a balance between output voltage stability and efficiency. In addition to affecting the output voltage and efficiency, the duty cycle also plays a role in overload protection and output waveform regulation. By adjusting the duty cycle, the output power of the Buck converter can be limited, thereby achieving overload protection. At the same time, a reasonable duty cycle selection can also reduce the ripple of the output waveform and improve stability.

[0091] Therefore, the duty cycle is a key control parameter in the Buck converter, which is essential to ensure stable operation of the system and meet performance requirements. By carefully selecting and adjusting the duty cycle, high efficiency and stability of the Buck converter can be achieved.

[0092] In this embodiment, the action space is discretized into a set of specific duty cycle values ​​to simplify the balance between exploration and exploitation. These duty cycle values ​​are selected as discrete values ​​between 0.3 and 0.7 with a step size of 0.01 so that the system can be effectively controlled within this range. t , the agent evaluates the pros and cons of the actions taken to guide its next move in the current environment.

[0093] In order to avoid overfitting of the neural network, this embodiment designs a relatively simple feedforward neural network, which means that its output does not affect its input. The network consists of six layers, including an input layer, an output layer, and three fully connected layers, including two hidden layers. One of the fully connected layers acts as an output layer, which is responsible for outputting the final result; the other two fully connected hidden layers each contain 64 neurons and use rectified linear units (ReLU) as activation functions to extract and learn the features of the input data.

[0094] In reinforcement learning, a mapping relationship is established between the output of a deep neural network and the actions taken by the agent. This relationship is continuously adjusted and optimized during the training process through interaction with the environment, so that the agent can learn to take the best actions to achieve a specific goal.

[0095] Specifically in the system, the corresponding duty cycle signal output from the action space is fed back to the PWM (pulse width modulation) generator. The PWM generator generates a control signal based on this to adjust the output voltage and output power of the converter. This feedback mechanism enables the system to dynamically adjust the control of the converter according to the action signal output by the DNN to achieve more efficient energy conversion and more stable system performance. Such a feedback loop not only improves the robustness of the system, but also makes it more adaptable and self-adaptive, enabling stable operation and efficient energy conversion in different environments.

[0096] S3. Adjust the parameters of the deep reinforcement learning DDQN algorithm;

[0097] In the parameter tuning of the DDQN algorithm, several key hyperparameters need to be focused on. First, select a suitable network structure (such as the number of layers, the number of neurons per layer, the activation function, etc.) to ensure that the model has sufficient expressive power but is not overfitting. Secondly, optimize the learning rate and its scheduling strategy to avoid divergence or slow convergence during training. The discount factor (γ) is set to 0.9 to balance the weights of short-term and long-term rewards. The exploration rate (ε) should be adjusted dynamically, gradually decreasing from a higher initial value to reduce random exploration and increase the utilization of the strategy. The size of the experience buffer should be large enough to ensure the diversity of the experience samples, and the update frequency of the target network should be moderate to balance the training stability and efficiency. The specific parameters are shown in Table 2.

[0098] Table 2. Parameters of deep reinforcement learning controller

[0099]

[0100] Transfer learning is a machine learning technique that uses knowledge from one field to improve learning tasks in another related field. The core idea of ​​this method is to transfer the knowledge or model learned on one task to another related task, thereby accelerating the learning process and improving performance. In the field of deep reinforcement learning, once the model is transferred to a real environment, the gap between the simulated world and the real world will be significant. In order to solve this gap, there are currently a variety of methods such as domain randomization, domain adaptation, imitation learning, etc. These methods have helped the deep reinforcement learning model to better migrate from the simulation environment to the real environment, and also improved the generalization of the model, which can effectively cope with complex changes in the real environment.

[0101] When applying domain adaptation to deep reinforcement learning, the data distribution may be slightly different from other fields, which involves the interaction between the agent and the environment, and different environments may lead to differences in data distribution. From the Markov decision process (E = S, A, R, P), it can be seen that elements in different environments may face different distributions. In the migration process of deep reinforcement learning, since the control objectives are the same, there is no need to perform some kind of mapping on the action space. In addition, this paper adopts a structure in which states are not shared but similar, that is, each state input is normalized, which can ensure data consistency and make the model more sensitive to environmental changes.

[0102] Specifically, in the state space of the DC microgrid, the bus voltage v0 will inevitably fluctuate due to environmental changes, resulting in different state inputs, while the tracking error e is calculated by the formula. Environmental changes will inevitably lead to changes in e, which results in different state inputs in the source domain and the target domain. The normalization method can minimize the gap as much as possible. Even if there are slight state space differences between the source domain and the target domain, the normalization process provides a basis for the generalization and adaptability of the model.

[0103] Since the reward function is calculated by e in real time, the sub-goal distribution in the reward function also needs to be adjusted in real time. When the output reward value is too large, the penalty term may need to be appropriately increased. On the contrary, referring to the reward curve obtained by source domain training, when the output reward value is small, the reward term should be appropriately reduced.

[0104] In addition to the state S and reward R, since the input voltage and the desired output voltage remain unchanged, the output action A is the duty cycle d = v0 / V in It always remains unchanged, so no action needs to be processed. P, as the transition probability function of the environment, is used to give the transition probability of the state of the agent at the next moment. It cannot be changed, so no change is made. The specific process of domain adaptation is as follows: Figure 4 shown.

[0105] S4, using the deep reinforcement learning DDQN algorithm and the reward function to train the transformer and reconstruct the mapping function;

[0106] A new interactive training method based on transfer learning is proposed. The detailed training process is as follows:

[0107] First, we can consider the DC microgrid as a combination of n DC-DC Buck converters, which are then divided into n sub-converters;

[0108] Secondly, if it is the first time to use deep reinforcement learning training, directly train the first sub-converter, and then use the voltage and current dual closed-loop PI to control the remaining converters;

[0109] After training is complete, the first agent weights and biases are extracted and reconstructed into a mapping function as shown below:

[0110]

[0111] Where w is the weight vector, which determines the impact of input x on the output, b is the bias term, which is used to adjust the output and enhance the flexibility of the model, x is the input feature vector, which contains the data received by the network, a is the weighted sum, which represents the result after linear transformation, and y is the final output of the neuron. The ReLU activation function (max(0, a)) is used to introduce nonlinearity to ensure that the output is non-negative.

[0112] If the training is for the ,th converter, then first apply the first j-1 mapping functions to the corresponding sub-converters, and then use deep reinforcement learning to train converter j. For the remaining converters, the voltage and current dual closed-loop PI control is still used;

[0113] Repeat the above steps until all PI controllers are replaced by deep reinforcement learning mapping functions.

[0114] S5. Input the action space into the mapping function, output the duty cycle into PWM, and adjust the output voltage and output power of the converter by dynamically adjusting the duty cycle to achieve model-free control of the DC microgrid.

[0115] After all controllers are trained, they are used as black box systems to directly generate corresponding PWM duty cycle signals through states. The compiler compiles the reconstructed neural network into C code and automatically writes it into the dSPACE MicroLabBox with real-time control function to achieve model-free control.

[0116] In the simulation model, three Buck converters are connected in parallel to a constant power load. Compared with the double closed-loop PI control, the simulation results are as follows: Figure 5 and Figure 6 As shown, Figure 5 The constant power load is initially set to 300W. At 0.14 seconds, the constant power load rises to 450W and then returns to 300W. Figure 5 As shown in (5a), although the constant power load is constantly changing, the bus voltage can always quickly track and return to a relatively stable state. And the peak current is also well maintained, as shown in Figure 5 as shown in (5b).

[0117] When the droop coefficient changes, it is equivalent to adjusting the power distribution of each converter, such as Figure 6 As shown, the droop coefficient of one of the converters is adjusted to 0.02, as shown in Figure 6 As shown in (6a), the controller proposed in this embodiment can still track quickly, and the current ratio of each converter is inversely proportional to the set droop coefficient, as shown in Figure 6 as shown in (6b).

[0118] Example 2

[0119] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a model-free microgrid control method based on transfer learning can be implemented.

[0120] The rest is the same as in Example 1.

[0121] Example 3

[0122] This embodiment provides a model-free microgrid intelligent control system based on transfer learning, including:

[0123] A simulation model building module, used to divide the DC microgrid into multiple converters and build a converter simulation model;

[0124] A deep reinforcement learning controller design module, used to design a deep reinforcement learning controller in combination with the converter simulation model, implement a deep reinforcement learning DDQN algorithm, construct a reward function, an action space and a state space, and normalize the state space;

[0125] An algorithm parameter adjustment module, used to adjust the parameters of the deep reinforcement learning DDQN algorithm;

[0126] A transformer training module, used for training the transformer using deep reinforcement learning, reconstructing a mapping function, and replacing the controller with the mapping function;

[0127] The model-free control module is used to input the state space in the mapping function and output the duty cycle to the PWM to achieve model-free control.

[0128] The rest is the same as in Example 1.

[0129] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A model-free microgrid control method based on transfer learning, characterized in that: The following steps are involved: Divide the DC microgrid into multiple converters and build a converter simulation model; Implementing a deep reinforcement learning DDQN algorithm in combination with the converter simulation model, constructing a reward function, an action space, and a state space, and normalizing the state space; Adjusting the parameters of the deep reinforcement learning DDQN algorithm; Use the deep reinforcement learning DDQN algorithm and the reward function to train the transformer and reconstruct the mapping function; The action space is input into the mapping function, the duty cycle is output into PWM, and the output voltage and output power of the converter are adjusted by dynamically adjusting the duty cycle to achieve model-free control of the DC microgrid.

2. The model-free microgrid control method based on transfer learning according to claim 1 is characterized in that: The deep reinforcement learning DDQN algorithm parameters include the number of training rounds, learning rate, experience buffer length, smoothing factor, discount factor, reward function reward and penalty terms, and reward function sub-goals.

3. The model-free microgrid control method based on transfer learning according to claim 1 is characterized in that: The reward function is as follows: Where, e(t) is the tracking error between the output voltage and the bus reference voltage, α 1～4 Represent the reward items of the corresponding subtasks, τ 1～5 They represent the critical range of the decomposed subtasks, and β represents the penalty term of each subtask.

4. The model-free microgrid control method based on transfer learning according to claim 3 is characterized in that: The bus reference voltage is obtained by droop control, and the calculation formula is as follows: V droop =V outref -k di P i Where V droop is the bus reference voltage, V outref To specify the reference voltage, P i is the output power of the ith converter, k di is the droop coefficient of the i-th converter; The calculation formula of the tracking error is as follows: e(t)=V droop -v0(t) Wherein, v0(t) is the output voltage of the converter.

5. The model-free microgrid control method based on transfer learning according to claim 1 is characterized in that: The state space is as follows: In the formula, S t is the state space, is the delayed signal of the output voltage, v ′ 0(t) is the derivative of the output voltage with respect to time, e delay (t) is the delayed signal of tracking error, r ′ (t) is the time derivative of the tracking error.

6. The model-free microgrid control method based on transfer learning according to claim 1, characterized in that: The state space is normalized, and the expression is as follows: In the formula, S i ′ is the state output after normalization, S i is the unnormalized state of the input, S min is the minimum value of the current input state space, S max is the maximum value of the current input state space.

7. The model-free microgrid control method based on transfer learning according to claim 1 is characterized in that: The training steps of the transformer using the deep reinforcement learning DDQN algorithm are as follows: The DC microgrid is considered as a combination of n converters, which are then divided into n converters; If it is the first time to use deep reinforcement learning training, directly train the first converter, and then use the voltage and current double closed-loop PI to control the remaining converters; after the training is completed, extract the weights and biases of the first agent and reconstruct them into a mapping function; If the jth converter is to be trained, the first j-1 mapping functions are first applied to the corresponding converter, and then the deep reinforcement learning DDQN algorithm is used to train converter j. For the remaining converters, the voltage and current dual closed-loop PI control is still used; The above steps are repeated until all PI controllers are replaced by the mapping function.

8. The model-free microgrid control method based on transfer learning according to claim 7 is characterized in that: The mapping function is as follows: In the formula, w is the weight vector, b is the bias term, x is the input feature vector, a is the weighted sum, y is the final output of the neuron, and max(0,a) is the ReLU activation function.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the model-free microgrid control method based on transfer learning as described in any one of claims 1 to 8 can be implemented.

10. A model-free microgrid intelligent control system based on transfer learning, characterized in that: include: A simulation model building module, used for dividing the DC microgrid into multiple converters and building a converter simulation model; A deep reinforcement learning controller design module, used to implement a deep reinforcement learning DDQN algorithm in combination with the converter simulation model, construct a reward function, an action space and a state space, and normalize the state space; An algorithm parameter adjustment module, used to adjust the parameters of the deep reinforcement learning DDQN algorithm; A transformer training module, used to train the transformer using the deep reinforcement learning DDQN algorithm and the reward function to reconstruct the mapping function; The model-free control module is used to input the action space in the mapping function, output the duty cycle to the PWM, and adjust the output voltage and output power of the converter by dynamically adjusting the duty cycle to achieve model-free control of the DC microgrid.

Images