A power system dispatch optimization method based on deep transfer reinforcement learning
By adopting deep transfer reinforcement learning methods in power system scheduling, expanding the input and output channels and combining DDPG algorithms, the problem of long training time of traditional deep reinforcement learning algorithms is solved, the reuse of historical scheduling knowledge is realized, and the efficiency of scheduling optimization of power system is improved.
Patent Information
- Application Number
- CN202310163931.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-24
- Publication Date
- 2025-09-02
- Estimated Expiration
- 2043-02-24
AI Technical Summary
Traditional deep reinforcement learning algorithms have a long training time when targeting specific scheduling tasks, and it is difficult to effectively utilize the historical scheduling knowledge of elastic resources on both the source and load, resulting in low efficiency in scheduling optimization of power system.
Using a method based on deep transfer reinforcement learning, the input and output channels are expanded on the basis of the original scheduling task network, the source task neural network parameters are migrated to the target task, and combined with the DDPG algorithm optimization solution, the reuse of historical scheduling knowledge is realized.
It effectively reduces the training cost of new power scheduling tasks, improves learning optimization efficiency, shortens training time, and improves the optimization efficiency of new scheduling tasks.
Smart Images
Figure CN115965224B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of power systems, and more specifically, relates to a power system scheduling optimization method based on deep transfer reinforcement learning. Background Art
[0002] The continuous integration of large-scale intermittent renewable energy sources into the grid has also led to increasing uncertainty in the power grid, placing higher demands on power system dispatch. With the development and construction of deep peak regulation, demand response, and integrated energy systems, the dispatchability of both the source and the load side has continued to increase, and the grid's flexibility has continued to improve. On the power supply side, by flexibly transforming traditional thermal power units to enable them to participate in deep peak regulation, the power supply's regulation potential can be effectively enhanced and the absorption of renewable energy can be promoted. On the load side, with the development and improvement of smart grid technology, flexible loads have emerged that can adjust their own electricity demand based on compensation prices. Using flexible loads to supplement power generation dispatch on the power supply side is an effective means of smoothing out fluctuations in renewable energy generation and load, and maintaining the safe and stable operation of the power grid.
[0003] With the introduction of flexible resources on both the source and load sides, the dimensionality of system states and actions increases, making accurate modeling more difficult and the optimization space larger. This poses significant challenges to the application of traditional scheduling optimization methods. Data-driven deep reinforcement learning algorithms offer advantages in solving such problems. However, since deep reinforcement learning algorithms are often targeted at specific scheduling tasks, they require learning when tasks change, resulting in lengthy training times. Therefore, in-depth research on transfer learning methods based on deep reinforcement learning with the intervention of flexible resources has important theoretical and practical value. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this paper proposes a method for optimizing power system dispatching based on deep transfer reinforcement learning. This method utilizes the proposed input-output expansion method to expand the input and output channels of the existing dispatching task decision network. The network parameters are then transferred to the new task decision network to address the knowledge reuse issue when the state-space dimension of the new dispatching task increases. This method can effectively reduce the training cost of deep reinforcement learning in new business scenarios.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] A power system dispatch optimization method based on deep transfer reinforcement learning is characterized in that the method comprises the following steps:
[0007] Step 1: Identify the various resources included in the power system and establish a mathematical model for them;
[0008] Step 2: Determine the optimization objective of the power system real-time dispatch task and establish its learning optimization model;
[0009] Step 3: Use the neural network input and output dimension expansion method to transfer the source task neural network parameters to the target task;
[0010] Step 4: After network parameter migration, the DDPG algorithm is used to further optimize and solve the problem to obtain a real-time scheduling plan.
[0011] This technical solution is further optimized. The power system described in step 1 includes a source task power system and a target task power system. The internal resources of the source task power system include Taiwan thermal power units (all conventional peak-shaving units), wind farms, PV power stations and non-adjustable rigid loads; the internal resources of the target task power system include Taiwan thermal power group (including The units are conventional peak-shaving units. (1 unit is a deep peak-shaving unit) wind farms, A photovoltaic power station, an unadjustable rigid load and an adjustable flexible load.
[0012] This technical solution is further optimized. The elastic resources on both the source and load sides of the target power system in step 1 include deep peak-shaving units and flexible loads. The specific model is as follows:
[0013] Deep peak-shaving unit model:
[0014] The peak regulation process of deep peak regulation units can be divided into three stages: conventional peak regulation, deep peak regulation without oil injection, and deep peak regulation with oil injection. Deep peak-shaving units at all times Operating costs According to its operating status and energy consumption characteristics, it can be uniformly expressed as:
[0015]
[0016] in, 、 and For thermal power units exist The coal consumption cost, unit loss cost and oil investment cost at each moment, For thermal power units exist Always make an effort, 、 、 and Thermal power units The maximum technical output, minimum technical output, minimum peak-shaving output without oil injection and minimum peak-shaving output with oil injection.
[0017] Flexible load model:
[0018] The electricity demand of flexible loads can be reduced within a certain range, and the part that can participate in grid dispatching presents elastic and adjustable characteristics related to the incentive price. Always incentivizing price Under this condition, the proportion of the curtailable load that can participate in grid dispatching to the total flexible load is It can be expressed as:
[0019]
[0020] in, and are the minimum incentive price and the maximum incentive price of the response interval, 、 are the elastic coefficients, and They are The maximum amount of flexible load that can be reduced and the total amount of flexible load at any time, Always incentivizing price Maximum dispatching capacity of flexible load It can be expressed as:
[0021]
[0022] but Always incentivizing price Actual reduction of flexible load , its compensation cost It can be expressed as:
[0023]
[0024] This technical solution is further optimized. The constraints of the power system model described in step 1 include power balance constraints, upper and lower output limits of conventional thermal power units, upper and lower output limits of deep peak-shaving units, thermal power unit ramping constraints, upper and lower limits of load reduction, and line flow constraints. The specific constraints are as follows:
[0025] The power balance constraint:
[0026]
[0027] in 、 and They are Thermal power units , wind farms and photovoltaic fields contribution; for Total load at any moment.
[0028] The output upper and lower limits of conventional thermal power units are as follows:
[0029]
[0030] in It is a collection of conventional thermal power units.
[0031] The upper and lower limits of the output of the deep peak-shaving units are as follows:
[0032]
[0033] in It is a collection of deep peak-shaving units.
[0034] The thermal power unit climbing constraint:
[0035]
[0036] in and They are thermal power units in unit dispatch period Maximum upward and downward climbing rates.
[0037] The upper and lower limits of the load reduction amount are as follows:
[0038]
[0039] The line power flow constraints are:
[0040]
[0041] in 、 、 and Thermal power units , wind farms , photovoltaic field and load Node to line The power transmission distribution coefficient of 、 and They are Thermal power units , wind farms and photovoltaic fields contribution; For the line The upper limit of the trend; is the total number of grid nodes; After the flexible load participates in the dispatch, Time Node The load value satisfies
[0042]
[0043] This technical solution is further optimized, and the optimization objectives of the power system real-time optimization task described in step 2 are as follows:
[0044] The optimization goal of the power system real-time optimization task is to minimize the operating cost while ensuring the safe and stable operation of the system. Specifically:
[0045]
[0046] in is the total daily operating cost, is the total number of scheduling cycles in a day, Scheduling period The running cost can be expressed as:
[0047]
[0048] in 、 、 、 and Scheduling period The operating costs of thermal power units, flexible load reduction compensation costs, wind and solar curtailment costs, load shedding costs and power flow over-limit penalty costs, 、 、 、 and is the weight coefficient of each cost.
[0049] This technical solution is further optimized, and the state space and action space of the learning optimization model described in step 2 are as follows:
[0050] Power systems with flexible resources System operation state vector at time By time , output of each thermal power unit , photovoltaic power station power , wind farm power , actual load value and the maximum reduction of flexible load Composition can be expressed as:
[0051]
[0052] in is the state space set of the power system with elastic resources.
[0053] No. Moment motion vector The output adjustment of each unit and the reduction of flexible load Composition can be expressed as:
[0054]
[0055] in is the action space set of the power system with elastic resources.
[0056] The state vector and action vector of the source task power system are similar to those of the power system with elastic resources and can be expressed as:
[0057]
[0058] in , is the state space set of the source task power system.
[0059]
[0060] in is the action space set of the source task power system.
[0061] This technical solution is further optimized, and the neural network input and output dimension expansion method described in step 3 is as follows:
[0062] Step 3.1: Divide the state vector into Divided into and Two parts, of which for Zhongyu The most similar vectors have the same dimensions, but the value ranges of the variables in the vectors are different. Divided into and Two parts, where the vector for Zhongyu The most similar vectors have the same dimensions, but the value ranges of the variables in the vectors are different.
[0063] Step 3.2: Add the new state vector or motion vector Based on the source task network structure, network input channels are added to make it suitable for the target task. represents the source task neural network weights, Indicates the weight of the neural network with newly added input channels, Represents the bias of the source task neural network, then the network output after adding the input channel It can be expressed as:
[0064]
[0065] in and Represents the network input. For Actor Network and Target Actor Network, , . As for the Critic network and the target Critic network, , .
[0066] Step 3.3: Add the new state vector action vector Based on the source task network structure, network output channels are added to make it suitable for the target task. and represents the source task neural network weights and biases, and Represents the weight and bias of the neural network with the newly added output channel. The network output after adding the output channel can be expressed as:
[0067]
[0068] in Represents the output of the previous layer of neural network. For the Critic network and the target Critic network, , .
[0069] Step 3.4: Target Task Actor Network Input is State , the output is action Therefore, both its input and output channels need to be expanded. Extract the network structure of the source task's Actor Network and expand it according to steps 3.2 and 3.3. This operation only partially changes the network structure. The connection weights of the reused network nodes can be derived from the connection weights of the nodes in the source task, while the newly added network parameters are obtained using a random initialization method. The target task's target Actor Network migration method is the same as the Actor Network.
[0070] Step 3.5: The target task Critic network input is output is , the output is in state Execute action when The Q value of the target task is increased while the output dimension remains unchanged. We only need to expand the input dimension using the method in step 3.2. This operation also only causes a local change in the network structure. The network parameters are initialized using the same method in step 3.4. The target task target critic network migration method is the same as the critic network.
[0071] This technical solution is further optimized, and the DDPG optimization solution in step 4 is specifically as follows:
[0072] Step 4.1: Divide the day into equal parts scheduling period, in the The reinforcement learning agent obtains the current operating status of the power grid Finally, a dispatch plan is generated based on dispatch objectives such as economy and new energy consumption rate. , and then the power system executes the dispatch plan After the current scheduling period ends, the system will Period operation cost Feedback is given to the agent, and the system state is determined by Transfer to A state transfer process can be completed with a four-tuple Indicates that the quadruple is stored in the experience pool as a training sample.
[0073] Step 4.2: Perform small batch sampling on the experience pool to obtain a batch size of State transition sample , 、 、 and They are the sampled system state, action, running cost and system state at the next moment respectively.
[0074] Step 4.3: By Calculated via the target Actor network , and then by and Calculated by target critic network , which can be calculated for:
[0075]
[0076] For a batch of State transition sample, Critic network Loss function It can be expressed as:
[0077]
[0078] Step 4.4: By and Calculated by the Critic network As the Actor network gradient, for a batch of size The state transition sample Actor network gradient is specifically:
[0079]
[0080] Step 4.5: The target actor network and target critic network are updated using soft update:
[0081]
[0082]
[0083] Different from existing technologies, the main beneficial effects are as follows: For the problem of real-time dispatch optimization of power systems, the present invention provides a power system dispatch optimization method based on deep transfer reinforcement learning, which realizes the utilization of historical dispatch knowledge through network migration. Compared with the migration of general deep reinforcement learning algorithms, the present invention uses network parameter migration as a knowledge reuse method. Based on the original dispatch task network structure, according to the dimensions of the state space and action space of the new dispatch task after the intervention of elastic resources, knowledge transfer is achieved by adding input and output channels, effectively improving the learning optimization efficiency of the new power dispatch task. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] Figure 1 This is a schematic diagram of power system knowledge transfer;
[0085] Figure 2 Deep transfer reinforcement learning for power system dispatch optimization flow chart;
[0086] Figure 3 Expand the diagram for the Actor Network and the target Actor Network;
[0087] Figure 4 Expanded diagram of the Critic network and target Critic network. DETAILED DESCRIPTION
[0088] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.
[0089] The present invention discloses a power system scheduling optimization method based on deep transfer reinforcement learning. Taking into account that deep reinforcement learning algorithms are usually targeted at specific scheduling tasks, after the intervention of elastic resources on both the source and load sides, the state space and action space dimensions of the scheduling task change, and historical scheduling knowledge cannot be used, so retraining is required, which takes a long time. Therefore, the present invention proposes a power system scheduling optimization method based on deep transfer reinforcement learning, which improves the optimization efficiency of new scheduling tasks by utilizing historical scheduling knowledge.
[0090] The present invention is specifically described by taking the provincial power grid as an example. The present invention will be further described in detail below in conjunction with specific embodiments and accompanying drawings. Figure 1 As shown in the figure, the schematic diagram of power system knowledge transfer is shown. The source task power system is a power system without elastic resources, and the target task power system is a power system with elastic resources. The purpose of knowledge transfer is to transfer the scheduling knowledge in the source task power system agent to the target task, thereby accelerating the training speed of the target task agent.
[0091] See for example Figure 2 As shown in the figure, the deep transfer reinforcement learning power system dispatch optimization flow chart includes the following steps:
[0092] Step 1: Identify the various resources included in the power system and establish a mathematical model for them:
[0093] The source mission power system consists of Taiwan thermal power units (all conventional peak-shaving units), wind farms, The target task power system consists of a photovoltaic power station and rigid loads; Thermal power units (including The units are conventional peak-shaving units. (1 unit is a deep peak-shaving unit) wind farms, It consists of a photovoltaic power station, rigid loads and flexible loads.
[0094] The elastic resources on both the source and load sides of the target power system include deep peaking units and flexible loads. The model of deep peaking units is as follows:
[0095] The peak regulation process of deep peak regulation units can be divided into three stages: conventional peak regulation, deep peak regulation without oil injection, and deep peak regulation with oil injection. Deep peak-shaving units at all times Operating costs According to its operating status and energy consumption characteristics, it can be uniformly expressed as:
[0096]
[0097] in, 、 and For thermal power units exist The coal consumption cost, unit loss cost and oil investment cost at each moment, For thermal power units exist Always make an effort, 、 、 and Thermal power units The maximum technical output, minimum technical output, minimum peak-shaving output without oil injection and minimum peak-shaving output with oil injection.
[0098] The flexible load model is as follows:
[0099] The electricity demand of flexible loads can be reduced within a certain range, and the part that can participate in grid dispatching presents elastic and adjustable characteristics related to the incentive price. Always incentivizing price Under this condition, the proportion of the curtailable load that can participate in grid dispatching to the total flexible load is It can be expressed as:
[0100]
[0101] in, and are the minimum incentive price and the maximum incentive price of the response interval, 、 are the elastic coefficients, and They are The maximum amount of flexible load that can be reduced and the total amount of flexible load at any time, Always incentivizing price Maximum dispatching capacity of flexible load It can be expressed as:
[0102]
[0103] but Always incentivizing price Actual reduction of flexible load , its compensation cost It can be expressed as:
[0104]
[0105] The constraints in the power system model include power balance constraints, upper and lower output limits of conventional thermal power units, upper and lower output limits of deep peak-shaving units, thermal power unit ramping constraints, upper and lower limits of curtailable load reduction, and line flow constraints, as shown in the following equation:
[0106] The power balance constraint:
[0107]
[0108] in 、 and They are Thermal power units , wind farms and photovoltaic fields contribution; for Total load at any moment.
[0109] The output upper and lower limits of conventional thermal power units are as follows:
[0110]
[0111] in It is a collection of conventional thermal power units.
[0112] The upper and lower limits of the output of the deep peak-shaving units are as follows:
[0113]
[0114] in It is a collection of deep peak-shaving units.
[0115] The thermal power unit climbing constraint:
[0116]
[0117] in and They are thermal power units in unit dispatch period Maximum upward and downward climbing rates.
[0118] The upper and lower limits of the load reduction amount are as follows:
[0119]
[0120] The line power flow constraints are:
[0121]
[0122] in 、 、 and Thermal power units , wind farms , photovoltaic field and load Node to line The power transmission distribution coefficient of 、 and They are Thermal power units , wind farms and photovoltaic fields contribution; For the line The upper limit of the trend; is the total number of grid nodes; After the flexible load participates in the dispatch, Time Node The load value satisfies
[0123]
[0124] Step 2: Determine the optimization objective of the power system real-time dispatch task and establish its learning optimization model:
[0125] The optimization goal of the power system real-time optimization task is to minimize the operating cost while ensuring the safe and stable operation of the system. It can be specifically expressed as:
[0126]
[0127] in is the total daily operating cost, is the total number of scheduling cycles in a day, Scheduling period The running cost can be expressed as:
[0128]
[0129] in 、 、 、 and Scheduling period The operating costs of thermal power units, flexible load reduction compensation costs, wind and solar curtailment costs, load shedding costs and power flow over-limit penalty costs, 、 、 、 and is the weight coefficient of each cost.
[0130] The state space and action space of the learning optimization model of the source power system and the target power system can be expressed as:
[0131] Power systems with flexible resources System operation state vector at time By time , thermal power unit output , photovoltaic power station power , wind farm power , actual load value and the maximum reduction of flexible load Composition can be expressed as:
[0132]
[0133] in is the state space set of the power system with elastic resources.
[0134] No. Moment motion vector The output adjustment of each unit and the reduction of flexible load Composition can be expressed as:
[0135]
[0136] in is the action space set of the power system with elastic resources.
[0137] The state vector and action vector of the power system without elastic resources are similar to those of the power system with elastic resources and can be expressed as:
[0138]
[0139] in , is the state space set of the power system without elastic resources.
[0140]
[0141] in is the set of action spaces of the power system without elastic resources.
[0142] Step 3: Use the neural network input and output dimension expansion method to transfer the source task neural network parameters to the target task.
[0143] Step 3.1: Divide the state vector into Divided into and Two parts, of which for Zhongyu The most similar vectors have the same dimensions, but the value ranges of the variables in the vectors are different. Divided into and Two parts, where the vector for Zhongyu The most similar vectors have the same dimensions, but the value ranges of the variables in the vectors are different.
[0144] Step 3.2: Add the new state vector or motion vector Based on the source task network structure, network input channels are added to make it suitable for the target task. represents the source task neural network weights, Indicates the weight of the neural network with newly added input channels, Represents the bias of the source task neural network, then the network output after adding the input channel It can be expressed as:
[0145]
[0146] in and Represents the network input. For Actor Network and Target Actor Network, , . As for the Critic network and the target Critic network, , .
[0147] Step 3.3: Add the new state vector action vector Based on the source task network structure, network output channels are added to make it suitable for the target task. and represents the source task neural network weights and biases, and Represents the weight and bias of the neural network with the newly added output channel. The network output after adding the output channel can be expressed as:
[0148]
[0149] in Represents the output of the previous layer of neural network. For the Critic network and the target Critic network, , .
[0150] Step 3.4: Target Task Actor Network Input is State , the output is action Therefore, both its input and output channels need to be expanded. Extract the network structure of the source task's Actor Network and expand it according to steps 3.2 and 3.3. This operation only partially changes the network structure. The connection weights of the reused network nodes can be derived from the connection weights of the nodes in the source task, while the newly added network parameters are obtained using a random initialization method. The target task's target Actor Network migration method is the same as the Actor Network.
[0151] Step 3.5: The target task Critic network input is output is , the output is in state Execute action when The Q value of the target task is increased while the output dimension remains unchanged. We only need to expand the input dimension using the method in step 3.2. This operation also only causes a local change in the network structure. The network parameters are initialized using the same method in step 3.4. The target task target critic network migration method is the same as the critic network.
[0152] Step 4: After network parameter migration, the DDPG algorithm is used to further optimize and solve the problem to obtain a real-time scheduling plan.
[0153] Step 4.1: Divide the day into equal parts scheduling period, in the The reinforcement learning agent obtains the current operating status of the power grid Finally, a dispatch plan is generated based on dispatch objectives such as economy and new energy consumption rate. , and then the power system executes the dispatch plan After the current scheduling period ends, the system will Period operation cost Feedback is given to the agent, and the system state is determined by Transfer to A state transfer process can be completed with a four-tuple Indicates that the quadruple is stored in the experience pool as a training sample.
[0154] Step 4.2: Perform small batch sampling on the experience pool to obtain a batch size of State transition sample , 、 、 and They are the sampled system state, action, running cost and system state at the next moment respectively.
[0155] Step 4.3: By Calculated via the target Actor network , and then by and Calculated by target critic network , which can be calculated for:
[0156]
[0157] For a batch of State transition sample, Critic network Loss function It can be expressed as:
[0158]
[0159] Step 4.4: By and Calculated by the Critic network As the Actor network gradient, for a batch of size The state transition sample Actor network gradient is specifically:
[0160]
[0161] Step 4.5: The target actor network and target critic network are updated using soft update:
[0162]
[0163]
[0164] This paper addresses the issue of traditional parameter transfer methods being unable to perform transfers due to the increased dimensionality of the state and action spaces after the introduction of flexible resources on both the source and load sides. It proposes a deep transfer reinforcement learning method based on the expansion of neural network input and output channels. By reusing historical scheduling knowledge, this method accelerates deep reinforcement learning training for new scheduling tasks, effectively reducing the time cost of training.
[0165] It should be noted that, in this document, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "include," "comprise," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. Without further limitation, elements defined by the phrase "include..." or "comprising..." do not exclude the presence of additional elements in the process, method, article, or terminal device comprising the elements. Furthermore, in this document, "greater than," "less than," "exceeding," etc., are understood to exclude the number itself; "above," "below," "within," etc., are understood to include the number itself.
[0166] Although the above embodiments have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present invention.
Claims
1. A power system dispatch optimization method based on deep transfer reinforcement learning, characterized in that: The scheduling optimization method comprises the following steps: Step 1: Determine the various resources included in the power system and establish a mathematical model for them, wherein the mathematical model is a deep peak-shaving unit model and a flexible load model; Step 2: Determine the optimization objective of the power system real-time dispatch task and establish its learning optimization model; The optimization objectives of the power system real-time dispatch task in step 2 are as follows: in is the total daily operating cost, is the total number of scheduling cycles in a day, Scheduling period The running cost can be expressed as: in 、 、 、 and Scheduling period The operating costs of thermal power units, flexible load reduction compensation costs, wind and solar curtailment costs, load shedding costs and power flow over-limit penalty costs, 、 、 、 and is the weight coefficient of each cost; Step 2 learns the state space and action space of the optimization model as follows: Power systems with flexible resources System operation state vector at time By time , output of each thermal power unit , photovoltaic power station power , wind farm power , actual load value and the maximum reduction of flexible load Composition can be expressed as: in is the state space set of the power system with elastic resources; No. Moment motion vector The output adjustment of each unit and the reduction of flexible load Composition can be expressed as: in is the set of action spaces of the power system with elastic resources; The state vector and action vector of the source task power system can be expressed as: in , is the state space set of the source task power system; in is the action space set of the source task power system; Step 3: Use the neural network input and output dimension expansion method to transfer the source task neural network parameters to the target task; The neural network input and output dimension expansion method in step 3 is as follows: Step 3.1: Transform the state vector Divided into and Two parts, of which for Zhongyu The most similar vector will be Divided into and Two parts, where the vector for Zhongyu Most similar vectors; Step 3.2: Add the new state vector or motion vector Increase network input channels, represents the source task neural network weights, Indicates the weight of the neural network with newly added input channels, represents the source task neural network bias, then the network output It can be expressed as: in and Represents network input, for Actor network and target Actor network, , , and for the Critic network and the target Critic network, , ; Step 3.3: Increase network output channels, and represents the source task neural network weights and biases, and Represents the weights and biases of the newly added output channel neural network, and the network output can be expressed as: in Represents the output of the previous layer of neural network. For the Critic network and the target Critic network, , ; Step 3.4: Target Task Actor Network Input is State , the output is action , perform channel expansion according to steps 3.2 and 3.3; Step 3.5: The target task Critic network input is output is , the output is in state Execute action when Q value, use step 3.2 to expand the input dimension, and use the same method as step 3.4 to initialize the network parameters; Step 4: After network parameter migration, the DDPG algorithm is used to further optimize and solve the problem to obtain a real-time scheduling plan.
2. The power system dispatch optimization method based on deep transfer reinforcement learning according to claim 1, characterized in that: The power system in step 1 includes a source task power system and a target task power system. The source task power system consists of Taiwan thermal power units, wind farms, It consists of a photovoltaic power station and rigid loads; The target power system is a power system with flexible resources, consisting of Taiwan thermal power units, wind farms, It consists of photovoltaic power stations, rigid loads and flexible loads, among which The units are conventional peak-shaving units. The unit is a deep peak-shaving unit.
3. The power system dispatch optimization method based on deep transfer reinforcement learning according to claim 1, characterized in that: The elastic resources on both the source and load sides of the target power system in step 1 include deep peak-shaving units and flexible loads. The specific model is as follows: Deep peak-shaving unit model: The peak regulation process of deep peak regulation units can be divided into three stages: conventional peak regulation, deep peak regulation without oil injection, and deep peak regulation with oil injection. Deep peak-shaving units at all times Operating costs According to its operating status and energy consumption characteristics, it can be uniformly expressed as: in, 、 and For thermal power units exist The coal consumption cost, unit loss cost and oil investment cost at each moment, For thermal power units exist Always make an effort, 、 、 and Thermal power units The maximum technical output, minimum technical output, minimum output for peak load regulation without oil injection and minimum output for peak load regulation with oil injection; Flexible load model: Flexible load electricity demand can be reduced within a certain range, and the part that can participate in grid dispatching presents elastic and adjustable characteristics related to incentive prices. Always incentivizing price Under this condition, the proportion of the curtailable load that can participate in grid dispatching to the total flexible load is It can be expressed as: in, and are the minimum incentive price and the maximum incentive price of the response interval, 、 are the elastic coefficients, and They are The maximum amount of flexible load that can be reduced and the total amount of flexible load at any time, Always incentivizing price Maximum dispatching capacity of flexible load It can be expressed as: but Always incentivizing price Actual reduction of flexible load , its compensation cost It can be expressed as: 。 4. The power system dispatch optimization method based on deep transfer reinforcement learning according to claim 1, characterized in that: The constraints of the power system model in step 1 include power balance constraints, upper and lower output limits of conventional thermal power units, upper and lower output limits of deep peak-shaving units, thermal power unit ramping constraints, upper and lower limits of load reduction, and line flow constraints, which are as follows: The power balance constraint: in 、 and They are Thermal power units , wind farms and photovoltaic fields contribution; for Total load at any moment; The output upper and lower limits of conventional thermal power units are as follows: in It is a collection of conventional thermal power units; The upper and lower limits of the output of the deep peak-shaving units are as follows: in It is a collection of deep peak-shaving units; The thermal power unit climbing constraint: in and They are thermal power units in unit dispatch period Maximum uphill and downhill climbing rates; The upper and lower limits of the load reduction amount are as follows: The line power flow constraints are: in 、 、 and Thermal power units , wind farms , photovoltaic field and load Node to line The power transmission distribution coefficient of 、 and They are Thermal power units , wind farms and photovoltaic fields contribution; For the line The upper limit of the trend; is the total number of grid nodes; After the flexible load participates in the dispatch, Time Node The load value satisfies 。 5. The power system dispatch optimization method based on deep transfer reinforcement learning according to claim 1, characterized in that: The DDPG optimization solution in step 4 is as follows: Step 4.1: Divide the day into equal parts scheduling period, in the At the decision moment, the deep reinforcement learning agent obtains the current operating status of the power grid Finally, a dispatch plan is generated based on the dispatch targets of economy and new energy consumption rate. , and then the power system executes the dispatch plan , after the current scheduling period ends, the system will Period operation cost Feedback is given to the agent, and the system state is determined by Transfer to , a state transfer process can be completed with a four-tuple Indicates that the quadruple is stored in the experience pool as a training sample; Step 4.2: Perform small batch sampling on the experience pool to obtain a batch size of State transition sample , 、 、 and are the sampled system state, action, running cost and system state at the next moment respectively; Step 4.3: By Calculated via the target Actor network , and then by and Calculated by target critic network , which can be calculated for: For a batch of State transition sample, Critic network Loss function It can be expressed as: Step 4.4: By and Calculated by the Critic network As the Actor network gradient, for a batch of size The state transition sample Actor network gradient is specifically: Step 4.5: The target actor network and target critic network are updated using soft update: 。