Method, system and device for large-scale load demand response strategy based on meta-reinforcement learning
Through the meta reinforcement learning method, a common initial model is obtained, and then trained for specific needs, solving the problem of high time and economic costs in large-scale load demand responses, and realizing the problem of high time and economic costs in existing reinforcement learning algorithms. Fast and efficient load demand response strategies.
Patent Information
- Application Number
- CN202111284855.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-01
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When existing reinforcement learning algorithms respond to large-scale load demands for multiple users and multiple power load devices, the training process requires a lot of interaction with real scenarios, resulting in high time and economic costs.
Using a method based on meta-reinforcement learning, the optimization model is constructed and converted into a Markov decision-making process. The initial model with the lowest electricity cost is obtained through meta-reinforcement learning training, and the load demand response of multiple users or multiple devices is used to utilize multiple Optimize the similarity of the model for parameterized training, reducing separate training for specific needs.
It effectively saves the time and economic costs of large-scale user demand response training, accelerates the independent training process of each user through a common model, and reduces computing power consumption and user data use.
Smart Images

Figure CN114004497B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power grid demand response, and in particular to a method, system and device for a large-scale load demand response strategy based on meta-reinforcement learning. Background Art
[0002] The tight supply of electricity has become one of the main contradictions in economic and social life. Ensuring the balance of electricity supply and demand and achieving a safe, reliable, economical, clean and efficient power system have become important guarantees for economic development and improvement of people's living standards. Relying on the advancement of market-oriented reforms and the adjustment of national energy policies, the coordination and exchange between the power network and the power market, and between the power network and power users have become increasingly close. Demand-side resource management has become an important measure to reduce power consumption, alleviate power shortage pressure, and reduce power supply and power consumption costs under the same power consumption function. The introduction of demand response (DR) in the power market competition has become an important means to strengthen the role of demand-side resources in the power market through price signals and incentive mechanisms. Auxiliary frequency regulation, ensuring safe system operation, and completing other power system auxiliary services through demand response have gradually become the development trend of the future power market.
[0003] There are many papers that have conducted in-depth discussions on the incentive-based direct load control (DLC) strategy. Since the emergence of reinforcement learning algorithms, their data-driven characteristics have helped the rapid development of power demand response.
[0004] However, considering multiple users and various power load devices, the training process of existing reinforcement learning algorithms requires a lot of interaction with real scenarios, which undoubtedly greatly increases the time and economic costs of using reinforcement learning algorithms, especially when facing large-scale load demand response with multiple users and various power load devices. Summary of the invention
[0005] In order to solve the problem of increasing the time cost of reinforcement learning in the above-mentioned prior art, the present invention provides a method for large-scale load demand response strategy based on meta-reinforcement learning, which can effectively save and reduce the time cost and economic cost of training for large-scale user demand response.
[0006] The present invention provides a method for large-scale load demand response strategy based on meta-reinforcement learning, comprising:
[0007] Constructing an optimization model and transforming the optimization model to obtain an initial model with the lowest electricity cost;
[0008] Performing meta-reinforcement learning training on the initial model to obtain a response model;
[0009] A response strategy for large-scale load demand is obtained through the response model to respond to load demands of multiple users or multiple devices;
[0010] The optimization model is constructed according to the constraints of the power load equipment and the power cost in discrete time, wherein the constraints include power balance constraints and power load equipment constraints. The optimization model is:
[0011] ;
[0012] in, , is the grid load power, for The electricity cost at each moment, and for A continuous control signal for identifying the electrical load at the moment;
[0013] The optimization model is transformed into a Markov decision process, which is expressed as ;
[0014] in, is the state set, is a collection of actions, is the state transfer function of the optimization model, is the reward function of the optimization model, The reward function at the time is the inverse of the electricity cost;
[0015] The initial model with the lowest electricity cost is obtained through the Markov decision process. The initial model is: ;
[0016] At each Markov decision moment, first obtain the current state And according to the status and the initial model selects an action from the action set , and then according to the state transfer function Get Status and rewards ;
[0017] At least one optimization model is constructed, the number of the optimization models is set to be multiple, the optimization models have similarities, the initial model is first parameterized and then the initial model is trained by using the meta-reinforcement learning;
[0018] The parameterized initial model is , training the initial model by meta-reinforcement learning includes the following steps:
[0019] Step 1: Initialize parameters , and randomly select several optimization models;
[0020] Step 2, obtain the first state and according to the parameterized initial model Randomly select the first action;
[0021] Step 3, execute the first action and obtain the second state and reward;
[0022] Step 4: Archive the state transfer process to the database ;
[0023] Step 5, calculate And update the parameters according to the gradient ;
[0024] Step 6: Repeat steps 2 to 5 to update the parameters. .
[0025] In one embodiment, when training the optimization model,
[0026] Step 1: Load parameters is the parameter of the meta-reinforcement learning ;
[0027] Step 2, obtain the first state and parameterize the initial model according to Randomly select the first action;
[0028] Step 3, execute the first action and obtain the second state and reward;
[0029] Step 4, calculate And update the parameters according to the gradient .
[0030] In one embodiment, the loss of the parameterized initial model is
[0031] ,
[0032] The loss is used to calculate the gradient.
[0033] The present invention also provides a system for a method of adopting a large-scale load demand response strategy based on meta-reinforcement learning, comprising a construction module, a training module and a response module;
[0034] The construction module is used to construct an optimization model and transform the optimization model to obtain an initial model with the lowest electricity cost;
[0035] The training module is used to perform meta-reinforcement learning training on the initial model to obtain a response model;
[0036] The response module is used to obtain a response strategy for large-scale load demand through the response model to respond to load demands of multiple users or multiple devices.
[0037] The present invention also provides a large-scale load demand response device based on meta-reinforcement learning, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor to enable the processor to execute a method for a large-scale load demand response strategy based on meta-reinforcement learning as described in any of the above items.
[0038] Based on the above, compared with the prior art, the method, system and device for large-scale load demand response strategy based on meta-reinforcement learning provided by the present invention, by introducing meta-reinforcement learning, do not directly train for a specific demand, but train on multiple demand responses at the same time, so as to obtain an initial model for the commonality of all demands, and then train separately for specific special demand responses on the basis of the initial model, thereby accelerating the training process, effectively reducing computing power consumption and user data usage, thereby saving time and economic costs in training large-scale user demand responses.
[0039] Other features and beneficial effects of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other beneficial effects of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The positional relationships described in the drawings in the following description are based on the directions in which the components are drawn in the diagrams, unless otherwise specified.
[0041] Figure 1 A flow chart of a method for large-scale load demand response strategy provided by the present invention;
[0042] Figure 2 An architectural diagram of a method for adopting a large-scale load demand response strategy provided by the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments; the technical features designed in different implementation modes of the present invention described below can be combined with each other as long as they do not conflict with each other; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0044] In the description of the present invention, it should be noted that all terms (including technical terms and scientific terms) used in the present invention have the same meanings as those generally understood by ordinary technicians in the field to which the present invention belongs, and should not be understood as limitations on the present invention; it should be further understood that the terms used in the present invention should be understood to have the same meanings as these terms in the context of this specification and in the relevant fields, and should not be understood in an idealized or overly formal sense, unless explicitly defined in this invention.
[0045] In terms of demand response, direct load control methods have played an important role in peak shaving and valley filling. Their data-driven characteristics have helped the rapid development of power demand response. However, the existing reinforcement learning training process requires a large amount of interaction in real scenarios, which undoubtedly greatly increases the time and economic costs of reinforcement learning training, which is more obvious when solving large-scale load demand response.
[0046] To this end, the present invention provides a method for a large-scale load demand response strategy based on meta-reinforcement learning, including constructing an optimization model and transforming the optimization model to obtain an initial model with the lowest electricity cost; performing meta-reinforcement learning training on the initial model to obtain a response model; and obtaining a response strategy for large-scale load demand through the response model to respond to load demands of multiple users or multiple devices.
[0047] When implementing it, Figure 1 As shown, the optimization model is first constructed based on the constraints of the power load equipment and the electricity cost in discrete time, wherein the constraints include power balance constraints and power load equipment constraints, wherein the power load equipment includes at least basic power load equipment and adjustable power load equipment.
[0048] Specifically, Figure 2 As shown, this architecture has users, and the user set is recorded as Each user has basic power load equipment, as well as adjustable power load equipment such as air conditioners and electric vehicles. Each user can choose to obtain an appropriate amount of electricity from the power grid or perform internal demand response based on real-time electricity costs and electricity demand.
[0049] When all users are at discrete time points The operation is performed within , T is the time span, then the power balance constraint is ;in, For users exist Base load power, air conditioning power, electric vehicle power and grid power at all times;
[0050] Air conditioning related constraints:
[0051] Specifically, users The transfer characteristics of the indoor temperature of the air conditioner are: ;in, They are Moments and Users Related indoor temperature, outdoor temperature, and disturbance variables; For users The transfer function of indoor temperature is determined by the current indoor temperature, outdoor temperature, air conditioning working power and interference variables.
[0052] In order to ensure the comfort of users, the indoor temperature is controlled within the range In which, For users The minimum and maximum temperatures accepted by the settings;
[0053] The air conditioner can be switched between off and full power Therefore, the air conditioning power is ,in, for Moment User Continuous control signal for the air conditioner.
[0054] Restrictions on electric vehicles:
[0055] Specifically, the electric vehicle constraint is
[0056]
[0057] , when the electric car arrives and departure time Between, must satisfy the user Charging needs , since the charging power of electric vehicles can also be, for example, from 0 to the maximum charging power of air conditioners Therefore, the power of the electric vehicle is ,in, for Moment User Continuous control signal for electric vehicles.
[0058] Taking into account the electricity cost of each user, the power grid The electricity price at that time is , then the optimization model is constructed as
[0059] ,
[0060] In this embodiment, multiple optimization models can be constructed, corresponding to User, User The optimization model is recorded as .
[0061] Next, the optimization model is transformed into a Markov decision process. In order to adapt to meta-reinforcement learning training, the optimization model Transformed into a Markov decision process (MDP). The Markov decision process is expressed as ;in, is the state set, is a collection of actions, is the state transfer function of the optimization model, For the optimization model The reward function is is the set of real numbers.
[0062] Specifically, the state set may include time, electricity cost at that moment, basic load power, indoor and outdoor temperature, electric vehicle power, and remaining charging time, etc.; the action set may include control signals of electric vehicles and air conditioners, etc.
[0063] At each decision moment, the user Receive current status , after which the user According to its status and the initial model In action set Internal selection action , and then according to the state transfer function Get Status and rewards .
[0064] The purpose of the Markov decision process is to obtain an initial model , and make Maximum to minimize electricity cost.
[0065] Specifically, users exist The received status can be , then the user According to the initial model Select Action ; The reward function at the moment is the opposite of the electricity cost, that is, .
[0066] Next, the initial model is trained by meta-reinforcement learning to obtain the response model. The similarities between them are that during training, all optimization models are first optimized simultaneously. Train and parameterize the initial model, the parameterized initial model is , the user selects an action based on the initial model Since any model has losses, in this embodiment, the loss of the parameterized initial model is
[0067] ,
[0068] The loss can be used to calculate the gradient.
[0069] For all optimization models The training is carried out through the following steps:
[0070] Step 1: Initialize parameters , and randomly select several optimization models;
[0071] Step 2, obtain the first state and according to the parameterized initial model Randomly select the first action;
[0072] Step 3, execute the first action and obtain the second state and reward;
[0073] Step 4: Archive the state transfer process to the database ;
[0074] Step 5, calculate And update the parameters according to the gradient ;
[0075] Step 6: Repeat steps 2 to 5 to update the parameters. .
[0076] Specifically,
[0077] First initialize the parameters ;
[0078] For episode=1,2,…,N do;
[0079] Randomly select several optimization models ;
[0080] For all do;
[0081] For do;
[0082] Get Status ;
[0083] According to the model In action set Internal selection action ;
[0084] Execute an action And get the next state and reward ;
[0085] The transfer process Archive to database ;
[0086] calculate ;
[0087] Update parameters based on gradients ;
[0088] For do;
[0089] Get Status ;
[0090] According to the model In action set Internal selection action ;
[0091] Execute an action And get the next state and reward ;
[0092] The transfer process Archive to database ;
[0093] Update Parameters .
[0094] The initial model is obtained through the above training , for each optimization model The training is carried out through the following steps:
[0095] Step 1: Load parameters is the parameter of the meta-reinforcement learning ;
[0096] Step 2, obtain the first state and parameterize the initial model according to Randomly select the first action;
[0097] Step 3, execute the first action and obtain the second state and reward;
[0098] Step 4, calculate And update the parameters according to the proposed .
[0099] Specifically, the loading parameters is the parameter of the meta-reinforcement learning ;
[0100] For episode=1,2,…,N do;
[0101] For do;
[0102] Get Status ;
[0103] According to the model In action set Internal selection action ;
[0104] Execute an action And get the next state and reward ;
[0105] calculate ;
[0106] Update parameters based on gradient .
[0107] According to [1], when using reinforcement learning methods to solve large-scale demand response for multiple users and multiple power load devices, it is necessary to train demand response strategies for each user separately, which places extremely high demands on computing resources and data resources. However, the present invention introduces meta-learning training to obtain the optimal initial strategy, which greatly accelerates the independent training process for each user and reduces computing power consumption and user data usage.
[0108] Among them, [1] is B. Wang, Y. Li, W. Ming, and S. Wang, "Deep ReinforcementLearning Method for Demand Response Management of Interruptible Load," IEEETransactions on Smart Grid, vol. 11, no. 4, pp. 3146-3155, July 2020.
[0109] The method of large-scale load demand response strategy based on meta-reinforcement learning provided by the present invention does not directly train for a specific demand, but trains on multiple demand responses at the same time by introducing meta-reinforcement learning, so as to obtain an initial model for the commonality of all demands, and then trains separately for specific special demand responses on the basis of the initial model, thereby accelerating the training process and effectively reducing the computing power consumption and the use of user data, thereby saving the time cost and economic cost of training for large-scale user demand response.
[0110] The present invention also provides a system for a method of adopting a large-scale load demand response strategy based on meta-reinforcement learning, comprising a building module, a training module and a response module. The building module, the training module and the response module can implement the method of the large-scale load demand response strategy based on meta-reinforcement learning described in the above embodiment.
[0111] In specific implementation, the construction module is used to construct an optimization model and transform the optimization model to obtain an initial model with the lowest electricity cost; the training module is used to perform meta-reinforcement learning training on the initial model to obtain a response model; the response module is used to obtain a response strategy for large-scale load demand through the response model, so as to respond to load demands of multiple users or multiple devices.
[0112] The present invention also provides an electronic device, which includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the processor to execute the method of large-scale load demand response strategy based on meta-reinforcement learning in the above-mentioned embodiment.
[0113] In specific implementation, the number of processors may be one or more, and the processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0114] The memory and the processor may be communicatively connected via a bus or other means, and the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor so that the processor executes the method for large-scale load demand response strategy based on meta-reinforcement learning in the above embodiment.
[0115] Based on the above, compared with the prior art, the method, system and device for large-scale load demand response strategy based on meta-reinforcement learning provided by the present invention, by introducing meta-reinforcement learning, do not directly train for a specific demand, but train on multiple demand responses at the same time, so as to obtain an initial model for the commonality of all demands, and then train separately for specific special demand responses on the basis of the initial model, thereby accelerating the training process, effectively reducing computing power consumption and user data usage, thereby saving time and economic costs in training large-scale user demand responses.
[0116] In addition, those skilled in the art should understand that, although there are many problems in the prior art, each embodiment or technical solution of the present invention can be improved in only one or several aspects, without having to solve all the technical problems listed in the prior art or background technology at the same time. Those skilled in the art should understand that the content not mentioned in a claim should not be used as a limitation on the claim.
[0117] Finally, it should be noted that the terms "first", "second", etc. (if any) in the description and claims of the embodiments of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for large-scale load demand response strategy based on meta-reinforcement learning, characterized by: include Constructing an optimization model and transforming the optimization model to obtain an initial model with the lowest electricity cost; Performing meta-reinforcement learning training on the initial model to obtain a response model; A response strategy for large-scale load demand is obtained through the response model to respond to load demands of multiple users or multiple devices; The optimization model is constructed according to the constraints of the power load equipment and the power cost in discrete time, wherein the constraints include power balance constraints and power load equipment constraints. The optimization model is: ; in, , is the grid load power, for The electricity cost at each moment, and for A continuous control signal for identifying the electrical load at the moment; The optimization model is transformed into a Markov decision process, which is expressed as ; in, is the state set, is a collection of actions, is the state transfer function of the optimization model, is the reward function of the optimization model, The reward function at the time is the inverse of the electricity cost; The initial model with the lowest electricity cost is obtained through the Markov decision process. The initial model is: ; At each Markov decision moment, first obtain the current state And according to the status and the initial model selects an action from the action set , and then according to the state transfer function Get Status and rewards ; At least one optimization model is constructed, the number of the optimization models is set to be multiple, the optimization models have similarities, the initial model is first parameterized and then the initial model is trained by using the meta-reinforcement learning; The parameterized initial model is , training the initial model by meta-reinforcement learning includes the following steps: Step 1: Initialize parameters , and randomly select several optimization models; Step 2, obtain the first state and according to the parameterized initial model Randomly select the first action; Step 3, execute the first action and obtain the second state and reward; Step 4: Archive the state transfer process to the database ; Step 5, calculate And update the parameters according to the gradient ; Step 6: Repeat steps 2 to 5 to update the parameters. .
2. The method for large-scale load demand response strategy based on meta-reinforcement learning according to claim 1 is characterized in that: When training the optimization model, Step 1: Load parameters is the parameter of the meta-reinforcement learning ; Step 2, obtain the first state and parameterize the initial model according to Randomly select the first action; Step 3, execute the first action and obtain the second state and reward; Step 4, calculate And update the parameters according to the gradient .
3. The method for large-scale load demand response strategy based on meta-reinforcement learning according to claim 1, characterized in that: The loss of the parameterized initial model is , The loss is used to calculate the gradient.
4. A system using the method of large-scale load demand response strategy based on meta-reinforcement learning as claimed in claim 1, characterized in that: Includes building module, training module and response module; The construction module is used to construct an optimization model and transform the optimization model to obtain an initial model with the lowest electricity cost; The training module is used to perform meta-reinforcement learning training on the initial model to obtain a response model; The response module is used to obtain a response strategy for large-scale load demand through the response model to respond to load demands of multiple users or multiple devices.
5. A large-scale load demand response device based on meta-reinforcement learning, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor to enable the processor to execute the method for large-scale load demand response strategy based on meta-reinforcement learning as described in any one of claims 1-3.
