Load partitioning and q-learning based method for electric demand response resource allocation

Abstract

The application discloses a kind of based on load division and Q-learning's electric energy demand response resource allocation method, it is related to demand response scheduling technical field.The application includes the following steps: S1, obtains historical load data from intelligent electric meter, and daily peak load is normalized to each user's daily load curve, and user type is obtained using SOM based on relative load curve clustering;S2, using Q-learning, energy is distributed to each residential user according to the daily peak load of user;S3, collects the electric energy demand of each residential user, and adjusts residential user distribution electric energy according to electric energy demand;S4, according to the electric energy of distribution and real-time electricity price, dispatch appliance operation.The application aims at solving the technical problem that existing method cannot minimize the total energy cost of residential community without relying on user equipment information, based only on the historical load data of intelligent electric meter.

Description

Technical Field

[0001] This invention relates to the field of demand response scheduling technology, and more specifically to a method for allocating power demand response resources based on load partitioning and Q-learning. Background Technology

[0002] Demand response (DR) refers to the proactive adjustment of electricity consumption behavior by users in response to rising wholesale electricity prices or threats to grid reliability, upon receiving compensation notices from electricity suppliers regarding load reduction or price increases. This adjustment involves reducing or shifting electricity consumption during off-peak hours to respond to demand, thereby maintaining grid stability and mitigating sharp short-term price increases. In a smart grid environment, DR enables the controlled and intelligent transmission of electricity from generation to active users on the demand side. Given that residential electricity consumption accounts for 30% to 40% of global total energy consumption, research on demand response in residential communities has both profound theoretical value and significant practical implications.

[0003] In recent years, users have placed increasing emphasis on privacy protection. Traditional methods often rely on users proactively providing detailed information such as device operating status, appliance types, and energy preferences to achieve refined modeling and personalized scheduling. However, some users, for data security and privacy reasons, refuse to share information about their household appliances, resulting in incomplete user-side information obtained by the scheduling model, thus affecting the effectiveness of the scheduling strategy.

[0004] To address these issues, some current research attempts to introduce privacy protection mechanisms, such as federated learning and differential privacy algorithms, or to employ information anonymization techniques for model training. While these methods can reduce reliance on raw device data to some extent, they often lead to increased model complexity and computational resource consumption. Currently, there is no method to minimize the total energy cost of a residential community without relying on user device information, based solely on historical load data from smart meters. Summary of the Invention

[0005] In view of this, the present invention provides a power demand response resource allocation method based on load partitioning and Q-learning, which aims to solve the technical problem that existing methods cannot minimize the total energy cost of residential communities without relying on user equipment information and only based on historical load data of smart meters.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A power demand response resource allocation method based on load partitioning and Q-learning includes the following steps:

[0008] S1. Obtain historical load data from smart meters, normalize the daily load curve of each user by daily peak load, and use SOM to cluster based on relative load curves to obtain user type;

[0009] S2. Use Q-learning to allocate energy to each residential user based on the user's daily peak load;

[0010] S3. Collect the electricity demand of each residential user and adjust the electricity allocation to residential users according to the electricity demand;

[0011] S4. Dispatch the operation of electrical appliances according to the allocated power and real-time electricity price.

[0012] Optionally, in S1, the specific process is as follows:

[0013] S11. Obtain historical load curves from smart meters and normalize the user's daily load curves using daily load peak values;

[0014] S12, Best Matching Unit This refers to the mapping unit with the smallest distance to the current input vector x during SOM training. The distance is calculated using Euclidean distance, as shown in the following expression:

[0015] ;

[0016] Where x represents the input vector; j represents the mapping unit, and each mapping unit j can be represented as a prototype vector. , where d is the input dimension;

[0017] S13. In each iteration, the prototype vector is updated. This process is repeated a predetermined number of times, with adaptive coefficients... and neighborhood radius The value of decreases over time, as defined below:

[0018] ;

[0019] ;

[0020] Where k represents the number of iterations of SOM; This represents the neighborhood kernel centered on the best-matching unit; f Indicates the location of neurons in the SOM grid;

[0021] S14. After obtaining the clustering type, the center curve expressions for the relative load curves of different user types are as follows:

[0022] ;

[0023] Where u represents the user category; This represents the relative load curve for user type u.

[0024] Optional, for each residential user i According to the following formula, it is in the time slot t The historical load data is normalized, and the normalized value is... For values ​​between 0 and 1, the calculation formula is as follows:

[0025] ;

[0026] This represents the historical load data of residential user i in time slot t; This represents the peak daily load of residential user i in time slot t.

[0027] Optionally, in S2, after receiving the user type, the power allocation for each user i of each type u in the community is calculated via DR; different agents adopt different allocation strategies for different types of users, and the agents observe the state in time slot t. And select actions based on strategy π The current reward is obtained based on the reward function. The agent then observes the new state. .

[0028] Optionally, the specific modeling method for the intelligent agent is as follows:

[0029] S21. State Modeling: The current state is defined by the real-time price and the median value for this user type, expressed as follows:

[0030] ;

[0031] in, This represents the center value of the daily normalized load for this type of user;

[0032] S22. Action Modeling: Actions represent the relative load assigned to users of the corresponding type; the normalized load distribution takes discrete values, i.e., [0.1, 0.2, ..., 0.9, 1], with corresponding indices of [0, 1, ..., 8, 9]; the minimum normalized load is 0.1; an ε-greedy strategy is used to select the current state. The following action The ε-greedy strategy is defined as follows:

[0033] ;

[0034] in, It is the Q-value of the state-action pair, and ε is the probability of choosing an action policy;

[0035] S23. Incentive Modeling: Incentive modeling comprises three parts: electricity cost reduction, comfort distance reduction, and load allocation rationality, which are respectively user electricity cost rewards. User comfort rewards Reasonable rewards for load allocation User electricity cost reward The definition is as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] in, This indicates the relative load allocated to this type of user at time t; This represents the minimum relative load allocated to this type of user at time t; This represents the maximum relative load allocated to this type of user at time t;

[0040] User comfort reward The definition is as follows:

[0041] ;

[0042] ;

[0043] The reasonable load allocation reward is calculated using cosine similarity to ensure that the allocated energy is similar to the normalized load center curve of this type of user. It is the value of the normalized cosine phase, and the reasonable load allocation reward. The definition is as follows:

[0044] ;

[0045] ;

[0046] ;

[0047] in, for The maximum value; for The minimum value;

[0048] award Rewards based on user electricity costs User comfort rewards Reasonable rewards for load allocation Composition, its expression is as follows:

[0049] ;

[0050] in, , , These are normalization parameters;

[0051] S24, Q-learning algorithm: In the context of user energy consumption management, Q-learning simulates users' electricity consumption behavior, allocates electricity to users, and optimizes electricity consumption strategies;

[0052] The goal of energy scheduling is to find the optimal strategy. That is, determining the optimal allocation of relative loads to maximize the action-value function; the basic mechanism of the Q-learning algorithm is to construct a Q-value table, where the Q-value of each state-action pair is... The algorithm updates in each iteration until the convergence condition is met; in this way, the best action with the best Q value in each state is selected; the best Q value The definition is as follows:

[0053] ;

[0054] The Q-value is updated based on the reward, learning rate, and discount factor, and its expression is as follows:

[0055] ;

[0056] Where θ represents the learning rate. At that time, the intelligent agent only uses prior information, while This means that the agent only considers the current estimate and ignores prior information; based on the power allocation results of different agents' Q-learning and the user's daily load peak, power is allocated to each user in each type. .

[0057] As can be seen from the above technical solution, compared with the prior art, this invention provides a method for energy demand response resource allocation based on load partitioning and Q-learning. First, historical load data from user smart meters is sent to a load aggregator. The load aggregator normalizes the load data based on the user's daily load peak and then partitions users using the System of Utility Models (SOM) to distinguish users with different electricity consumption habits. Then, based on the user partitioning results and real-time electricity prices, the load aggregator allocates electricity to different types of users in the community. Finally, the electricity consumption behavior of users after scheduling generates new energy consumption data, which is then fed back to the load aggregator by the smart meters. This invention provides a complete method for demand response resource allocation in residential communities, characterized by reducing the community's total electricity cost and peak-to-average power ratio. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the overall framework for power demand response resource allocation based on load partitioning and Q-learning;

[0060] Figure 2 This is a flowchart of the power demand response resource allocation method based on load partitioning and Q-learning provided by the present invention;

[0061] Figure 3 This is a comparison chart of the load allocated to the six types of users and the original load;

[0062] Figure 4 This is a comparison chart of electricity consumption before and after residential user 1 participated in demand response;

[0063] Figure 5 This is a comparison chart of electricity consumption before and after a residential community participated in demand response. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] This invention discloses a power demand response resource allocation method based on load partitioning and Q-learning, comprising the following steps:

[0066] S1. Obtain historical load data from smart meters, normalize the daily load curve of each user by daily peak load, and use SOM to cluster based on relative load curves to obtain user type;

[0067] S2. Use Q-learning to allocate energy to each residential user based on the user's daily peak load;

[0068] S3. Collect the electricity demand of each residential user and adjust the electricity allocation to residential users according to the electricity demand;

[0069] S4. Dispatch the operation of electrical appliances according to the allocated power and real-time electricity price.

[0070] like Figure 1 As shown, this embodiment provides a power demand response resource allocation method based on load partitioning and Q-learning, applied to demand response scheduling in residential communities. It comprises two key parts. One part is user partitioning, where the load aggregator partitions users into categories based on historical load curves (historical energy consumption data) using a System of Origin (SOM). The other part is power allocation, where the load aggregator calculates the power allocation for each user in the community using Q-learning based on user type and real-time electricity price. For example... Figure 2 As shown, the method of the present invention includes the following steps:

[0071] Step S1: The load aggregator obtains historical load data from smart meters and normalizes the daily load curve of each user through daily peak load. It then uses SOM to cluster based on relative load curves to obtain user types.

[0072] The specific process for this step is as follows:

[0073] S11. The load aggregator first obtains historical load curves (historical energy consumption data) from smart meters, and uses daily load peak values ​​to normalize the daily load curves of users to ensure that the daily load curve size of different users is within the range of 0 to 1, so as to eliminate the inability of clustering of load curves between different users due to differences in electricity consumption scale to reflect users' electricity consumption habits.

[0074] residential users For example, for each residential user According to the following formula, it is in the time slot The historical load data is normalized, and the normalized value is... Between 0 and 1.

[0075] ;

[0076] S12 and SOM are unsupervised learning algorithms, typically represented as mapping units in a two-dimensional grid, used to map high-dimensional data to a low-dimensional space while preserving the data's topological structure. Optimal matching unit. This refers to the mapping unit with the smallest distance to the current input vector x during SOM training. The distance is calculated using Euclidean distance, as shown in the following expression:

[0077] ;

[0078] Where x represents the input vector; j represents the mapping unit, and each mapping unit j can be represented as a prototype vector. , where d is the input dimension.

[0079] S11 is equivalent to data preprocessing. First, the dataset is normalized, and then SOM is used for clustering. The reason for normalization is mentioned in S11—to eliminate the fact that clustering of load curves between different users due to differences in electricity consumption scale cannot reflect users' electricity consumption habits.

[0080] S12 and S13 introduce the basic principles of SOM. S12 explains that SOM in this invention finds the minimum matching unit (the mapping unit with the smallest distance to the current input vector x) based on Euclidean distance—the SOM clustering process can be understood as the process of continuously assigning the input vector to the minimum matching unit as the algorithm iterates. S13 explains the process of SOM learning data features and updating clustering results. The classification process does not end immediately after the input vector finds the minimum mapping unit. SOM continues to learn and update this unit and its neighborhood to gradually optimize the clustering structure. As the iteration progresses, the adaptive coefficients... and neighborhood radius It will gradually decrease, and the algorithm will gradually converge.

[0081] S13. In each iteration, the prototype vector will be updated. This process will be repeated a predetermined number of iterations, with adaptive coefficients... and neighborhood radius The value of decreases over time, as defined below:

[0082] ;

[0083] ;

[0084] Where k represents the number of iterations of SOM; This represents the neighborhood kernel centered on the best-matching unit; f This indicates the location of neurons in the SOM grid.

[0085] S14. After obtaining the clustering type, the center curve expressions for the relative load curves of different user types are as follows:

[0086] ;

[0087] Where u represents the user category; This represents the relative load curve for user type u.

[0088] Step S2: The load aggregator uses Q-learning to distribute energy to each residential user based on the user's daily peak load.

[0089] After receiving the user type, the load aggregator calculates the power allocation for each user i of each type u in the community using the DR (Digital Distribution Calculation). Different agents adopt different allocation strategies for different types of users, and the agents observe the state during time slot t. And select actions based on strategy π The current reward is obtained based on the reward function. After this action is taken, the intelligent entity observes the new state. This process continues until all services have been assigned and the process terminates. Below is an example of modeling a single agent; other agents are modeled using the same method.

[0090] S21. State Modeling: The current state is defined by the real-time price and the median value for this user type, expressed as follows:

[0091] ;

[0092] in, This represents the center value of the daily normalized load for this type of user.

[0093] S22. Action Modeling: An action represents the relative load allocated to a user of that type. The normalized load distribution takes discrete values, i.e., [0.1, 0.2, ..., 0.9, 1], with corresponding indices of [0, 1, ..., 8, 9]. The minimum normalized load is 0.1 because each user has unschedulable devices, so the normalized load can never be 0 at any given time. During strategy selection, an ε-greedy strategy is used to select the current state. The following action The ε-greedy strategy is defined as follows:

[0094] ;

[0095] in, ε is the Q-value of the state-action pair, and ε is the probability of choosing an action policy.

[0096] S23. Incentive Modeling: Incentive modeling comprises three parts: electricity cost reduction, comfort distance reduction, and load allocation rationality, which are respectively user electricity cost rewards. User comfort rewards Reasonable rewards for load allocation User electricity cost reward The definition is as follows:

[0097] ;

[0098] ;

[0099] ;

[0100] in, This indicates the relative load allocated to this type of user at time t; This represents the minimum relative load allocated to this type of user at time t; This represents the maximum relative load allocated to this type of user at time t.

[0101] User comfort reward The definition is as follows:

[0102] ;

[0103] ;

[0104] The reasonable load allocation reward is calculated using cosine similarity to ensure that the allocated energy is similar to the normalized load center curve of this type of user. It is the value of the normalized cosine phase, and the reasonable load allocation reward. The definition is as follows:

[0105] ;

[0106] ;

[0107] ;

[0108] in, for The maximum value; for The minimum value.

[0109] award Rewards based on user electricity costs User comfort rewards Reasonable rewards for load allocation Composition, its expression is as follows:

[0110] ;

[0111] in, , , For normalization parameters, .

[0112] S24. Q-learning Algorithm: Q-learning is a reinforcement learning algorithm that learns the optimal policy through interaction with the environment, that is, choosing the action that maximizes the cumulative reward in a given state. In the context of user energy consumption management, Q-learning can be used to simulate users' electricity consumption behavior and allocate electricity to users, optimizing electricity consumption strategies.

[0113] The goal of energy scheduling is to find the optimal strategy. The algorithm aims to determine the optimal allocation of relative loads to maximize the action-value function. The basic mechanism of this algorithm is to construct a Q-value table, where the Q-value of each state-action pair is... The value is updated in each iteration until the convergence condition is met. In this way, the optimal action with the best Q-value in each state can be selected. (Optimal Q-value) The definition is as follows:

[0114] ;

[0115] The Q-value can be updated based on the reward, learning rate, and discount factor, and its expression is as follows:

[0116] ;

[0117] Where θ represents the learning rate, which takes a value between 0 and 1. At that time, the intelligent agent only uses prior information, while This means the agent only considers the current estimate and ignores prior information. Based on the energy allocation results from different agents' Q-learning and the user's daily load peak, energy is allocated to each user in each type. .

[0118] Step S3: The load aggregator receives the electricity demand of each residential user from the home energy management system. If the electricity allocated to a residential user is less than the demand, the load aggregator will adjust the electricity allocation for that user.

[0119] Each residential user's electricity demand is different, depending on the user's previous energy consumption data. The load aggregator allocates more energy to users who consume more electricity and less energy to users who consume less electricity.

[0120] Step S4: The home energy management system schedules the operation of appliances based on the electricity allocated by the load aggregator and the real-time electricity price.

[0121] Home energy management systems compare real-time electricity prices with the price the user is willing to pay. If the current electricity price is higher than the user's willingness to pay, the current price is considered high; if the current electricity price is lower than or equal to the user's willingness to pay, the current price is considered low.

[0122] residential users i For example, residential users The home energy management system decides the residential user's energy based on the received high / low price information. i Starting and stopping household appliances. (For residential users) i Speaking of time slots tIf electricity prices are high, the home energy management system may suspend the operation of some dispatchable devices and reduce the power of some adjustable devices, while retaining non-dispatchable devices. For residential users... Speaking of time slots t If the electricity price is low, then the home energy management system may activate some dispatchable devices and increase the power of some adjustable devices.

[0123] The following analysis uses specific practical examples to demonstrate the effectiveness of the present invention.

[0124] like Figure 3 The diagram shows the load allocation and original load for the six user groups obtained through SOM clustering. It can be seen that each user group achieves reasonable optimization of power allocation, specifically: power allocation decreases during periods when the electricity price is higher than the average price; power allocation increases during periods when the electricity price is lower than the average price; and the allocated power is roughly the same as that without the DR strategy. The method of this invention is abbreviated as the DR-SOM-Q algorithm. It can be seen that using the DR-SOM-Q algorithm can help users reduce electricity costs without significantly altering their electricity consumption habits.

[0125] Figure 4 This comparison examines the electricity consumption of User 1 using the DR-SOM-Q algorithm versus not using a demand response strategy. The DR-SOM-Q algorithm increases electricity consumption during periods when electricity prices are lower than the average price and decreases consumption during periods when prices are higher than the average price. The electricity consumption trends of the DR-SOM-Q algorithm and the non-demand response strategy are similar and show minimal changes, indicating that the DR-SOM-Q algorithm hardly alters the user's electricity consumption habits.

[0126] Figure 5 This comparison examines the load trends of a community using the DR-SOM-Q algorithm versus one without a demand response strategy. The DR-SOM-Q algorithm increases electricity consumption during periods when electricity prices are below average and decreases consumption during periods when prices are above average. The load trends of the DR-SOM-Q algorithm and the non-demand response strategy show similar and minimal changes, indicating that the DR-SOM-Q algorithm helps the community reduce electricity costs while minimizing disruption to user comfort.

[0127] To further verify the effectiveness of the algorithm, peak-to-average ratio (PAR) is introduced.

[0128] ;

[0129] This represents the sum of energy consumption of all residential community users, that is, the electricity purchased by the residential community from the grid.

[0130] Table 1 shows the performance changes of the DR-SOM-Q algorithm and the DR-SOM-MDP algorithm (demand response scheduling algorithms based on SOM and Markov decision-making) relative to the lack of a DR strategy. The difference between the DR-SOM-Q and DR-SOM-MDP algorithms stems from their different uncertainty resolution strategies. In terms of power consumption, both are roughly the same as those without a DR strategy; in terms of power cost, the DR-SOM-Q algorithm reduces costs more significantly than the DR-SOM-MDP algorithm; in terms of peak-to-average power ratio (PAPR), the DR-SOM-Q algorithm reduces costs slightly more significantly than the DR-SOM-MDP algorithm; and in terms of user comfort distance, the DR-SOM-Q algorithm is slightly higher than the DR-SOM-MDP algorithm, but this is within an acceptable range and will not have a significant impact on the user's electricity experience.

[0131] Table 1 Performance changes of DR-SOM-Q algorithm and DR-SOM-MDP algorithm

[0132]

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0134] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A power demand response resource allocation method based on load partitioning and Q-learning, characterized in that, Includes the following steps: S1. Obtain historical load data from smart meters, normalize the daily load curve of each user by daily peak load, and use SOM to cluster based on relative load curves to obtain user type; S2. Use Q-learning to allocate energy to each residential user based on the user's daily peak load; In S2, after receiving the user type, the power allocation for each user i of each type u in the community is calculated using DR; different agents adopt different allocation strategies for different types of users, and the agents observe the state in time slot t. And select actions based on strategy π The current reward is obtained based on the reward function. The agent then observes the new state. The specific modeling method for intelligent agents is as follows: S21. State Modeling: Use the real-time price and the central value of this type of user as the current state; S22. Action Modeling: Actions represent the relative load assigned to users of the corresponding user type; S23. Incentive Modeling: Incentive modeling comprises three parts: electricity cost reduction, comfort distance reduction, and load allocation rationality, which are respectively user electricity cost rewards. User comfort rewards Reasonable rewards for load allocation ; The reasonable load allocation reward is calculated using cosine similarity to ensure that the allocated energy is similar to the normalized load center curve of this type of user. It is the value of the normalized cosine phase, and the reasonable load allocation reward. The definition is as follows: ； ； ； in, for The maximum value; for The minimum value; This represents the center value of the daily normalized load for this type of user. This indicates the relative load allocated to this type of user at time t; award Rewards based on user electricity costs User comfort rewards Reasonable rewards for load allocation Composition, its expression is as follows: ； in, , , These are normalization parameters; S24, Q-learning algorithm: In the context of user energy consumption management, Q-learning simulates users' electricity consumption behavior, allocates electricity to users, and optimizes electricity consumption strategies; S3. Collect the electricity demand of each residential user and adjust the electricity allocation to residential users according to the electricity demand; S4. Dispatch the operation of electrical appliances according to the allocated power and real-time electricity price.

2. The energy demand response resource allocation method based on load partitioning and Q-learning according to claim 1, characterized in that, In S1, the specific process is as follows: S11. Obtain historical load curves from smart meters and normalize the user's daily load curves using daily load peak values; S12, Best Matching Unit This refers to the mapping unit with the smallest distance to the current input vector x during SOM training. The distance is calculated using Euclidean distance, as shown in the following expression: ； Where x represents the input vector; j represents the mapping unit, and each mapping unit j is represented as a prototype vector. , where d is the input dimension; S13. In each iteration, the prototype vector is updated. This process is repeated a predetermined number of times, with adaptive coefficients... and neighborhood radius The value of decreases over time, as defined below: ； ； Where k represents the number of iterations of SOM; This represents the neighborhood kernel centered on the best-matching unit; f Indicates the location of neurons in the SOM grid; S14. After obtaining the clustering type, the center curve expressions for the relative load curves of different user types are as follows: ； Where u represents the user category; This represents the relative load curve for user type u.

3. The energy demand response resource allocation method based on load partitioning and Q-learning according to claim 2, characterized in that, For each residential user i According to the following formula, it is in the time slot t The historical load data is normalized, and the normalized value is... For values ​​between 0 and 1, the calculation formula is as follows: ； This represents the historical load data of residential user i in time slot t; This represents the peak daily load of residential user i in time slot t.

4. The energy demand response resource allocation method based on load partitioning and Q-learning according to claim 1, characterized in that, S21. The expression for state modeling is as follows: ； S22. Action modeling specifically involves the normalized load distribution taking discrete values, namely [0.1, 0.2, ..., 0.9, 1], with corresponding indices of [0, 1, ..., 8, 9]. The normalized load minimum is 0.1; the current state is selected using an ε-greedy strategy. The following action The ε-greedy strategy is defined as follows: ； in, It is the Q-value of the state-action pair, and ε is the probability of choosing an action policy; S23, User Electricity Cost Rewards in Reward Modeling The definition is as follows: ； ； ； in, This represents the minimum relative load allocated to this type of user at time t; This represents the maximum relative load allocated to this type of user at time t; User comfort reward The definition is as follows: ； ； The specific steps of the S24 and Q-learning algorithms are as follows: The goal of energy scheduling is to find the optimal strategy. That is, determining the optimal allocation of relative loads to maximize the action-value function; the basic mechanism of the Q-learning algorithm is to construct a Q-value table, where the Q-value of each state-action pair is... The algorithm updates in each iteration until the convergence condition is met; in this way, the best action with the best Q value in each state is selected; the best Q value The definition is as follows: ； The Q-value is updated based on the reward, learning rate, and discount factor, and its expression is as follows: ； Where θ represents the learning rate. At that time, the intelligent agent only uses prior information, while This means that the agent only considers the current estimate and ignores prior information; based on the power allocation results of different agents' Q-learning and the user's daily load peak, power is allocated to each user in each type. .

Images