A method for establishing an evaluation model for user peak load behavior, an evaluation method and a device
By constructing an evaluation model for user peak shaving behavior, using Markov decision-making process and reverse learning technology to dynamically estimate user peak shaving behavior, the problem of difficult prediction of user load complexity and time domain coupling in the existing technology is solved, and the problem of unpredictable user load complexity and time domain coupling in the existing technology is realized. The load balance of the power system and the support of new energy consumption.
Patent Information
- Application Number
- CN202111000182.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-08-27
AI Technical Summary
The existing power peak shaving auxiliary services are difficult to effectively utilize user resources, especially due to the complexity of user load and time-domain coupling, it is impossible to accurately predict and dynamically estimate the user's peak shaving behavior, resulting in insufficient adjustment methods of the power system.
Using the evaluation model establishment method of user peak shaking behavior, the triple Markov decision-making process is constructed by obtaining user data, the feature function and weight parameters are acquired inverse learning, the return function and quadruple Markov decision-making process is constructed, and the evaluation model is trained. Dynamically estimate the user's peak shaving behavior.
It realizes dynamic estimation of peak shaving behavior based on the real-time state of users, improves power load balance, promotes the economic operation of the power system, provides accurate means of estimating peak shaving behavior on the user side, alleviates supply and demand balance and promotes new energy consumption.
Smart Images

Figure CN113706018B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of demand response of power systems, and in particular to a method for establishing an evaluation model, an evaluation method and a device for user peak load regulation behavior. Background Art
[0002] The essence of peak load regulation is to achieve a balance of power by matching power generation output with load changes through short-term power regulation. With the large-scale grid connection of new energy sources, the problem of insufficient power system regulation has become increasingly prominent, and the original peak load regulation service based on thermal power can no longer meet the needs of power grid operation. In recent years, third-party entities based on user resources have also been increasingly involved in power peak load regulation services, providing different peak load regulation services such as peak shaving and valley filling for the power grid.
[0003] Unlike thermal power units, when user resources participate in peak load regulation, the peak load regulation model is more complicated. This is because, on the one hand, it is difficult to establish an analytical economic operation model for load demand like thermal power units. On the other hand, user load also has obvious time domain coupling. The duration of peak load regulation and the completed peak load regulation mileage will affect the user's electricity consumption and peak load regulation potential. Therefore, the internal state changes of user resources must be considered.
[0004] At present, conventional statistical models and data-driven methods tend to predict peak-shaving behavior from the surface data of user peak-shaving. They lack sufficient consideration for the time domain correlation of user peak-shaving behavior and the inherent decision-making mechanism of user behavior, and are unable to dynamically estimate user peak-shaving behavior based on the user's real-time status. Summary of the invention
[0005] The purpose of the present invention is to provide a method for establishing an evaluation model of user peak-shaving behavior, an evaluation method and a device, which can dynamically estimate the user's peak-shaving behavior according to the user's real-time status, so as to better improve the power load and promote the economic operation of the power system.
[0006] In order to achieve the above object, the present invention provides a method for establishing an evaluation model of user peak load shaving behavior, comprising:
[0007] Obtain user data, the user data including: actual load data, baseline load data and peak-shaving price time-sharing data.
[0008] A triple Markov decision process of the user's peak-shaving behavior is constructed according to the user data, and a first example trajectory of the user's peak-shaving behavior is obtained according to the triple Markov decision process; the triple Markov decision process includes: the Markov state of the user in time period t, the user's peak-shaving behavior and the state transferred to time period t+1.
[0009] A plurality of effective feature functions are obtained according to the peak-shaving behavior of the user, weight parameters corresponding to the plurality of effective feature functions are obtained through reverse learning of the first example trajectory, and a reward function is constructed according to the plurality of effective feature functions and the weight parameters corresponding to the plurality of effective feature functions, wherein the reward function is a linear mapping of the plurality of effective feature functions.
[0010] A quaternary Markov decision process is obtained according to the reward function and the ternary Markov decision process, and a second example trajectory of the user's peak-shaving behavior is obtained according to the quaternary Markov decision process.
[0011] The second example trajectory is learned to obtain an evaluation model of the user's peak load shaving behavior.
[0012] Furthermore, the triplet Markov decision process is constructed according to the user data, and a first example trajectory of the user peak load shaving behavior is obtained according to the triplet Markov decision process, specifically:
[0013] Assume that the actual load of user i in period t is The baseline load is The peak-shaving price time-sharing data is The peak load is in,
[0014] Constructing a triplet Markov decision process for user peak-shaving behavior t = t ,a t ,s′ t >, where s t represents the Markov state of the user in period t, a t represents the user's peak load behavior, s′ t represents the state of the transfer in the t+1 period; wherein, the s t Including the peak price of the current period Baseline load of users Peak load behavior of users in the past 24 hours make And assume that the user's state space is S i ,Right now The peak load regulation behavior of users is specifically peak load regulation, that is, and A i is the action space of user i’s peak load behavior.
[0015] According to the triple Markov decision process, the first example trajectory of the peak-shaving behavior of user i is obtained, and the first example trajectory is recorded as τ m , the τ m The specific representation is as follows:
[0016]
[0017] Wherein, m=1, 2, ..., M, m represents the number of trajectories, and T represents the total duration of the trajectory.
[0018] Furthermore, the method of obtaining multiple effective characteristic functions according to the user's peak load shaving behavior is specifically as follows:
[0019] An effective characteristic function is constructed based on the peak-shaving behavior that the user has performed and the peak-shaving benefits of the peak-shaving behavior. The effective characteristic function includes: the peak-shaving benefits of the user in the current period Maximum user adjustable peak capacity The peak mileage that the user has completed in the past 24 hours Duration of effective peak load regulation with users
[0020] The specific expression of the effective characteristic function is as follows:
[0021]
[0022]
[0023]
[0024]
[0025] Furthermore, the reverse learning of the first example trajectory obtains weight parameters corresponding to multiple valid feature functions, and constructs a reward function according to the multiple valid feature functions and the weight parameters corresponding to the multiple valid feature functions, wherein the reward function is a linear mapping of the multiple valid feature functions, specifically:
[0026] Assume that The weight parameter is θ1, The weight parameter is θ2, The weight parameter is θ3, The weight parameter is θ4.
[0027] According to the linear mapping of the reward function to the effective feature function, the reward function can be obtained, which is recorded as Said Specifically:
[0028]
[0029] in, represents the effective feature function, and θ=[θ1,θ2,θ3,θ4] represents the weight parameter of each effective feature function.
[0030] Define the feature expectation f(π) as the effective feature function on a first example trajectory τ m The cumulative expectation in is as follows:
[0031]
[0032] Among them, π represents the strategy selected by the user, and γ represents the attenuation coefficient of future period returns;
[0033] Based on the first example trajectories of the peak-shaving behaviors of M users, the experience value of the feature expectation is obtained, which is specifically:
[0034]
[0035] in, Represents the feature expectation of the first example trajectory of the peak behavior of the mth user;
[0036] When the weight parameter of the reward function is θ, and the total value of the feature expectation obtained by the reinforcement learning training of multiple first example trajectories is equal to the empirical value of the feature expectation of multiple first example trajectory data, it is determined that the trained simulation model has the same decision preference as the user, and the specific form is as follows:
[0037]
[0038] Among them, Pr(τ m ) represents each behavior trajectory τ m probability;
[0039] When the weight parameter of the reward function is θ, the probability of trajectory occurrence is exponentially related to the value of the reward function, as shown below:
[0040]
[0041] Among them, s j represents the trajectory τ belonging to the first example m A state in , Z(θ) is the partition function that normalizes the total probability, and the expression is as follows:
[0042]
[0043] The optimization model based on maximum entropy can be expressed as:
[0044]
[0045]
[0046] Expressing it as Lagrangian relaxation yields:
[0047]
[0048] Among them, λ1 and λ0 are Lagrangian relaxation conditions, which can be obtained by subgradient method. Solve and obtain θ.
[0049] Furthermore, the Markov decision process of the quaternary is obtained according to the reward function and the Markov decision process of the ternary, and the second example trajectory of the user peak-shaving behavior is obtained according to the Markov decision process of the quaternary, which is specifically:
[0050] Add the reward function to the three-tuple Markov decision process to obtain the four-tuple Markov decision process d t = t ,a t ,r t ,s′t>;
[0051] According to the four-tuple Markov decision process, the second example trajectory of the peak-shaving behavior of user i is obtained, and the second example trajectory is recorded as τ′ m , the τ′ m The specific representation is as follows:
[0052]
[0053] Furthermore, the second example trajectory is learned to obtain an evaluation model of the user's peak load shaving behavior, specifically:
[0054] The intelligent agent is constructed to learn the second example trajectory. The goal of the intelligent agent learning is to By optimizing To maximize the expected return, the expected return is
[0055] Define the following Q function:
[0056]
[0057] Among them, Q π (s,a) represents the expected return of Markov state s and peak-shaving behavior a under a given strategy π; the specific connotation of strategy π is that when the input is Markov state s, the output is the probability distribution π(s,a) of a peak-shaving behavior a;
[0058] The optimal strategy is to make Q π The (s,a) function satisfies the following Bellman equation:
[0059]
[0060] The Q function is trained according to the second example trajectory to obtain an evaluation model of the user's peak shaving behavior.
[0061] Furthermore, the intelligent agent includes: Q learning, Sarsa learning and deep Q network.
[0062] The present invention also discloses a method for evaluating user peak load-shaving behavior, wherein the evaluation model established by the above evaluation model establishment method is used to evaluate the user's peak load-shaving behavior, and the evaluation method comprises:
[0063] Get the user's current Markov state.
[0064] The user's current Markov state is input into the preset evaluation model to obtain the user's peak load behavior.
[0065] The invention also discloses an evaluation device for household peak load shaving behavior, comprising: an acquisition module and an evaluation module.
[0066] The acquisition module is used to acquire the current Markov state of the user.
[0067] The evaluation module is used to input the current Markov state of the user into a preset evaluation model to obtain the peak load shaving behavior of the user.
[0068] Compared with the prior art, the method for establishing an evaluation model, an evaluation method and an apparatus for user peak-shaving behavior disclosed in the present invention have the following beneficial effects: the present invention can be applied in the field of demand-side response of power systems, so that the dispatching agency can accurately estimate the peak-shaving behavior of the user side in a data-driven manner without understanding the analytical model of the user load, and evaluate the user's subsequent peak-shaving behavior according to the user's Markov state at different times, thereby providing an accurate estimation method for the user side to participate in power auxiliary services, which has application value in alleviating supply and demand balance and promoting the consumption of new energy. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 It is a flow chart of a method for establishing an evaluation model of user peak load shaving behavior according to the present invention;
[0070] Figure 2 It is a structural schematic diagram of a device for evaluating household peak load shaving behavior according to the present invention. DETAILED DESCRIPTION
[0071] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0072] Embodiment 1:
[0073] Refer to the attached Figure 1 The present invention discloses a method for establishing an evaluation model of user peak load behavior, which mainly includes the following steps:
[0074] Step S1, obtaining user data, wherein the user data includes: actual load data, baseline load data and peak load price time-sharing data.
[0075] Step S2, constructing a triple Markov decision process of the user's peak-shaving behavior based on the user data, and obtaining a first example trajectory of the user's peak-shaving behavior based on the triple Markov decision process; the triple Markov decision process includes: the Markov state of the user in time period t, the user's peak-shaving behavior and the state transferred to time period t+1.
[0076] Step S3, obtaining multiple effective feature functions according to the user's peak-shaving behavior, reverse learning the first example trajectory to obtain weight parameters corresponding to the multiple effective feature functions, and constructing a reward function according to the multiple effective feature functions and the weight parameters corresponding to the multiple effective feature functions, wherein the reward function is a linear mapping of the multiple effective feature functions.
[0077] Step S4, obtaining a quaternary Markov decision process according to the reward function and the ternary Markov decision process, and obtaining a second example trajectory of the user's peak-shaving behavior according to the quaternary Markov decision process.
[0078] Step S5: learning the second example trajectory to obtain an evaluation model of the user's peak load shaving behavior.
[0079] In step S1 and step S2, the user data obtained and used are the same. Suppose the actual load of user i in time period t is The baseline load is The peak-shaving price time-sharing data is The peak load is in,
[0080] In this embodiment, A positive value represents valley-filling load, and a negative value represents peak-cutting load. Assume that the peak-shaving price time-sharing data is That is, the compensation price of the peak load ancillary service at time t. When users are required to participate in peak load regulation, they adjust their own electricity consumption to fill valleys or cut peaks. This method can be applied to the estimation of user peak cutting behavior and user valley filling behavior.
[0081] In this embodiment, the baseline load is measured using historical data, and generally, the historical average load of several days of the same type is selected as the baseline load.
[0082] Those skilled in the art know that the Markov decision process is generally in the form of a four-tuple: d t = t ,a t ,r t ,s′ t >, where st represents the Markov state of user t period, a t represents the user peak load behavior, r t represents the reward function, s′ t Represents the state of the next time period. Among them, the user's reward function represents the rewards and penalties of the user's peak-shaving behavior, such as the compensation benefits obtained by the user from the peak-shaving, the adverse effects of the user's peak-shaving behavior on his own production and life, etc.
[0083] However, if the user-side resources are not fully modeled, the reward function cannot be directly obtained from the user's load data. If the internal model of the user side can be established, the objective function and constraints of the user's peak load behavior can be known. However, in reality, these user internal model architectures and model parameters are difficult to obtain. Therefore, it is necessary to improve the method of obtaining the reward function and obtain the reward function from other channels.
[0084] In order to obtain the reward function, the present application first establishes a triple Markov decision process. In step S2, the triple Markov decision process is constructed according to the user data, and the first example trajectory of the user's peak-shaving behavior is obtained according to the triple Markov decision process, which is specifically:
[0085] Assume that the actual load of user i in period t is The baseline load is The peak-shaving price time-sharing data is The peak load is in,
[0086] Constructing a triplet Markov decision process for user peak-shaving behavior t = t ,a t ,s′ t >, where s t represents the Markov state of the user in period t, a t represents the user's peak load behavior, s′ t Represents the state of the next period of time transfer; wherein, the s t Including the peak price of the current period Baseline load of users Peak load behavior of users in the past 24 hours make And assume that the user's state space is S i ,Right now The peak load regulation behavior of users is specifically peak load regulation, that is, and A i is the action space of user i’s peak load behavior.
[0087] In this embodiment, the peak-shaving behavior of the user in the past 24 hours is selected as the peak-shaving behavior related to the current Markov state. The selection can be made according to actual needs without affecting the Markov property of the user's peak-shaving behavior.
[0088] According to the triple Markov decision process, the first example trajectory of the peak-shaving behavior of user i is obtained, and the first example trajectory is recorded as τ m , the τ m The specific representation is as follows:
[0089]
[0090] Wherein, m=1, 2, ..., M, m represents the number of trajectories, and T represents the total duration of the trajectory.
[0091] In step S3, a plurality of effective characteristic functions are obtained according to the user's peak load shaving behavior, specifically:
[0092] An effective characteristic function is constructed based on the peak-shaving behavior that the user has performed and the peak-shaving benefits of the peak-shaving behavior. The effective characteristic function includes: the peak-shaving benefits of the user in the current period Maximum user adjustable peak capacity The peak mileage that the user has completed in the past 24 hours Duration of effective peak load regulation with users
[0093] The specific expression of the effective characteristic function is as follows:
[0094]
[0095]
[0096]
[0097]
[0098] In this embodiment, it is set that the user's peak load shaving (valley filling) behavior must exceed 10% of the baseline load to be effective, otherwise it is regarded as an invalid response, that is, the user does not actually perform the peak load shaving behavior.
[0099] In this embodiment, different effective characteristic functions represent the internal mechanisms that users consider when making peak load decision, such as Consider the economic benefits of peak load shaving on behalf of users, Etc. represent the impact of the user's current or past peak load regulation behavior on their electricity consumption behavior.
[0100] In this embodiment, the weight parameters of the effective feature function are calculated using the maximum entropy-based inverse reinforcement learning method: The maximum entropy principle means that when predicting the probability distribution of a random event, the prediction should meet all known conditions and no subjective assumptions can be made about unknown situations. In this case, the probability distribution is the most uniform and the prediction risk is the smallest, because the information entropy of the probability is the largest at this time, so it is called the maximum entropy model.
[0101] In step S3, the first example trajectory is reversely learned to obtain weight parameters corresponding to multiple valid feature functions, and a reward function is constructed according to the multiple valid feature functions and the weight parameters corresponding to the multiple valid feature functions. The reward function is a linear mapping of the multiple valid feature functions, specifically:
[0102] Said The weight parameter is θ1, The weight parameter is θ2, The weight parameter is θ3, The weight parameter is θ4;
[0103] According to the linear mapping of the reward function to the effective feature function, the reward function can be obtained, which is recorded as Said Specifically:
[0104]
[0105] in, represents the effective feature function, θ=[θ1,θ2,θ3,θ4] represents the weight parameter of each effective feature function, and θ is unknown for different users.
[0106] Define the feature expectation f(π) as the effective feature function on a first example trajectory τ m The cumulative expectation in is as follows:
[0107]
[0108] Among them, π represents the strategy selected by the user, and γ represents the attenuation coefficient of returns in future periods.
[0109] Based on the first example trajectories of the peak-shaving behaviors of M users, the experience value of the feature expectation is obtained, which is specifically:
[0110]
[0111] in, Represents the feature expectation of the first example trajectory of the peak behavior of the mth user;
[0112] If and only if the weight parameter of the reward function is θ, and the total value of the feature expectation obtained by the reinforcement learning training based on the multiple first example trajectories is equal to the empirical value of the feature expectation of the multiple first example trajectory data, it is determined that the trained simulation model has the same decision preference as the user, and the specific form is as follows:
[0113]
[0114] Among them, Pr(τ m ) represents each behavior trajectory τ m probability;
[0115] When the simulation model has the same θ, the probability of trajectory occurrence is exponentially related to the value of the reward function, as shown below:
[0116]
[0117] Among them, s j represents the trajectory τ belonging to the first example m A state in , Z(θ) is the partition function that normalizes the total probability, and the expression is as follows:
[0118]
[0119] The optimization model based on maximum entropy can be expressed as:
[0120]
[0121]
[0122] Expressing it as Lagrangian relaxation yields:
[0123]
[0124] Among them, λ1 and λ0 are Lagrangian relaxation conditions, which can be obtained by subgradient method. Solve and obtain θ.
[0125] In this embodiment, the reward function can be obtained according to the solved θ and the effective characteristic function.
[0126] In step S4, the Markov decision process of the quaternary is obtained according to the reward function and the Markov decision process of the ternary, and the second example trajectory of the user's peak load behavior is obtained according to the Markov decision process of the quaternary, which is specifically:
[0127] Add the reward function to the three-tuple Markov decision process to obtain the four-tuple Markov decision process d t = t ,a t ,r t ,s′ t >
[0128] According to the four-tuple Markov decision process, the second example trajectory of the peak-shaving behavior of user i is obtained, and the second example trajectory is recorded as τ′ m , the τ′ m The specific representation is as follows:
[0129]
[0130] In step S5, the second example trajectory is learned to obtain an evaluation model of the user's peak load behavior, specifically:
[0131] Construct an agent to learn the second example trajectory. The goal of the agent learning is to By optimizing To maximize the expected return, the expected return is
[0132] Define the following Q function:
[0133]
[0134] Among them, Q π (s,a) represents the expected return of Markov state s and peak-shaving behavior a under a given strategy π; the specific connotation of strategy π is that when the input is Markov state s, the output is the probability distribution π(s,a) of a peak-shaving behavior a;
[0135] The optimal strategy is to make Q π The (s,a) function satisfies the following Bellman equation:
[0136]
[0137] The Q function is trained according to the second example trajectory to obtain an evaluation model of the user's peak shaving behavior.
[0138] In this embodiment, the intelligent agent includes: Q learning, Sarsa learning and deep Q network.
[0139] In this embodiment, it should be noted that r must be provided for training the Q function, otherwise the algorithm is difficult to converge, and it is difficult to obtain r directly from the user's peak-shaving behavior data. This is also the significance of the present invention using inverse reinforcement learning to obtain r.
[0140] The following is a method for training a user peak shaving evaluation model using a deep Q network method.
[0141] First, we build a deep neural network model of the Q function. Assuming that the dimension of s is u, a is a discrete variable, and a has v possible values, and assuming that the depth of the model is 3 layers, the Q function represented by the deep neural network model is as follows:
[0142] Q β (s,a)=oneshot(a)*(A3*σ(A2*σ(A1*s+b1)+b2)+b3);
[0143] Among them, β represents all the parameters to be trained, such as A1, A2, A3, b1, b2 and b3. A1, A2 and A3 are u×v-dimensional parameter matrices, b1, b2 and b3 are v×1-dimensional parameter vectors, and σ represents the activation function. ReLU() is generally used as the activation function to add nonlinear factors to the neural network. The oneshot(a) function converts a into a 1×v-dimensional vector of the form [0 0 1 … 0], where the kth element is 1, indicating that the value of a is the kth possible value, and the other elements are 0.
[0144] Two Q networks with the same structure are established, represented by Q and Q' respectively, and are called the predicted Q network and the target Q network respectively. The structures of the two networks are exactly the same, and the corresponding weight coefficients are β and β' respectively.
[0145] Next, we will train the parameters of the Q network. Assume that all data constitute a historical transaction data set. The loss function of the parameter training model is as follows:
[0146]
[0147] Among them, y j Represents the target value of the Q network function, and the expression is as follows:
[0148]
[0149] Among them, γ represents the discount rate, and generally γ can be taken as 0.95.
[0150] In the kth training, a batch containing u samples (s j ,a j ,s′ j ,r j ), j = 1, 2, ..., u, and the loss function Loss(β) can be calculated by substituting the sample into the above formula. It can be seen that the loss function contains the four-tuple data of each sample.
[0151] The parameters of the prediction Q network are updated using the gradient descent method as follows:
[0152]
[0153] Among them, α represents the learning rate of parameter training, which is generally 0.01. After a certain number of training times, the parameters of the predicted Q network are copied to the target Q network, as follows:
[0154] β′←β;
[0155] After several trainings, the predicted Q network can approximate the real Q function and be used to simulate the user's decision. When the user makes a decision, he chooses the value a that maximizes the Q function based on the given s.
[0156] Embodiment 2:
[0157] On the basis of implementation 1, the present invention further discloses a method for evaluating user peak load shaving behavior, wherein the evaluation model established by the evaluation model establishment method of implementation 1 is applied to evaluate the user's peak load shaving behavior, and the evaluation method comprises:
[0158] Get the user's current Markov state.
[0159] The user's current Markov state is input into the preset evaluation model to obtain the user's peak load behavior.
[0160] Embodiment 3:
[0161] On the basis of Example 2, the present invention further discloses a device for evaluating household peak load shaving behavior, including: an acquisition module 101 and an evaluation module 102 .
[0162] The acquisition module 101 is used to acquire the current Markov state of the user.
[0163] The evaluation module 102 is used to input the current Markov state of the user into a preset evaluation model to obtain the peak load shaving behavior of the user.
[0164] In summary, the embodiments of the present invention provide a method for establishing an evaluation model, an evaluation method and a device for user peak-shaving behavior, and the beneficial effects are: the present invention can be applied in the field of demand-side response of power systems, so that the dispatching agency can accurately estimate the peak-shaving behavior of the user side in a data-driven manner without understanding the analytical model of the user load, and evaluate the user's subsequent peak-shaving behavior according to the user's Markov state at different times, thereby providing an accurate estimation method for the user side to participate in power auxiliary services, which has application value in alleviating supply and demand balance and promoting the consumption of new energy.
[0165] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.
Claims
1. A method for establishing an evaluation model for user peak load shaving behavior, characterized in that: include: Acquire user data, the user data including: actual load data, baseline load data and peak load price time-sharing data; A triple Markov decision process of the user's peak-shaving behavior is constructed according to the user data, and a first example trajectory of the user's peak-shaving behavior is obtained according to the triple Markov decision process; the triple Markov decision process includes: the Markov state of the user in time period t, the user's peak-shaving behavior, and the state of the transfer in time period t+1; Acquire multiple effective feature functions according to the user's peak shaving behavior, perform reverse learning on the first example trajectory to acquire weight parameters corresponding to the multiple effective feature functions, and construct a reward function according to the multiple effective feature functions and the weight parameters corresponding to the multiple effective feature functions, wherein the reward function is a linear mapping of the multiple effective feature functions; The method of obtaining multiple effective characteristic functions according to the user peak load shaving behavior is specifically as follows: An effective characteristic function is constructed based on the peak-shaving behavior that the user has performed and the peak-shaving benefits of the peak-shaving behavior. The effective characteristic function includes: the peak-shaving benefits of the user in the current period Maximum user adjustable peak capacity The peak mileage that the user has completed in the past 24 hours Duration of effective peak load regulation with users The specific expression of the effective characteristic function is as follows: in, It is the time-sharing data of peak-shaving price; is the actual load of user i in period t; is the baseline load of user i in period t; i represents the user number; The reverse learning of the first example trajectory obtains weight parameters corresponding to multiple valid feature functions, and constructs a reward function according to the multiple valid feature functions and the weight parameters corresponding to the multiple valid feature functions. The reward function is a linear mapping of the multiple valid feature functions, specifically: Assume that The weight parameter is θ1, The weight parameter is θ2, The weight parameter is θ3, The weight parameter is θ4; According to the linear mapping of the reward function to the effective feature function, the reward function can be obtained, which is recorded as Said Specifically: in, represents the effective feature function, θ=[θ1,θ2,θ3,θ4] represents the weight parameter of each effective feature function; is the Markov state of user i in period t; is the peak load-shaving behavior of user i in period t; Define the feature expectation f(π) as the effective feature function on a first example trajectory τ m The cumulative expectation in is as follows: Among them, π represents the strategy selected by the user, and γ represents the attenuation coefficient of future period returns; Based on the first example trajectories of the peak-shaving behaviors of M users, the experience value of the feature expectation is obtained, which is specifically: in, Represents the feature expectation of the first example trajectory of the peak shaving behavior of the mth user; When the weight parameter of the reward function is θ, and the total value of the feature expectation obtained by the reinforcement learning training of multiple first example trajectories is equal to the empirical value of the feature expectation of multiple first example trajectory data, it is determined that the trained simulation model has the same decision preference as the user, and the specific form is as follows: Among them, Pr(τ m ) represents each behavior trajectory τ m probability; When the weight parameter of the reward function is θ, the probability of trajectory occurrence is exponentially related to the value of the reward function, as shown below: Among them, s j represents the trajectory τ belonging to the first example m A state in , Z(θ) is the partition function that normalizes the total probability, and the expression is as follows: The optimization model based on maximum entropy can be expressed as: Expressing it as Lagrangian relaxation yields: Among them, λ1 and λ0 are Lagrangian relaxation conditions, which can be obtained by subgradient method. Solve and obtain θ; A quaternary Markov decision process is obtained according to the reward function and the ternary Markov decision process, and a second example trajectory of the user's peak load-shaving behavior is obtained according to the quaternary Markov decision process; The second example trajectory is learned to obtain an evaluation model of the user's peak load shaving behavior.
2. The method for establishing an evaluation model for user peak load shaving behavior according to claim 1, characterized in that: The method of constructing a triplet Markov decision process according to the user data and obtaining a first example trajectory of the user's peak load shaving behavior according to the triplet Markov decision process is specifically as follows: Assume that the actual load of user i in period t is The baseline load is The peak-shaving price time-sharing data is The peak load is in, Constructing a triplet Markov decision process for user peak-shaving behavior t = t ,a t ,s t ′ >, where s t represents the Markov state of the user in period t, a t represents the user's peak load behavior, s t ′ represents the state of the transfer in the t+1 period; wherein, the s t Including the peak price of the current period Baseline load of users Peak load behavior of users in the past 24 hours make And assume that the user's state space is S i ,Right now The peak load regulation behavior of users is specifically peak load regulation, that is, and A i is the action space of user i’s peak load behavior; According to the triple Markov decision process, the first example trajectory of the peak-shaving behavior of user i is obtained, and the first example trajectory is recorded as τ m , the τ m The specific representation is as follows: Wherein, m=1,2,…,M, m represents the number of trajectories, and T represents the total duration of the trajectory.
3. The method for establishing an evaluation model for user peak load shaving behavior according to claim 1, characterized in that: The Markov decision process of the four-tuple is obtained according to the reward function and the Markov decision process of the three-tuple, and the second example trajectory of the user's peak-shaving behavior is obtained according to the four-tuple Markov decision process, which is specifically: Add the reward function to the three-tuple Markov decision process to obtain the four-tuple Markov decision process d t = t ,a t ,r t ,s t ′ > According to the four-tuple Markov decision process, the second example trajectory of the peak-shaving behavior of user i is obtained, and the second example trajectory is recorded as τ ′ m , the τ ′ m The specific representation is as follows:
4. The method for establishing an evaluation model for user peak load behavior according to claim 1, characterized in that: The evaluation model of the user peak load behavior is obtained by learning the second example trajectory, which is: The intelligent agent is constructed to learn the second example trajectory. The goal of the intelligent agent learning is to By optimizing To maximize the expected return, the expected return is Define the following Q function: Among them, Q π (s,a) represents the expected return of Markov state s and peak-shaving behavior a under a given strategy π; the specific connotation of strategy π is that when the input is Markov state s, the output is the probability distribution π(s,a) of a peak-shaving behavior a; The optimal strategy is to make Q π The (s,a) function satisfies the following Bellman equation: The Q function is trained according to the second example trajectory to obtain an evaluation model of the user's peak shaving behavior.
5. The method for establishing a user peak load behavior evaluation model according to claim 4, characterized in that: The intelligent agents include: Q learning, Sarsa learning and deep Q network.
6. A method for evaluating user peak load shaving behavior, characterized in that: The evaluation model established by the evaluation model establishment method described in any one of claims 1 to 5 is used to evaluate the peak load shaving behavior of the user, wherein the evaluation method comprises: Get the user's current Markov state; The user's current Markov state is input into the preset evaluation model to obtain the user's peak load behavior.
7. A device for evaluating user peak load shaving behavior, characterized in that: include: Acquisition modules and assessment modules; The acquisition module is used to obtain the current Markov state of the user; The evaluation module is used to input the user's current Markov state into the evaluation model established by applying the evaluation model establishment method described in any one of claims 1 to 5 to obtain the user's peak shaving behavior.
