Energy management method and system based on Markov decision process

Abstract

The present invention discloses an energy management method and system based on a Markov decision process. The system models a smart home and establishes a unique cost calculation system. At the same time, a measure of the degree of privacy protection in the system is defined, a rechargeable battery is introduced for staggered charging and discharging, and the user's real-time electricity consumption information is concealed. An optimization problem that weighs privacy protection and cost savings is constructed using the Markov decision process, and the Sarsa algorithm is used to solve the problem. Ultimately, the most appropriate energy management strategy can be found according to the user's personalized needs, while minimizing system costs while improving privacy security, and achieving optimal resource allocation. The present invention designs an energy management system and energy management method that take into account both privacy protection and cost savings, which can satisfy users' personalized privacy protection while minimizing system costs.

Description

Technical Field

[0001] The present invention mainly relates to the field of smart meters, and specifically to an energy management method and system based on a Markov decision process, which achieves both privacy protection of smart meter data and system cost savings. Background Art

[0002] With the transition from traditional power grids to smart grids, smart grids are becoming increasingly popular. As data collection and transmission devices, smart meters are an indispensable part of the development of modern smart grids: they provide energy suppliers with real-time information on users' electricity usage with outstanding reliability, anti-interference capabilities, and high cost-effectiveness. The collected data can be used for dynamic pricing and more accurate load forecasts. However, this data contains a large amount of personal information, and this sensitive information is likely to pose a threat to consumer privacy. For example, non-invasive load monitoring methods can easily obtain consumers' real-time electricity usage, thereby inferring their living habits or current living conditions. These legitimate or illegitimate data recipients may exploit or leak this data, thereby affecting users' lives.

[0003] To this end, scholars both domestically and internationally have proposed various methods to protect smart meter data. One approach involves modifying the original smart meter information through methods such as obfuscation, anonymization, and homomorphic encryption to achieve privacy protection. However, encryption reduces data accuracy. Differential privacy is another common method for preventing privacy leaks. Differential privacy algorithms typically use Laplace or exponential mechanisms to introduce noise into query results, ensuring that the required statistical information is not significantly altered by individual deviations in the entry database while simplifying mathematical modeling and analysis.

[0004] Adding noise to the data reduces data accuracy and increases system costs to a certain extent. To this end, we introduced rechargeable batteries, a technology that is becoming increasingly mature, to compensate for this shortcoming and protect privacy at a physical level. Numerous studies have previously demonstrated the advantages and feasibility of rechargeable batteries in this regard. By utilizing rechargeable batteries, the actual duration and specific time periods of household electricity usage can be effectively masked, while actual device energy consumption can be obscured by the discrepancy between energy supply and user demand. Furthermore, considering the peak and valley periods of electricity prices, rechargeable batteries can also be charged and discharged according to user load demand and real-time electricity prices, reducing the system's electricity costs.

[0005] As privacy protection increases, system costs also increase. Finding a balance between the two, minimizing system costs while ensuring privacy, and designing an optimal energy allocation solution for users, is an important and challenging issue. Summary of the Invention

[0006] To address the deficiencies in the existing technology, the present invention proposes an energy management method and system based on the Markov decision process. The purpose is to use the Markov decision process to construct an optimization problem and apply the Sarsa algorithm to solve the proposed problem. Ultimately, a unique optimal energy management strategy is found based on the personalized needs of different users to resolve the trade-off between privacy and cost.

[0007] The technical solutions adopted in the present invention are as follows:

[0008] An energy management system based on Markov decision process, including a Markov decision module, an energy management module, a cost calculation module and a privacy protection module;

[0009] The energy management module includes an energy demand side and an energy management unit. The energy demand side includes electrical appliances and rechargeable batteries. The energy supplier connects to the energy demand side through a smart meter to provide electricity to the household. The energy management unit is connected to the electrical appliances and rechargeable batteries through signal lines. The energy management unit monitors electricity prices and total user demand in real time and controls the charge and discharge time of the rechargeable battery.

[0010] The privacy protection module divides the data collected by the smart meter within a time period into several groups and uses a differential privacy algorithm to perform noise processing on the data groups; the privacy protection level is defined in the privacy protection module, and different privacy protection levels are quantitatively analyzed;

[0011] The cost calculation module is connected to the energy management module and the privacy protection module through signals. The cost calculation module combines the real-time electricity price, battery usage and privacy budget to calculate the system cost of the energy management system;

[0012] The Markov decision module is connected to the cost calculation module and the energy management module through signals, and the system cost of the energy management system and the degree of privacy protection are input into the Markov decision model as parameters, and the optimal degree of privacy protection and system cost of the energy management system are output and input into the energy management module.

[0013] An energy management method based on a Markov decision process comprises the following steps:

[0014] Step 1: Based on the user's personalized privacy protection requirements, select the privacy protection level K of the differential privacy algorithm;

[0015] Step 2: Using one day as the basic unit, traverse each possible protection level k, k≤K; at fixed time intervals, during the process of transmitting data from the smart meter to the energy supplier, use the differential privacy algorithm to encrypt the original data;

[0016] Step 3: Calculate the privacy protection level and system cost of the energy management system;

[0017] Step 4: Input the system cost and privacy protection level into the Markov decision model of this module to construct an optimization problem, and use the Sarsa algorithm to solve the optimization problem and perform the Markov decision process to obtain the complete value table Q;

[0018] Step 5: Traverse the entire value table Q and find the privacy protection level k that best suits the user * and the corresponding cost of that level of protection And feedback to the user; if no privacy protection level k that meets the conditions is found * and the corresponding cost of that level of protection Then return to step 3.

[0019] Furthermore, in step 2, a differential privacy algorithm is used to identify the data transmitted by the smart meter to the energy supplier at a fixed time interval of 6 (11-k). The original data is encrypted;

[0020] Furthermore, the method for calculating the privacy protection level of the energy management system in step 3 is:

[0021]

[0022] Among them, P sm (t) and P′ sm (t) represents the original data collected by the smart meter and the encrypted data, respectively, and k represents the number of times the privacy protection is performed on the original data set.

[0023] Furthermore, the system cost of the energy management system in step 3 includes the operating cost of the rechargeable battery C b , electricity price cost C e and the privacy budget C p , so the system cost is expressed as: C π =C e +C b +C p .

[0024] Further, calculate the operating cost of the rechargeable battery:

[0025] The amortized cost q of the battery is defined by dividing the cost of the rechargeable battery by the estimated number of times the battery can be charged and discharged:

[0026]

[0027] Define s(t) to represent the working state of the rechargeable battery in the time period Δt. If the rechargeable battery is charging or discharging, s(t) = 1; if the rechargeable battery is not in use, s(t) = 0. Thus, the operating cost of the rechargeable battery is obtained:

[0028]

[0029] Where k is the privacy protection level, that is, within a fixed time period T, it is divided into k time periods Δt according to the current privacy protection level k, k = 1, 2, ..., K; then Add up the battery cost in each time period to get C b .

[0030] Further, calculate the electricity price cost:

[0031]

[0032] Monitor the real-time electricity price V through the control terminal t , according to the meter reading P sm (t) can directly calculate the user's electricity price cost in the time period Δt, and add the real-time electricity prices in each time period to get C e .

[0033] Furthermore, the privacy budget under different user privacy protection levels k is expressed as:

[0034] C p =k∈

[0035] Where ∈ is the privacy budget.

[0036] Further, the process of step 4 is as follows:

[0037] Step 4.1: Initialize the state set S(D,d,y,P sm ,Cπ), where D is the privacy protection degree, d is the user's privacy protection requirement, is an element of the privacy protection degree set under all levels, y is a binary parameter, y = 0 means that the current privacy protection degree D does not meet the privacy protection requirements, y = 1 means that the current privacy protection degree D meets the privacy protection requirements; P sm is the original data collected by the smart meter, C π Represents the system cost in the current state;

[0038] Step 4.2: Based on the ξ-greedy algorithm, a possible action a is selected in the current state, and the next state s′ is updated according to the meter data at the next moment. At the same time, an immediate reward r is obtained and the next action a′ is determined;

[0039] Step 4.3: Update the value Q(s, a) of the previous moment based on the action and state pair Q(s′, a′) of the next moment, and repeat this cycle until all current actions at this level are completed;

[0040] Step 4.4: Repeat the above process to obtain the value of all state-action pairs under protection level k;

[0041] Step 4.5: Consider all possible k that meet the user's needs, traverse each protection level k, and obtain a complete value table Q.

[0042] Furthermore, based on the system cost and privacy protection level, the following reward function is designed, and the reward r is determined according to the reward function:

[0043]

[0044] Among them, R is the reward set, r - Is a negative constant indicating that when the system cost exceeds the cost budget When , negative immediate rewards have a significant impact on the current value. The coefficients β and δ are the weights of privacy protection and system cost respectively. When the system cost is For the current state, there are two possible rewards: If the current privacy protection degree D does not meet the privacy protection requirements, that is, y = 0, the reward is -βD-δC π ; The current privacy protection level D meets the privacy protection requirements, that is, y = 1, and the reward is r + -δC π , r + is a positive constant.

[0045] Beneficial effects of the present invention:

[0046] The functions and working principles of each part of this system are as follows:

[0047] 1. Model a smart home and introduce rechargeable batteries. Split the electricity provided by the energy supplier into two parts: supplying the battery and powering the devices. Under the control of the energy management unit, the battery is charged and discharged at off-peak times to mask the user's original load curve.

[0048] 2. Based on the original data in the smart meter and the personalized requirements of the user, a differential privacy algorithm is used to add noise to the data during data transmission. The degree of privacy protection is classified according to the amount of noisy data. Based on this, a precision measurement standard for data protection is defined, and different degrees of privacy protection are quantitatively analyzed and introduced into the Markov decision model.

[0049] 3. Considering the case of rechargeable batteries, a unique cost calculation system is established in the model based on different levels of privacy protection budget and real-time electricity prices. The specific cost after using the system is calculated and brought into the Markov decision model as a parameter.

[0050] 4. This method uses the Markov decision process to transform the trade-off between privacy and cost into an optimization problem, and uses the Sarsa algorithm to solve and find the optimal strategy. This method can measure the level of privacy protection and system cost, and can also formulate personalized requirements according to user needs, find the most appropriate energy management strategy, and optimally balance system cost and data privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a model diagram of an energy management system designed by the present invention that takes into account both privacy protection and cost savings.

[0052] Figure 2 This is a flow chart of an energy management system designed by the present invention that takes into account both privacy protection and cost savings.

[0053] Figure 3 This is a comparison chart of the original data and the privacy-protected data when k=1, k=3, and k=10. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] like Figure 1 As shown, this application proposes an energy management system based on a Markov decision process, which includes a Markov decision module, an energy management module, a cost calculation module, and a privacy protection module. In this energy management system, both the energy management module and the privacy protection module play a role in protecting the user's electricity usage data, as follows:

[0056] The energy management module consists of an energy demand side and an energy management unit. The energy demand side includes electrical appliances and rechargeable batteries. Energy suppliers connect to the energy demand side through smart meters. A portion of the electricity provided by the energy supplier to households directly powers electrical appliances, while the remainder is stored in rechargeable batteries. Signal lines connect the energy management unit to the electrical appliances, rechargeable batteries, and smart meters. The energy management unit monitors electricity prices and total user demand in real time, enabling the rechargeable battery's charge and discharge times to be appropriately controlled. The energy management module uses the rechargeable battery's charge and discharge status at any given moment to mask a user's actual electricity usage, thus providing a form of physical privacy protection.

[0057] The privacy protection module divides the data collected by smart meters over a period of time into several groups and applies a differential privacy algorithm to the data groups, adding noise to prevent attackers from stealing important information from each group during transmission. The privacy protection module defines privacy protection levels and quantitatively analyzes different levels of privacy protection.

[0058] The cost calculation module is connected to the energy management module and the privacy protection module through signals. The cost calculation module combines real-time electricity prices, battery usage and privacy budget to calculate the system cost of the energy management system.

[0059] As users' demands for privacy protection increase, system costs also increase. Therefore, this application designs a Markov decision module to balance privacy and cost, finding an optimal energy management strategy for users. The Markov decision module is connected to the cost calculation module and the energy management module via signals. The system cost of the energy management system and the degree of privacy protection are input as parameters into the Markov decision model to construct an optimization problem. The Sarsa algorithm is used to solve the optimization problem, outputting the optimal degree of privacy protection and the system cost of the energy management system, which are then input into the energy management module.

[0060] Based on the energy management system based on the Markov decision process designed in this application, this application also proposes an energy management method based on the Markov decision process, which includes the following steps:

[0061] Step 1: Based on the user's personalized privacy protection requirements, select the privacy protection level K of the differential privacy algorithm;

[0062] Step 2: Take one day as the basic unit and traverse each possible protection level k, k≤K; take 6(11-k) as the fixed time interval, and use the differential privacy algorithm to identify the data in the process of smart meter transmitting data to energy supplier. The original data is encrypted;

[0063] Step 3: Calculate the privacy protection level and system cost of the energy management system;

[0064] Step 3.1, privacy protection level:

[0065] The privacy protection module specifically uses the differential privacy algorithm of the Laplace mechanism to add noise points to different numbers of data in the original data set according to the user's personalized privacy needs.

[0066] According to the definition of differential privacy, for any two adjacent data sets X, X′, the result S of each query and the random mechanism M satisfy:

[0067] Pr[M(X)∈S]≤exp(∈)×Pr[M(X′)∈S];

[0068] Where Pr[.] is the probability of an event occurring, and ∈ is the privacy budget. By introducing the parameter ∈, we can control the degree of privacy protection of the dataset before and after modification. As the privacy budget decreases, the degree of data protection increases.

[0069] For any query f:X→R, the Laplace mechanism M has:

[0070] M(X)=f(X)+Y;

[0071] Where Y is random noise, Y~Lap(Δf / ∈), Δf is the sensitivity of the query result f. The sensitivity of the function describes the possible changes brought about by a single individual in the worst case. The definition of Δf is:

[0072]

[0073] In the privacy protection module, in order to accurately measure the degree of privacy protection, the concept of statistical distance is referred to to define the accuracy of privacy in the system:

[0074]

[0075] Among them, P sm (t) and P′ sm (t) represents the original data and modified data collected by the smart meter, respectively. k represents the number of times the privacy protection is performed on the original data set. The larger k is, the better the protection of user data is.

[0076] Step 3.2, System Cost

[0077] The cost of the system consists of three parts: the operating cost of the rechargeable battery, the electricity cost of electrical appliances under real-time electricity prices, and a personalized privacy budget.

[0078] (1) The operating cost of the rechargeable battery is defined as the amortized cost q of the battery divided by the estimated number of charge and discharge cycles of the battery:

[0079]

[0080] Define s(t) to represent the operating state of the rechargeable battery during the time period Δt. If the rechargeable battery is charging or discharging, then s(t) = 1; if the rechargeable battery is not in use, then s(t) = 0. From this, the operating cost of the battery is obtained:

[0081]

[0082] (2) Electricity price cost, monitor the real-time electricity price V through the control terminal t , according to the meter reading P sm (t) can directly calculate the user's electricity price cost in the time period Δt:

[0083]

[0084] (3) Privacy budget: Use differential privacy algorithm to protect user data and determine the degree of personalized privacy protection according to the needs of different users. According to the definition of differential privacy, the privacy budget of each data with noise is ∈, and the privacy budget C under different user privacy protection levels k is p The calculation formula is as follows:

[0085] C p =k∈;

[0086] Finally, the sum of each Δt segment is accumulated, and the personalized cost system in the system is summarized as follows:

[0087]

[0088] Step 4: Input the system cost and privacy accuracy D into the Markov decision model of this module, perform the Markov decision process, and obtain the complete value table Q; the process is as follows:

[0089] Step 4.1: Define the triplet M = (S, A, R), where S represents the state set, A represents the action set, and R represents the reward set. The details are as follows:

[0090] 1) State set: Use S(d,y,P sm ,C π ,D) is used to represent the state space, which defines all possible states. In the state set, d is the user's privacy protection requirement, which is an element of the privacy protection level set under all levels. y is a binary parameter used to reflect whether the current accuracy D meets the user's expected needs. π Indicates the system cost in the current state, P sm For the data collected by smart meters, D represents the set of all desirable accuracies;

[0091] 2) Action Set: The action set A consists of all possible actions after the state set S. The specific actions in this system involve modifying the original smart meter data using a differential privacy algorithm. Different degrees of modification may result in different levels of system cost and data accuracy. Based on the user's personalized privacy protection requirements, the defined action space contains at most K choices, which means that at most K elements in the original dataset may be modified.

[0092] 3) Reward Set: Based on the designed personalized system cost and privacy accuracy, the reward function is expressed as follows:

[0093]

[0094] Among them, r - is a constant representing a large negative value, and the coefficients β and δ are the weights of data accuracy and system cost respectively. π and D is the system cost and privacy accuracy in the current state. When the system cost exceeds the cost budget When the system cost is For the current state, there are two possible rewards: If the current accuracy D does not meet the privacy protection requirements, that is, y = 0, the reward is -βD-δC π ; The current accuracy D meets the privacy protection requirements, that is, y = 1, and the reward is r + -δC π , r + is a positive constant.

[0095] Step 4.2: Based on the ξ-greedy algorithm, select a possible action a in the current state, and update the next state s′ according to the meter data at the next moment. At the same time, obtain an immediate reward r1 and determine the next action a′.

[0096] Step 4.3: Update the value Q(s, a) of the previous moment according to the action state of the next moment, and repeat this cycle until all current actions at this level are completed.

[0097] Step 4.4: Repeat the above process to obtain the value of all state-action pairs under the protection level k.

[0098] Step 4.5: Consider all possible k that meet the user's needs, traverse each protection level k, and obtain a complete value table Q.

[0099] Step 5: Query the entire value table Q and find the privacy protection level k that best suits the user * and the corresponding cost of that level of protection Feedback to the user; if no privacy protection level k that meets the conditions is found * and the corresponding cost of that level of protection Then it returns to ask the user whether to change the initial conditions and repeats the above process.

[0100] like Figure 2 The following is a further explanation of the specific process of applying the Sarsa algorithm to the Markov model:

[0101]

[0102]

[0103] The details of Sarsa are shown in Algorithm 1. Lines 1-17 describe the learning process of Sarsa. We first learn the data P collected by the smart meter. sm Initialize the state set S, and we loop K times to evaluate the sequence value at each protection level k. In each loop, we traverse all possible action-state pairs to obtain all value sets at that level. Specifically, we first randomly select an element from the set D of all possible privacy accuracy, and according to the definition of the state set and the original data, we can get the first state S1. In each iteration, we select our action based on the ξ-greedy algorithm. Specifically, we randomly select an action with a probability of ξ in the action set. According to the determined state s and action a, we can update the next state s'. At the same time, we can get a timely reward r1, and the next action a' is also determined. After determining the value of the next action-state pair Q(s',a'), we update the value Q(s,a) at the previous moment. Repeat After that, we can get the entire Q table under the current protection level.

[0104] The above table describes the application of the Sarsa algorithm. By searching the entire Q table, we can calculate the optimal privacy protection level k * We then compare the current privacy accuracy with the customer's requirements. If the requirements are met, we output them; if not, we send a request to the user to determine whether to change the requirements.

[0105] Combined with attachment Figure 3 , combined with the comparison diagrams of the original data and the privacy-protected data when k = 1, k = 3 and k = 10, it can be seen that when k = 1, we observe that the difference between the two load curves in Figure (a) is very small, indicating that the degree of privacy protection is very low; when k = 5, it can be seen from Figure (b) that there is a significant difference between the original load curve and the modified curve, and the user's real-time electricity consumption information is masked to a certain extent, which better protects the user's privacy; when k = 10, the encrypted curve in Figure (c) becomes smoother, making it difficult for criminals to steal or use this data, thereby effectively achieving the purpose of protecting user privacy.

[0106] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. An energy management method based on Markov decision process, characterized in that: Based on an energy management system based on Markov decision process, the system includes a Markov decision module, an energy management module, a cost calculation module and a privacy protection module; The energy management module includes an energy demand side and an energy management unit, wherein the energy demand side includes electrical appliances and rechargeable batteries; the energy supplier connects to the energy demand side through a smart meter to provide electricity to the household; the energy management unit is connected to the electrical appliances and the rechargeable battery through a signal line, and the energy management unit monitors the electricity price and the total demand of users in real time, and controls the charging and discharging time of the rechargeable battery; The privacy protection module divides the data collected by the smart meter within a period of time into several groups, and uses a differential privacy algorithm to perform noise processing on the data groups; Define privacy protection levels in the privacy protection module and quantify and analyze different levels of privacy protection; The cost calculation module is connected to the energy management module and the privacy protection module through signals, and the system cost of the energy management system is calculated by combining the real-time electricity price, battery usage and privacy budget in the cost calculation module; The Markov decision module is connected to the cost calculation module and the energy management module through signals, and the system cost of the energy management system and the degree of privacy protection are input into the Markov decision model as parameters, and the optimal degree of privacy protection and system cost of the energy management system are output and input into the energy management module; The method comprises the following steps: Step 1: Based on the user's personalized privacy protection requirements, select the privacy protection level K of the differential privacy algorithm; Step 2: Taking one day as the basic unit, traverse each possible protection level k, k≤K; at fixed time intervals, when the smart meter transmits data to the energy supplier, use the differential privacy algorithm to encrypt the original data; Step 3, calculate the privacy protection degree and system cost of the energy management system; Step 4: Input the system cost and privacy protection level into the Markov decision model of this module to construct an optimization problem, and use the Sarsa algorithm to solve the optimization problem, perform the Markov decision process, and obtain the complete value table Q; Step 5: Traverse the entire value table Q and find the privacy protection level k that best suits the user * and the corresponding cost of that level of protection And feedback to the user; if no privacy protection level k that meets the conditions is found * and the corresponding cost of that level of protection Then return to step 3; In step 2, 6 (11-k) is used as a fixed time interval. During the process of transmitting data from the smart meter to the energy supplier, the differential privacy algorithm is used to The original data is encrypted; The method for calculating the privacy protection degree of the energy management system in step 3 is: Among them, P sm (t) and P′ sm (t) represents the original data collected by the smart meter and the encrypted data, respectively, and k represents the number of times the privacy protection is performed on the original data set; The system cost of the energy management system in step 3 includes the operating cost of the rechargeable battery C b , electricity price cost C e and the privacy budget C p , so the system cost is expressed as: C π =C e +C b +C p ; Calculate rechargeable battery operating costs: The amortized cost of the battery, q, is defined by dividing the cost of the rechargeable battery by the estimated number of times the battery can be charged and discharged: Define s(t) to represent the working state of the rechargeable battery in the time period Δt. If the rechargeable battery is being charged or discharged, s(t) = 1; if the rechargeable battery is not being used, s(t) = 0. Thus, the operating cost of the rechargeable battery is obtained: Where k is the privacy protection level, that is, within a fixed time period T, it is divided into k time periods Δt according to the current privacy protection level k, k = 1, 2, ..., K; then Add up the battery cost in each time period to get C b ; Calculate electricity price cost: Monitor the real-time electricity price V through the control terminal t , according to the meter reading P sm (t) can directly calculate the user's electricity price cost in the time period Δt, and add the real-time electricity price in each time period to get C e ; The privacy budget under different user privacy protection levels k is expressed as: C p =k∈ Among them, ∈ is the privacy budget.

2. The energy management method based on Markov decision process according to claim 1, characterized in that: Step 4 process is as follows: Step 4.1: Initialize the state set S(D,d,y,P) of Markov decision sm ,Cπ), where D is the privacy protection degree, d is the user's privacy protection requirement, is an element of the privacy protection degree set under all levels, y is a binary parameter, y = 0 means that the current privacy protection degree D does not meet the privacy protection requirements, y = 1 means that the current privacy protection degree D meets the privacy protection requirements; P sm is the original data collected by the smart meter, C π Represents the system cost in the current state; Step 4.2: Based on the ξ-greedy algorithm, select a possible action a in the current state, and update the next state s′ according to the meter data at the next moment. At the same time, get an immediate reward r and determine the next action a′; Step 4.3: Update the value Q(s, a) of the previous moment according to the action and state pair Q(s′, a′) of the next moment, and repeat this cycle until all current actions at this level are completed; Step 4.4: Repeat the above process to obtain the value of all state-action pairs under the protection level k; Step 4.5: Consider all possible k that meet user requirements, traverse each protection level k, and obtain a complete value table Q.

3. The energy management method based on Markov decision process according to claim 1, characterized in that: According to the system cost and privacy protection level, the following reward function is designed, and the reward r is confirmed according to the reward function: Among them, R is the reward set, r - Is a negative constant, indicating that when the system cost exceeds the cost budget When , negative immediate rewards have a significant reduction on the current value. The coefficients β and δ are the weights of the degree of privacy protection and system cost respectively. When the system cost is If the current privacy protection level D does not meet the privacy protection requirements, that is, y = 0, the reward is -βD-δC π ; The current privacy protection level D meets the privacy protection requirements, that is, y = 1, and the reward is r + -δC π , r + is a normal number.

Images