Deep reinforcement learning interpretable rule extraction method for community energy management

By adopting a decision tree ensemble model based on separation-aggregation mechanism and a knowledge distillation method of joint interaction and action value evaluation in community energy management, the problem of insufficient interpretability of DRL model in community energy management is solved, and an efficient strategy is achieved Improved extraction and decision performance.

CN119089998AActive Publication Date: 2024-12-06HEBEI UNIV OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411033882.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-12-06
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing technology has insufficient interpretability of deep reinforcement learning (DRL) models in community energy management, resulting in unpredictable and unexplainable decision-making processes, making it difficult to evaluate the safety and rationality of decision-making. Existing methods such as SHAP and strategy imitation methods are There are limitations in improving the interpretability of DRL rules.

Method used

Using a decision tree ensemble model (DADT) based on the separation-aggregation mechanism, combining knowledge distillation methods of joint interaction and action value evaluation, the value of expert strategies is evaluated through RF loss, and the strategy extraction of student models is optimized to ensure the interpretability of decision trees. and fitting effect.

Benefits of technology

It improves the decision-making performance and interpretability of DRL rules, enhances the fitting ability of students' models, ensures the safety and effectiveness of the extracted strategies, and improves generalization and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089998A_ABST
    Figure CN119089998A_ABST
Patent Text Reader

Abstract

The invention relates to a deep reinforcement learning interpretable rule extraction method for community energy management, and the method comprises the following steps: S10, training an expert model, and extracting an expert strategy; s20, constructing a student model DADT explained by a decision tree set based on a separation-aggregation mechanism; s30, extracting an expert strategy through a knowledge distillation method based on joint interaction and action value evaluation, deploying the expert model obtained in the S10 and the student model obtained in the S20 into a control environment at the same time, performing multi-round interaction iteration with the environment, and in the process of interaction iteration with the environment, extracting the expert strategy; evaluating the values of the expert model and the student model in real time by using an action value function Q function, introducing RF loss to realize optimal updating of a student model DADT strategy, and retraining to obtain a new student model DADT; and S40, extracting decision tree rules. The method can improve the decision-making performance of typical rules, and ensures the interpretability of the typical rules.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of interpretable rule extraction, and relates to a deep reinforcement learning interpretable rule extraction method, and in particular to a deep reinforcement learning interpretable rule extraction method for community energy management. Background Art

[0002] In recent years, deep reinforcement learning (DRL) has achieved significant energy-saving and consumption-reducing effects in the field of community energy management with its powerful learning ability. However, DRL is a black box, and its decision-making process is unpredictable and unexplainable, making it difficult to evaluate the rationality and safety of decisions, and there are security risks in practical applications.

[0003] In reality, energy management controllers often use rule-based controllers (RBC), whose control rules are transparent and set according to human expertise and experience, usually in the form of "if-then". However, there is still a lot of room for improvement in the artificially set RBC in terms of tapping the potential of control systems, and the complexity and black-box nature of the DRL model make it difficult to provide effective guidance for RBC strategies. Therefore, improving the interpretability of DRL rules is of great significance to the trusted decision-making and strategy optimization of community energy management.

[0004] Currently, there are few interpretable methods for DRL in the energy field, and existing technologies mostly use the Shapley Adaptive Explanation (SHAP) method or strategy imitation method.

[0005] The existing technology [1] uses the Deep Q-Networks (DQN) algorithm to perform system undervoltage load shedding control for the delayed voltage recovery problem caused by power grid faults, and uses the SHAP method to generate a feature importance summary diagram of the DQN decision process. The SHAP method is a model-independent interpretability technology, so in the field of deep reinforcement learning, a fully trained deep neural network (DNN) can use the SHAP method to obtain the contribution of its input features to the output. However, as a post-explanation method, SHAP only analyzes the feature importance of the DRL strategy, which cannot make the DRL strategy transparent, cannot obtain a certain strategy execution result before executing the strategy, and cannot guarantee the safety and rationality of the strategy. Therefore, the SHAP method cannot be used as a direct method for extracting interpretable rules for DRL in the field of community energy management.

[0006] The idea of ​​the policy imitation method comes from the imitation learning method, and its core is to use a small policy network to fit the fully trained DNN policy. Many existing technologies extract DRL decision strategies into student models based on decision trees (DTs) through policy imitation to achieve an effective method for extracting interpretable rules from DRL.

[0007] The existing technology [2] deploys the Deep Q-learning (DQL) model to the field of energy management, and then uses the traditional imitation learning idea to make the control examples of the expert model (DQL) into a training set to train the student model (DT) to achieve strategy extraction from the expert model. However, the traditional decision tree has poor fitting ability, which is significantly weaker than the DRL model. Therefore, some methods focus on how to improve the fitting ability of the decision tree model.

[0008] The existing technology [3] uses differentiable decision trees (DDT) to extract DQN strategies and realize the extraction of control rules for battery energy storage in home energy management systems. This method uses DDT to fit the execution probability of actions in the action space under specific states in the DQN strategy, achieving a control effect close to that of the expert model. However, this method can only be used in discrete action spaces and cannot adapt to more complex continuous action space scenarios. In addition, DDT uses soft splitting, which makes the global structure of the model more complex and it is not easy to explain the rules through simple rules or paths.

[0009] The prior art [4] proposed a weighted oblique decision tree based on information gain rate, which was applied in the field of power system emergency control to extract the control strategy of DQN to solve the problem of insufficient fitting ability of traditional decision trees. However, since the oblique decision tree uses the linear combination of features as the tree bifurcation criterion, the rules are complex and not intuitive.

[0010] The existing technology [5] uses a soft decision tree as a student model to clone the control strategy of the Proximal Policy Optimization (PPO) algorithm. However, since the soft decision tree introduces fuzzy logic when the tree bifurcates, it uses probability distribution to represent the probability of samples passing through different paths, and its rules cannot be intuitively explained before deployment.

[0011] In summary, there is a contradiction between the interpretability and decision-making performance of the student model based on the decision tree. The above method weakens the interpretability of the extracted rules in the process of improving the decision tree. At the same time, the above method fails to evaluate the value of expert strategies during the knowledge distillation process, resulting in the student model failing to focus on high-value strategies, reducing the effectiveness of rule extraction.

[0012] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0013] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a deep reinforcement learning interpretable rule extraction method for community energy management, which can improve the decision-making performance of typical rules while ensuring their interpretability.

[0014] The present invention solves the practical problem by adopting the following technical solutions:

[0015] A deep reinforcement learning interpretable rule extraction method for community energy management, comprising the following steps:

[0016] S10: train expert model and extract expert strategy;

[0017] S20: Construct an interpretable student model DADT based on a set of decision trees based on the separation-aggregation mechanism. The student model consists of multiple decision trees. The control strategy of each device is separated from the expert strategy obtained in S10. Each decision tree is responsible for learning and simulating the control strategy of a specific device, and then aggregated into a complete student strategy to form the student model DADT.

[0018] S30: Extract expert strategies through the knowledge distillation method based on joint interaction and action value evaluation, deploy the expert model obtained in S10 and the student model obtained in S20 into the control environment at the same time, and perform multiple rounds of interaction iterations with the environment. In the process of interaction iterations with the environment, use the action value function Q function to evaluate the value of the expert model and the student model in real time, introduce RF loss to achieve the optimal update of the student model DADT strategy, and retrain to obtain a new student model DADT;

[0019] S40: Extract decision tree rules, input the new student model DADT trained after S30, traverse all nodes of each decision tree model, and effectively derive the control rules of each device by analyzing the splitting characteristics and thresholds of each node.

[0020] Moreover, the specific steps of step S10 include:

[0021] S11: Training of the expert model: In the community energy management task, there are m types of controllable devices involved. The expert model gives m corresponding control actions for m types of devices at the same time. After the training phase, an expert strategy is established, denoted as π e :s→p(a e ), where s is the state of the control system, is a vector of m actions given by the expert model, and p is a e The probability distribution of

[0022] S12: Extraction of expert strategy: In order to obtain π e Based on the behavioral patterns in different states, the trained expert model is redeployed to the community energy management task; in this process, the state s at each moment and the actions taken by the expert are observed and recorded. Then sort out the strategy trajectory Extract expert strategies;

[0023] Moreover, the specific steps of step S20 include:

[0024] S21: Training of the student model: At this stage, the expert strategy π e It is divided into a separate control strategy for each device, denoted as Extract the corresponding control actions from the strategy trajectory D Combine it with the state s to form m strategy trajectories D1,…,D m ,in Then, the classification and regression tree algorithm was used to k Train m decision trees separately to obtain the control strategy of the student model for each device Then aggregate into student strategies

[0025] S22: In the decision-making phase, the student model DADT uses each sub-strategy in parallel When a state s is observed, each decision tree independently follows its own sub-strategy Generate control actions for corresponding devices These control actions are eventually combined into a complete action vector

[0026] Moreover, the specific steps of step S30 include:

[0027] S31: The student model and the expert model interact with the environment for multiple rounds of iterations at the same time; in the i-th iteration, a hybrid policy is used to interact with the environment:

[0028]

[0029] where β i The value of determines the mixed strategy Middle School Student Strategies and the expert strategy π e The relative weight of

[0030] S32: Calculate RF loss, considering the expert strategy π e does not always outperform the student policy π in the distribution of environment states c; In order to quantitatively evaluate π under a specific state e and π c The impact of , introduce the action value function:

[0031]

[0032] The Q function represents the state s t Take action a t And the expected reward obtained by following the strategy π at a later moment; where R is the environmental reward function, p is the environmental state transition probability, and V is the state value function:

[0033]

[0034] The V function represents the transition from state s t The expected reward from starting and following policy π at a later time;

[0035] Based on this, the RF loss is used to decide whether to adopt the expert’s example at the current time step t:

[0036]

[0037] in

[0038]

[0039] Formula (5) measures the t Next, expert action Action with students Differences in expected rewards;

[0040] S33: Record strategy trajectories. The student model will only accept the expert's example if the expected value of adopting the expert's action is higher. In each round of interactive iteration, observe and record the state s at each moment and the actions taken by the expert. and RF losses

[0041] S34: Update of student strategy. At the end of each round of iteration, the strategy trajectory obtained in S33 is organized into a training set of the student model DADT. Using l as the sample weight of each sample in the training set, the new student model DADT is retrained.

[0042] Advantages and beneficial effects of the present invention:

[0043] 1. The present invention proposes a deep reinforcement learning interpretable rule extraction method for community energy management, which can improve the decision-making performance of typical rules while ensuring its interpretability. In terms of student models, a decision tree set interpretable model based on the Decomposition-Aggregation (DA) mechanism is proposed to enhance the fitting effect of the student model while maintaining extremely strong interpretability. In terms of learning strategies, a knowledge distillation strategy based on joint interaction and action value evaluation (Knowledge Distillation based on Joint Interaction and Action Value Evaluation, DIE) is designed. The value of the expert strategy is evaluated through the proposed rational follower (RF) loss, so that the student model pays more attention to high-value expert examples under the premise of limited learning ability. Through the joint interaction of experts and student models on the environment, the student model is updated to the best, and the effectiveness of interpretable rule extraction is improved.

[0044] 2. The present invention uses the traditional decision tree as the basis of the student model and retains the structure of the original decision tree. The tree structure of the traditional decision tree is intuitive and clear, the decision path is clear, and the feature importance is obvious. Users can clearly see the splitting conditions of each node and the path of the final decision. It is highly interpretable, which is also the advantage of the present invention, and effectively ensures the security of the extracted strategy.

[0045] 3. The present invention constructs a student model based on the separation-aggregation mechanism, and separates the control strategy of each device in the expert model, so that each decision tree can focus on learning and simulating the regulation strategy of a specific device. This division of labor and cooperation approach can more effectively utilize relevant features and directly capture decision logic, which can reduce learning complexity and increase decision accuracy to a certain extent.

[0046] 4. The present invention adopts a hybrid strategy to interact with the environment, and dynamically adjusts the hybrid strategy weight β in each round of interaction iteration of DIE i , the weight ratio of the expert strategy and the student strategy when interacting with the environment can be adjusted, which effectively enriches the distribution of environmental states in the DIE process, enables the student model to learn expert examples in more states, and increases the generalization of the student model strategy to a certain extent.

[0047] 5. The present invention uses the Q function to evaluate the pros and cons of the expert strategy and the student strategy, and calculates in real time the expected total reward value (Q value) obtained by the actions obtained using the expert strategy and the student strategy in the current state, effectively measuring the value of different strategies.

[0048] 6. The present invention uses RF loss as the sample weight of the strategy trajectory. When the student model is better than the expert model, the RF loss is 0, and the expert example at the current moment is not used; when the expert model is better than the student model, the value of RF loss is the value difference between the student's current action and the expert example. When the difference between the two is large, the corresponding expert example has a larger sample weight, which promotes the student model to learn high-value actions first and realize the optimization update of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Extracting a framework diagram for typical interpretable rules in deep reinforcement learning;

[0050] Figure 2 This is the training flow chart of the expert model;

[0051] Figure 3 Flowchart for extracting expert strategies;

[0052] Figure 4 Initialize the flow chart for the decision tree ensemble student model (DADT) based on the DA mechanism;

[0053] Figure 5 Schematic diagram of the knowledge distillation strategy (DIE) based on joint interaction and action value evaluation;

[0054] Figure 6 A comparison chart of the effects of DADT with different DIE iterations at different DT depths;

[0055] Figure 7 This is a comparison chart of the effects of DADT-10E and MDT at different depths;

[0056] Figure 8(a) is a tree diagram of a typical interpretable decision rule - chilled water storage equipment;

[0057] Figure 8(b) is a dendrogram of a typical interpretable decision rule - domestic hot water storage equipment;

[0058] Figure 8(c) is a dendrogram of a typical interpretable decision rule - battery device. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0060] A deep reinforcement learning interpretable rule extraction method for community energy management, such as Figure 1 As shown, the following steps are included:

[0061] S10: train expert model and extract expert strategy;

[0062] The expert model (deep reinforcement learning model) is deployed to the community energy management task, interacts with the environment for multiple episodes, and trains the expert model. After the model training is completed, the imitation learning method is used to extract the decision-making strategy of the expert model.

[0063] The specific steps of step S10 include:

[0064] S11: Training of expert model: In the community energy management task, there are m types of controllable devices involved. The expert model can give m corresponding control actions for m types of devices at the same time. After the training phase, the expert strategy is established, denoted as π e :s→p(a e ). This process is as follows Figure 2 As shown. Where s is the state of the control system, is a vector of m actions given by the expert model, and p is a e The probability distribution of .

[0065] S12: Extraction of expert strategy: In order to obtain π e Based on the behavior patterns in different states, we redeploy the trained expert model to the community energy management task. In this process, we observe and record the state s at each moment and the actions taken by the expert. Then sort out the strategy trajectory Extract expert strategies, such as Figure 3 shown.

[0066] S20: Construct an interpretable student model DADT based on a set of decision trees based on the separation-aggregation mechanism. The student model consists of multiple decision trees, which separate the control strategy of each device from the expert strategy obtained in S10. Each decision tree is responsible for learning and simulating the control strategy of a specific device, and then aggregated into a complete student strategy to form the student model DADT.

[0067] The specific steps of step S20 include:

[0068] S21: Training of the student model: At this stage, the expert strategy π e It is divided into a separate control strategy for each device, denoted as Specifically, the corresponding control actions are extracted from the strategy trajectory D Combine it with the state s to form m strategy trajectories D1,…,D m ,in 1≤k≤m. Then, the Classification and Regression Trees (CAR T) algorithm was used to kTrain m decision trees separately to obtain the control strategy of the student model for each device Then aggregate into student strategies like Figure 4 shown.

[0069] S22: In the decision-making phase, the student model DADT uses each sub-strategy in parallel When a state s is observed, each decision tree independently follows its own sub-strategy Generate control actions for corresponding devices These control actions are eventually combined into a complete action vector

[0070] S30: Extract the expert strategy through the knowledge distillation method based on joint interaction and action value evaluation, deploy the expert model obtained in S10 and the student model obtained in S20 to the control environment at the same time, and perform multiple rounds of interaction iterations with the environment. The specific process is shown in Algorithm 1. In the process of interaction iteration with the environment, the action value function (Q function) is used to evaluate the value of the expert model and the student model in real time, and the RF loss is introduced to achieve the optimal update of the student model DADT strategy, and the trained student model DADT is obtained;

[0071] The specific steps of step S30 include:

[0072] S31: The student model and the expert model interact with the environment for multiple rounds of iterations. In the i-th iteration, a hybrid policy is used to interact with the environment:

[0073]

[0074] where β i The value of determines the mixed strategy Middle School Student Strategies and the expert strategy π e relative weight of .

[0075] S32: Calculate RF loss, considering the expert strategy π e does not always outperform the student policy π in the distribution of environment states c In order to quantitatively evaluate the e and π c In order to reduce the impact of the action, we introduce the action-value function:

[0076]

[0077] The Q function represents the state s t Take action a tAnd the expected reward obtained by following the strategy π at a later time. Among them, R is the environment reward function, p is the environment state transition probability, and V is the state value function (State-ValueFunction):

[0078]

[0079] The V function represents the transition from state s t The expected reward from starting and following policy π at a later time.

[0080] Based on this, RF loss is used to decide whether to adopt the expert’s example at the current time step t:

[0081]

[0082] in

[0083]

[0084] Formula (5) measures the t Next, expert action Action with students The difference in expected rewards.

[0085] S33: Record strategy trajectories. The student model will only accept the expert's example if the expected value of adopting the expert's action is higher. In each round of interactive iteration, observe and record the state s at each moment and the actions taken by the expert. and RF losses

[0086] S34: Update of student strategy. At the end of each iteration, the strategy trajectory obtained in S33 is organized into the training set of the student model DADT. Using l as the sample weight of each sample in the training set, the new student model DADT is retrained. The process is as follows: Figure 5 shown.

[0087] S40: Extract decision tree rules, input the student model DADT trained after S30, traverse all nodes of each decision tree model, and effectively derive the control rules of each device by analyzing the splitting characteristics and threshold of each node.

[0088] In this embodiment, the knowledge distillation method based on joint interaction and action value evaluation is as follows:

[0089]

[0090] Among them, π e is the expert model strategy, is the DADT strategy directly output by S20 (without DIE), D is the strategy trajectory, N is the number of DIE iterations, β=[β1,…,β N ]The hybrid strategy parameter vector, β i (1≤i≤N) is the mixed strategy parameter in each round of DIE iteration, T i is the time step of each iteration.

[0091] The present invention is further described below by means of specific examples:

[0092] Embodiment 1:

[0093] The typical interpretable decision rule extraction method of this embodiment is as follows: Figure 1 shown.

[0094] The community energy management task of this embodiment is carried out on the CityLearn platform, which provides simulation models of buildings and various energy equipment (heat pumps, batteries, etc.). The data set adopts the data set of CityLearn Challenge 2020, which has 4 similar communities. Each community has a time span of 1 year and a time interval of 1 hour. It contains data such as weather, partial load usage, and real-time electricity prices. The reinforcement learning model can regulate three energy storage devices: chilled water, domestic hot water, and batteries. The continuous action space is set to [-1,1]. The action value represents the proportion of the energy charged (or discharged) of the corresponding device in the next hour. The experimental goal is to minimize the total electricity expenditure in the community (unit: thousand US dollars), and the reinforcement learning environment reward is set to:

[0095] R t =-E t ×C t (6)

[0096] Where R t is the reward value at time t, E t is the net power consumption at the current moment (kWh), C t is the electricity cost at the current moment ($ / kWh).

[0097] Step S11: Train the expert model. Configure the expert model control algorithm as soft actor-critic (SAC), adopt a decentralized control strategy, configure a SAC model for each building, and each SAC value network has two soft Q networks. The strategy network and soft Q network both have two hidden layers, with 400 and 300 neurons respectively. Deploy the expert model to community 1 for 100 rounds of training (the average reward obtained by the expert model in each round has stabilized), and obtain the expert strategy π e :s→p(a e ).

[0098] Step S12: Extract expert strategy. The expert model trained in S11 is redeployed to community 1 to interact with the environment, observe and record the state of the environment s at each moment and the actions taken by the expert. Then sort out the strategy trajectory

[0099] Step S21: Initialize the student model DADT. Extract the control actions for the three energy storage devices of refrigeration, domestic hot water and battery from the strategy trajectory D. Combine it with state s to form three strategy trajectories D1, D2, and D3, where Configure 3 decision tree regression models for DADT, corresponding to 3 types of control devices, and limit the depth of each decision tree to n depth Then, the CART algorithm is used to train the three decision trees on D1, D2, and D3 respectively, and then the student strategy is obtained.

[0100] Step S30: Perform knowledge distillation strategy (DIE) based on joint interaction and action value evaluation. Set the hybrid strategy weight β i =0.5 i In the first round of iteration, the weight of the student model and the expert model strategy that interact with the environment is 1:1. Then, as the student model is gradually updated, the weight of the expert strategy is reduced exponentially. Set the number of DIE iterations to n DIE .

[0101] Step S31: When the student model and the expert model observe the environment state s at time t t After that, according to their respective decision strategies π e With π c Give the control action of the device and Then use a mixed strategy Further interaction with the environment.

[0102] Steps S32 and S33: Use s in S31 t , Calculate RF loss l t , and Recorded in strategy trajectory D.

[0103] Step S34: At the end of each DIE iteration, use a method similar to S21 to reconstruct the three DTs. t Set as sample weights and use the CART algorithm to train these three DTs to obtain a new DADT.

[0104] Step S40: traverse all nodes of each decision tree model of DADT, and derive the control rules of each device by analyzing the splitting characteristics and threshold of each node.

[0105] Embodiment 2:

[0106] This embodiment compares the effects of DADT directly output by S21 (without DIE, denoted as DADT-0E) and DADT with different rounds of DIE (DIE iterations from 1 to 10, denoted as DADT-1E to DADT-10E) to verify the effect of DIE proposed in the present invention. Figure 6 As shown, where the y-axis is the total electricity expenditure in community 1 (unit: thousand US dollars), and the x-axis is the depth limit of each DT in the DADT.

[0107] Depend on Figure 6 It can be seen that under any depth limit, the DADT model after different rounds of DIE iterative updates is always better than the DADT-0E model. With the increase of the number of DIE update rounds, the model performance has been significantly improved, but the degree of increase tends to saturation. It is worth noting that the shallower the tree depth, the fewer the number of rules generated and the stronger the interpretability of the model. At a lower tree depth, the proposed method significantly improves the performance of the student model. This shows that under the condition that the learning ability of the student model is limited, the DIE strategy effectively promotes the student model to prioritize learning high-value expert examples, realizes the optimal update of the student model, and improves the effectiveness of interpretable rule extraction.

[0108] Embodiment 3:

[0109] This example sets up a comparison between DADT-10E and traditional decision trees to verify the effect of the DA mechanism. The traditional decision tree is a regression tree of scikit-learn (because it fits multiple actions, it is called a multi-output decision tree regressor, Multi-output Decision Tree Regressor, MDT), with the environment state s in the strategy trajectory D as the feature, and the expert example of 3 devices The number of leaf nodes of the limiter MDT is the sum of the number of leaf nodes of the three DTs in DADT-10E to ensure the same interpretability (with the same number of rules). In addition, this embodiment also provides a rule-based controller configured according to human expertise and experience as a baseline, and its control rules are shown in Table 1.

[0110] Table 1 Baseline control rules

[0111]

[0112]

[0113] The final effect is compared Figure 7 As shown, where the y-axis is the total electricity expenditure in community 1 (unit: thousand US dollars), and the x-axis is the depth limit of each DT in the DADT.

[0114] It can be seen that when the depth of DADT-10E is shallow, the reduction in total electricity bill expenditure of DADT-10E is more significant compared with MDT. In addition, when the depth is 3, DADT-10E achieves a 13.84% reduction in expenditure relative to the baseline. As the depth increases, the performance of DADT-10E and MDT gradually improves and approaches the expert model. This shows that under the conditions of fewer rules and enhanced interpretability, the student model DADT constructed by the DA mechanism proposed in the present invention effectively improves the fitting performance of the decision tree. DADT enhances the fitting effect of the student model while maintaining extremely strong interpretability.

[0115] Embodiment 4:

[0116] In order to test the performance of DADT with fewer nodes and the generalization of its rules, this embodiment compares its performance with Random Forest (RF) and XGBoost (XGB), which are decision tree ensemble models with excellent fitting capabilities, in 4 communities (trained only in community 1). To maintain the same number of leaf nodes, all models are set with 3 DTs of depth 3, and each DT is limited to 5 leaf nodes. Under this configuration, DADT has 15 leaf nodes (only 15 rules), recorded as DADT-15. Their total electricity expenditure in the 4 communities is shown in Table 2.

[0117] Table 2 Comparison of the effects of different models in different communities

[0118]

[0119] It can be seen that compared with RF and XGB, DADT-15 reduces electricity costs by 3.84% and 8.19%, and is more intuitively interpretable. At the same time, it can be seen that DADT-15 can still effectively reduce the total electricity costs of untrained communities, proving the generalization performance of the extracted rules.

[0120] DADT-15 contains three DTs, corresponding to the three energy storage devices of refrigeration, domestic hot water, and battery. Each DT has only five rules, and the total number of rules is 15, which is the same as the number of rules in Baseline. However, the average electricity cost is reduced by 11.81%, which will provide important guidance for the optimization of RBC strategy. Figure 8(a) , 8(b) , 8(c), the rules are extracted as follows:

[0121] (1) If the temperature forecast for the next 12 hours is lower than or equal to 22.2°C before 14:30, the chilled water storage equipment will release 2% of its total capacity;

[0122] (2) If the time is earlier than (equal to) 14:30, the 12-hour temperature forecast exceeds 22.2°C, and the 12-hour electricity price forecast is lower than (equal to) 0.31$ / kWh, the chilled water storage equipment releases 6% of its total capacity;

[0123] (3) If the time is before (or equal to) 14:30, the 12-hour temperature forecast exceeds 22.2°C, and the 12-hour electricity price forecast is higher than 0.31$ / kWh, the chilled water storage equipment releases 13% of its total capacity;

[0124] (4) If the time is later than 14:30 and earlier than (equal to) 19:30, the chilled water storage equipment stores 17% of its total capacity;

[0125] (5) If the time is later than 19:30, the chilled water storage equipment stores 5% of its total capacity;

[0126] (6) If the 6-hour electricity price forecast is less than (equal to) $0.31 / kWh, the domestic hot water storage is less than (equal to) 3% of the total capacity, and the uncontrolled equipment load is less than (equal to) 3.9 kWh, the domestic hot water storage equipment stores 1% of its total capacity;

[0127] (7) If the 6-hour electricity price forecast is less than (equal to) $0.31 / kWh, the domestic hot water storage is less than (equal to) 3% of the total capacity, and the uncontrolled equipment load is greater than 3.9 kWh, the domestic hot water storage equipment stores 5% of its total capacity;

[0128] (8) If the 6-hour electricity price forecast is less than (equal to) $0.31 / kWh and the domestic hot water storage is greater than 3% of the total capacity, the domestic hot water equipment stores 13% of its total capacity;

[0129] (9) If the 6-hour electricity price forecast is higher than $0.31 / kWh and the domestic hot water storage is less than (equal to) 1% of the total capacity, the domestic hot water storage equipment will not be operated;

[0130] (10) If the 6-hour electricity price forecast is higher than $0.31 / kWh and the domestic hot water storage is greater than 1% of the total capacity, the domestic hot water storage equipment releases 8% of its total capacity;

[0131] (11) If the time is earlier than (or equal to) 14:30, the battery has stored 0 energy, and the battery does not take any action;

[0132] (12) If the time is earlier than (equal to) 7:30 and the battery has stored energy greater than 0, then the battery is charged to 6% of its total capacity;

[0133] (13) If the time is later than 7:30 and earlier than (equal to) 14:30, and the battery has stored energy greater than 0, then the battery is charged to 14% of its total capacity;

[0134] (14) If the time is later than 14:30 and earlier than (equal to) 19:30, the battery is discharged by 15% of its total capacity;

[0135] (15) If the time is later than 19:30, the battery is discharged by 8% of its total capacity.

[0136] The total number of DADT rules in this embodiment is the same as that of the baseline, but the total electricity expense is reduced by 12.92% relative to the baseline.

[0137] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning interpretable rule extraction method for community energy management, characterized by: The following steps are involved: S10: train expert model and extract expert strategy; S20: Construct an interpretable student model DADT based on a set of decision trees based on the separation-aggregation mechanism. The student model consists of multiple decision trees. The control strategy of each device is separated from the expert strategy obtained in S10. Each decision tree is responsible for learning and simulating the control strategy of a specific device, and then aggregated into a complete student strategy to form the student model DADT. S30: Extract expert strategies through the knowledge distillation method based on joint interaction and action value evaluation, deploy the expert model obtained in S10 and the student model obtained in S20 into the control environment at the same time, and perform multiple rounds of interaction iterations with the environment. In the process of interaction iterations with the environment, use the action value function Q function to evaluate the value of the expert model and the student model in real time, introduce RF loss to achieve the optimal update of the student model DADT strategy, and retrain to obtain a new student model DADT; S40: Extract decision tree rules, input the new student model DADT trained after S30, traverse all nodes of each decision tree model, and effectively derive the control rules of each device by analyzing the splitting characteristics and thresholds of each node.

2. The deep reinforcement learning interpretable rule extraction method for community energy management according to claim 1, characterized in that: The specific steps of step S10 include: S11: Training of the expert model: In the community energy management task, there are m types of controllable devices involved. The expert model gives m corresponding control actions for m types of devices at the same time. After the training phase, an expert strategy is established, denoted as π e :s→p(a e ), where s is the state of the control system, is a vector of m actions given by the expert model, and p is a e The probability distribution of S12: Extraction of expert strategy: In order to obtain π e Based on the behavioral patterns in different states, the trained expert model is redeployed to the community energy management task; in this process, the state s at each moment and the actions taken by the expert are observed and recorded. Then sort out the strategy trajectory Extract expert strategies.

3. The deep reinforcement learning interpretable rule extraction method for community energy management according to claim 1, characterized in that: The specific steps of step S20 include: S21: Training of the student model: At this stage, the expert strategy π e It is divided into a separate control strategy for each device, denoted as Extract the corresponding control actions from the strategy trajectory D Combine it with the state s to form m strategy trajectories D1,…,D m ,in Then, the classification and regression tree algorithm was used to k Train m decision trees separately to obtain the control strategy of the student model for each device Then aggregate into student strategies S22: In the decision-making phase, the student model DADT uses each sub-strategy in parallel When a state s is observed, each decision tree independently follows its own sub-strategy Generate control actions for corresponding devices These control actions are eventually combined into a complete action vector 4. The deep reinforcement learning interpretable rule extraction method for community energy management according to claim 1, characterized in that: The specific steps of step S30 include: S31: The student model and the expert model interact with the environment for multiple rounds of iterations at the same time; in the i-th iteration, a hybrid policy is used to interact with the environment: where β i The value of determines the mixed strategy Middle School Student Strategies and the expert strategy π e The relative weight of S32: Calculate RF loss, considering the expert strategy π e does not always outperform the student policy π in the distribution of environment states c ; In order to quantitatively evaluate π under a specific state e and π c The impact of , introduce the action value function: The Q function represents the state s t Take action a t And the expected reward obtained by following the strategy π at a later moment; where R is the environmental reward function, p is the environmental state transition probability, and V is the state value function: The V function represents the transition from state s t The expected reward from starting and following policy π at a later time; Based on this, the RF loss is used to decide whether to adopt the expert’s example at the current time step t: in Formula (5) measures the t Next, expert action Action with students Differences in expected rewards; S33: Record strategy trajectories. The student model will only accept the expert's example if the expected value of adopting the expert's action is higher. In each round of interactive iteration, observe and record the state s at each moment and the actions taken by the expert. and RF losses S34: Update of student strategy. At the end of each round of iteration, the strategy trajectory obtained in S33 is organized into a training set of the student model DADT. Using l as the sample weight of each sample in the training set, the new student model DADT is retrained.

Citation Information

Patent Citations

  • Active target tracking method and system based on expert strategy guidance

    CN115439510A

  • Defect detection model interpretation method and device, equipment and storage medium

    CN116187452A

  • Server and method for obtaining key performance index rapid adaptation artificial intelligence model

    CN117917057A

  • Hybrid decision-making method and device for autonomous driving and computer storage medium

    US20220388540A1