A deep reinforcement learning interpretable rule extraction method for community energy management

Abstract

The application relates to a deep reinforcement learning interpretable rule extraction method for community energy management, comprising the following steps: S10, training an expert model and extracting an expert strategy; S20, constructing a decision tree set interpretable student model DADT based on a separation-aggregation mechanism; S30, extracting the expert strategy through a knowledge distillation method based on joint interaction and action value evaluation, simultaneously deploying the expert model obtained in S10 and the student model obtained in S20 to a control environment, performing multiple rounds of interaction iteration with the environment, in the process of interaction iteration with the environment, using an action value function Q function to evaluate the value of the expert model and the student model in real time, introducing an RF loss to realize optimal updating of the student model DADT strategy, and retraining to obtain a new student model DADT; and S40, extracting a decision tree rule. The application can improve the decision performance of typical rules and ensure the interpretability.

Description

Technical Field

[0001] This invention belongs to the field of interpretable rule extraction technology, and relates to deep reinforcement learning interpretable rule extraction methods, particularly a deep reinforcement learning interpretable rule extraction method for community energy management. Background Technology

[0002] In recent years, Deep Reinforcement Learning (DRL) has achieved significant energy conservation and emission reduction effects in the field of community energy management due to its powerful learning capabilities. However, DRL is a black box, and its decision-making process is unpredictable and unexplainable, making it difficult to assess the rationality and safety of the decisions, and posing security risks in practical applications.

[0003] In reality, energy management controllers often employ rule-based controllers (RBCs), whose control rules are transparent, set based on human expertise and experience, and generally in an "if-then" format. However, manually configured RBCs have significant room for improvement in unlocking the potential of control systems, and the complexity and black-box nature of DRL models make it difficult to provide effective guidance for RBC strategies. Therefore, improving the interpretability of DRL rules is crucial for credible decision-making and strategy optimization in community energy management.

[0004] Currently, there are few interpretable methods for DRL in the energy field, and existing technologies mostly adopt the Shapley Adaptive Explanation (SHAP) method or strategy imitation method.

[0005] The existing technology [1] uses the Deep Q-Networks (DQN) algorithm to perform system undervoltage load shedding control for the delayed voltage recovery problem caused by power grid faults, and uses the SHAP method to generate a feature importance summary map of the DQN decision process. The SHAP method is a model-independent interpretability technique. Therefore, in the field of deep reinforcement learning, a fully trained deep neural network (DNN) can use the SHAP method to obtain the contribution of its input features to the output. However, as a post-interpretation method, SHAP only analyzes the feature importance of the DRL policy, which cannot make the DRL policy transparent, cannot obtain a definite policy execution result before the policy is executed, and cannot guarantee the safety and rationality of the policy. Therefore, the SHAP method cannot be used as a direct extraction method for interpretable DRL rules in the field of community energy management.

[0006] The idea behind policy imitation methods originates from imitation learning methods. Its core principle is to use a small policy network to fit a fully trained DNN policy. Many existing techniques use policy imitation to extract DRL decision policies into a decision tree (DT)-based student model, thus achieving an effective method for extracting interpretable rules from DRL.

[0007] Existing technologies [2] deploy Deep Q-learning (DQL) models in the field of energy management, and then use the traditional imitation learning approach to make the regulation examples of the expert model (DQL) into a training set to train the student model (DT) and realize the extraction of the expert model's strategy. However, the traditional decision tree has poor fitting ability and is significantly weaker than the DRL model. Therefore, some methods mainly focus on how to improve the fitting ability of the decision tree model.

[0008] The existing technology [3] uses Differentiable Decision Trees (DDT) to extract DQN strategies and realize the extraction of control rules for battery energy storage in home energy management systems. This method uses DDT to fit the execution probability of actions in the action space for a specific state in the DQN strategy, and achieves a control effect close to that of expert models. However, this method can only be used in discrete action spaces and cannot adapt to more complex continuous action space scenarios. Moreover, DDT uses soft splitting, and the global structure of the model becomes more complex, making it difficult to interpret the rules through simple rules or paths.

[0009] The prior art [4] proposes a weighted skew decision tree based on information gain ratio, which is applied in the field of emergency control of power systems to extract the control strategy of DQN in order to solve the problem of insufficient fitting ability of traditional decision trees. However, since the skew decision tree uses the linear combination of features as the tree branching criterion, the rules are complex and not intuitive.

[0010] The existing technology [5] uses a soft decision tree as the student model and clones the control strategy of the Proximal Policy Optimization (PPO) algorithm. However, since the soft decision tree introduces fuzzy logic when the tree branches, it uses probability distribution to represent the probability of a sample passing through different paths, and its rules cannot be intuitively explained before deployment.

[0011] In summary, there is a trade-off between the interpretability and decision performance of decision tree-based student models. The methods described above weaken the interpretability of the extracted rules while improving the decision tree. Furthermore, these methods fail to evaluate the value of expert strategies during knowledge distillation, causing the student model to miss high-value strategies and reducing the effectiveness of rule extraction.

[0012] A search revealed no publicly available literature of the same or similar prior art as this invention. Summary of the Invention

[0013] The purpose of this invention is to overcome the shortcomings of the prior art and propose a deep reinforcement learning interpretable rule extraction method for community energy management, which can improve the decision performance of typical rules while ensuring their interpretability.

[0014] The present invention solves its practical problem by adopting the following technical solution:

[0015] A deep reinforcement learning-based interpretable rule extraction method for community energy management includes the following steps:

[0016] S10: Train the expert model and extract expert strategies;

[0017] S20: Construct a decision tree set interpretable student model DADT based on the separation-aggregation mechanism. The student model consists of multiple decision trees. The control strategy of each device is separated from the expert strategy obtained in S10. Each decision tree is specifically responsible for learning and simulating the control strategy of a specific device, and then they are aggregated into a complete student strategy to form the student model DADT.

[0018] S30: Expert policies are extracted using a knowledge distillation method based on joint interaction and action value evaluation. The expert model obtained in S10 and the student model obtained in S20 are simultaneously deployed to the control environment and interact with the environment for multiple rounds of iteration. During the interaction and iteration process, the value of the expert model and the student model is evaluated in real time using the action value function Q function. RF loss is introduced to achieve the optimization update of the student model DADT policy and retraining to obtain a new student model DADT.

[0019] S40: Extract decision tree rules. Input the new student model DADT trained after S30, traverse all nodes of each decision tree model, and effectively derive the control rules for each device by analyzing the splitting features and thresholds of each node.

[0020] Furthermore, the specific steps of step S10 include:

[0021] S11: Expert Model Training: In a community energy management task, m types of controllable devices are involved. The expert model provides m corresponding control actions for each of the m devices simultaneously. After the training phase, an expert strategy is established, denoted as π. e :s→p(a e ); where s represents the state of the control system, It is a vector consisting of m actions given by the expert model, where p is a e The probability distribution;

[0022] S12: Extraction of expert strategy: In order to obtain π e By analyzing behavioral patterns under different states, the trained expert model is redeployed to community energy management tasks. During this process, the state 's' at each moment and the actions taken by the expert are observed and recorded. Then, the strategy trajectory can be sorted out. Extract expert strategies;

[0023] Furthermore, the specific steps of step S20 include:

[0024] S21: Training of the student model: In this stage, the expert policy π e The control strategy is divided into separate policies for each device, denoted as... Extract the corresponding control actions from the strategy trajectory D. Combined with state s, this forms m policy trajectories D1,…,D m ,in Subsequently, classification and regression tree algorithms were used in D k Train m decision trees on each device to obtain the student model's control strategy for each device. This then converges into student strategies.

[0025] S22: During the decision-making phase, the student model DADT utilizes each sub-policy in parallel. When state s is observed, each decision tree independently determines its sub-policy. Generate control actions for the corresponding devices These control actions will eventually combine into a complete action vector.

[0026] Furthermore, the specific steps of step S30 include:

[0027] S31: The student model and the expert model simultaneously interact with the environment in multiple rounds of iteration; in the i-th iteration, a hybrid policy is used to interact with the environment:

[0028]

[0029] Where β i The value determines the mixing strategy Strategies for Middle School Students and expert strategy π e The relative weights;

[0030] S32: Calculate the RF loss, taking into account the expert policy π. e In the distribution of environmental states, it is not always superior to student strategy π. cIn order to quantitatively evaluate π under specific conditions e and π c The influence of this is introduced by the action value function:

[0031]

[0032] The Q function represents the state s. t Take action a at the time t And at subsequent times, follow the expected reward obtained by policy π; where R is the environment reward function, p is the environment state transition probability, and V is the state value function:

[0033]

[0034] The V function represents the state s t The expected reward obtained by starting and following strategy π at subsequent moments;

[0035] Based on this, here's an example of using RF loss to determine whether to adopt an expert at the current time step t:

[0036]

[0037] in

[0038]

[0039] Formula (5) measures the performance of a given state s. t Next, expert actions Student actions The difference in expected rewards;

[0040] S33: Record the policy trajectory. The student model will only accept the expert's example if the expected value of the expert's action is higher. During each round of interaction iteration, observe and record the state s at each time step and the action taken by the expert. and RF loss

[0041] S34: Update the student policy. At the end of each iteration, organize the policy trajectory obtained in S33 into the training set of the student model DADT, and use l as the sample weight of each sample in the training set to retrain the new student model DADT.

[0042] Advantages and beneficial effects of the present invention:

[0043] 1. This invention proposes a deep reinforcement learning-based interpretable rule extraction method for community energy management, which improves the decision-making performance of typical rules while ensuring their interpretability. Regarding the student model, a decision tree ensemble interpretable model based on the decomposition-aggregation (DA) mechanism is proposed, enhancing the student model's fitting effect while maintaining strong interpretability. In terms of the learning strategy, a knowledge distillation strategy based on joint interaction and action value evaluation (DIE) is designed. The value of expert strategies is evaluated using the proposed rational follower (RF) loss, enabling the student model to focus more on high-value expert examples despite limited learning capacity. Through the joint interaction between experts and the student model with the environment, the student model achieves optimization updates, improving the effectiveness of interpretable rule extraction.

[0044] 2. This invention uses a traditional decision tree as the basis for the student model, preserving the original decision tree structure. The tree structure of a traditional decision tree is intuitive and clear, with well-defined decision paths and obvious feature importance. Users can clearly see the splitting conditions of each node and the final decision path, making it highly interpretable. This is also an advantage of this invention, effectively ensuring the security of the extracted strategy.

[0045] 3. This invention constructs a student model based on a separation-aggregation mechanism, separating the expert model for the control strategy of each device, so that each decision tree can focus on learning and simulating the control strategy of a specific device. This division of labor and cooperation can more effectively utilize relevant features and directly capture the decision logic, which can reduce the learning complexity and increase the accuracy of decision-making to a certain extent.

[0046] 4. This invention employs a hybrid strategy to interact with the environment, dynamically adjusting the weight β of the hybrid strategy in each round of DIE interaction iterations. i It can adjust the weight ratio of expert strategies and student strategies when interacting with the environment, effectively enriching the distribution of environmental states in the DIE process, enabling the student model to learn from expert examples in more states, and increasing the generalization of the student model's strategy to a certain extent.

[0047] 5. This invention uses the Q function to evaluate the merits of expert strategies and student strategies, and calculates in real time the expected total reward value (Q value) obtained by using expert strategies and student strategies in the current state, effectively measuring the value of different strategies.

[0048] 6. This invention uses RF loss as the sample weights for the policy trajectory. When the student model outperforms the expert model, the RF loss is 0, and the expert example at the current time step is not used. When the expert model outperforms the student model, the value of the RF loss is the value difference between the student's current action and the expert example. When the difference is large, the corresponding expert example has a larger sample weight, which encourages the student model to prioritize learning high-value actions and achieve optimal updating of the student model. Attached Figure Description

[0049] Figure 1 A framework diagram for extracting typical interpretable rules in deep reinforcement learning;

[0050] Figure 2 This is a flowchart of the training process for an expert model.

[0051] Figure 3 A flowchart for extracting expert strategies;

[0052] Figure 4 Flowchart for initializing the DA-based decision tree ensemble student model (DADT);

[0053] Figure 5 This is a schematic diagram of a knowledge distillation (DIE) strategy based on joint interaction and action value assessment.

[0054] Figure 6 A comparison chart showing the performance of DADT with different DIE iteration rounds at different DT depths;

[0055] Figure 7 Comparison of DADT-10E and MDT at different depths;

[0056] Figure 8(a) is a tree diagram of typical interpretable decision rules for a chilled water storage device;

[0057] Figure 8(b) is a tree diagram of typical interpretable decision rules for domestic hot water storage equipment;

[0058] Figure 8(c) is a tree diagram of a typical interpretable decision rule for a battery device. Detailed Implementation

[0059] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings:

[0060] A deep reinforcement learning-based interpretable rule extraction method for community energy management, such as... Figure 1 As shown, it includes the following steps:

[0061] S10: Train the expert model and extract expert strategies;

[0062] The expert model (deep reinforcement learning model) is deployed to the community energy management task and interacts with the environment in multiple episodes to train the expert model. After the model is trained, the decision-making strategy of the expert model is extracted using imitation learning.

[0063] The specific steps of step S10 include:

[0064] S11: Expert Model Training: In community energy management tasks, m types of controllable devices are involved. The expert model can simultaneously provide m corresponding control actions for each of the m devices. After the training phase, an expert strategy is established, denoted as π. e :s→p(a e This process is as follows: Figure 2 As shown. Where s represents the state of the control system. It is a vector consisting of m actions given by the expert model, where p is a e The probability distribution.

[0065] S12: Extraction of expert strategy: In order to obtain π e Based on the behavioral patterns under different states, we redeploy the trained expert model to the community energy management task. During this process, we observe and record the state 's' at each time step and the actions taken by the expert. Then, the strategy trajectory can be sorted out. Extracting expert strategies, such as Figure 3 As shown.

[0066] S20: Construct a decision tree set interpretable student model DADT based on a separation-aggregation mechanism. This student model consists of multiple decision trees, separating the control strategy of each device from the expert strategy obtained in S10. Each decision tree is specifically responsible for learning and simulating the control strategy of a specific device, and then aggregating them into a complete student strategy to form the student model DADT.

[0067] The specific steps of step S20 include:

[0068] S21: Training of the student model: In this stage, the expert policy π e The control strategy is divided into separate policies for each device, denoted as... Specifically, the corresponding control actions are extracted from the strategy trajectory D. Combined with state s, this forms m policy trajectories D1,…,D m ,in 1≤k≤m. Then, the Classification and Regression Trees (CAR T) algorithm is used in D. kTrain m decision trees on each device to obtain the student model's control strategy for each device. This then converges into student strategies. like Figure 4 As shown.

[0069] S22: During the decision-making phase, the student model DADT utilizes each sub-policy in parallel. When state s is observed, each decision tree independently determines its sub-policy. Generate control actions for the corresponding devices These control actions will eventually combine into a complete action vector.

[0070] S30: Expert policies are extracted using a knowledge distillation method based on joint interaction and action value evaluation. The expert model obtained in S10 and the student model obtained in S20 are simultaneously deployed to the control environment for multiple rounds of interactive iteration with the environment, as shown in Algorithm 1. During the interactive iteration with the environment, the values ​​of the expert model and the student model are evaluated in real time using the action value function (Q function). RF loss is introduced to achieve the optimization update of the student model's DADT policy, resulting in the trained student model DADT.

[0071] The specific steps of step S30 include:

[0072] S31: The student model and the expert model simultaneously interact with the environment in multiple rounds of iteration. In the i-th iteration, a hybrid policy is used to interact with the environment:

[0073]

[0074] Where β i The value determines the mixing strategy Strategies for Middle School Students and expert strategy π e The relative weights.

[0075] S32: Calculate the RF loss, taking into account the expert policy π. e In the distribution of environmental states, it is not always superior to student strategy π. c To quantitatively evaluate π under specific conditions e and π c To address the impact of this, we introduce the Action-Value Function:

[0076]

[0077] The Q function represents the state s. t Take action a at that time tAnd then, at subsequent times, follow the expected reward obtained by policy π. Here, R is the environment reward function, p is the environment state transition probability, and V is the state-value function:

[0078]

[0079] The V function represents the state s t The expected reward obtained by starting and following strategy π at subsequent moments.

[0080] Based on this, here's an example of using RF loss to determine whether to adopt an expert approach at the current time step t:

[0081]

[0082] in

[0083]

[0084] Formula (5) measures the performance of a given state s. t Next, expert actions Student actions The difference in expected rewards.

[0085] S33: Record the policy trajectory. The student model will only accept the expert's example if the expected value of the expert's action is higher. During each round of interaction iteration, observe and record the state s at each time step and the action taken by the expert. and RF loss

[0086] S34: Student policy update. At the end of each iteration, the policy trajectory obtained in S33 is organized into the training set of the student model DADT. Using l as the sample weight of each sample in the training set, a new student model DADT is retrained. The process is as follows: Figure 5 As shown.

[0087] S40: Extract decision tree rules. Input the student model DADT trained after S30, traverse all nodes of each decision tree model, and effectively derive the control rules of each device by analyzing the splitting features and thresholds of each node.

[0088] In this embodiment, the knowledge distillation method based on joint interaction and action value assessment is as follows:

[0089]

[0090] Where, π e For expert model strategies, The policy of DADT is directly output by S20 (without going through DIE), D is the policy trajectory, N is the number of DIE iterations, and β = [β1, ..., β]. N Hybrid strategy parameter vector, β i (1≤i≤N) represents the hybrid strategy parameters in each round of DIE iteration, T i This represents the time step for each iteration.

[0091] The invention will be further illustrated below with specific examples:

[0092] Example 1:

[0093] The typical interpretable decision rule extraction method in this embodiment is as follows: Figure 1 As shown.

[0094] This embodiment's community energy management task is conducted on the CityLearn platform, which provides simulation models of buildings and various energy devices (heat pumps, batteries, etc.). The dataset used is the CityLearn Challenge 2020 dataset, which contains four similar communities, each spanning one year with one-hour intervals, and includes data on weather, partial load usage, and real-time electricity prices. The reinforcement learning model can control three types of energy storage devices: cooling water, domestic hot water, and batteries. The continuous action space is set to [-1, 1], where the action value represents the proportion of energy charged (or discharged) by the corresponding device in the next hour. The experimental objective is to minimize the total electricity expenditure (in thousands of US dollars) in the community, and the reinforcement learning environment reward is set as follows:

[0095] R t =-E t ×C t (6)

[0096] Where R t Let E be the reward value at time t. t C represents the net electricity consumption (kWh) at the current moment. t The electricity cost at the current moment ($ / kWh).

[0097] Step S11: Train the expert model. Configure the expert model control algorithm as Soft Actor-Critic (SAC), employing a distributed control strategy. One SAC model is configured for each building. Each SAC's value network has two soft Q-networks, and both the policy network and the soft Q-network have two hidden layers with 400 and 300 neurons respectively. Deploy the expert model to Community 1 for 100 rounds of training (the average reward obtained by the expert model per round has stabilized), obtaining the expert policy π. e :s→p(a e ).

[0098] Step S12: Extract expert policies. The expert model trained in S11 is redeployed to community 1 to interact with the environment, observing and recording the environment state s at each time step and the actions taken by the expert. Then, the strategy trajectory can be sorted out.

[0099] Step S21: Initialize the student model DADT. Extract control actions for the three energy storage devices (cooling, domestic hot water, and batteries) from the policy trajectory D. Combined with state s, this forms three policy trajectories D1, D2, and D3, where Configure DADT with 3 decision tree regression models, corresponding to 3 types of control devices, with each decision tree limited to a depth of n. depth Subsequently, the CART algorithm was used to train these three decision trees on D1, D2, and D3 respectively, thereby obtaining the student's strategy.

[0100] Step S30: Implement a knowledge distillation strategy (DIE) based on joint interaction and action value assessment. Set the hybrid strategy weights β. i =0.5 i In the first iteration, the student model and expert model policies interacting with the environment have a 1:1 weight ratio. Subsequently, as the student model is updated, the weight of the expert policy is reduced exponentially. The number of iterations in the DIE is set to n. DIE .

[0101] Step S31: When the student model and the expert model observe the environmental state s at time t... t Then, based on their respective decision-making strategies π e With π c Provide the equipment control actions and Therefore, a hybrid strategy is used. To further interact with the environment.

[0102] Steps S32 and S33: Use s in S31 t , Calculate RF loss l t and will Record it in the strategy trajectory D.

[0103] Step S34: At the end of each DIE iteration, reconstruct the three DTs using a method similar to S21, and then... t The three DTs are trained using the CART algorithm with sample weights to obtain a new DADT.

[0104] Step S40: Traverse all nodes of each decision tree model in DADT, and derive the control rules for each device by analyzing the splitting characteristics and thresholds of each node.

[0105] Example 2:

[0106] This embodiment compares the effects of the DADT directly output by S21 (without DIE, denoted as DADT-0E) with DADTs that have undergone different numbers of DIE iterations (DIE iterations from 1 to 10, denoted as DADT-1E to DADT-10E) to verify the effectiveness of the DIE proposed in this invention. Figure 6 As shown, the y-axis represents the total electricity cost in Community 1 (in thousands of dollars), and the x-axis represents the depth limit of each DT in the DADT.

[0107] Depend on Figure 6 It is evident that, under any depth constraint, the DADT model, updated through different numbers of DIE iterations, consistently outperforms the DADT-0E model. As the number of DIE update rounds increases, model performance significantly improves, but the rate of increase tends to saturate. Notably, shallower tree depths result in fewer generated rules and stronger model interpretability. At lower tree depths, the proposed method significantly improves the performance of the student model. This demonstrates that, under conditions of limited student model learning capacity, the DIE strategy effectively promotes the student model's priority learning of high-value expert examples, achieving optimal updates and improving the effectiveness of interpretable rule extraction.

[0108] Example 3:

[0109] This embodiment compares DADT-10E with a traditional decision tree to verify the effectiveness of the DA mechanism. The traditional decision tree is a scikit-learn regression tree (called a Multi-output Decision Tree Regressor, MDT because it fits multiple actions), featuring the environmental state s in the policy trajectory D, and using an expert example with three devices. For the label, the number of leaf nodes of the limiter MDT is the sum of the number of leaf nodes of the three DTs in DADT-10E to ensure the same interpretability (with the same number of rules). In addition, this embodiment also sets up a rule-based controller configured according to human expertise and experience as a baseline, and its control rules are shown in Table 1.

[0110] Table 1 Baseline Control Rules

[0111]

[0112]

[0113] Final effect for example Figure 7 As shown, the y-axis represents the total electricity cost in Community 1 (in thousands of dollars), and the x-axis represents the depth limit of each DT in the DADT.

[0114] It can be seen that when the DADT-10E depth is shallow, the total electricity cost reduction is more significant compared to MDT. Furthermore, at a depth of 3, DADT-10E achieves a 13.84% cost reduction relative to the baseline. With increasing depth, the performance of DADT-10E and MDT gradually improves and approaches that of the expert model. This indicates that, under conditions of fewer rules and enhanced interpretability, the student model DADT constructed by the DA mechanism proposed in this invention effectively improves the fitting performance of the decision tree. DADT enhances the fitting effect of the student model while maintaining extremely strong interpretability.

[0115] Example 4:

[0116] To test the performance of DADT with fewer nodes and the generalization ability of its rules, this embodiment compares its performance with ensemble decision tree models such as Random Forest (RF) and XGBoost (XGB), which have excellent fitting capabilities, in four communities (trained only in community 1). To maintain the same number of leaf nodes, all models were configured with three DTs of depth 3, each limited to five leaf nodes. In this configuration, DADT has 15 leaf nodes (only 15 rules), denoted as DADT-15. Their total electricity consumption in the four communities is shown in Table 2.

[0117] Table 2 Comparison of the effects of different models in different communities

[0118]

[0119] As can be seen, compared to RF and XGB, DADT-15 reduces electricity costs by 3.84% and 8.19%, respectively, and also offers more intuitive interpretability. Furthermore, it can be seen that DADT-15 still effectively reduces the total electricity costs of the untrained community, demonstrating the generalization performance of the extracted rules.

[0120] DADT-15 comprises three Data Centers (DTs), corresponding to cooling, domestic hot water, and battery energy storage devices. Each DT has only five rules, for a total of 15 rules, the same as the Baseline. However, it reduces electricity costs by an average of 11.81%, providing important guidance for optimizing RBC strategies. The three DTs of DADT-15 are as follows: Figure 8(a) , 8(b) As shown in 8(c), the extraction rules are as follows:

[0121] (1) If the time is earlier than (or equal to) 14:30 and the 12-hour temperature forecast is lower than (or equal to) 22.2℃, the chilled water storage equipment will release 2% of its total capacity;

[0122] (2) If the time is earlier than (or equal to) 14:30, the 12-hour temperature forecast exceeds 22.2℃, and the 12-hour electricity price forecast is lower than (or equal to) $0.31 / kWh, then the chilled water storage equipment releases 6% of its total capacity;

[0123] (3) If the time is earlier than (or equal to) 14:30, the 12-hour temperature forecast exceeds 22.2℃, and the 12-hour electricity price forecast is higher than $0.31 / kWh, then the chilled water storage equipment releases 13% of its total capacity;

[0124] (4) If the time is later than 14:30 and earlier than (or equal to) 19:30, the chilled water storage equipment stores 17% of its total capacity;

[0125] (5) If the time is later than 19:30, the chilled water storage equipment will store 5% of its total capacity;

[0126] (6) If the electricity price forecast for 6 hours is less than (or equal to) $0.31 / kWh, the domestic hot water storage is less than (or equal to) 3% of the total capacity, and the uncontrollable equipment load is less than (or equal to) 3.9kWh, then the domestic hot water storage equipment stores 1% of its total capacity.

[0127] (7) If the electricity price forecast for 6 hours is less than (or equal to) $0.31 / kWh, the domestic hot water storage is less than (or equal to) 3% of the total capacity, and the uncontrollable equipment load is greater than 3.9kWh, then the domestic hot water storage equipment shall store 5% of its total capacity.

[0128] (8) If the electricity price forecast for 6 hours is less than (or equal to) $0.31 / kWh, and the domestic hot water storage is greater than 3% of the total capacity, then the domestic hot water equipment stores 13% of its total capacity;

[0129] (9) If the electricity price forecast for 6 hours is higher than $0.31 / kWh and the domestic hot water storage is less than (or equal to) 1% of the total capacity, the domestic hot water storage equipment will not operate.

[0130] (10) If the electricity price forecast for 6 hours is higher than $0.31 / kWh and the domestic hot water storage is greater than 1% of the total capacity, then the domestic hot water storage equipment will release 8% of its total capacity.

[0131] (11) If the time is earlier than (or equal to) 14:30, the battery has stored 0 energy, and the battery will not operate.

[0132] (12) If the time is earlier than (or equal to) 7:30 and the battery has stored more than 0 energy, then the battery is charged to 6% of its total capacity.

[0133] (13) If the time is later than 7:30 and earlier than (or equal to) 14:30, and the battery has stored more than 0 energy, then the battery is charged to 14% of its total capacity.

[0134] (14) If the time is later than 14:30 and earlier than (or equal to) 19:30, the battery will discharge 15% of its total capacity.

[0135] (15) If the time is later than 19:30, the battery will discharge 8% of its total capacity.

[0136] In this embodiment, the total number of DADT rules is the same as the baseline, but the total electricity cost is reduced by 12.92% relative to the baseline.

[0137] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A deep reinforcement learning-based interpretable rule extraction method for community energy management, characterized in that: Includes the following steps: S10: Train the expert model and extract expert strategies; S20: Construct a decision tree set interpretable student model DADT based on the separation-aggregation mechanism. The student model consists of multiple decision trees. The control strategy of each device is separated from the expert strategy obtained in S10. Each decision tree is specifically responsible for learning and simulating the control strategy of a specific device, and then they are aggregated into a complete student strategy to form the student model DADT. S30: Expert policies are extracted using a knowledge distillation method based on joint interaction and action value evaluation. The expert model obtained in S10 and the student model obtained in S20 are simultaneously deployed to the control environment and interact with the environment for multiple rounds of iteration. During the interaction and iteration process, the value of the expert model and the student model is evaluated in real time using the action value function Q function. RF loss is introduced to achieve the optimization update of the student model DADT policy and retraining to obtain a new student model DADT. S40: Extract decision tree rules, input the new student model DADT trained after S30, traverse all nodes of each decision tree model, and effectively derive the control rules of each device by analyzing the splitting features and thresholds of each node; The specific steps of step S10 include: S11: Training of expert models: In community energy management tasks, it involves A type of controllable device, with expert models targeting the same target at the same time. This equipment provides A corresponding regulatory action; after the training phase, an expert strategy is established and recorded. ;in, To control the state of the system. It is given by the expert model. A vector composed of actions, for The probability distribution; S12: Extraction of Expert Strategies: In order to obtain By analyzing behavioral patterns under different states, the trained expert model is redeployed to community energy management tasks; during this process, the state at each moment is observed and recorded. And the actions taken by experts This leads to the analysis of the strategy trajectory. Extract expert strategies; The specific steps of step S30 include: S31: The student model and the expert model simultaneously interact and iterate with the environment in multiple rounds; in the... In this iteration, a hybrid policy is used to interact with the environment: (1) in The value determines the mixing strategy Strategies for Middle School Students and expert strategies The relative weights; S32: Calculate the RF loss, taking into account the expert policy. In the distribution of environmental states, it is not always superior to student strategies. In order to quantitatively assess the situation under specific conditions and The influence of this is introduced by the action value function: (2) Functions represent states Take action at the time And follow the strategy in the following moments The expected reward obtained; among which, It is the environmental reward function. Let be the probability of environmental state transition. State value function: (3) The function represents the state. Start and follow the strategy in the moments that follow. The expected reward obtained; Based on this, the RF loss is used to determine the current time step. Should we use an expert's example? (4) in (5) Formula (5) measures the state Next, expert actions Student actions The difference in expected rewards; S33: Record the policy trajectory. The student model will only accept the expert's example if the expected value of the expert's action is higher. During each round of interaction iteration, observe and record the state at each moment. Actions taken by experts and RF loss ; S34: Student policy update. At the end of each iteration, the policy trajectory obtained in S33 is organized into the training set of the student model DADT, and then used... As the sample weights for each sample in the training set, a new student model DADT is retrained.

2. The deep reinforcement learning interpretable rule extraction method for community energy management according to claim 1, characterized in that: The specific steps of step S20 include: S21: Training the student model: In this stage, the expert strategy... The control strategy is divided into separate policies for each device, denoted as... From the strategy trajectory Extract the corresponding control actions to the state Combine, form Each strategy trajectory ,in , Subsequently, classification and regression tree algorithms were used in... Training separately A decision tree is used to obtain the control strategy for each device in the student model. This leads to the aggregation of student strategies. ; S22: During the decision-making phase, the student model DADT utilizes each sub-policy in parallel. When the state is observed At that time, each decision tree independently determines its sub-strategy. Generate control actions for the corresponding devices These control actions will eventually combine into a complete action vector. .

Images