Community integrated energy optimization scheduling method and system capable of explaining reinforcement learning

Abstract

The invention discloses a community integrated energy optimization scheduling method and system capable of explaining reinforcement learning, and the method comprises the steps: building an energy management model of a community integrated energy system based on a Markov decision process according to the model architecture of the community integrated energy system, and training the model through a collaborative reinforcement learning algorithm, secondly, extracting a training decision rule of the algorithm and important environment characteristics influencing the decision by creating an interpretable artificial intelligence framework, taking the training decision rule and the important environment characteristics as new experience to act on the training process of the algorithm again, and finally setting an iteration threshold value, and continuously updating the decision rule and the important environment characteristics of the algorithm. And obtaining an optimal energy scheduling strategy of the community integrated energy system. Rich energy resources in a community are fully utilized, the energy supply and demand conditions of the community are accurately grasped through real-time monitoring and analysis of energy, the generalization ability and robustness of an energy system model are improved through an interpretable artificial intelligence framework, and efficient management and scheduling of energy are achieved on the community level.

Description

Technical Field

[0001] The present invention belongs to the field of energy system optimization decision-making, and specifically relates to a community comprehensive energy optimization scheduling method and system that can explain reinforcement learning. Background Art

[0002] According to statistics, the energy consumption of the construction industry accounts for about one-third of the global energy consumption, and this proportion is expected to increase with the advancement of urbanization. The progress of the power and energy sectors has promoted the development of the power grid towards an energy internet, in which the storage of distributed energy and the development of renewable energy must be coordinated. By using advanced technologies, a balance between building comfort and environmental sustainability can be achieved.

[0003] Applying distributed coordinated control to achieve coordinated optimization of community renewable energy systems has become a feasible approach. Currently, various methods, including deep reinforcement learning, have been used to optimize the energy use of buildings and power systems, but there are still difficulties in ensuring the safety and understandability of these AI-based solutions. Among them, many reinforcement learning models are called "black box" models because their internal decision-making processes are unexplainable to humans. In community energy systems, some decisions may affect the stability of the system or cause waste of resources, but due to the unpredictability of the model, it is difficult to effectively prevent these problems, which means that although the model can give good results, users cannot understand why such a decision was made, and reinforcement learning models may cause some safety issues due to their unforeseen behavior. Summary of the invention

[0004] The purpose of the present invention is to provide a community comprehensive energy optimization scheduling method and system based on explainable reinforcement learning in order to solve the above problems existing in the prior art.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention proposes a community comprehensive energy optimization scheduling method based on explainable reinforcement learning, comprising:

[0007] S1. According to the model architecture of the community integrated energy system and based on the Markov decision process, an energy management model of the community integrated energy system is established, wherein the model architecture includes residential areas, commercial areas, industrial areas, photovoltaic power generation systems, energy storage systems and dispatchable resource systems;

[0008] S2. Use a collaborative reinforcement learning algorithm to train the energy management model of the community integrated energy system to obtain the energy dispatch strategy of the community integrated energy system;

[0009] S3. Create an explainable artificial intelligence framework to extract the decision rules and important environmental features that affect the decision of the collaborative reinforcement learning algorithm to train the energy scheduling strategy, wherein the explainable artificial intelligence framework includes a decision tree model for extracting the algorithm decision rules and an under-sampled deep Shapley additive interpretation method for analyzing the important environmental features that affect the decision;

[0010] S4, taking the extracted decision rules and important environmental features that affect the decision as new experience of the collaborative reinforcement learning algorithm, and executing steps S2-S3 again to update the decision rules and important environmental features that affect the decision;

[0011] S5. Determine whether the accuracy of the decision tree after the decision rule is updated reaches the set threshold, and whether the correlation between the important environmental features that affect the decision after the update is lower than the set value. If the accuracy of the decision tree after the update reaches the set threshold, and the correlation between the important environmental features that affect the decision after the update is lower than the set value, then output the optimal energy scheduling strategy for the community integrated energy system obtained at this time, otherwise return to S4 for the next iteration.

[0012] In S1, the overall objective function of the energy management model of the community integrated energy system is:

[0013] F = min(λ1F1+λ2F2+λ3F3);

[0014]

[0015] In the above formula, λ1 is the weight coefficient of the residential area, λ2 is the weight coefficient of the commercial area, λ3 is the weight coefficient of the industrial area, F1 is the objective function of the residential area, F2 is the objective function of the commercial area, and F3 is the objective function of the industrial area; T is the total time gap, t is the t moment in the time gap, I is the residential area, commercial area or industrial area, i is the i-th building in each area, The cost of purchasing electricity from the power grid, Energy cost for HVAC operation, is the loss cost of the energy storage device, Cost of diesel generators;

[0016] The constraints of the energy management model of the community integrated energy system include:

[0017]

[0018] v comm ≥ v ;

[0019] In the above formula, is the amount of electricity purchased from the power grid at time t, is the output power of distributed photovoltaic at time t, is the output power of the diesel generator at time t, is the discharge power of the energy storage system at time t, is the electricity load of buildings in the community at time t, is the charging power of the energy storage system at time t, is the power consumption of the HVAC system in the community at time t; v comm is the voltage of the common bus, v It is the lower limit of the voltage of the common bus.

[0020] The S2 includes:

[0021] S21, creating a customized multi-agent community integrated energy system as environment E, wherein the agent is an independent building i in each district;

[0022] S22. Design a multi-agent reinforcement learning algorithm with progressive task context in environment E to continuously train the energy scheduling strategy in the task flow of the community integrated energy system, including:

[0023] S221, abstract the tasks under environment E into context information and build a model extractor h θ , with the trajectory of the agent in the task τ=(s0,…,s N ) is the input, s N The trajectory of strategy N, the extracted Gaussian distribution as contextual information;

[0024] The following formula is used to calculate the parts in the Gaussian distribution:

[0025]

[0026] In the above formula, μ θ (τ) is the mean of the context information extracted by the model, N is the number of strategies, μ t is the average energy demand at time t, σ t is the volatility of energy demand at time t, The variance of the context information extracted by the model;

[0027] S222. Design a feature extractor that can share task knowledge. Each agent uses K trajectories to calculate the context mean and uses the following formula to select the relevant strategy head, that is, the most relevant energy scheduling strategy under the changing task environment:

[0028]

[0029] In the above formula, m ＊i is the most relevant strategy head, M is the number of strategy heads designed in N strategies, K is the number of trajectories, is the context mean of agent i on the kth trajectory, bs is the number of samples in each trajectory, is the mean of all sample feature extraction values ​​in strategy head m.

[0030] The S3 includes:

[0031] S31, creating a buffer to collect behavior data when the agent uses a collaborative reinforcement learning algorithm to perform tasks in the environment, wherein the behavior data includes current state data, action data taken, reward data received, and a new state quadruple converted to;

[0032] S32. Based on the behavior data of the agent when performing tasks in the environment using the collaborative reinforcement learning algorithm, a decision tree model is constructed to guide the agent to transform the original environmental feature data into a state representation that is more conducive to learning and decision-making, including:

[0033] S321, splitting the feature behavior data with information selected by the decision tree, and outputting the key feature vector of the model structure by analyzing the structure of the decision tree and the branch conditions on the nodes;

[0034] S322. According to the value range of the key feature vector output by the decision tree, the four-tuple of state action, reward, and new state matching with it is screened, and the state vector is constructed using the following formula to represent the extracted rules that affect the final decision:

[0035]

[0036] In the above formula, s is the state vector, n v is the feature of each node, V is the number of features, h′ v is the feature vector obtained from the decision tree;

[0037] S33. Design a deep Shapley additive interpretation method based on undersampling, and form an interpretable artificial intelligence framework with the decision tree model to analyze and extract important environmental features that affect decision-making, including:

[0038] S331. Use the trained energy management model of the community integrated energy system to perform a series of decision-making tasks, collect the state action data of the model when the current optimal strategy is executed as the environmental data set, and classify the data set into fault environmental state samples and normal environmental state samples according to whether the state observed in the power system is normal;

[0039] S332, balancing the fault environment state samples and the normal environment state samples by minimizing the undersampling objective function, and identifying the fault environment state samples and the normal environment state samples with the highest purity;

[0040] The undersampling objective function is:

[0041]

[0042] In the above formula, S f is the fault status data in the sample, S n is the normal state data in the sample, θ is the sampling weight of the fault sample, is the sampling weight of normal samples;

[0043] S333. Select the best features from the fault environment state samples and the normal environment state samples with the highest purity, use the following formula to calculate the average SHAP value of the best features in each environment state to indicate the importance of the environment features in different environment states to the decision, and select the environment features with the largest average SHAP value as the important environment features that affect the decision:

[0044]

[0045]

[0046] In the above formula, is the average SHAP value of the optimal feature, K is the total number of sample data, is the gradient of the predicted value relative to the i-th eigenvalue of the k-th sample data in the data set, is the change in the i-th eigenvalue of the k-th sample data during the iteration, Y k is the predicted value, that is, the action corresponding to the kth sample data state, is the i-th eigenvalue of the k-th sample data.

[0047] The weight coefficients of residential areas, commercial areas, and industrial areas in the objective function of the energy management model of the community integrated energy system are set using the flexible load dispatching cost, including:

[0048] A. Use the following formula to calculate the flexible load dispatching cost of each area:

[0049]

[0050] In the above formula, is the flexible load dispatching cost of power in area I, where I is a residential area, commercial area or industrial area. is the total compensation cost for load reduction in zone I, is the total compensation cost for movable loads in zone I, is the total compensation cost for the transferable load in Area I; cut is the compensation cost per unit load reduction, is the load reduction weight factor at time t, is the load-reducible power before load reduction, is the load-reducible power after load reduction; shift is the compensation cost per movable load, is a specific starting time point, t1 and t2 are the first and second moments respectively, is the weight factor of the transferable load at time t, is the power shifted at time t; trans is the compensation cost of transferring load at time t, tr- is the beginning of a specific time, tr+ is the end of a specific time, is the transferable load weight factor at time t, is the power delivered at time t;

[0051] B. Normalize the flexible load dispatching cost of each area, and use the following formula to express the relationship between the weight coefficient and the flexible load dispatching cost:

[0052]

[0053] In the above formula, λ I is the weight coefficient of each area, is the S-shaped curve of the normalized flexible load dispatching cost, n is the total number of zones, a is the parameter controlling the slope of the curve, and x0 is the center position of the curve.

[0054] In the second aspect, the present invention proposes a community comprehensive energy optimization scheduling system with interpretable reinforcement learning, including a model building module, a model training module, an intelligent framework creation module, a decision updating module, and a loop judgment module;

[0055] The model building module is used to establish an energy management model of the community integrated energy system based on the model architecture of the community integrated energy system and the Markov decision process, wherein the model architecture includes residential areas, commercial areas, industrial areas, photovoltaic power generation systems, energy storage systems and dispatchable resource systems;

[0056] The model training module is used to train the energy management model of the community integrated energy system using a collaborative reinforcement learning algorithm to obtain an energy dispatch strategy for the community integrated energy system;

[0057] The intelligent framework creation module is used to create an explainable artificial intelligence framework to extract the decision rules of the collaborative reinforcement learning algorithm to train the energy scheduling strategy and the important environmental features that affect the decision. The explainable artificial intelligence framework includes a decision tree model for extracting the algorithm decision rules and an under-sampled deep Shapley additive interpretation method for analyzing the important environmental features that affect the decision.

[0058] The decision updating module is used to use the extracted decision rules and important environmental features that affect the decision as new experience of the collaborative reinforcement learning algorithm, and to perform the operations of the model training module and the intelligent framework building module again to update the decision rules and important environmental features that affect the decision;

[0059] The loop judgment module is used to judge whether the accuracy of the decision tree after the decision rule is updated reaches the set threshold, and whether the correlation between the important environmental features that affect the decision after the update is lower than the set value. If the accuracy of the decision tree after the update reaches the set threshold, and the correlation between the important environmental features that affect the decision after the update is lower than the set value, the optimal energy scheduling strategy of the community integrated energy system obtained at this time is output, otherwise it returns to the decision update module for the next iteration.

[0060] The model construction module includes an objective function construction unit and a constraint condition construction unit;

[0061] The objective function construction unit is used to construct the following overall objective function of the energy management model of the community integrated energy system:

[0062] F = min(λ1F1+λ2F2+λ3F3);

[0063]

[0064] In the above formula, λ1 is the weight coefficient of the residential area, λ2 is the weight coefficient of the commercial area, λ3 is the weight coefficient of the industrial area, F1 is the objective function of the residential area, F2 is the objective function of the commercial area, and F3 is the objective function of the industrial area; T is the total time gap, t is the t moment in the time gap, I is the residential area, commercial area or industrial area, i is the i-th building in each area, The cost of purchasing electricity from the power grid, Energy cost for HVAC operation, is the loss cost of the energy storage device, Cost of diesel generators;

[0065] The constraint condition construction unit is used to construct the constraint conditions of the energy management model of the following community integrated energy system:

[0066]

[0067] v comm ≥ v ;

[0068] In the above formula, is the amount of electricity purchased from the power grid at time t, is the output power of distributed photovoltaic at time t, is the output power of the diesel generator at time t, is the discharge power of the energy storage system at time t, is the electricity load of buildings in the community at time t, is the charging power of the energy storage system at time t, is the power consumption of the HVAC system in the community at time t; v comm is the voltage of the common bus, v It is the lower limit of the voltage of the common bus.

[0069] The model training module includes an environment creation unit and a strategy training unit;

[0070] The environment creation unit is used to create a customized multi-agent community integrated energy system as the environment E, and the agent is an independent building i in each district;

[0071] The strategy training unit is used to design a multi-agent reinforcement learning algorithm with progressive task context in environment E, and continuously train the energy scheduling strategy in the task flow under the community integrated energy system, including a model extractor construction subunit and a feature extractor design subunit;

[0072] The model extractor construction subunit is used to abstract the tasks under the environment E into context information and construct a model extractor h θ , with the trajectory of the agent in the task τ=(s0,…,s N ) is the input, s N The trajectory of strategy N, the extracted Gaussian distribution as contextual information;

[0073] The following formula is used to calculate the parts in the Gaussian distribution:

[0074]

[0075] In the above formula, μ θ (τ) is the mean of the context information extracted by the model, N is the number of strategies, μ t is the average energy demand at time t, σ t is the volatility of energy demand at time t, The variance of the context information extracted by the model;

[0076] The feature extractor design subunit is used to design a feature extractor that can share task knowledge. Each agent uses K trajectories to calculate the context mean and selects the relevant strategy head, that is, the most relevant energy scheduling strategy under the changing task environment, using the following formula:

[0077]

[0078] In the above formula, m ＊iis the most relevant strategy head, M is the number of strategy heads designed in N strategies, K is the number of trajectories, is the context mean of agent i on the kth trajectory, bs is the number of samples in each trajectory, is the mean of all sample feature extraction values ​​in strategy head m.

[0079] The intelligent framework creation module includes a data collection unit, a decision tree construction unit, and an important feature analysis unit;

[0080] The data collection unit is used to create a buffer to collect behavior data when the agent uses a collaborative reinforcement learning algorithm to perform tasks in the environment, wherein the behavior data includes current state data, action data taken, reward data received, and a new state quadruple converted to;

[0081] The decision tree construction unit is used to construct a decision tree model based on the behavior data of the agent when performing tasks in the environment using the collaborative reinforcement learning algorithm, and guide the agent to convert the original environmental feature data into a state representation that is more conducive to learning and decision-making, including a key feature output subunit and a state vector construction subunit;

[0082] The key feature output subunit is used to split the feature behavior data with information selected by the decision tree, and output the key feature vector of the model structure by analyzing the structure of the decision tree and the branch conditions on the nodes;

[0083] The state vector construction subunit is used to screen the matching state action, reward, and new state quadruple according to the value range of the key feature vector output by the decision tree, and construct the state vector using the following formula to represent the extracted rules that affect the final decision:

[0084]

[0085] In the above formula, s is the state vector, n v is the feature of each node, V is the number of features, h′ v is the feature vector obtained from the decision tree;

[0086] The important feature analysis unit is used to design a deep Shapley additive interpretation method based on undersampling, which forms an explainable artificial intelligence framework with the decision tree model to analyze and extract important environmental features that affect decision-making, including an environmental data collection subunit, a sample identification subunit, and an average SHAP value calculation subunit;

[0087] The environmental data collection subunit is used to use the trained energy management model of the community integrated energy system to perform a series of decision-making tasks, collect the state action data of the model when the current optimal strategy is executed as the environmental data set, and classify the data set into fault environment state samples and normal environment state samples according to whether the state observed in the power system is normal;

[0088] The sample identification subunit is used to balance the fault environment state samples and the normal environment state samples by minimizing the undersampling objective function, and identify the fault environment state samples and the normal environment state samples with the highest purity;

[0089] The undersampling objective function is:

[0090]

[0091] In the above formula, S f is the fault status data in the sample, S n is the normal state data in the sample, θ is the sampling weight of the fault sample, is the sampling weight of normal samples;

[0092] The average SHAP value calculation subunit is used to select the best features in the fault environment state samples and normal environment state samples with the highest purity, and the average SHAP value of the best features in each environment state is calculated using the following formula to indicate the importance of the environment features in different environment states to the decision, and the environment features with larger average SHAP values ​​are selected as the important environment features affecting the decision:

[0093]

[0094] In the above formula, is the average SHAP value of the optimal feature, K is the total number of sample data, is the gradient of the predicted value relative to the i-th eigenvalue of the k-th sample data in the data set, is the change in the i-th eigenvalue of the k-th sample data during the iteration, Y k is the predicted value, that is, the action corresponding to the kth sample data state, is the i-th eigenvalue of the k-th sample data.

[0095] The weight coefficients of residential areas, commercial areas, and industrial areas in the objective function of the energy management model of the community integrated energy system are set using the flexible load dispatching cost, including:

[0096] A. Use the following formula to calculate the flexible load dispatching cost of each area:

[0097]

[0098] In the above formula, is the flexible load dispatching cost of power in area I, where I is a residential area, commercial area or industrial area. is the total compensation cost for load reduction in zone I, is the total compensation cost for movable loads in zone I, is the total compensation cost for the transferable load in Area I; cut is the compensation cost per unit load reduction, is the load reduction weight factor at time t, is the load-reducible power before load reduction, is the load-reducible power after load reduction; shift is the compensation cost per movable load, is a specific starting time point, t1 and t2 are the first and second moments respectively, is the weight factor of the transferable load at time t, is the power shifted at time t; trans is the compensation cost of transferring load at time t, tr- is the beginning of a specific time, tr+ is the end of a specific time, is the transferable load weight factor at time t, is the power delivered at time t;

[0099] B. Normalize the flexible load dispatching cost of each area, and use the following formula to express the relationship between the weight coefficient and the flexible load dispatching cost:

[0100]

[0101] In the above formula, λ I is the weight coefficient of each area, is the S-shaped curve of the normalized flexible load dispatching cost, n is the total number of zones, a is the parameter controlling the slope of the curve, and x0 is the center position of the curve.

[0102] Compared with the prior art, the present invention has the following beneficial effects:

[0103] 1. The present invention proposes a community integrated energy optimization scheduling method and system based on interpretable reinforcement learning. According to the model architecture of the community integrated energy system and based on the Markov decision process, the method establishes an energy management model of the community integrated energy system, uses a collaborative reinforcement learning algorithm to train the energy management model of the community integrated energy system, and obtains the energy scheduling strategy of the community integrated energy system. Then, an interpretable artificial intelligence framework is created to extract the decision rules and important environmental features that affect the decision of the collaborative reinforcement learning algorithm for training the energy scheduling strategy, and re-apply them to the training process of the algorithm as new experience of the algorithm. Finally, the accuracy threshold of the decision tree and the value of the correlation between the important environmental features that affect the decision are set, and the optimal energy scheduling strategy of the community integrated energy system is obtained by continuously updating the decision rules and environmental features of the algorithm. On the one hand, this method jointly models residential areas, commercial areas, industrial areas, photovoltaic power generation systems, energy storage systems, and dispatchable resource systems to solve the complexity and uncertainty problems of community integrated energy systems, making full use of the richness and diversity of energy resources in the community, as well as the flexibility of energy storage facilities. Through real-time monitoring and analysis of these energy sources, it accurately grasps the current energy supply and demand situation in the community, provides reliable data support for decision-making, and improves the operating efficiency of the community integrated energy system while reducing the overall cost of the energy system and maintaining the thermal comfort of the building. On the other hand, an interpretable artificial intelligence framework including a decision tree model and an under-sampling deep Shapley additive interpretation method is designed to explain the decision-making process of the reinforcement learning algorithm. By analyzing the structure of the decision tree and the nodes The branch conditions of each node are extracted, the rules corresponding to each node are extracted, and the intelligent agent is guided to transform the original environmental feature data into a state that is more conducive to learning and decision-making. The behavior patterns and possible abnormal situations of the intelligent agent are identified, and the optimization strategy is adjusted in time. The important environmental features that affect the decision-making are analyzed through the under-sampling deep Shapley additive interpretation method. The optimization scheduling strategy made by the intelligent agent under specific environmental features is analyzed in combination with the actual situation, so as to improve the learning efficiency of the algorithm and the quality of the training strategy; and the extracted decision rules and important environmental features are used as new experiences of the collaborative reinforcement learning algorithm to update the optimization scheduling strategy, which improves the generalization ability and robustness of the model, while minimizing the total cost of energy consumption, maximizing the satisfaction of community energy needs, and enhancing the sustainability and stability of the energy system.

[0104] 2. The present invention proposes a community comprehensive energy optimization scheduling method and system based on explainable reinforcement learning. According to the heterogeneous nature of the buildings in the community, the method divides the buildings in the community into three different types: residential areas, commercial areas, and industrial areas, to construct an energy management model of the community comprehensive energy system. Through the refined classification of the community structure, the system can more accurately understand the energy demand characteristics and usage patterns of different types of building complexes, thereby formulating a more practical energy scheduling strategy, reducing energy waste, and improving overall energy utilization efficiency; for the setting of weight coefficients for residential areas, commercial areas, and industrial areas, the importance of each part of the area is judged by the flexible load scheduling cost, and a nonlinear function is introduced to model the relationship between the weight coefficient and the flexible load scheduling cost, further refining the adjustment accuracy of the weight coefficient, making the scheduling strategy more refined and efficient, thereby ensuring that the community energy demand is met, giving priority to scheduling loads with lower costs, minimizing energy costs, and in the face of emergencies such as equipment failures, the system can quickly adjust the scheduling strategy based on the established energy management model and weight coefficient to reduce the impact of emergencies on the community energy supply. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 This is a model architecture diagram of the community integrated energy system described in Example 1.

[0106] Figure 2 The figure is an overall flow chart of the method of the present invention.

[0107] Figure 3 The structure diagram of the system of the present invention is shown in FIG. DETAILED DESCRIPTION

[0108] The present invention is further described in detail below in conjunction with specific implementations and drawings.

[0109] The present invention proposes a community comprehensive energy optimization scheduling method and system with explainable reinforcement learning. The method utilizes the richness and diversity of various energy resources in the community to establish a community comprehensive energy system model including renewable energy, traditional energy, energy storage equipment and other components. According to the energy management model of the community comprehensive energy system, a multi-agent reinforcement learning decision algorithm is designed to realize energy allocation and scheduling. Through a multi-objective scheduling framework, the intelligent agent can learn how to maximize energy utilization efficiency under different conditions, thereby improving the reliability of the community energy system and promoting the utilization and popularization of renewable energy. By realizing efficient energy management and scheduling at the community level, a solid foundation is laid for building a clean and low-carbon energy future, and technological innovation in the field of sustainable development of the community and energy is promoted.

[0110] Embodiment 1:

[0111] This example takes a community as the research object, and the model architecture of its integrated energy system is as follows: Figure 1 As shown, area 1 is a residential area, area 2 is a commercial area, area 3 is an industrial area, and area 4 is equipped with a 500kW photovoltaic power generation system and a 10MW·h energy storage system for the residential and commercial areas. In addition, considering the stability of the power system, dispatchable resource systems in areas 5 and 6 are equipped for the commercial and industrial areas. Among them, area 5 is a commercial area equipped with a 400kW diesel generator and a 10MW·h energy storage system, and area 6 is an industrial area equipped with a 400kW diesel generator and a 20MW·h energy storage system.

[0112] like Figure 2 As shown in the figure, a community comprehensive energy optimization scheduling method based on interpretable reinforcement learning is carried out in the following steps:

[0113] 1. According to the model architecture of the community integrated energy system and based on the Markov decision process, an energy management model of the community integrated energy system is established;

[0114] According to the heterogeneous nature of buildings in the community, the buildings in the community are divided into three different types: residential area, commercial area, and industrial area. Since the community comprehensive energy system usually operates under the synergy of energy supply, consumption, storage and conversion, each area is also equipped with renewable energy power generation and heating devices and energy storage devices. HVAC systems are usually installed in residential and commercial buildings. Commercial and industrial areas are usually equipped with diesel generators as dispatchable resources.

[0115] The community integrated energy system is taken as the environment E, and the independent buildings in each area are taken as an intelligent agent. The scheduling process of the community integrated energy system is described as a Markov decision process, and an energy management model of the community integrated energy system is constructed. The goal of the intelligent agent is to minimize the operating cost of the building by optimizing the management of the community energy management system components. Through the energy management model of the multi-objective scheduling framework, the intelligent agent can learn how to maximize energy utilization efficiency under different conditions to achieve energy scheduling and allocation.

[0116] The objective functions of the energy management model of the community integrated energy system include:

[0117] F = min(λ1F1+λ2F2+λ3F3);

[0118]

[0119] In the above formula, λ1 is the weight coefficient of the residential area, λ2 is the weight coefficient of the commercial area, λ3 is the weight coefficient of the industrial area, F1 is the objective function of the residential area, F2 is the objective function of the commercial area, and F3 is the objective function of the industrial area; T is the total time gap, t is the t moment in the time gap, I is the residential area, commercial area or industrial area, i is the i-th building in each area, The cost of purchasing electricity from the power grid, Energy cost for HVAC operation, is the loss cost of the energy storage device, Cost of diesel generators;

[0120] When designing the objective function of the energy management model of the community integrated energy system, the richness and diversity of energy resources in the community and the flexibility of energy storage facilities are fully utilized. Through real-time monitoring of energy, the current consumption and supply and demand of the energy system are accurately grasped, providing reliable data support for scheduling decisions;

[0121] In the energy management process of the community integrated energy system, each energy device meets the following energy supply and demand balance constraints:

[0122]

[0123] In the above formula, is the amount of electricity purchased from the power grid at time t, is the output power of distributed photovoltaic at time t, is the output power of the diesel generator at time t, is the discharge power of the energy storage system at time t, is the electricity load of buildings in the community at time t, is the charging power of the energy storage system at time t, is the power consumption of the HVAC system in the community at time t;

[0124] In the case of a sudden drop in PV power generation when the system load level is high, if the three buildings connected to the same bus in the distribution system only care about their own goals, the voltage at the common bus may be lower than the limit. Therefore, the common voltage must meet the following constraints:

[0125] v comm ≥ v ;

[0126] In the above formula, v comm is the voltage of the common bus, v is the lower limit of the voltage of the common bus;

[0127] Taking into account the complex relationship between different energy components as well as factors such as energy conversion efficiency, equipment utilization, and overall system stability, the energy management model structure of the community integrated energy system and the system constraints in scheduling are designed to ensure the effective allocation and utilization of community energy, promote environmental protection, and achieve the goal of sustainable development.

[0128] 2. For the weight coefficient setting of residential areas, commercial areas, and industrial areas, the importance of each part is judged by the flexible load dispatching cost, including:

[0129] The following formula is used to calculate the flexible load dispatching cost of each district:

[0130]

[0131] In the above formula, is the flexible load dispatching cost of power in area I, where I is a residential area, commercial area or industrial area. is the total compensation cost for load reduction in zone i, is the total compensation cost for movable loads in zone I, is the total compensation cost for the transferable load in Area I; cut is the compensation cost per unit load reduction, is the load reduction weight factor at time t, is the load-reducible power before load reduction, is the load-reducible power after load reduction; shift is the compensation cost per movable load, is a specific starting time point, t1 and t2 are the first and second moments respectively, is the weight factor of the transferable load at time t, is the power shifted at time t; trans is the compensation cost of transferring load at time t, tr- is the beginning of a specific time, tr+ is the end of a specific time, is the transferable load weight factor at time t, is the power delivered at time t;

[0132] Consider introducing a nonlinear model to more accurately reflect the actual dynamic relationship between cost and weight. Assuming that there are n flexible loads in the community, the flexible load dispatching cost of each area is normalized:

[0133]

[0134] In the above formula, is the normalized flexible load dispatching cost of power in area I, are the maximum and minimum values ​​of flexible load dispatching costs in all regions respectively;

[0135] Substitute the normalized cost into the S-curve and apply the Sigmoid nonlinear function to model the nonlinear relationship between the weight coefficient and the flexible load scheduling cost:

[0136]

[0137] In the above formula, λ I is the weight coefficient of each area, is the S-curve of normalized flexible load dispatching cost, n is the total number of zones, a is the parameter controlling the slope of the curve, x0 is the center position of the curve, The sum of the normalized inverse costs of all regions is used to ensure that the sum of all weight coefficients is 1.

[0138] 3. Use collaborative reinforcement learning algorithm to train the energy management model of the community integrated energy system and obtain the energy dispatch strategy of the community integrated energy system;

[0139] A customized multi-agent community integrated energy system heterogeneous multi-agent environment centered on the power system is created as environment E. The agent is an independent building i in each district. A multi-agent reinforcement learning algorithm with progressive task context is designed under environment E to continuously train the energy scheduling strategy in the task flow of the community integrated energy system, including:

[0140] Abstract the tasks under environment E into context information and build a model extractor h θ , with the trajectory of the agent in the task τ=(s0,…,s N ) is the input, s N The trajectory of strategy N, the extracted Gaussian distribution as contextual information;

[0141] The following formula is used to calculate the parts in the Gaussian distribution:

[0142]

[0143] In the above formula, μ θ (τ) is the mean of the context information extracted by the model, which can be regarded as the center point or central tendency of the context information, N is the number of strategies, μ t is the average energy demand at time t, σ t is the volatility of energy demand at time t, The variance of the context information extracted by the model indicates the degree of dispersion or dispersion of the data and measures the average distance between the data points and the mean.

[0144] Design a feature extractor that can share task knowledge. Each agent explores the environment with K trajectories, randomly selects a strategy head to calculate the context mean, and uses the following formula to select the relevant strategy head, that is, the most relevant energy scheduling strategy under the changing task environment:

[0145]

[0146] In the above formula, m ＊i is the most relevant strategy head, M is the number of strategy heads designed in N strategies, K is the number of trajectories, is the context mean of agent i on the kth trajectory, bs is the number of samples in each trajectory, is the mean of all sample feature extraction values ​​in strategy head m;

[0147] Through the multi-agent reinforcement learning algorithm based on progressive task context, the continuously occurring task flow in the community is continuously optimized, the energy scheduling strategy of the community integrated energy system is trained to minimize the objective function (i.e., minimize the total cost of energy consumption). While maximizing the satisfaction of community energy needs, the environmental friendliness and economy of energy production are achieved, which not only improves the efficiency and reliability of the community energy system, but also helps promote the utilization and popularization of renewable energy.

[0148] 4. Establish a decision tree model to collaboratively train the decision rules of energy scheduling strategy with reinforcement learning algorithm;

[0149] Create a buffer to collect behavior data of the agent when performing tasks in the environment using a collaborative reinforcement learning algorithm, wherein the behavior data includes current state data, action data taken, reward data received, and a new state quadruple converted to;

[0150] Based on the behavior data of the agent using the collaborative reinforcement learning algorithm to perform tasks in the environment, a decision tree model is constructed to extract the decision rules of the collaborative reinforcement learning algorithm to explain how to affect the decision-making process in reinforcement learning, guide the agent to transform the original environmental feature data into a state representation that is more conducive to learning and decision-making, and store this state representation as a new experience, including:

[0151] Analyze the information gain between different features, split the feature behavior data with information selected by the decision tree to reduce the uncertainty of the data, and understand the rules corresponding to each node by analyzing the structure of the decision tree and the branch conditions on the nodes, identify the behavior pattern of the agent and possible abnormal situations, so as to adjust the strategy in time and output the key feature vector of the model structure. The key feature vector contains rich information about the current environment. These feature vectors can be used as part of the state vector to extract the rules that affect the final decision;

[0152] According to the value range of the key feature vector output by the decision tree, the matching state action, reward, and new state quadruplets are screened, and the data that does not conform to experience is eliminated. The following formula is used to construct a more representative and effective state vector to represent the extracted rules that affect the final decision, thereby improving the generalization ability and robustness of the model:

[0153]

[0154] In the above formula, s is the state vector, n v is the feature of each node, V is the number of features, h′ v is the feature vector obtained from the decision tree.

[0155] 5. Design a deep Shapley additive interpretation method based on undersampling, and form an interpretable artificial intelligence framework with the decision tree model to analyze and extract important environmental features that affect decision-making, and understand the process of reinforcement learning decision-making, including:

[0156] Use the trained reinforcement learning model to perform a series of decision-making tasks, collect the state action data of the model when the current optimal strategy is executed as the environmental data set, and classify the data set into fault environmental state samples and normal environmental state samples according to whether the observed state in the power system is normal;

[0157] By minimizing the undersampling objective function, the faulty environment state samples and the normal environment state samples are balanced, the proportion of valid samples is increased, and the faulty environment state samples and the normal environment state samples with the highest purity are identified, so that more accurate Shapley addition values ​​can be obtained from fewer samples;

[0158] The undersampling objective function is:

[0159]

[0160] In the above formula, S f is the fault status data in the sample, S n is the normal state data in the sample, θ is the sampling weight of the fault sample, is the sampling weight of normal samples;

[0161] Select the best features from the highest purity fault environment state samples and normal environment state samples, use the following formula to calculate the average SHAP value of the best features in each environment state to indicate the importance of environment features in different environment states to decision making, and select the environment features with larger average SHAP values ​​as the important environment features that affect decision making:

[0162]

[0163] In the above formula, is the average SHAP value of the optimal feature, K is the total number of sample data, is the gradient of the predicted value relative to the i-th eigenvalue of the k-th sample data in the data set, is the change in the i-th eigenvalue of the k-th sample data during the iteration, Y k is the predicted value, that is, the action corresponding to the kth sample data state, is the i-th eigenvalue of the k-th sample data;

[0164] Through feature importance analysis based on undersampling, users can understand how the intelligent agent considers environmental factors when making decisions, explain which factors in the environment have a greater impact on the agent's decision-making, and analyze why the intelligent agent is more inclined to make corresponding decisions under certain conditions based on actual conditions, thereby improving learning efficiency and strategy quality.

[0165] 6. The decision rules extracted by the explainable artificial intelligence framework (i.e., the more representative and effective state vector data screened by the decision tree model) and the important environmental features that affect the decision (i.e., the environmental feature data with a large average SHAP value obtained by the under-sampling deep Shapley additive interpretation method) are used as new experiences for the collaborative reinforcement learning algorithm, and the energy management model of the community integrated energy system is trained again to update the decision rules of the collaborative reinforcement learning algorithm and the important environmental features that affect the decision;

[0166] In reinforcement learning, experience is referred to as a four-tuple data including state, action, reward, and new state. The state vector data extracted by the decision tree and the under-sampling deep SHAP additive interpretation method can identify the environmental features that have the greatest impact on the decision, so that these important environmental features can be integrated into the state representation of the collaborative reinforcement learning algorithm. Finally, more representative and effective states are selected in training and the corresponding actions, rewards, and new states are used as new experiences.

[0167] The decision-making process of deep reinforcement learning is explained through an explainable artificial intelligence framework. The interpretability of reinforcement learning enables system administrators and users to clearly understand the decision logic of the system, enhance their trust in the system behavior, improve the efficiency and reliability of community energy systems, and help promote the utilization and popularization of renewable energy.

[0168] 7. Determine whether the accuracy of the decision tree after the decision rule is updated reaches the set threshold, and whether the correlation between the important environmental features that affect the decision after the update is lower than the set value. If the accuracy of the decision tree after the update reaches the set threshold, and the correlation between the important environmental features that affect the decision after the update is lower than the set value, then output the optimal energy scheduling strategy of the community integrated energy system at this time; otherwise, return to step 6 for the next iteration;

[0169] The accuracy of the decision tree is measured by the accuracy index, which is calculated using the following formula:

[0170]

[0171] In the above formula, Acc is the accuracy index of the decision tree, TP is the true positive example, TN is the true negative example, FP is the false positive example, and FN is the false negative example;

[0172] The updated correlation between important environmental features that affect decision-making is calculated using the Pearson correlation coefficient:

[0173]

[0174] In the above formula, R is the correlation between the two environmental characteristics. When the correlation coefficient is 1, it means that the two environmental characteristics are completely positively correlated. When the correlation coefficient is -1, it means that the two environmental characteristics are completely negatively correlated. When the correlation coefficient is 0, it means that there is no linear correlation between the two environmental characteristics. n is the total number of samples, x is i is the value of environmental feature x at sample i, y i is the value of environmental feature y at sample i, are the means of environmental features x and y respectively.

[0175] Embodiment 2:

[0176] like Figure 3 As shown, a community comprehensive energy optimization scheduling system with interpretable reinforcement learning includes a model building module, a model training module, an intelligent framework creation module, a decision updating module, and a loop judgment module;

[0177] The model building module is used to establish an energy management model of the community integrated energy system based on the model architecture of the community integrated energy system and the Markov decision process, wherein the model architecture includes residential areas, commercial areas, industrial areas, photovoltaic power generation systems, energy storage systems and dispatchable resource systems;

[0178] The model training module is used to train the energy management model of the community integrated energy system using a collaborative reinforcement learning algorithm to obtain an energy dispatch strategy for the community integrated energy system;

[0179] The intelligent framework creation module is used to create an explainable artificial intelligence framework to extract the decision rules of the collaborative reinforcement learning algorithm to train the energy scheduling strategy and the important environmental features that affect the decision. The explainable artificial intelligence framework includes a decision tree model for extracting the algorithm decision rules and an under-sampled deep Shapley additive interpretation method for analyzing the important environmental features that affect the decision.

[0180] The decision updating module is used to use the extracted decision rules and important environmental features that affect the decision as new experience of the collaborative reinforcement learning algorithm, and to perform the operations of the model training module and the intelligent framework building module again to update the decision rules and important environmental features that affect the decision;

[0181] The loop judgment module is used to judge whether the accuracy of the decision tree after the decision rule is updated reaches the set threshold, and whether the correlation between the important environmental features that affect the decision after the update is lower than the set value. If the accuracy of the decision tree after the update reaches the set threshold, and the correlation between the important environmental features that affect the decision after the update is lower than the set value, the optimal energy scheduling strategy of the community integrated energy system obtained at this time is output, otherwise it returns to the decision update module for the next iteration.

[0182] The model construction module includes an objective function construction unit and a constraint condition construction unit;

[0183] The objective function construction unit is used to construct the following overall objective function of the energy management model of the community integrated energy system:

[0184] F = min(λ1F1+λ2F2+λ3F3);

[0185]

[0186] In the above formula, λ1 is the weight coefficient of the residential area, λ2 is the weight coefficient of the commercial area, λ3 is the weight coefficient of the industrial area, F1 is the objective function of the residential area, F2 is the objective function of the commercial area, and F3 is the objective function of the industrial area; T is the total time gap, t is the t moment in the time gap, I is the residential area, commercial area or industrial area, i is the i-th building in each area, The cost of purchasing electricity from the power grid, Energy cost for HVAC operation, is the loss cost of the energy storage device, Cost of diesel generators;

[0187] The constraint condition construction unit is used to construct the constraint conditions of the energy management model of the following community integrated energy system:

[0188]

[0189] vcomm ≥ v ;

[0190] In the above formula, is the amount of electricity purchased from the power grid at time t, is the output power of distributed photovoltaic at time t, is the output power of the diesel generator at time t, is the discharge power of the energy storage system at time t, is the electricity load of buildings in the community at time t, is the charging power of the energy storage system at time t, is the power consumption of the HVAC system in the community at time t; v comm is the voltage of the common bus, v It is the lower limit of the voltage of the common bus.

[0191] The weight coefficients of residential areas, commercial areas, and industrial areas in the objective function of the energy management model of the community integrated energy system are set using the flexible load dispatching cost, including:

[0192] A. Use the following formula to calculate the flexible load dispatching cost of each area:

[0193]

[0194] In the above formula, is the flexible load dispatching cost of power in area I, where I is a residential area, commercial area or industrial area. is the total compensation cost for load reduction in zone I, is the total compensation cost for movable loads in zone I, is the total compensation cost for the transferable load in Area I; cut is the compensation cost per unit load reduction, is the load reduction weight factor at time t, is the load-reducible power before load reduction, is the load-reducible power after load reduction; shift is the compensation cost per movable load, is a specific starting time point, t1 and t2 are the first and second moments respectively, is the weight factor of the transferable load at time t, is the power shifted at time t; trans is the compensation cost of transferring load at time t, tr- is the beginning of a specific time, tr+ is the end of a specific time, is the transferable load weight factor at time t, is the power delivered at time t;

[0195] B. Normalize the flexible load dispatching cost of each area, and use the following formula to express the relationship between the weight coefficient and the flexible load dispatching cost:

[0196]

[0197] In the above formula, λ I is the weight coefficient of each area, is the S-shaped curve of the normalized flexible load dispatching cost, n is the total number of zones, a is the parameter controlling the slope of the curve, and x0 is the center position of the curve.

[0198] The model training module includes an environment creation unit and a strategy training unit;

[0199] The environment creation unit is used to create a customized multi-agent community integrated energy system as the environment E, and the agent is an independent building i in each district;

[0200] The strategy training unit is used to design a multi-agent reinforcement learning algorithm with progressive task context in environment E, and continuously train the energy scheduling strategy in the task flow under the community integrated energy system, including a model extractor construction subunit and a feature extractor design subunit;

[0201] The model extractor construction subunit is used to abstract the tasks under the environment E into context information and construct a model extractor h θ , with the trajectory of the agent in the task with any strategy τ=(s0,...,s N ) is the input, S N The trajectory of strategy N, the extracted Gaussian distribution as contextual information;

[0202] The following formula is used to calculate the parts in the Gaussian distribution:

[0203]

[0204] In the above formula, μ θ (τ) is the mean of the context information extracted by the model, N is the number of strategies, μ t is the average energy demand at time t, σ t is the volatility of energy demand at time t, The variance of the context information extracted by the model;

[0205] The feature extractor design subunit is used to design a feature extractor that can share task knowledge. Each agent uses K trajectories to calculate the context mean and selects the relevant strategy head, that is, the most relevant energy scheduling strategy under the changing task environment, using the following formula:

[0206]

[0207] In the above formula, m ＊i is the most relevant strategy head, M is the number of strategy heads designed among n strategies, K is the number of trajectories, is the context mean of agent i on the kth trajectory, bs is the number of samples in each trajectory, is the mean of all sample feature extraction values ​​in strategy head m.

[0208] The intelligent framework creation module includes a data collection unit, a decision tree construction unit, and an important feature analysis unit;

[0209] The data collection unit is used to create a buffer to collect behavior data when the agent uses a collaborative reinforcement learning algorithm to perform tasks in the environment, wherein the behavior data includes current state data, action data taken, reward data received, and a new state quadruple converted to;

[0210] The decision tree construction unit is used to construct a decision tree model based on the behavior data of the agent when performing tasks in the environment using the collaborative reinforcement learning algorithm, and guide the agent to convert the original environmental feature data into a state representation that is more conducive to learning and decision-making, including a key feature output subunit and a state vector construction subunit;

[0211] The key feature output subunit is used to split the feature behavior data with information selected by the decision tree, and output the key feature vector of the model structure by analyzing the structure of the decision tree and the branch conditions on the nodes;

[0212] The state vector construction subunit is used to screen the matching state action, reward, and new state quadruple according to the value range of the key feature vector output by the decision tree, and construct the state vector using the following formula to represent the extracted rules that affect the final decision:

[0213]

[0214] In the above formula, s is the state vector, n v is the feature of each node, V is the number of features, h′ v is the feature vector obtained from the decision tree;

[0215] The important feature analysis unit is used to design a deep Shapley additive interpretation method based on undersampling, which forms an explainable artificial intelligence framework with the decision tree model to analyze and extract important environmental features that affect decision-making, including an environmental data collection subunit, a sample identification subunit, and an average SHAP value calculation subunit;

[0216] The environmental data collection subunit is used to use the trained energy management model of the community integrated energy system to perform a series of decision-making tasks, collect the state action data of the model when the current optimal strategy is executed as the environmental data set, and classify the data set into fault environment state samples and normal environment state samples according to whether the state observed in the power system is normal;

[0217] The sample identification subunit is used to balance the fault environment state samples and the normal environment state samples by minimizing the undersampling objective function, and identify the fault environment state samples and the normal environment state samples with the highest purity;

[0218] The undersampling objective function is:

[0219]

[0220] In the above formula, S f is the fault status data in the sample, S n is the normal state data in the sample, θ is the sampling weight of the fault sample, is the sampling weight of normal samples;

[0221] The average SHAP value calculation subunit is used to select the best features in the fault environment state samples and normal environment state samples with the highest purity, and the average SHAP value of the best features in each environment state is calculated using the following formula to indicate the importance of the environment features in different environment states to the decision, and the environment features with larger average SHAP values ​​are selected as the important environment features affecting the decision:

[0222]

[0223] In the above formula, is the average SHAP value of the optimal feature, K is the total number of sample data, is the gradient of the predicted value relative to the i-th eigenvalue of the k-th sample data in the data set, is the change in the i-th eigenvalue of the k-th sample data during the iteration, Y k is the predicted value, that is, the action corresponding to the kth sample data state, is the i-th eigenvalue of the k-th sample data.

Claims

1. A community comprehensive energy optimization scheduling method based on explainable reinforcement learning, characterized in that: The method comprises: S1. According to the model architecture of the community integrated energy system and based on the Markov decision process, an energy management model of the community integrated energy system is established, wherein the model architecture includes residential areas, commercial areas, industrial areas, photovoltaic power generation systems, energy storage systems and dispatchable resource systems; S2. Use a collaborative reinforcement learning algorithm to train the energy management model of the community integrated energy system to obtain the energy dispatch strategy of the community integrated energy system; S3. Create an explainable artificial intelligence framework to extract the decision rules and important environmental features that affect the decision of the collaborative reinforcement learning algorithm to train the energy scheduling strategy, wherein the explainable artificial intelligence framework includes a decision tree model for extracting the algorithm decision rules and an under-sampled deep Shapley additive interpretation method for analyzing the important environmental features that affect the decision; S4, taking the extracted decision rules and important environmental features that affect the decision as new experience of the collaborative reinforcement learning algorithm, and executing steps S2-S3 again to update the decision rules and important environmental features that affect the decision; S5. Determine whether the accuracy of the decision tree after the decision rule is updated reaches the set threshold, and whether the correlation between the important environmental features that affect the decision after the update is lower than the set value. If the accuracy of the decision tree after the update reaches the set threshold, and the correlation between the important environmental features that affect the decision after the update is lower than the set value, then output the optimal energy scheduling strategy for the community integrated energy system obtained at this time, otherwise return to S4 for the next iteration.

2. According to the method of community comprehensive energy optimization scheduling based on interpretable reinforcement learning according to claim 1, it is characterized in that: In S1, the overall objective function of the energy management model of the community integrated energy system is: F = min(λ1F1+λ2F2+λ3F3); In the above formula, λ1 is the weight coefficient of the residential area, λ2 is the weight coefficient of the commercial area, λ3 is the weight coefficient of the industrial area, F1 is the objective function of the residential area, F2 is the objective function of the commercial area, and F3 is the objective function of the industrial area; T is the total time gap, t is the t moment in the time gap, I is the residential area, commercial area or industrial area, i is the i-th building in each area, The cost of purchasing electricity from the power grid, Energy cost for HVAC operation, is the loss cost of the energy storage device, Cost of diesel generators; The constraints of the energy management model of the community integrated energy system include: v comm ≥ v ； In the above formula, is the amount of electricity purchased from the power grid at time t, is the output power of distributed photovoltaic at time t, is the output power of the diesel generator at time t, is the discharge power of the energy storage system at time t, is the electricity load of buildings in the community at time t, is the charging power of the energy storage system at time t, is the power consumption of the HVAC system in the community at time t; v comm is the voltage of the common bus, v It is the lower limit of the voltage of the common bus.

3. According to the method of community comprehensive energy optimization scheduling based on interpretable reinforcement learning according to claim 1, it is characterized in that: The S2 includes: S21, creating a customized multi-agent community integrated energy system as environment E, wherein the agent is an independent building i in each district; S22. Design a multi-agent reinforcement learning algorithm with progressive task context in environment E to continuously train the energy scheduling strategy in the task flow of the community integrated energy system, including: S221, abstract the tasks under environment E into context information and build a model extractor h θ , with the trajectory of the agent in the task τ=(s0,…,s N ) is the input, s N The trajectory of strategy N, the extracted Gaussian distribution as contextual information; The following formula is used to calculate the parts in the Gaussian distribution: In the above formula, μ θ (τ) is the mean of the context information extracted by the model, N is the number of strategies, μ t is the average energy demand at time t, σ t is the volatility of energy demand at time t, is the variance of the context information extracted by the model; S222. Design a feature extractor that can share task knowledge. Each agent uses K trajectories to calculate the context mean and uses the following formula to select the relevant strategy head, that is, the most relevant energy scheduling strategy under the changing task environment: In the above formula, m ＊i is the most relevant strategy head, M is the number of strategy heads designed in N strategies, K is the number of trajectories, is the context mean of agent i on the kth trajectory, bs is the number of samples in each trajectory, is the mean of all sample feature extraction values ​​in strategy head m.

4. According to the method of community comprehensive energy optimization scheduling based on interpretable reinforcement learning according to claim 1, it is characterized in that: The S3 includes: S31, creating a buffer to collect behavior data when the agent uses a collaborative reinforcement learning algorithm to perform tasks in the environment, wherein the behavior data includes current state data, action data taken, reward data received, and a new state quadruple converted to; S32. Based on the behavior data of the agent when performing tasks in the environment using the collaborative reinforcement learning algorithm, a decision tree model is constructed to guide the agent to transform the original environmental feature data into a state representation that is more conducive to learning and decision-making, including: S321, splitting the feature behavior data with information selected by the decision tree, and outputting the key feature vector of the model structure by analyzing the structure of the decision tree and the branch conditions on the nodes; S322. According to the value range of the key feature vector output by the decision tree, the four-tuple of state action, reward, and new state matching with it is screened, and the state vector is constructed using the following formula to represent the extracted decision rule: In the above formula, s is the state vector, n v is the feature of each node, V is the number of features, h′ v is the feature vector obtained from the decision tree; S33. Design a deep Shapley additive interpretation method based on undersampling, and form an interpretable artificial intelligence framework with the decision tree model to analyze and extract important environmental features that affect decision-making, including: S331. Use the trained energy management model of the community integrated energy system to perform a series of decision-making tasks, collect the state action data of the model when the current optimal strategy is executed as the environmental data set, and classify the data set into fault environmental state samples and normal environmental state samples according to whether the state observed in the power system is normal; S332, balancing the fault environment state samples and the normal environment state samples by minimizing the undersampling objective function, and identifying the fault environment state samples and the normal environment state samples with the highest purity; The undersampling objective function is: In the above formula, S f is the fault status data in the sample, S n is the normal state data in the sample, θ is the sampling weight of the fault sample, is the sampling weight of normal samples; S333. Select the best features from the fault environment state samples and the normal environment state samples with the highest purity, use the following formula to calculate the average SHAP value of the best features in each environment state to indicate the importance of the environment features in different environment states to the decision, and select the environment features with the largest average SHAP value as the important environment features that affect the decision: In the above formula, is the average SHAP value of the optimal feature, K is the total number of sample data, is the gradient of the predicted value relative to the i-th eigenvalue of the k-th sample data in the data set, is the change in the i-th eigenvalue of the k-th sample data during the iteration, Y k is the predicted value, that is, the action corresponding to the kth sample data state, is the i-th eigenvalue of the k-th sample data.

5. According to claim 2, a community comprehensive energy optimization scheduling method based on explainable reinforcement learning is characterized in that: The weight coefficients of residential areas, commercial areas, and industrial areas in the objective function of the energy management model of the community integrated energy system are set using the flexible load dispatching cost, including: A. Use the following formula to calculate the flexible load dispatching cost of each area: In the above formula, is the flexible load dispatching cost of power in area I, where I is a residential area, commercial area or industrial area. is the total compensation cost for load reduction in zone I, is the total compensation cost for movable loads in zone I, is the total compensation cost for the transferable load in Area I; cut is the compensation cost per unit load reduction, is the load reduction weight factor at time t, is the load-reducible power before load reduction, is the load-reducible power after load reduction; shift is the compensation cost per movable load, is a specific starting time point, t1 and t2 are the first and second moments respectively, is the transferable load weight factor at time t, is the power shifted at time t; trans is the compensation cost of transferring load at time t, tr- is the beginning of a specific time, tr+ is the end of a specific time, is the transferable load weight factor at time t, is the power delivered at time t; B. Normalize the flexible load dispatching cost of each area, and use the following formula to express the relationship between the weight coefficient and the flexible load dispatching cost: In the above formula, λ I is the weight coefficient of each area, is the S-curve of the normalized flexible load dispatching cost, n is the total number of zones, a is the parameter controlling the slope of the curve, and x o is the center position of the curve.

6. A community comprehensive energy optimization scheduling system based on explainable reinforcement learning, characterized in that: The system includes a model building module, a model training module, an intelligent framework creation module, a decision updating module, and a loop judgment module; The model building module is used to establish an energy management model of the community integrated energy system based on the model architecture of the community integrated energy system and the Markov decision process, wherein the model architecture includes residential areas, commercial areas, industrial areas, photovoltaic power generation systems, energy storage systems and dispatchable resource systems; The model training module is used to train the energy management model of the community integrated energy system using a collaborative reinforcement learning algorithm to obtain an energy dispatch strategy for the community integrated energy system; The intelligent framework creation module is used to create an explainable artificial intelligence framework to extract the decision rules of the collaborative reinforcement learning algorithm to train the energy scheduling strategy and the important environmental features that affect the decision. The explainable artificial intelligence framework includes a decision tree model for extracting the algorithm decision rules and an under-sampled deep Shapley additive interpretation method for analyzing the important environmental features that affect the decision. The decision updating module is used to use the extracted decision rules and important environmental features that affect the decision as new experience of the collaborative reinforcement learning algorithm, and to perform the operations of the model training module and the intelligent framework building module again to update the decision rules and important environmental features that affect the decision; The loop judgment module is used to judge whether the accuracy of the decision tree after the decision rule is updated reaches the set threshold, and whether the correlation between the important environmental features that affect the decision after the update is lower than the set value. If the accuracy of the decision tree after the update reaches the set threshold, and the correlation between the important environmental features that affect the decision after the update is lower than the set value, the optimal energy scheduling strategy of the community integrated energy system obtained at this time is output, otherwise it returns to the decision update module for the next iteration.

7. The community comprehensive energy optimization scheduling system based on interpretable reinforcement learning according to claim 6 is characterized in that: The model construction module includes an objective function construction unit and a constraint condition construction unit; The objective function construction unit is used to construct the following overall objective function of the energy management model of the community integrated energy system: F = min(λ1F1+λ2F2+λ3F3); In the above formula, λ1 is the weight coefficient of the residential area, λ2 is the weight coefficient of the commercial area, λ3 is the weight coefficient of the industrial area, F1 is the objective function of the residential area, F2 is the objective function of the commercial area, and F3 is the objective function of the industrial area; T is the total time gap, t is the t moment in the time gap, I is the residential area, commercial area or industrial area, i is the i-th building in each area, The cost of purchasing electricity from the power grid, Energy cost for HVAC operation, is the loss cost of the energy storage device, Cost of diesel generators; The constraint condition construction unit is used to construct the constraint conditions of the energy management model of the following community integrated energy system: v comm ≥ v ； In the above formula, is the amount of electricity purchased from the power grid at time t, is the output power of distributed photovoltaic at time t, is the output power of the diesel generator at time t, is the discharge power of the energy storage system at time t, is the electricity load of buildings in the community at time t, is the charging power of the energy storage system at time t, is the power consumption of the HVAC system in the community at time t; v comm is the voltage of the common bus, v It is the lower limit of the voltage of the common bus.

8. The community comprehensive energy optimization scheduling system based on interpretable reinforcement learning according to claim 6 is characterized in that: The model training module includes an environment creation unit and a strategy training unit; The environment creation unit is used to create a customized multi-agent community integrated energy system as the environment E, and the agent is an independent building i in each district; The strategy training unit is used to design a multi-agent reinforcement learning algorithm with progressive task context in environment E, and continuously train the energy scheduling strategy in the task flow under the community integrated energy system, including a model extractor construction subunit and a feature extractor design subunit; The model extractor construction subunit is used to abstract the tasks under the environment E into context information and construct a model extractor h θ , with the trajectory of the agent in the task τ=(s0,…,s N ) is the input, s N The trajectory of strategy N, the extracted Gaussian distribution as contextual information; The following formula is used to calculate the parts in the Gaussian distribution: In the above formula, μ θ (τ) is the mean of the context information extracted by the model, N is the number of strategies, μ t is the average energy demand at time t, σ t is the volatility of energy demand at time t, is the variance of the context information extracted by the model; The feature extractor design subunit is used to design a feature extractor that can share task knowledge. Each agent uses K trajectories to calculate the context mean and selects the relevant strategy head, that is, the most relevant energy scheduling strategy under the changing task environment, using the following formula: In the above formula, m ＊i is the most relevant strategy head, M is the number of strategy heads designed in N strategies, M is the number of trajectories, is the context mean of agent i on the kth trajectory, bs is the number of samples in each trajectory, is the mean of all sample feature extraction values ​​in strategy head m.

9. The community comprehensive energy optimization scheduling system based on interpretable reinforcement learning according to claim 6 is characterized in that: The intelligent framework creation module includes a data collection unit, a decision tree construction unit, and an important feature analysis unit; The data collection unit is used to create a buffer to collect behavior data when the agent uses a collaborative reinforcement learning algorithm to perform tasks in the environment, wherein the behavior data includes current state data, action data taken, reward data received, and a new state quadruple converted to; The decision tree construction unit is used to construct a decision tree model based on the behavior data of the agent when performing tasks in the environment using the collaborative reinforcement learning algorithm, and guide the agent to convert the original environmental feature data into a state representation that is more conducive to learning and decision-making, including a key feature output subunit and a state vector construction subunit; The key feature output subunit is used to split the feature behavior data with information content selected by the decision tree, and output the key feature vector of the model structure by analyzing the structure of the decision tree and the branch conditions on the nodes; The state vector construction subunit is used to screen the matching state action, reward, and new state quadruple according to the value range of the key feature vector output by the decision tree, and construct the state vector using the following formula to represent the extracted rules that affect the final decision: In the above formula, s is the state vector, n v is the feature of each node, V is the number of features, h′ v is the feature vector obtained from the decision tree; The important feature analysis unit is used to design a deep Shapley additive interpretation method based on undersampling, which forms an explainable artificial intelligence framework with the decision tree model to analyze and extract important environmental features that affect decision-making, including an environmental data collection subunit, a sample identification subunit, and an average SHAP value calculation subunit; The environmental data collection subunit is used to use the trained energy management model of the community integrated energy system to perform a series of decision-making tasks, collect the state action data of the model when the current optimal strategy is executed as the environmental data set, and classify the data set into fault environment state samples and normal environment state samples according to whether the state observed in the power system is normal; The sample identification subunit is used to balance the fault environment state samples and the normal environment state samples by minimizing the undersampling objective function, and identify the fault environment state samples and the normal environment state samples with the highest purity; The undersampling objective function is: In the above formula, S f is the fault status data in the sample, S n is the normal state data in the sample, θ is the sampling weight of the fault sample, is the sampling weight of normal samples; The average SHAP value calculation subunit is used to select the best features in the fault environment state samples and normal environment state samples with the highest purity, and the average SHAP value of the best features in each environment state is calculated using the following formula to indicate the importance of the environment features in different environment states to the decision, and the environment features with larger average SHAP values ​​are selected as the important environment features affecting the decision: In the above formula, is the average SHAP value of the optimal feature, K is the total number of sample data, is the gradient of the predicted value relative to the i-th eigenvalue of the k-th sample data in the data set, is the change in the i-th eigenvalue of the k-th sample data during the iteration, Y k is the predicted value, that is, the action corresponding to the kth sample data state, is the i-th eigenvalue of the k-th sample data.

10. The community comprehensive energy optimization scheduling system based on interpretable reinforcement learning according to claim 7 is characterized in that: The weight coefficients of residential areas, commercial areas, and industrial areas in the objective function of the energy management model of the community integrated energy system are set using the flexible load dispatching cost, including: A. Use the following formula to calculate the flexible load dispatching cost of each area: In the above formula, is the flexible load dispatching cost of power in area I, where I is a residential area, commercial area or industrial area. is the total compensation cost for load reduction in zone I, is the total compensation cost for movable loads in zone I, is the total compensation cost for the transferable load in Area I; X cut is the compensation cost per unit load reduction, is the load reduction weight factor at time t, is the load-reducible power before load reduction, is the load-reducible power after load reduction; shift is the compensation cost per movable load, is a specific starting time point, t1 and t2 are the first and second moments respectively, is the weight factor of the transferable load at time t, is the power shifted at time t; trans is the compensation cost of transferring load at time t, tr- is the beginning of a specific time, tr+ is the end of a specific time, is the transferable load weight factor at time t, is the power delivered at time t; B. Normalize the flexible load dispatching cost of each area, and use the following formula to express the relationship between the weight coefficient and the flexible load dispatching cost: In the above formula, λ I is the weight coefficient of each area, is the S-shaped curve of the normalized flexible load dispatching cost, n is the total number of zones, a is the parameter controlling the slope of the curve, and x0 is the center position of the curve.

Images