Intelligent business building cluster collaborative operation method based on improved reinforcement learning

Abstract

The invention relates to a smart business building cluster collaborative operation method based on improved reinforcement learning, and the method comprises the following steps: collecting the operation data of a smart business building cluster participating in a carbon transaction, the smart business building cluster comprising a source side device, a load side device and a storage side device; based on the operation data, considering load, output and a carbon transaction mechanism, and establishing an energy management model of the smart business building; based on the energy management model, an improved reinforcement learning algorithm is adopted for solving, cooperative operation is carried out according to a solving result, in the improved reinforcement learning algorithm, each source side device, each load side device and each storage side device serve as an intelligent agent, and each intelligent agent interacts with the environment under a game mechanism. Compared with the prior art, the system has the advantages of high energy utilization efficiency, capability of meeting dynamic load requirements and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of commercial buildings, and in particular to a collaborative operation method of a smart commercial building cluster based on improved reinforcement learning. Background Art

[0002] As the world pays more attention to sustainable development, commercial buildings, as the most important typical users in urban public activities, are increasingly concerned about energy management and carbon emissions. Commercial buildings consume a lot of electricity in the process, and also generate considerable regenerative braking energy. How to effectively utilize this regenerative braking energy, improve energy efficiency, and reduce operating costs has become a technical problem that needs to be solved urgently. At the same time, the introduction of the carbon trading mechanism provides a new idea for energy management in commercial buildings, promoting low-carbon operations through economic incentives.

[0003] The existing implementation schemes that are most similar to the present invention mainly include smart building energy management systems based on virtual power plant technology. These systems achieve coordinated scheduling and energy optimization configuration among buildings by integrating the hybrid energy storage and electric vehicle charging loads of each building. In addition, existing studies have explored how to combine carbon trading mechanisms with energy management of building rooftop photovoltaics and air conditioners to maximize economic and environmental benefits by dynamically adjusting energy use strategies.

[0004] Distributed energy in commercial buildings mainly includes distributed power sources and distributed loads. Distributed power sources are composed of photovoltaics and wind turbines (mainly considering rooftop photovoltaic power generation), and distributed loads mainly consider electric vehicle charging as loads. Commercial buildings involve sensitive energy data and equipment control, and must pay attention to issues such as security and privacy. Kebriaei et al. used the Q-learning algorithm to simulate the decision-making behavior of power generators under incomplete information. In the game process, power generators do not rely on their opponents' profit functions or relevant information about the market clearing mechanism to participate in decision-making. Second, they choose action strategies based on historical data or their own experience. Therefore, their game model is closer to the competition process between power plants in reality.

[0005] However, the existing technologies in energy management of smart commercial buildings lack effective load and output coordination, resulting in inefficient energy utilization. Traditional energy storage systems (such as flywheels and batteries) have limitations in short-term and long-term storage and cannot meet the dynamic needs of smart commercial buildings. Summary of the invention

[0006] The purpose of the present invention is to provide a collaborative operation method of a smart commercial building cluster based on improved reinforcement learning to improve energy utilization efficiency.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A method for collaborative operation of a smart commercial building cluster based on improved reinforcement learning, comprising the following steps:

[0009] Collecting operational data of smart business building clusters participating in carbon trading, wherein the smart business building clusters include source measurement, load-side and storage-side equipment;

[0010] Based on the operation data, an energy management model of a smart commercial building is established taking into account load and output as well as a carbon trading mechanism;

[0011] Based on the energy management model, an improved reinforcement learning algorithm is used to solve the problem, and collaborative operation is performed according to the solution results. In the improved reinforcement learning algorithm, each source measurement, load side and storage side equipment is regarded as an intelligent agent, and each intelligent agent interacts with the environment under a game mechanism.

[0012] Furthermore, the energy management model aims to minimize the operating cost of the smart commercial building. The energy management model includes a cost objective function and corresponding constraints, wherein the expression of the cost objective function is:

[0013] C total =C T +C S

[0014] In the formula, C total is the total operating cost, C T is the carbon trading cost, C S The cost of purchasing and selling electricity;

[0015] The constraints include:

[0016] 1) Load constraints:

[0017]

[0018] Where P pv，i，t is the output of distributed unit photovoltaic at time t, N pv is the number of photovoltaic cells, P ev，i，t is the output of electric vehicles participating in the operation of smart business building clusters at time t, N ev is the number of electric vehicles, P hess，i，t is the hybrid energy storage output at time t in the station; N hess is the amount of hybrid energy storage, P market，t represents the power purchased by the smart business building cluster at time t, P load，i，t Refers to the total site load at time t, including air conditioning load and various flexible loads, N load is the load quantity;

[0019] 2) Photovoltaic output constraints:

[0020] P pv，i，t ≤P pvmax，i，t

[0021] Where P pvmax，i，t The maximum photovoltaic output corresponds to time t;

[0022] 3) Hybrid energy storage output constraints:

[0023] P hess，i，t ≤P hess，max

[0024] Where P hess，max It is the upper limit of the hybrid energy storage output within the station;

[0025] 4) Charge constraints of hybrid energy storage:

[0026] SOC min ≤SOC i，t ≤SOC max

[0027] In the formula, SOC min is the minimum value of hybrid energy storage charge, SOC max is the maximum value of the hybrid energy storage charge; SOC i，t is the mixed energy storage charge at time t;

[0028] 5) Electric vehicle output constraints:

[0029] P ev，i，t ≤P ev，max

[0030] Where P ev，max It is the upper limit of the output of electric vehicles participating in the operation of smart commercial building clusters;

[0031] 6) Constraints of smart business building clusters:

[0032] P train，i，t ≤P train max，i，t

[0033] Where Ptrainmax,i,t is the maximum value of the smart business building cluster load at time t.

[0034] Furthermore, the carbon trading cost is calculated by determining the total carbon emissions of different loads or outputs in the smart business building cluster. The calculation expression of the carbon trading cost is:

[0035] C T =λ CP [∑(f cbuy +f n-carbon )]

[0036]

[0037] In the formula, C T is the total carbon transaction cost of the virtual power plant, λ CP is the carbon trading price coefficient, f cbuy is the carbon emissions from electricity purchase, f n-ctarbon represents the carbon trading cost, F buy is the unit carbon emission factor, P gbuy is the amount of electricity purchased from the power grid, is the carbon trading quota per unit of electricity, P pv is the photovoltaic output, and T is the dispatching time.

[0038] Furthermore, the electricity purchase and sale cost includes electricity purchase cost and electricity sale income, wherein the calculation expression of the electricity purchase and sale cost is:

[0039] C S =C bs +C ss

[0040] In the formula, C ss is the electricity purchase cost, C ss For income from electricity sales;

[0041] The calculation expression of the electricity purchase cost is:

[0042]

[0043] In the formula, T is the scheduling time, C buy (t) is the unit price of electricity, P buy (t) is the amount of electricity purchased;

[0044] The calculation expression of the electricity sales revenue is:

[0045]

[0046] In the formula, C sell (t) is the unit price of electricity, P sell (t) is the electricity sales amount.

[0047] Furthermore, the method further includes the following steps: before solving the problem using the improved reinforcement learning algorithm, the energy management model is transformed using an embedded augmented Lagrangian function method to convert the constrained optimization problem into an unconstrained optimization problem, specifically including the following steps:

[0048] Construct the augmented Lagrangian function:

[0049] The augmented Lagrangian multiplier is introduced to construct the augmented Lagrangian function, and the expression of the augmented Lagrangian function is:

[0050] L=f(x)-λh(x)

[0051] Where L is the augmented Lagrangian function, f(·) is the objective function of the energy management model, λ is the augmented Lagrangian multiplier, h(·) is the constraint of the energy management model, and x is the decision-making independent variable of the smart business building cluster;

[0052] Compute the first-order partial derivative and set it to 0:

[0053] For the augmented Lagrangian function, the first-order partial derivatives of the decision variable x and the augmented Lagrangian multiplier λ are calculated and set to 0, and the result is:

[0054]

[0055] Modify the system of equations and solve:

[0056] According to the calculation results of the first-order partial derivative, the equation group of the optimality condition is obtained, and the following formula is solved by back substitution method:

[0057]

[0058] Where H is the Hessian matrix, is the transpose of the gradient of the constraint function, i.e., the row vector, is the gradient of the constraint function with respect to the variable x, Δx is the change in the variable x, Δλ is the Lagrange multiplier, and L x is the objective function value, L λ is the partial derivative of the Lagrangian function with respect to λ. According to the solution of the equations, the approximate optimal solution at the kth iteration is obtained, which is expressed as:

[0059]

[0060] In the formula, x (k+1) is the parameter vector in the k+11th iteration, x (k) is the parameter vector in the kth iteration, p (k) is the learning rate in the kth iteration, is the gradient of function f with respect to parameter x;

[0061] The solution is solved through continuous iteration until the convergence condition is met.

[0062] Furthermore, the expression of the decision variable x of the smart business building cluster is:

[0063]

[0064] In the formula, x is the decision-making independent variable of the smart business building cluster, Q vpp , Q se , Q pv and Qfl are the decision variables of the smart business building virtual power plant cluster, supercapacitor, photovoltaic and flywheel energy storage, Q i The subscript i in the table is vpp, se, pv, train, in, out, and fl, respectively. i , b i With c i Represents the eigenvalues ​​of the profit function of different objects, q i is the characteristic variable.

[0065] Furthermore, the improved reinforcement learning algorithm is an improved Q-learning algorithm.

[0066] Furthermore, the step of solving the problem using the improved reinforcement learning algorithm includes:

[0067] 1) Build a reinforcement learning environment for each agent, define action rewards, and design reward function R(x, q) i , Q i ) to approximately reflect the competition or cooperation relationship between the intelligent agents of different source measurement, load side and storage side equipment, where x is the decision-making independent variable of the smart business building cluster, Q i Represents the eigenvalues ​​of the profit function of different objects, q i is the characteristic variable;

[0068] 2) Generate a Q table for each agent and set the number of iterations;

[0069] 3) Generate a random number e, set the exploration probability ε of the greedy strategy, and determine whether e is greater than ε. If so, the agent makes a random action with a probability of ε and expands the Q table. If not, the agent selects the current optimal action strategy according to the Q table for execution, where the random number e is:

[0070] e∈[0,1]

[0071] In the formula, e is a random number;

[0072] 4) The building virtual power plant cluster submits power generation information to the external power grid and settles the revenue;

[0073] 5) The environment updates the state according to the agent's execution actions and provides instant rewards to each agent. Each agent calculates the Q value based on the received reward and the new state and updates the Q table;

[0074] 6) Repeat steps 3)-5) for iterative learning until the maximum number of iterations is reached.

[0075] Furthermore, the current optimal action strategy is:

[0076]

[0077] In the formula, A i (t) is the action strategy at the i-th iteration at time t, a is the current action, s is the current state, and Q(·) is the Q function.

[0078] Furthermore, the update formula of the Q value in the Q table is:

[0079]

[0080] Where a is the current action, s is the current state, Q(·) is the Q function, α is the learning rate, γ is the discount factor, a′ is the next action, s′ is the next state, and R(·) is the reward function.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] (1) The present invention starts from the perspective of energy management of smart business buildings, according to the characteristics of smart business buildings participating in carbon trading, and taking into account the operation of source measurement, load side and storage side inside the smart business building, an energy management model of the smart business building is established, and the solution is obtained by improving the reinforcement learning algorithm. During the solution process, each intelligent agent interacts with the environment under the game mechanism, continuously seeks the optimal solution, improves the coordination of energy management of source measurement, load side and storage side, and improves energy utilization efficiency.

[0083] (2) The present invention uses the augmented Lagrangian function method to convert the complex constraint problems existing in smart buildings into equivalent unconstrained problems, which not only improves the ability of the reinforcement learning algorithm to solve problems, but also improves its adaptability. Combining it with the improved reinforcement learning algorithm provides a new solution path for the game decision-making problem in the improved reinforcement learning algorithm, overcomes the limitations of the improved reinforcement learning algorithm in dealing with constraint problems, and ensures that each iterative solution can more accurately approach the global optimal solution, and ultimately further improves the coordination of energy management in smart business buildings and improves energy utilization efficiency.

[0084] (3) The energy management model of the present invention aims to minimize the operating costs of smart commercial buildings and achieve a dual improvement in energy coordination and operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0086] Figure 2 It is the agent-environment interaction diagram in the Q-learning reinforcement learning algorithm of the present invention;

[0087] Figure 3 It is a solution flow chart of the Q-learning reinforcement learning algorithm of the present invention. DETAILED DESCRIPTION

[0088] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0089] Example 1

[0090] This embodiment provides a method for collaborative operation of smart commercial building clusters based on improved reinforcement learning. By establishing a comprehensive load modeling system, considering the energy consumption characteristics of elevators, escalators and other equipment, and combining the dynamic changes of electric vehicle charging and swapping characteristics, accurate energy demand forecasting is performed. This will provide important reference data for the energy management system and optimize energy distribution and scheduling strategies. Figure 1 As shown, the method comprises the following steps:

[0091] 1. Collect operating data of smart business building clusters

[0092] Under the carbon trading mechanism, energy consumption data and carbon emission data of different users participating in carbon trading are collected.

[0093] 2. Establish an energy management model for smart commercial buildings

[0094] (1) Building a carbon emissions trading model based on operating data

[0095] This embodiment establishes a carbon emission trading model by determining the total carbon emissions of different loads or outputs of a smart business building cluster and calculating the economic cost based on the carbon emissions.

[0096]

[0097] The formula for purchasing electricity from the grid can be seen in (1), where f cbuy is the carbon emissions from electricity purchase, F buy is the unit carbon emission factor, P gbuy The amount of electricity purchased from the power grid.

[0098]

[0099] For photovoltaics, as a new energy source, it does not need to consider carbon emissions. The statistical formula is shown in 2-19, where f n-carbon represents its carbon trading cost, is the carbon trading quota per unit of electricity, P pv Power for photovoltaics.

[0100] C T =λ CP [∑(f cbuy +f n-carbon )] (3)

[0101] The total calculation formula of carbon trading cost is shown in formula (3), where CT is the total carbon trading cost of the virtual power plant, λ CP is the carbon trading price coefficient.

[0102] (2) Electricity purchasing and selling cost model for smart commercial building clusters

[0103] By combining the above-mentioned distributed loads and outputs, smart business building clusters have the ability to adjust distributed devices within a certain range. By combining with the carbon trading mechanism, smart business building clusters can participate in more scenarios based on the purchase and sale of electricity, which can enhance the smart business building clusters' absorption of new energy to a certain extent.

[0104] C S =C bs +C ss (4)

[0105] The electricity purchase and sales cost model includes electricity purchase cost and electricity sales revenue, where C S The cost of purchasing and selling electricity.

[0106]

[0107] The electricity purchase cost of the smart business building cluster is shown in formula (5), where C buy (t) is the unit price of electricity, in yuan / kW·h, P buy (t) is the amount of electricity purchased, in (kW·h).

[0108]

[0109] The electricity sales revenue of the smart business building cluster is shown in formula (6), where C sell (t) is the unit price of electricity sold, in (yuan / kW·h), P sell (t) is the electricity sales amount, unit is (kW·h).

[0110] Combining the carbon emission trading model and the electricity purchase and sales cost model, we get the energy management model of smart commercial buildings:

[0111] C total =C T +C S (7)

[0112] In the formula, C total is the total operating cost, C T is the carbon trading cost, C S The cost of purchasing and selling electricity.

[0113] (3) Energy management constraints

[0114] The core of energy management of smart business building clusters is to achieve the optimal use of energy during operation. By comprehensively considering the capacity limitations of the system, the power status of energy storage components, and site characteristics, the operating costs of smart business building clusters can be effectively reduced, and carbon emissions can be reduced. When considering electric vehicles as a collaborative operation in the energy management process of smart business building clusters, it is necessary to combine factors such as electricity-carbon coupling and time-of-use electricity prices for analysis, and participate in the operation of smart business building clusters as demand-side output or load.

[0115]

[0116] The energy management in the smart commercial building cluster integrates multiple types of loads and outputs, and is analyzed with 1 hour as the minimum time scale, as shown in formula (8), where P pv，i，t is the output of the distributed unit photovoltaic at time t, P ev，i,t is the output of electric vehicles participating in the operation of smart business building clusters at time t. At the same time, electric vehicles can also participate in the operation as loads. P hess，i，t is the hybrid energy storage output at time t in the station, P market，t represents the power purchased by the smart business building cluster at time t, P load，i,t Refers to the total site load at time t, including air-conditioning load and various flexible loads.

[0117] The photovoltaic output constraint is shown in formula (9), where P pvmax，i，t The corresponding time is the maximum photovoltaic output.

[0118] P pv，i，t ≤P pvmax，i，t (9)

[0119] The hybrid energy storage output constraint is shown in formula (10), where Phess,max is the maximum hybrid energy storage output corresponding to that time.

[0120] P hess，i，t ≤P hess，max (10)

[0121] The charge constraint of hybrid energy storage is shown in formula (11), where SOC min SOC is the minimum charge of the hybrid energy storage, and SOCmax is the maximum charge of the hybrid energy storage.

[0122] SOC min ≤SOC i，t ≤SOC max (11)

[0123] The output constraint of electric vehicles is shown in formula (12), where P ev，max It is the maximum output value of the electric vehicle corresponding to that time.

[0124] Pev，i，t ≤P ev，max (12)

[0125] The constraints of smart business building clusters are shown in formula (13), where P load_max，t The maximum value of the smart business building cluster load at that time.

[0126] P load，i，t ≤P load_max，t (13)

[0127] The optimized scheduling results obtained through game operation are used as a reference. Combined with the constraints of the output and load of the given smart business building cluster, the energy management of the smart business building cluster can be carried out on a time scale of 1 hour, and the energy management analysis can be carried out in multiple scenarios in combination with the carbon trading mechanism.

[0128] 3. Q-learning reinforcement learning algorithm solution

[0129] In the framework of game theory solution strategies for multi-agent systems, reinforcement learning shows its unique adaptability and effectiveness in scenarios with incomplete information. The modeling process of this algorithm consists of two parts: agent and environment. Figure 2 As shown in the figure, the agent is responsible for collecting the information state provided by the environment and taking exploratory random actions in the presence of any "game rules"; the environment provides corresponding rewards based on the agent's behavior. In this game mechanism, the agent explores and searches for the action with the maximum reward through continuous trial and error and iteration. In this embodiment, each source measurement, load side, and storage side device is regarded as an agent.

[0130] Greedy strategy generation in Q-learning reinforcement learning algorithm:

[0131] e∈[0,1] (14)

[0132] Optimal action strategy selection in Q-learning reinforcement learning algorithm:

[0133]

[0134] The Q table in the Q-learning reinforcement learning algorithm is updated as follows:

[0135]

[0136] Figure 3A deep Q-learning-based agent is implemented for training and optimization. By combining image analysis, it is concluded that the agent does not have direct data contact with the training subject, but is implemented in a random action situation through multiple training iterations, and learns the strategy of selecting the best action in different states, and achieves the goals of power adjustment and cost optimization by maximizing the cumulative reward. During the training process, the agent continuously updates the Q table through interaction with the environment, and selects the best action based on the Q value.

[0137] In view of the complex constraint problems existing in smart buildings, it is proposed to improve the Q-1earning algorithm by using the embedded augmented Lagrangian function method. The constraint terms in the original optimization problem are converted into additional penalty functions, which are combined with the objective function to become augmented Lagrangian functions, and the constrained optimization problem is converted into an equivalent unconstrained problem. On the one hand, it can improve the function solving ability, and on the other hand, it can improve adaptability. This strategy is effectively integrated into the Q-learning algorithm, providing a new solution path for dealing with power market game decision-making problems. It overcomes the limitations of Q-1earning in dealing with constraint problems. This ensures that each iterative solution can more accurately approach the global optimal solution.

[0138] Decision-making independent variable x of smart business building cluster:

[0139]

[0140] The parameters in formula (1) are the decision-making independent variables within the smart business building cluster. The profit function of the internal unit of the virtual power plant is modeled as follows, where the characteristic value a of the profit function of different objects is i , b i With c i It depends on their respective characteristics.

[0141] The constructed Lagrangian function is shown in (18), where λ is the Lagrangian multiplier.

[0142] L=f(x)-λh(x) (18)

[0143] As shown in formula (19), the first-order partial derivative of the decision variable x is zero and the first-order partial derivative of the Lagrangian function is zero, and the local optimal solution of a single variable is obtained.

[0144]

[0145] According to the calculation results of the first-order partial derivative, the optimality condition equation system is obtained. The matrix form of the equation system is modified and the equation (20) is solved by back substitution:

[0146]

[0147] As shown in formula (21), the optimal solution for the kth iteration is approximately expressed as follows:

[0148]

[0149] According to the transformation of the above complex constraint problem, Figure 3 As shown, the Q-learning solution process is:

[0150] 1) Build a reinforcement learning environment for each agent, define action rewards, and design reward function R(x, q) i , Q i ) to approximately reflect the competition or cooperation relationship between the intelligent agents of different source, load and storage devices;

[0151] 2) Generate a Q table for each agent and set the number of iterations;

[0152] 3) Generate a random number e, set the exploration probability ε of the greedy strategy, and determine whether e is greater than ε. If so, the agent makes a random action with a probability of ε and expands the Q table. If not, the agent selects the current optimal action strategy according to the Q table for execution, where the current optimal action strategy is formula (15);

[0153] 4) Submit electricity consumption and settle revenue;

[0154] 5) The environment updates the state according to the agent's execution action and provides an immediate reward to each agent. Each agent calculates the Q value based on the received reward and the new state and updates the Q table, which is updated according to formula (16);

[0155] 6) Repeat steps 3)-5) for iterative learning until the maximum number of iterations is reached.

[0156] In the process of Q-learning strategy iteration, it is necessary to make behavioral decisions on the agent and define its strategy space to ensure that its behavior is within the set upper and lower limits. The strategic game behavior of multiple power market participants is modeled and solved. Each participant is regarded as an autonomous agent. The specific actions taken by the agent are reflected in the strategic choice of power production. Countermeasures are formed by collecting market competition status information.

[0157] In summary, smart business buildings mainly integrate hybrid energy storage and electric vehicle charging and swapping information from various business buildings, and use each smart business building as a dispatchable node to determine whether the overall energy optimization configuration can be achieved through coordinated dispatch between buildings. If effective dispatch cannot be achieved, after prioritizing energy allocation between different buildings, the overall energy surplus or shortage information in the region is summarized and uploaded to the wide-area demand matching center by the central control platform for coordination. Considering the dynamic characteristics of electricity, a dynamic economic dispatch model for business buildings is established, with energy prices, energy purchased from the wide-area demand matching center, energy interaction with smart business buildings, and carbon trading costs as decision variables to meet the load demand and energy storage constraints of each building. The goal is to minimize the total operating cost of the smart business building cluster, including the charging and discharging costs of hybrid energy storage, the operating costs of rooftop photovoltaic units, and the costs of energy interaction with buildings, while effectively participating in the carbon trading mechanism to optimize economic benefits.

[0158] Example 2

[0159] This embodiment provides an electronic device, including one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the smart business building cluster collaborative operation method based on improved reinforcement learning as described in Example 1.

[0160] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0161] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for collaborative operation of smart commercial building clusters based on improved reinforcement learning, characterized in that: The following steps are involved: Collecting operational data of smart business building clusters participating in carbon trading, wherein the smart business building clusters include source measurement, load-side and storage-side equipment; Based on the operation data, an energy management model of a smart commercial building is established taking into account load and output as well as a carbon trading mechanism; Based on the energy management model, an improved reinforcement learning algorithm is used to solve the problem, and collaborative operation is performed according to the solution results, wherein the solution results are energy scheduling plans for source measurement, load side and storage side equipment. In the improved reinforcement learning algorithm, each source measurement, load side and storage side equipment is regarded as an intelligent agent, and each intelligent agent interacts with the environment under a game mechanism.

2. According to claim 1, a method for collaborative operation of smart commercial building clusters based on improved reinforcement learning, characterized in that: The energy management model aims to minimize the operating cost of smart commercial buildings. The energy management model includes a cost objective function and corresponding constraints, wherein the expression of the cost objective function is: C total =C T +C S In the formula, C totat is the total operating cost, C T is the carbon trading cost, C S Cost of purchasing and selling electricity; The constraints include: 1) Load constraints: Where P pv,i,t is the output of distributed unit photovoltaic at time t, N pv is the number of photovoltaic cells, P ev,i,t is the output of electric vehicles participating in the operation of smart business building clusters at time t, N ev is the number of electric vehicles, P hess,i,t is the hybrid energy storage output at time t in the station; N hess is the amount of hybrid energy storage, P market,t represents the power purchased by the smart business building cluster at time t, P load，i，t Refers to the total building load at time t, including air conditioning load and various flexible loads, N load is the load quantity; 2) Photovoltaic output constraints: P pv，i，t ≤P pvmax,i,t Where P pvmax,i,t The maximum photovoltaic output corresponds to time t; 3) Hybrid energy storage output constraints: P hess,i,t ≤P hess,max Where P hess,max It is the upper limit of the hybrid energy storage output within the station; 4) Charge constraints of hybrid energy storage: SOC min ≤SOC i,t ≤SOC max In the formula, SOC min is the minimum value of hybrid energy storage charge, SOC max is the maximum value of the hybrid energy storage charge; SOC i，t is the mixed energy storage charge at time t; 5) Electric vehicle output constraints: P ev,i,t ≤P ev,max Where P ev，max It is the upper limit of the output of electric vehicles participating in the operation of smart commercial building clusters; 6) Constraints of smart business building clusters: P load,i,t ≤P load_max,t Where P load_max,t is the maximum value of the smart business building cluster load at time t.

3. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 2, characterized in that: The carbon trading cost is calculated by determining the total carbon emissions of different loads or outputs in the smart business building cluster. The calculation expression of the carbon trading cost is: C T =λ CP [∑(f cbuy +f n-carbon )] In the formula, C T is the total carbon transaction cost of the virtual power plant, λ CP is the carbon trading price coefficient, f cbuy is the carbon emissions from electricity purchase, f n-carbon represents the carbon trading cost, F buy is the unit carbon emission factor, P gbuy is the amount of electricity purchased from the power grid, is the carbon trading quota per unit of electricity, P pv is the photovoltaic output, and T is the dispatching time.

4. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 2, characterized in that: The electricity purchase and sale cost includes electricity purchase cost and electricity sale income, wherein the calculation expression of the electricity purchase and sale cost is: C S =C bs +C ss In the formula, C ss is the electricity purchase cost, C ss For income from electricity sales; The calculation expression of the electricity purchase cost is: In the formula, T is the scheduling time, C buy (t) is the unit price of electricity, P buy (t) is the amount of electricity purchased; The calculation expression of the electricity sales revenue is: In the formula, C sell (t) is the unit price of electricity, P sell (t) is the electricity sales amount.

5. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 1, characterized in that: The following steps are also included: Before using the improved reinforcement learning algorithm to solve the problem, the energy management model is transformed using the embedded augmented Lagrangian function method to convert the constrained optimization problem into an unconstrained optimization problem, specifically including the following steps: Construct the augmented Lagrangian function: The augmented Lagrangian multiplier is introduced to construct the augmented Lagrangian function, and the expression of the augmented Lagrangian function is: L=f(x)-λh(x) Where L is the augmented Lagrangian function, f(·) is the objective function of the energy management model, λ is the augmented Lagrangian multiplier, h(·) is the constraint of the energy management model, and x is the decision-making independent variable of the smart business building cluster; Compute the first-order partial derivative and set it to 0: For the augmented Lagrangian function, the first-order partial derivatives of the decision variable x and the augmented Lagrangian multiplier λ are calculated and set to 0, and the result is: Modify the system of equations and solve: According to the calculation results of the first-order partial derivative, the equation group of the optimality condition is obtained, and the following formula is solved by back substitution method: Where H is the Hessian matrix, is the transpose of the gradient of the constraint function, i.e., the row vector, is the gradient of the constraint function with respect to the variable x, Δx is the change in the variable x, Δλ is the Lagrange multiplier, and L x is the objective function value, L λ is the partial derivative of the Lagrangian function with respect to λ; According to the solution results of the equation group, the approximate optimal solution at the kth iteration is obtained, which is expressed as: In the formula, x (k+1) is the parameter vector in the k+1th iteration, x (k) is the parameter vector in the kth iteration, p (k) is the learning rate in the kth iteration, is the gradient of function f with respect to parameter x; The solution is solved through continuous iteration until the convergence condition is met.

6. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 5, characterized in that: The expression of the decision variable x of the smart business building cluster is: Where x is the decision-making independent variable of the smart business building cluster, Q vpp , Q se , Q pv and Q fl are the decision variables of the smart business building virtual power plant cluster, supercapacitor, photovoltaic and flywheel energy storage, Q i The subscript i in the table is vpp, se, pv, train, in, out, and fl, respectively. i 、b i With c i Represents the eigenvalues ​​of the profit function of different objects, q i is the characteristic variable.

7. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 1, characterized in that: The improved reinforcement learning algorithm is an improved Q-learning algorithm.

8. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 7, characterized in that: The steps of solving the problem by using the improved reinforcement learning algorithm include: 1) Build a reinforcement learning environment for each agent, define action rewards, and design reward function R(x,q) i ,Q i ) to approximately reflect the competition or cooperation relationship between the intelligent agents of different source measurement, load side and storage side equipment, where x is the decision-making independent variable of the smart business building cluster, Q i Represents the eigenvalues ​​of the profit function of different objects, q i is the characteristic variable; 2) Generate a Q table for each agent and set the number of iterations; 3) Generate a random number e, set the exploration probability ε of the greedy strategy, and determine whether e is greater than ε. If so, the agent makes a random action with a probability of ε and expands the Q table. If not, the agent selects the current optimal action strategy according to the Q table for execution, where the random number e is: e∈[0,1] In the formula, e is a random number; 4) The building virtual power plant cluster submits power generation information to the external power grid and settles the revenue; 5) The environment updates the state according to the agent's execution actions and provides instant rewards to each agent. Each agent calculates the Q value based on the received reward and the new state and updates the Q table; 6) Repeat steps 3)-5) for iterative learning until the maximum number of iterations is reached.

9. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 8, characterized in that: The current optimal action strategy is: In the formula, A i (t) is the action strategy at the i-th iteration at time t, a is the current action, s is the current state, and Q(·) is the Q function.

10. The method for collaborative operation of smart commercial building clusters based on improved reinforcement learning according to claim 8, characterized in that: The update formula of the Q value in the Q table is: Where a is the current action, s is the current state, Q(·) is the Q function, α is the learning rate, γ is the discount factor, a′ is the next action, s′ is the next state, and R(·) is the reward function.

Images