An energy management optimization method based on adaptive exploration deep Q network
By using adaptive exploration deep Q-networks to optimize energy management, the prediction and coordination problems of traditional strategies are solved, achieving efficient energy management and power consumption optimization, and improving energy utilization and equipment lifespan.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2024-11-08
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional energy management strategies lack the ability to accurately predict future energy demand and make dynamic optimization decisions. They cannot continuously learn the behavior patterns of end users, and there is a lack of coordination between internal subsystems, making it difficult to handle high-dimensional and large-scale problems. This results in slow learning speed and difficulty in ensuring the optimality of the strategy.
An adaptive exploration deep Q-network is adopted, which combines real-time power data and deep reinforcement learning methods. The autoregressive interpolation method is used to complete the missing data values, and a deep Q-neural network agent is constructed. Reward rules are designed and the deep Q-neural network is trained through an adaptive ε-greedy policy to optimize the energy management strategy.
It can process high-dimensional, large-scale power data, generate real-time dynamic energy management optimization strategies, reduce power consumption, lower electricity costs, improve energy efficiency, and extend equipment life.
Smart Images
Figure CN119539988B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an energy management optimization method based on an adaptive exploratory deep Q-network. Background Technology
[0002] Because production status varies across different time periods within an enterprise, the uncertain and complex operating environment necessitates energy management strategies. Effective energy management allows enterprises to accurately grasp energy consumption and implement corresponding energy-saving measures, thereby reducing energy costs. Based on energy management strategies, the supply plans for both electrical resources and green energy can be rationally arranged, reducing electricity consumption for the enterprise. Furthermore, effective energy management can reduce the number of charge-discharge cycles for energy storage devices, effectively extending their lifespan and maximizing energy cost reduction for the enterprise, achieving optimal economic benefits and energy utilization. Traditional energy management strategies are based on remote monitoring and control. These control strategies employ simple "if-then" rules, are strict and passive, lack predictive capabilities, and cannot continuously learn end-user behavior patterns. Coordination between internal subsystems is also lacking. Moreover, due to complexity and uncertainty, existing control strategies struggle to meet evolving needs.
[0003] In recent years, with the rapid development of artificial intelligence technology, especially the widespread application of machine learning and deep learning technologies, new solutions have been provided for optimizing energy management strategies. Many scholars have begun to use machine learning methods, such as reinforcement learning (RL) and deep learning (DL), to improve the ability to process large-scale data and handle complex dynamic systems. The literature (S. Bahrami, YCChen and VWSWong, 2021. Deep Reinforcement Learning for Demand Response in Distribution Networks. 1496-1506.) utilizes machine learning methods such as deep reinforcement learning (DRL) and federated learning (FL) to optimize user electricity consumption behavior, thereby reducing energy costs and improving economic efficiency and energy utilization. The user load control problem is modeled as a Markov decision process (MDP), and DRL is applied to develop a demand response algorithm in the distribution network. An actor-critic-based reinforcement learning framework is used to determine the optimal neural network parameters. To implement the proposed learning method in a decentralized manner, federated learning techniques are applied, and the constraints of the power distribution network are considered. The problem of updating neural network parameters is transformed into a sequence of semidefinite programs (SDPs) to handle non-convex power flow constraints. However, DRL typically requires a large number of training samples to achieve a certain level of model performance, which can be a challenge in practical applications, especially for scenarios like energy systems that require rapid response and optimization. Moreover, DRL models may exhibit high instability during training; even with the same hyperparameters and random seeds, model performance can vary significantly. Furthermore, DRL models are prone to overfitting to the training environment, leading to performance degradation under other environments or conditions, which limits their widespread applicability in practical applications.
[0004] Traditional energy management strategies employ simple "if-then" rules, which are rigid and passive, lacking the ability to accurately predict future energy demand and dynamically optimize decision-making, and cannot continuously learn end-user behavior patterns. There is also a lack of coordination between internal subsystems. Furthermore, due to complexity and uncertainty, existing control strategies struggle to meet evolving needs. They may be limited by data processing and analysis capabilities, unable to process large amounts of data in real time and extract valuable information for optimization decisions. Traditional reinforcement learning cannot handle high-dimensional, large-scale problems. Faced with increasing complexity and uncertainty on the user side, the stability and convergence of traditional reinforcement learning may be problematic, leading to slow learning speeds in practical applications and difficulty in guaranteeing that the learned strategy is optimal. Summary of the Invention
[0005] The technical problem to be solved by this invention is to address the shortcomings of the existing technology by providing an energy management optimization method based on an adaptive exploratory deep Q-network. This method combines real-time power data and deep reinforcement learning to enable artificial intelligence technology to take a series of actions in response to dynamic environments to help customers reduce power consumption and improve energy efficiency without affecting daily production. It also helps businesses or households to rationally plan their electricity usage and energy dispatch, reduce operating costs, and improve the operating efficiency and stability of the power system.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an energy management optimization method based on an adaptive exploratory deep Q-network, comprising the following steps:
[0007] Step 1: Preprocess the power environment status, total power consumption data, and photovoltaic power generation data of each energy-consuming device; use autoregressive interpolation to complete the default values of the power environment status, total power consumption data, and photovoltaic power generation data of each energy-consuming device.
[0008] The specific formula for using the Lagrange interpolation method to complete the missing values in the autoregressive interpolation method is as follows:
[0009]
[0010] Where n represents the time of the power environment status of each energy-consuming device being processed, and there are a total of n+1 time points; i is the index of the known power environment status of each energy-consuming device currently being calculated; j is the index of the power environment status of other known energy-consuming devices; x i and y i The power environment status of each energy-consuming device at time i and time i' are respectively, l i (x) is the Lagrange basis function at time i; x0, x1, ..., x n For n+1 time points, including the time point containing the default value, y0, y1, ..., yn Let x0, x1, ..., x n The total electricity consumption data for each energy-consuming device, where L(x) is the function value of the Lagrange interpolation formula, i.e., the dependent variable;
[0011] Step 2: Construct an agent based on a deep Q-neural network, using the power environment state as the input to the agent, so that the agent can be trained in the power environment and obtain the corresponding total reward;
[0012] The agent includes a deep Q-neural network, an experience storage pool D, and experience for training, to achieve the training and output of the deep Q-neural network; three hidden layers are constructed in the deep Q-neural network, and two linear rectified functions ReLU and one sigmoid function are used as activation functions of the deep Q-neural network;
[0013] Let the power environment state at time t be set as the input vector S in the deep Q neural network. t ={P tot ,P d ,P solar ,S SOC ,λ t}, where P tot P represents the total electrical load. d Let P be the load of the d-th energy-consuming device, d∈N, where N is the total number of energy-consuming devices. solar For photovoltaic power generation, S SOC In charging state, λ t Let be the electricity price at time t;
[0014] Use A t ={a t,d ,a t,cha ,a t,disc} represents the action vector taken at time t, where a t,d Let a be the operating action of the d-th energy-consuming device at time t. t,d =1 indicates that the energy-consuming equipment is in a power-consuming state during the time interval Δt, otherwise a t,d =0 indicates that the energy-consuming equipment is in an unpowered state during the time interval Δt; where a t,cha For the charging operation of the energy storage device at time t, a t,cha =1 indicates that the energy storage device is in a charging state during the time interval Δt, otherwise a t,cha =0 indicates that the energy storage device is in an uncharged state during the time interval Δt; where a t,disc Let a be the power consumption operation of the energy storage device at time t. t,disc =1 indicates that the energy storage device is in a power-consuming state during the time interval Δt, otherwise a t,disc=0 indicates that the energy storage device is in an unused state during the time interval Δt; and it is set that charging and power consumption cannot occur simultaneously.
[0015] The reward rule for a deep Q-neural network is designed as follows:
[0016] The specific formula for setting the total electricity cost bonus r1 is as follows:
[0017] r1=ξ1C (3)
[0018] Where C is the total electricity cost, and ξ1 is the weighting factor for the total electricity cost;
[0019] The charging status of the energy storage device is quantified, with the charging status set to 1 and the discharging status set to 0.
[0020] The frequency of energy storage device usage is reduced by obtaining a small negative reward for performing charging and discharging actions. The specific formula for setting the reward r2 of the energy storage device is as follows:
[0021]
[0022] Where N is the number of times the energy storage device is charged and discharged within a time interval, and ξ2 and ξ3 are the weighting factors of the operating cost of the energy storage device.
[0023] The specific formula for generating a reward r3 from photovoltaic power generation and consuming it locally is as follows:
[0024] r3=ξ4P use +(-ξ5P sell (5)
[0025] Among them, P use P represents the electricity consumption of photovoltaic power generation in energy storage devices. sell The amount of electricity generated by photovoltaic power generation and sold back to the power company, and which satisfies P. use +P sell =P solar ξ4 and ξ5 are the weighting factors for the operating cost of photovoltaic power generation;
[0026] The specific formula for the total reward r brought to the electrical environment by each action is as follows:
[0027] r = r1 + r2 + r3 (6)
[0028] Step 3: Train the deep Q-neural network to obtain the predicted Q-value of the Q-neural network and the target Q-value of the Target-Q neural network, and calculate the loss value using the loss function of the deep Q-neural network;
[0029] First, initialize the experience storage pool D, the Q-neural network parameters w, and the Target-Q neural network parameters w′; then, obtain an initial state s0 from the power environment state, schedule the user's energy consumption, and obtain the deep Q-neural network objective function, the specific formula of which is:
[0030]
[0031] Where, λ t Let P be the electricity price at time t, T be the total time, and P be the total electricity price at time t. t fixed For the unavoidable power consumption and losses, P t cha and P t disc These represent the charging power and discharging power of the energy storage device, respectively, P t solar Let be the power generated by solar photovoltaic power generation, and 'a' be the action vector.
[0032] Action a is selected using an adaptive ε-greedy strategy. t When selecting an action, use a greedy rate ε t The probability of choosing a random action is 1-ε. t The probability selection maximizes the predicted Q-value, and the specific formula is:
[0033]
[0034] Greed rate ε of the adaptive ε-greedy policy t The magnitude of the error dynamically depends on the temporal difference error of the deep Q-neuron; the historical records of the temporal difference error are set as {δ1,δ2,…,δ t}, then the current greed rate ε t The specific formula is as follows:
[0035] ε t =max(ε min ,ε inital ·exp(-α·mean(δ t ))) (9)
[0036] Where, ε min To minimize the exploration rate, ε inital The initial exploration rate is α, which is an adjustment coefficient used to control the sensitivity of the exploration rate to changes in time difference error. mean(δ) t () represents the average of all time difference errors up to time t;
[0037] Using action a t Update the deep Q-neural network to obtain the state s at the next time step. t+1 and select action a tThe reward r t+1 The experience gained at this time {s t ,a t ,s t+1 ,r t+1} Store the experience in the experience pool D, then perform experience replay, retrieve a new batch of experiences, and for each experience {s t ,a t ,s t+1 ,r t+1}, calculate the predicted value Q(s) of the Q-neural network. t ,a t ;w) and the target value y of the Target-Q neural network i Calculate the target value y i The formula is as follows:
[0038]
[0039] Where γ is the discount factor, s′ is the state of the Target-Q neural network, and a′ is the action taken by the Target-Q neural network;
[0040] Based on the target value y of the Target-Q neural network i The predicted value Q(s) of the Q-neural network t ,a t ;w), calculate the loss function L of the deep Q-neural network using formula (11):
[0041]
[0042] The gradient descent algorithm is used to update the parameters w of the Q-neural network using the loss function L, and the parameters w′=w of the Target-Q neural network are updated every N steps.
[0043] Step 4: Repeat steps 2-3 until the parameter w′ in the Target-Q neural network converges;
[0044] Step 5: Input the power environment status of each energy-consuming device, the photovoltaic power generation status, and the electricity price into the trained Target-Q neural network to obtain a series of action vectors as optimization strategies, and obtain updated power environment status after executing the optimization strategies;
[0045] The updated status of the power environment includes total power consumption, photovoltaic power generation, and the power consumption of each controllable electrical appliance.
[0046] The beneficial effects of adopting the above technical solution are as follows: The energy management optimization method based on adaptive exploratory deep Q-network provided by this invention, compared with the prior art, can process high-dimensional and large-scale power data. For complex energy scheduling and real-time dynamic power consumption environment, it generates energy management optimization strategies without affecting users' production and life, providing effective support for power supply scheduling and management. The optimization strategy derived by this invention can reduce power consumption, lower electricity costs, improve energy efficiency, and extend equipment life. Attached Figure Description
[0047] Figure 1 A flowchart of an energy management optimization method based on an adaptive exploratory deep Q-neural network is provided in this embodiment of the invention;
[0048] Figure 2 The intelligent agent structure diagram provided in the embodiments of the present invention;
[0049] Figure 3 The power consumption optimization algorithm framework diagram of the adaptive exploration depth Q neural network provided in the embodiments of the present invention is shown in the figure. Detailed Implementation
[0050] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0051] In this embodiment, an energy management optimization method based on an adaptive exploratory deep Q-neural network is described, such as... Figure 1 As shown, it includes the following steps:
[0052] Step 1: Preprocess the power environment status, total power consumption data, and photovoltaic power generation data of each energy-consuming device; use autoregressive interpolation to complete the default values of the power environment status, total power consumption data, and photovoltaic power generation data of each energy-consuming device.
[0053] The specific formula for using the Lagrange interpolation method to complete the missing values in the autoregressive interpolation method is as follows:
[0054]
[0055] Where n represents the time of the power environment status of each energy-consuming device being processed, and there are a total of n+1 time points; i is the index of the known power environment status of each energy-consuming device currently being calculated; j is the index of the power environment status of other known energy-consuming devices; x i and y i The power environment status of each energy-consuming device at time i and time i' are respectively, l i (x) is the Lagrange basis function at time i, where x0, x1, ..., xj is the function of the i-th time.n For n+1 time points, including the time point containing the default value, y0, y1, ..., y n Let x0, x1, ..., x n The total electricity consumption data for each energy-consuming device, where L(x) is the function value of the Lagrange interpolation formula, i.e., the dependent variable;
[0056] Step 2: Construct an agent based on a deep Q-neural network, using the power environment state as the input to the agent, so that the agent can be trained in the power environment and obtain the corresponding total reward;
[0057] The agent includes a deep Q-neural network, an experience storage pool D, and a small batch of experience for training, such as Figure 2 As shown, this is used to train and output a deep Q-neural network; three hidden layers are constructed in the deep Q-neural network, and two linear rectified functions ReLU and one sigmoid function are used as activation functions of the deep Q-neural network;
[0058] Let the power environment state at time t be set as the input vector S in the deep Q neural network. t ={P tot ,P d ,P solar ,S SOC ,λ t}, where P tot P represents the total electrical load. d Let P be the load of the d-th energy-consuming device, d∈N, where N is the total number of energy-consuming devices. solar For photovoltaic power generation, S SOC In charging state, λ t Let be the electricity price at time t;
[0059] Use A t ={a t,d ,a t,cha ,a t,disc} represents the action vector taken at time t, where a t,d Let a be the operating action of the d-th energy-consuming device at time t. t,d =1 indicates that the energy-consuming equipment is in a power-consuming state during the time interval Δt, otherwise a t,d =0 indicates that the energy-consuming equipment is in an unpowered state during the time interval Δt; where a t,cha For the charging operation of the energy storage device at time t, a t,cha =1 indicates that the energy storage device is in a charging state during the time interval Δt, otherwise a t,cha =0 indicates that the energy storage device is in an uncharged state during the time interval Δt; where a t,disc Let a be the power consumption operation of the energy storage device at time t. t,disc=1 indicates that the energy storage device is in a power-consuming state during the time interval Δt, otherwise a t,disc =0 indicates that the energy storage device is in an unused state during the time interval Δt, and the charging state and the power consumption state cannot be performed simultaneously;
[0060] In this embodiment, a reward rule for a deep Q-neural network is designed with the goal of reducing power consumption, lowering operating costs, and extending equipment lifespan.
[0061] To reduce electricity consumption, the total electricity cost incentive r1 is set using the following formula:
[0062] r1=ξ1C (3)
[0063] Where C is the total electricity cost, and ξ1 is the weighting factor for the total electricity cost;
[0064] The charging status of the energy storage device is quantified, with the charging status set to 1 and the discharging status set to 0.
[0065] In this embodiment, since the lifespan of the energy storage device is related to the number of charging and discharging states, the number of times the energy storage device is used is reduced by obtaining a small negative reward through performing charging and discharging actions. The specific formula for setting the reward r2 of the energy storage device is as follows:
[0066]
[0067] Where N is the number of times the energy storage device is charged and discharged within a time interval, and ξ2 and ξ3 are the weighting factors of the operating cost of the energy storage device.
[0068] The specific formula for the reward r3 generated by photovoltaic power generation and consumed locally (including storing renewable energy in local storage) is as follows:
[0069] r3=ξ4P use +(-ξ5P sell (5)
[0070] Among them, P use P represents the electricity consumption of photovoltaic power generation in energy storage devices. sell The amount of electricity generated by photovoltaic power generation and sold back to the power company, and which satisfies P. use +P sell =P solar ξ4 and ξ5 are the weighting factors for the operating cost of photovoltaic power generation;
[0071] The specific formula for the total reward r brought to the electrical environment by each action is as follows:
[0072] r = r1 + r2 + r3 (6)
[0073] Step 3: Train the deep Q-neural network to obtain the predicted Q-value of the Q-neural network and the target Q-value of the Target-Q neural network, and calculate the loss value using the loss function of the deep Q-neural network, such as... Figure 3 As shown;
[0074] First, initialize the experience storage pool D, the Q neural network parameters w, and the Target-Q neural network parameters w′; then, obtain an initial state s0 from the power environment state and schedule the user's energy consumption.
[0075] The main objective of this embodiment is to schedule users' energy consumption to help producers and consumers reduce electricity costs. Therefore, the objective function of the deep Q-neural network is obtained, and the specific formula is as follows:
[0076]
[0077] Where, λ t Let P be the electricity price at time t, T be the total time, and P be the total electricity price at time t. t fixed For the unavoidable power consumption and losses, P t cha and P t disc These represent the charging power and discharging power of the energy storage device, respectively, P t solar Let be the power generated by solar photovoltaic power generation, and 'a' be the action vector.
[0078] Action a is selected using an adaptive ε-greedy strategy. t When selecting an action, use a greedy rate ε t The probability of choosing a random action is 1-ε. t The probability selection maximizes the predicted Q-value, and the specific formula is:
[0079]
[0080] In this embodiment, the greed rate ε of the adaptive ε-greedy strategy is... t The magnitude of the error dynamically depends on the temporal difference error (TD error) of the deep Q-neuron; the historical records of the temporal difference error are set as {δ1, δ2, ..., δ t}, then the current greed rate ε t The specific formula is as follows:
[0081] ε t =max(ε min ,ε inital ·exp(-α·mean(δ t ))) (9)
[0082] Where, ε minTo minimize the exploration rate and prevent it from becoming too low, ε inital The initial exploration rate is α, which is an adjustment coefficient used to control the sensitivity of the exploration rate to changes in time difference error. mean(δ) t () represents the average of all time difference errors up to time t;
[0083] Using action a t Update the deep Q-neural network to obtain the state s at the next time step. t+1 and select action a t The reward r t+1 The experience gained at this time {s t ,a t ,s t+1 ,r t+1} Store the experience in the experience pool D, then perform experience replay, retrieve a new batch of experiences, and for each experience {s t ,a t ,s t+1 ,r t+1}, calculate the predicted value Q(s) of the Q-neural network. t ,a t ;w) and the target value y of the Target-Q neural network i Calculate the target value y i The formula is as follows:
[0084]
[0085] Where γ is the discount factor, s′ is the state of the Target-Q neural network, and a′ is the action taken by the Target-Q neural network;
[0086] Based on the target value y of the Target-Q neural network i The predicted value Q(s) of the Q-neural network t ,a t ;w), calculate the loss function L of the deep Q-neural network using formula (11):
[0087]
[0088] The gradient descent algorithm is used to update the parameters w of the Q-neural network using the loss function L, and the parameters w′=w of the Target-Q neural network are updated every N steps.
[0089] Step 4: Repeat steps 2-3 until the parameter w′ in the Target-Q neural network converges;
[0090] Step 5: Input the power environment status of each energy-consuming device, the photovoltaic power generation status, and the electricity price into the trained Target-Q neural network to obtain a series of action vectors as optimization strategies, and obtain the updated power environment status after executing the optimization strategies;
[0091] The updated status of the power environment includes total power consumption, photovoltaic power generation, and the power consumption of each controllable electrical appliance.
[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.
Claims
1. An energy management optimization method based on adaptive exploratory deep Q-networks, characterized in that: Includes the following steps: Step 1: Preprocess the power environment status, total power consumption data, and photovoltaic power generation data for each energy-consuming device; Autoregressive interpolation was used to complete the default values of the power environment status, total power consumption data, and photovoltaic power generation data for each energy-consuming device. Step 2: Construct an agent based on a deep Q-neural network, using the power environment state as the input to the agent, so that the agent can be trained in the power environment and obtain the corresponding total reward; Step 3: Train the deep Q-neural network to obtain the predicted Q-value of the Q-neural network and the target Q-value of the Target-Q neural network, and calculate the loss value using the loss function of the deep Q-neural network; First, initialize the experience storage pool and the parameters of the D and Q neural networks. and Target-Q neural network parameters Then, an initial state is obtained from the power environment state. The user's energy consumption is scheduled to obtain the objective function of the deep Q-neural network, and the specific formula is as follows: (7) in, Let be the electricity price at time t, and T be the total time. This refers to the unavoidable electricity consumption and losses. and These refer to the charging power and discharging power of the energy storage device, respectively. Let be the power generated by solar photovoltaic power generation, and 'a' be the action vector. Through adaptive Greedy strategy selects action When selecting actions, use a greedy rate. The probability of choosing a random action, in order to The probability choice maximizes the predicted Q-value, and the specific formula is: （8） Adaptive Greed rate of a greedy strategy The magnitude of the error dynamically depends on the temporal difference error of the deep Q-neural network; the historical record of the temporal difference error is set to... Then the current greed rate The specific formula is as follows: （9） in, To minimize the exploration rate, The initial exploration rate, The adjustment coefficient is used to control the sensitivity of the exploration rate to changes in time difference error. This is the average of all time difference errors up to time t; Using actions Update the deep Q-neural network to obtain the state at the next time step. and select action Rewards received The experience gained at this time Store the experience in the experience pool D, then perform experience replay to retrieve a new batch of experiences, for each experience... Calculate the predicted value of the Q-neural network. The target value of the Target-Q neural network Calculate the target value The formula is as follows: （10） in, As a discount factor, This represents the state of the Target-Q neural network; Actions taken by the Target-Q neural network; Based on the target value of the Target-Q neural network The predicted values of the Q-neural network The loss function L of the deep Q-neural network is calculated using formula (11): （11） The gradient descent algorithm is used to update the parameters of the Q neural network using the loss function L. Update the parameters of the Target-Q neural network every N steps. ; Step 4: Repeat steps 2-3 until the parameters in the Target-Q neural network are correct. convergence; Step 5: Input the power environment status of each energy-consuming device, the photovoltaic power generation status, and the electricity price into the trained Target-Q neural network to obtain a series of action vectors as optimization strategies, and obtain updated power environment status after executing the optimization strategies.
2. The energy management optimization method based on adaptive exploratory deep Q-network according to claim 1, characterized in that: The specific formula for using the Lagrange interpolation method to complete the missing values in the autoregressive interpolation method is as follows: （1） （2） Where n represents the time of the power environment status of each energy-consuming device being processed, and there are a total of n+1 times; 𝑖 is the index of the known power environment status of each energy-consuming device currently being calculated; j is the index of the power environment status of other known energy-consuming devices. and These represent the power environment status of each energy-consuming device at time i and time i , respectively. Let be the Lagrange basis function at time i; For n+1 time points, including the time point where the default value is located, for The total power consumption data for each energy-consuming device. The function value of the Lagrange interpolation formula is the dependent variable.
3. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 1, characterized in that: The agent includes a deep Q-neural network, an experience storage pool φ, and experience for training, to achieve training and output of the deep Q-neural network; three hidden layers are constructed in the deep Q-neural network, and two linear rectified functions ReLU and a sigmoid function are used as activation functions of the deep Q-neural network.
4. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 3, characterized in that: The specific method for step 2 is as follows: The power environment state at time t is set as the input vector in the deep Q neural network. ,in The total electrical load, Let d be the load of the energy-consuming equipment. N is the total number of energy-consuming devices. For photovoltaic power generation, In charging state. Let be the electricity price at time t; use Let represent the action vector taken at time t, where Let d be the operating action of the energy-consuming device at time t. For energy-consuming equipment at time intervals When it is in a state of power consumption, otherwise For energy-consuming equipment at time intervals It is in a state of no power use; among which The charging operation of the energy storage device at time t. For energy storage devices at time intervals The inside is in a charging state, and vice versa. For energy storage devices at time intervals The internal part is in an uncharged state; among which Let t represent the power consumption and operation of the energy storage device. For energy storage devices at time intervals The inside is in a state of being in use, and vice versa. For energy storage devices at time intervals The device is in an unused state; and it is set that charging and power consumption cannot occur simultaneously. The reward rule for a deep Q-neural network is designed as follows: Set up total electricity cost incentive The charging status of energy storage devices is quantified, with charging status set to 1 and discharging status set to 0. A small negative reward is awarded for performing charging and discharging actions on the energy storage devices to reduce their usage frequency; a reward system for the energy storage devices is set. ; Rewards are generated for photovoltaic power generation that generates local consumption. Then the total reward r that each action generates for the electrical environment is denoted as r.
5. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 4, characterized in that: The total electricity cost incentive The specific formula is set as follows: （3） Where C represents the total cost of electricity. This is the weighting factor for total electricity cost.
6. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 4, characterized in that: The reward for the energy storage device The specific formula is set as follows: （4） Where N represents the number of times the energy storage device charges and discharges within a time interval. and This is a weighting factor for the operating cost of energy storage equipment.
7. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 4, characterized in that: The photovoltaic power generation generates rewards The specific formula for local consumption is: （5） in, The electricity consumption of photovoltaic power generation in energy storage devices, The electricity generated by photovoltaic power generation is sold back to the power company, and meets the following requirements. , and This is a weighting factor for the operating cost of photovoltaic power generation.
8. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 7, characterized in that: The specific formula for the total reward r that each generated action brings to the electrical environment is as follows: （6）。 9. The energy management optimization method based on an adaptive exploratory deep Q-network according to claim 1, characterized in that: The updated status of the power environment includes total power consumption, photovoltaic power generation, and the power consumption of each controllable electrical appliance.