Energy management optimization method based on adaptive exploration depth Q network

Abstract

The invention provides an energy management optimization method based on an adaptive exploration depth Q network, and belongs to the technical field of artificial intelligence. The method comprises the following steps: preprocessing the power utilization environment state, the total power utilization data and the photovoltaic power generation data of each energy utilization device; complementing default values of the power utilization environment state, the total power utilization data and the photovoltaic power generation data of each energy utilization device by using an autoregressive interpolation method; constructing an intelligent agent based on a deep Q neural network and obtaining a corresponding total reward; training the deep Q neural network to obtain a predicted Q value of the Q neural network and a target Q value of the Target-Q neural network, and calculating a loss value through a loss function of the deep Q neural network; and inputting the power utilization environment state, the photovoltaic power generation state and the electricity price of each energy utilization device into the trained Target-Q neural network to obtain a series of action vectors as optimization strategies, and after the optimization strategies are executed, obtaining an updated power utilization environment state.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an energy management optimization method based on adaptive exploration of deep Q network. Background Art

[0002] Due to the different production status in different periods of time in the enterprise, the uncertain and complex operating environment requires energy management strategies to manage. Through effective energy management, enterprises can accurately grasp the energy consumption and take corresponding energy-saving measures to reduce energy costs. According to the energy management strategy, the power resources and green energy supply plan are reasonably arranged to reduce the consumption of electric energy for the enterprise. In addition, effective energy management can reduce the number of charge and discharge times of energy storage equipment, effectively extend the service life of energy storage equipment, reduce energy costs for enterprises to the greatest extent, and achieve the best economic benefits and energy utilization. Traditional energy management strategies are based on remote monitoring and remote control. The control strategy adopts a simple "if-then" rule, which is strict and passive, lacks prediction function, and cannot continuously learn the behavior patterns of end users. There is also a lack of coordination between internal subsystems. In addition, due to complexity and uncertainty, existing control strategies are difficult to meet the needs of development.

[0003] In recent years, with the rapid development of artificial intelligence technology, especially the widespread application of machine learning and deep learning technology, new solutions have been provided for the optimization of energy management strategies. Many scholars have begun to use machine learning methods, such as reinforcement learning (RL) and deep learning (DL), to improve the ability to process large-scale data and handle complex dynamic systems. The literature (S.Bahrami, YC Chen and VWSWong, 2021. Deep Reinforcement Learning for Demand Response in Distribution Networks. 1496-1506.) uses machine learning methods of deep reinforcement learning (DRL) and federated learning (FL) to optimize the user's electricity consumption behavior, thereby reducing energy costs and improving economic benefits and energy utilization. The user's load control problem is modeled as a Markov decision process (MDP), and DRL is applied to develop a demand response algorithm in the distribution network, and the actor-critic-based reinforcement learning framework is used to determine the optimal neural network parameters. In order to implement the proposed learning method in a decentralized manner, the federated learning technique is applied, and the constraints of the distribution network are considered to transform the update problem of the neural network parameters into a sequence of semidefinite programs (SDPs) that handle non-convex power flow constraints. However, DRL usually requires a large number of training samples to achieve a certain level of model performance, which may be a challenge in practical applications, especially for scenarios such as energy systems that require rapid response and optimization. Moreover, the DRL model may exhibit high instability during training, and the performance of the model may vary greatly even under the same hyperparameters and random seeds. In addition, the DRL model is prone to overfitting the training environment, resulting in performance degradation in other environments or conditions, which limits its wide applicability in practical applications.

[0004] Traditional energy management strategies use simple "if-then" rules, which are strict and passive. They lack the ability to accurately predict future energy demand and dynamically optimize decision-making, and cannot continuously learn the behavior patterns of end users. There is also a lack of coordination between internal subsystems. In addition, due to complexity and uncertainty, existing control strategies are difficult to meet the needs of development. It may be limited by data processing and analysis capabilities, and it is impossible to process large amounts of data in real time and extract valuable information for optimizing decisions. Traditional reinforcement learning cannot handle high-dimensional and large-scale problems. Faced with the increasing complexity and uncertainty on the user side, the stability and convergence of traditional reinforcement learning may be problematic, which leads to slow learning in practical applications and it is difficult to ensure that the learned strategy is optimal. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide an energy management optimization method based on an adaptive exploration deep Q network. This method combines real-time power data and deep reinforcement learning methods to enable artificial intelligence technology to take a series of actions in dynamic environments to help customers reduce power consumption without affecting daily production, improve energy utilization efficiency, help enterprises or households reasonably arrange power consumption plans and energy scheduling, reduce operating costs, and improve the operating efficiency and stability of the power system.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: an energy management optimization method based on adaptive exploration of deep Q network, comprising the following steps:

[0007] Step 1: Preprocess the power consumption environment status, total power consumption data and photovoltaic power generation data of each energy-consuming device; use the autoregressive interpolation method to fill in the default values ​​of the power consumption environment status, total power consumption data and photovoltaic power generation data of each energy-consuming device;

[0008] The specific formula for the autoregressive interpolation method to use the Lagrange interpolation method to complete the default value is:

[0009]

[0010] Where n is the moment of the power environment state of each energy-consuming device being processed, and there are n+1 moments in total; i is the index of the power environment state of each known energy-consuming device currently being calculated; j is the index of the power environment state of other known energy-consuming devices; x i and i are the power consumption environment states of each energy-consuming device at time i and i, respectively, i (x) is the Lagrangian basis function at the i-th moment; x 0 ,x 1 ,…,x nis n+1 time points including the time point of the default value, y 0 ,y 1 ,…,y n For x 0 ,x 1 ,…,x n The corresponding total power consumption data of each energy-consuming device, L(x) is the function value of the Lagrange interpolation formula, that is, the dependent variable;

[0011] Step 2: Build an intelligent agent based on a deep Q neural network and use the power environment status as the input of the intelligent agent so that the intelligent agent can be trained in the power environment and obtain the corresponding total reward;

[0012] The intelligent agent includes a deep Q neural network, an experience storage pool D, and experience for training to achieve training and output of the deep Q neural network; three hidden layers are constructed in the deep Q neural network, and two linear rectification functions ReLU and one Sigmoid function are used as activation functions of the deep Q neural network;

[0013] The power consumption state at time t is set as the input vector S in the deep Q neural network t = {P tot ,P d ,P solar ,S SOC ,λ t}, where P tot is the total power load, P d is the load of the dth energy-consuming device, d∈N, N is the total number of energy-consuming devices, P solar is the photovoltaic power generation, S SOC is the charging state, λ t is the electricity price at time t;

[0014] Use A t ={a t,d ,a t,cha ,a t,disc} represents the running action vector taken at time t, where a t,d is the operation action of the dth energy-consuming device at time t, a t,d =1 means that the energy-consuming equipment is in power consumption state during the time interval Δt, otherwise a t,d = 0 means that the energy-consuming device is in a non-powered state during the time interval Δt; where a t,cha is the charging operation of the energy storage device at time t, a t,cha =1 means the energy storage device is in charging state within the time interval Δt, otherwise a t,cha =0 means that the energy storage device is in an uncharged state within the time interval Δt; where a t,disc is the power consumption operation of the energy storage device at time t, a t,disc=1 means the energy storage device is in power consumption state within the time interval Δt, otherwise a t,disc =0 means that the energy storage device is in an unpowered state within the time interval Δt; and the charging state and the power consumption state cannot be set at the same time;

[0015] Design the reward rule for the deep Q neural network, specifically:

[0016] Total electricity cost reward 1 Set the specific formula as:

[0017] r 1 =ξ 1 C (3)

[0018] Among them, C is the total electricity cost, ξ 1 is the total electricity cost weight factor;

[0019] The charging state of the energy storage device is digitized, wherein the charging state is set to 1 and the discharging state is set to 0;

[0020] By performing the charging and discharging actions of the energy storage device, a small amount of negative reward is obtained to reduce the number of times the energy storage device is used. The reward of the energy storage device is r 2 Set the specific formula as:

[0021]

[0022] Where N is the number of times the energy storage device is charged and discharged within a time interval, ξ 2 and 3 is the weight factor of the energy storage equipment operation cost;

[0023] Photovoltaic power generation generates rewards 3 The specific formula for local consumption is:

[0024] r 3 =ξ 4 P use +(-ξ 5 P sell ) (5)

[0025] Among them, P use is the power consumption of photovoltaic power generation in energy storage equipment, P sell The amount of electricity sold back to the power company by photovoltaic power generation and meets P use +P sell =P solar ,ξ 4 and 5 is the weight factor of photovoltaic power generation operation cost;

[0026] The specific formula for the total reward r brought by each action to the power environment is:

[0027] r=r 1 +r 2 +r 3 (6)

[0028] Step 3: Train the deep Q neural network to obtain the predicted Q value of the Q neural network and the target Q value of the Target-Q neural network, and calculate the loss value through the loss function of the deep Q neural network;

[0029] First, initialize the experience storage pool D, Q neural network parameters w and Target-Q neural network parameters w′; then obtain an initial state s from the power environment state 0 , schedule the user's energy consumption and obtain the deep Q neural network objective function. The specific formula is:

[0030]

[0031] Among them, λ t is the electricity price at time t, T is the total time, P t fixed P is the power consumption for unavoidable power consumption and loss. t cha and P t disc are the charging power and discharging power of the energy storage device, P t solar is the power generated by solar photovoltaic power generation, a is the action vector;

[0032] Select action a through an adaptive ε-greedy strategy t ; When selecting actions, use the greedy rate ε t The probability of selecting a random action is 1-ε t The probability of selecting the action that maximizes the predicted Q value is:

[0033]

[0034] Greedy rate ε of the adaptive ε-greedy strategy t The size of the dynamics depends on the time difference error of the deep Q neural network; the historical record of the time difference error is set to {δ 1 ,δ 2 ,…,δ t}, then the current greed rate ε t The specific formula is:

[0035] ε t =max(ε min ,ε inital ·exp(-α·mean(δ t ))) (9)

[0036] Among them, ε min is the minimum exploration rate, ε inital is the initial exploration rate, α is the adjustment coefficient, which is used to control the sensitivity of the exploration rate to the time difference error, mean(δ t ) is the average value of all time difference errors up to time t;

[0037] Using action a t Update the deep Q neural network to get the state s at the next moment t+1 and select action a t The reward r t+1 ; The experience gained at this time t ,a t ,s t+1 ,r t+1} is stored in the experience pool D, and then the experience is replayed to take out a new batch of experience. For each experience {s t ,a t ,s t+1 ,r t+1}, calculate the predicted value Q(s) of the Q neural network t ,a t ; w) and the target value y of the Target-Q neural network i , calculate the target value y i The formula is as follows:

[0038]

[0039] Where γ is the discount factor, s′ is the state of the Target-Q neural network; a′ is the action taken by the Target-Q neural network;

[0040] According to the target value y of the Target-Q neural network i and the predicted value Q(s) of the Q neural network t ,a t ; w), use formula (11) to calculate the loss function L of the deep Q neural network:

[0041]

[0042] Use the gradient descent algorithm to update the Q neural network parameter w using the loss function L, and update the Target-Q neural network parameter w′=w every N steps;

[0043] Step 4: Repeat steps 2 and 3 until the parameter w′ in the Target-Q neural network converges;

[0044] Step 5: Input the power consumption environment status of each energy-consuming device, the photovoltaic power generation status and the electricity price into the trained Target-Q neural network to obtain a series of action vectors as the optimization strategy, and after executing the optimization strategy, obtain the updated power consumption environment status;

[0045] The updated state of the power consumption environment includes the total power consumption, photovoltaic power generation, and power consumption of each controllable electrical appliance.

[0046] The beneficial effects of adopting the above technical solution are: the energy management optimization method based on adaptive exploration of deep Q network provided by the present invention can process high-dimensional and large-scale power data compared with the prior art, and generate energy management optimization strategies for complex energy scheduling and real-time dynamic power consumption environment without affecting the production and life of users, thereby providing effective support for the scheduling and management of power supply; the optimization strategy obtained by the present invention can reduce the consumption of electric energy, reduce electricity costs, improve energy utilization efficiency, and extend the life cycle of equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of an energy management optimization method based on an adaptive exploration deep Q neural network provided by an embodiment of the present invention;

[0048] Figure 2 An intelligent agent structure diagram provided by an embodiment of the present invention;

[0049] Figure 3 A framework diagram of an adaptive exploration deep Q neural network power optimization algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] In this embodiment, an energy management optimization method based on adaptive exploration of deep Q neural network is provided. Figure 1 As shown, the following steps are included:

[0052] Step 1: Preprocess the power consumption environment status, total power consumption data and photovoltaic power generation data of each energy-consuming device; use the autoregressive interpolation method to fill in the default values ​​of the power consumption environment status, total power consumption data and photovoltaic power generation data of each energy-consuming device;

[0053] The specific formula for the autoregressive interpolation method to use the Lagrange interpolation method to complete the default value is:

[0054]

[0055] Where n is the moment of the power environment state of each energy-consuming device being processed, and there are n+1 moments in total, i is the index of the power environment state of each known energy-consuming device currently being calculated, j is the index of the power environment state of other known energy-consuming devices, x i and i are the power consumption environment states of each energy-consuming device at time i and i, respectively, i (x) is the Lagrangian basis function at the i-th moment, x 0 ,x 1 ,…,x n is n+1 time points including the time point of the default value, y 0 ,y 1 ,…,y n For x 0 ,x 1 ,…,x n The corresponding total power consumption data of each energy-consuming device, L(x) is the function value of the Lagrange interpolation formula, that is, the dependent variable;

[0056] Step 2: Build an intelligent agent based on a deep Q neural network and use the power environment status as the input of the intelligent agent so that the intelligent agent can be trained in the power environment and obtain the corresponding total reward;

[0057] The agent includes a deep Q neural network, an experience storage pool D, and a small batch of experiences for training, such as Figure 2 As shown, to realize the training and output of the deep Q neural network; construct three hidden layers in the deep Q neural network, and use two linear rectification functions ReLU and one Sigmoid function as the activation function of the deep Q neural network;

[0058] The power consumption state at time t is set as the input vector S in the deep Q neural network t = {P tot ,P d ,P solar ,S SOC ,λ t}, where P tot is the total power load, P d is the load of the dth energy-consuming device, d∈N, N is the total number of energy-consuming devices, P solar is the photovoltaic power generation, S SOC is the charging state, λ t is the electricity price at time t;

[0059] Use A t ={a t,d ,a t,cha ,a t,disc} represents the running action vector taken at time t, where a t,dis the operation action of the dth energy-consuming device at time t, a t,d =1 means that the energy-consuming equipment is in power consumption state during the time interval Δt, otherwise a t,d = 0 means that the energy-consuming device is in a non-powered state during the time interval Δt; where a t,cha is the charging operation of the energy storage device at time t, a t,cha =1 means the energy storage device is in charging state within the time interval Δt, otherwise a t,cha =0 means that the energy storage device is in an uncharged state within the time interval Δt; where a t,disc is the power consumption operation of the energy storage device at time t, a t,disc =1 means the energy storage device is in power consumption state within the time interval Δt, otherwise a t,disc =0 means that the energy storage device is in an unpowered state within the time interval Δt, and the charging state and the power consumption state cannot be set at the same time;

[0060] In this embodiment, the reward rules of the deep Q neural network are designed with the goal of reducing power consumption, lowering operating costs, and extending the service life of the equipment;

[0061] In order to reduce electricity consumption, the total electricity cost reward r 1 Set the specific formula as:

[0062] r 1 =ξ 1 C (3)

[0063] Among them, C is the total electricity cost, ξ 1 is the total electricity cost weight factor;

[0064] The charging state of the energy storage device is digitized, wherein the charging state is set to 1 and the discharging state is set to 0;

[0065] In this embodiment, since the life cycle of the energy storage device is related to the number of charging and discharging states, in this case, a small amount of negative rewards are obtained by performing the charging and discharging actions of the energy storage device to reduce the number of times the energy storage device is used. The reward r of the energy storage device 2 Set the specific formula as:

[0066]

[0067] Where N is the number of times the energy storage device is charged and discharged within a time interval, ξ 2 and 3 is the weight factor of the energy storage equipment operation cost;

[0068] Photovoltaic power generation generates rewards 3 The specific formula for local consumption (including storage of renewable energy in local storage) is:

[0069] r 3 =ξ 4 P use +(-ξ 5 P sell ) (5)

[0070] Among them, P use is the power consumption of photovoltaic power generation in energy storage equipment, P sell The amount of electricity sold back to the power company by photovoltaic power generation and meets P use +P sell =P solar ,ξ 4 and 5 is the weight factor of photovoltaic power generation operation cost;

[0071] The specific formula for the total reward r brought by each action to the power environment is:

[0072] r=r 1 +r 2 +r 3 (6)

[0073] Step 3: Train the deep Q neural network to obtain the predicted Q value of the Q neural network and the target Q value of the Target-Q neural network, and calculate the loss value through the loss function of the deep Q neural network, such as Figure 3 As shown;

[0074] First, initialize the experience storage pool D, Q neural network parameters w and Target-Q neural network parameters w′; then obtain an initial state s from the power environment state 0 , schedule the energy consumption of users;

[0075] The main goal of this embodiment is to schedule the energy consumption of users and help prosumers reduce electricity costs. Therefore, the deep Q neural network objective function is obtained. The specific formula is:

[0076]

[0077] Among them, λ t is the electricity price at time t, T is the total time, P t fixed P is the power consumption for unavoidable power consumption and loss. t cha and P t disc are the charging power and discharging power of the energy storage device, P t solar is the power generated by solar photovoltaic power generation, a is the action vector;

[0078] Select action a through an adaptive ε-greedy strategyt ; When selecting actions, use the greedy rate ε t The probability of selecting a random action is 1-ε t The probability of selecting the action that maximizes the predicted Q value is:

[0079]

[0080] In this embodiment, the greed rate ε of the adaptive ε greedy strategy t The size of the dynamics depends on the time difference error (TD error) of the deep Q neural network; the historical record of the time difference error is set to {δ 1 ,δ 2 ,…,δ t}, then the current greed rate ε t The specific formula is:

[0081] ε t =max(ε min ,ε inital ·exp(-α·mean(δ t ))) (9)

[0082] Among them, ε min is the minimum exploration rate to prevent the exploration rate from being too low, ε inital is the initial exploration rate, α is the adjustment coefficient, which is used to control the sensitivity of the exploration rate to the time difference error, mean(δ t ) is the average value of all time difference errors up to time t;

[0083] Using action a t Update the deep Q neural network to get the state s at the next moment t+1 and select action a t The reward r t+1 ; The experience gained at this time t ,a t ,s t+1 ,r t+1} is stored in the experience pool D, and then the experience is replayed to take out a new batch of experience. For each experience {s t ,a t ,s t+1 ,r t+1}, calculate the predicted value Q(s) of the Q neural network t ,a t ; w) and the target value y of the Target-Q neural network i , calculate the target value y i The formula is as follows:

[0084]

[0085] Among them, γ is the discount factor, s′ is the state of the Target-Q neural network, and a′ is the action taken by the Target-Q neural network;

[0086] According to the target value y of the Target-Q neural network i and the predicted value Q(s) of the Q neural network t ,a t ; w), use formula (11) to calculate the loss function L of the deep Q neural network:

[0087]

[0088] Use the gradient descent algorithm to update the Q neural network parameter w using the loss function L, and update the Target-Q neural network parameter w′=w every N steps;

[0089] Step 4: Repeat steps 2 and 3 until the parameter w′ in the Target-Q neural network converges;

[0090] Step 5: Input the power consumption environment status of each energy-consuming device, the photovoltaic power generation status and the electricity price into the trained Target-Q neural network to obtain a series of action vectors as the optimization strategy, and after executing the optimization strategy, obtain the updated power consumption environment status;

[0091] The updated state of the power consumption environment includes the total power consumption, photovoltaic power generation, and power consumption of each controllable electrical appliance.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. An energy management optimization method based on adaptive exploration of deep Q network, characterized by: The following steps are involved: Step 1: Preprocess the power consumption environment status, total power consumption data and photovoltaic power generation data of each energy-consuming device; The autoregressive interpolation method is used to complete the default values ​​of the power consumption environment status of each energy-consuming device, the total power consumption data, and the photovoltaic power generation data; Step 2: Build an intelligent agent based on a deep Q neural network and use the power environment status as the input of the intelligent agent so that the intelligent agent can be trained in the power environment and obtain the corresponding total reward; Step 3: Train the deep Q neural network to obtain the predicted Q value of the Q neural network and the target Q value of the Target-Q neural network, and calculate the loss value through the loss function of the deep Q neural network; Step 4: Repeat steps 2-3 until the parameter w′ in the Target-Q neural network converges; Step 5: Input the power consumption environment status of each energy-consuming device, the photovoltaic power generation status and the electricity price into the trained Target-Q neural network to obtain a series of action vectors as the optimization strategy, and after executing the optimization strategy, obtain the updated power consumption environment status.

2. The energy management optimization method based on adaptive exploration deep Q network according to claim 1 is characterized in that: The specific formula for the autoregressive interpolation method to use the Lagrange interpolation method to complete the default value is: Where n is the moment of the power environment state of each energy-consuming device being processed, and there are n+1 moments in total; i is the index of the power environment state of each known energy-consuming device currently being calculated; j is the index of the power environment state of other known energy-consuming devices; x i and i are the power consumption environment states of each energy-consuming device at time i and i, respectively, i (x) is the Lagrangian basis function at the i-th moment; x0, x1, …, x n is n+1 time points including the time point of the default value, y0, y1, …, y n for x0,x1,…,x n The corresponding total electricity consumption data of each energy-consuming equipment, L(x) is the function value of the Lagrange interpolation formula, that is, the dependent variable.

3. The energy management optimization method based on adaptive exploration deep Q network according to claim 1 is characterized in that: The intelligent agent includes a deep Q neural network, an experience storage pool D and experience for training to achieve training and output of the deep Q neural network; three hidden layers are constructed in the deep Q neural network, and two linear rectification functions ReLU and a Sigmoid function are used as activation functions of the deep Q neural network.

4. The energy management optimization method based on adaptive exploration deep Q network according to claim 3 is characterized by: The specific method of step 2 is: The power consumption state at time t is set as the input vector S in the deep Q neural network t = {P tot ,P d ,P solar ,S SOC ,λ t }, where P tot is the total power load, P d is the load of the dth energy-consuming device, d∈N, N is the total number of energy-consuming devices, P solar is the photovoltaic power generation, S SOC is the charging state, λ t is the electricity price at time t; Use A t ={a t,d ,a t,cha ,a t,disc } represents the running action vector taken at time t, where a t,d is the operation action of the dth energy-consuming device at time t, a t,d =1 means that the energy-consuming equipment is in power consumption state during the time interval Δt, otherwise a t,d = 0 means that the energy-consuming device is in a non-powered state during the time interval Δt; where a t,cha is the charging operation of the energy storage device at time t, a t,cha =1 means the energy storage device is in charging state within the time interval Δt, otherwise a t,cha =0 means that the energy storage device is in an uncharged state within the time interval Δt; where a t,disc is the power consumption operation of the energy storage device at time t, a t,disc =1 means the energy storage device is in power consumption state within the time interval Δt, otherwise a t,disc =0 means that the energy storage device is in an unpowered state within the time interval Δt; and the charging state and the power consumption state cannot be set at the same time; Design the reward rule for the deep Q neural network, specifically: Set the total electricity cost reward r1; digitize the charging state of the energy storage device, where the charging state is set to 1 and the discharging state is set to 0; obtain a small amount of negative rewards by executing the charging and discharging actions of the energy storage device to reduce the number of times the energy storage device is used, and set the reward r2 for the energy storage device; set the reward r3 for photovoltaic power generation generated locally; then the total reward r brought by each action to the electricity consumption environment.

5. The energy management optimization method based on adaptive exploration deep Q network according to claim 4 is characterized in that: The specific formula for setting the total electricity cost reward r1 is: r1=ξ1C (3) Where C is the total electricity cost and ξ1 is the weight factor of the total electricity cost.

6. The energy management optimization method based on adaptive exploration deep Q network according to claim 4 is characterized by: The specific formula for setting the reward r2 of the energy storage device is: Where N is the number of times the energy storage device is charged and discharged within a time interval, and ξ2 and ξ3 are the weight factors of the operating cost of the energy storage device.

7. The energy management optimization method based on adaptive exploration deep Q network according to claim 4 is characterized by: The specific formula for generating the reward r3 from photovoltaic power generation and consuming it locally is: r3=ξ4P use +(-ξ ξ5 P sell ) (5) Among them, P use is the power consumption of photovoltaic power generation in energy storage equipment, P sell The amount of electricity sold back to the power company by photovoltaic power generation and meets P use +P sell =P solar , ξ4 and ξ5 are weight factors of photovoltaic power generation operation cost.

8. The energy management optimization method based on adaptive exploration deep Q network according to claim 7 is characterized in that: The specific formula for the total reward r brought by each action to the power environment is: r＝r1+r2+r3 (6)。 9. The energy management optimization method based on adaptive exploration deep Q network according to claim 8, characterized in that: The specific method of step 3 is: First, the experience storage pool D, Q neural network parameter w and Target-Q neural network parameter w′ are initialized; then an initial state s0 is obtained from the power environment state, the user's energy consumption is scheduled, and the deep Q neural network objective function is obtained. The specific formula is: Among them, λ t is the electricity price at time t, T is the total time, P t fixed P is the power consumption for unavoidable power consumption and loss. t cha and P t disc are the charging power and discharging power of the energy storage device, P t solar is the power generated by solar photovoltaic power generation, a is the action vector; Select action a through an adaptive ε-greedy strategy t ; When selecting actions, use the greedy rate ε t The probability of selecting a random action is 1-ε t The probability of selecting the action that maximizes the predicted Q value is: Greedy rate ε of the adaptive ε-greedy strategy t The size of the dynamics depends on the time difference error of the deep Q neural network; the historical record of the time difference error is set to {δ1,δ2,…,δ t }, then the current greed rate ε t The specific formula is: e t =max(e min ,he inital ·exp(-α·mean(δ t ))) (9) Among them, ε min is the minimum exploration rate, ε inital is the initial exploration rate, α is the adjustment coefficient, which is used to control the sensitivity of the exploration rate to the time difference error, mean(δ t ) is the average value of all time difference errors up to time t; Using action a t Update the deep Q neural network to get the state s at the next moment t+1 and select action a t The reward r t+1 ; The experience gained at this time t ,a t ,s t+1 ,r t+1 } is stored in the experience pool D, and then the experience is replayed to take out a new batch of experience. For each experience {s t ,a t ,s t+1 ,r t+1 }, calculate the predicted value Q(s) of the Q neural network t ,a t ; w) and the target value y of the Target-Q neural network i , calculate the target value y i The formula is as follows: Where γ is the discount factor, s′ is the state of the Target-Q neural network; a′ is the action taken by the Target-Q neural network; According to the target value y of the Target-Q neural network i and the predicted value Q(s) of the Q neural network t ,a t ; w), use formula (11) to calculate the loss function L of the deep Q neural network: Use the gradient descent algorithm to update the Q neural network parameter w using the loss function L, and update the Target-Q neural network parameter w′=w every N steps.

10. The energy management optimization method based on adaptive exploration deep Q network according to claim 9 is characterized in that: The updated state of the power consumption environment includes the total power consumption, photovoltaic power generation, and power consumption of each controllable electrical appliance.

Images