Green electricity park source network load storage planning method based on deep reinforcement learning

Abstract

The invention discloses a green electricity park source network load storage planning method based on deep reinforcement learning, and relates to the technical field of park electric power planning. According to the method, a green electricity park planning optimization model is constructed based on historical data, and a source network load storage optimization configuration objective function, a micro gas turbine model and multi-dimensional constraint conditions are covered; building a reinforcement learning training environment including a state space, an action space and a reward function; and finally, training and optimizing the load storage optimization configuration of the source network through a DDPG algorithm fused with a scene memory mechanism to obtain an optimal scheme. According to the method, the historical optimal strategy is reserved through the scene memory mechanism, the coordination of cost optimization and income maximization in the whole life cycle of the green electricity park is realized, the timeliness and accuracy of planning are remarkably improved, and technical support is provided for fine planning of the green electricity park.

Description

Technical Field

[0001] This invention relates to the field of power planning technology in industrial parks, and in particular to a green energy park source-grid-load-storage planning method based on deep reinforcement learning. Background Technology

[0002] To achieve the goals of "carbon peaking and carbon neutrality," accelerating the construction of a new power system based on new energy sources has become a global consensus. Against the backdrop of my country's active promotion of green energy development and energy conservation and emission reduction, green power parks have become crucial for achieving green industrial transformation. Green power supply models for these parks include purchasing green certificates, direct green power supply, and self-built green power sources. How to select the correct green power supply model with the goal of optimizing the entire life cycle cost to maximize the benefits of the park has become a critical issue that urgently needs to be addressed.

[0003] Existing planning methods are prone to getting stuck in local optima when dealing with multiple constraints and nonlinear conditions, and are difficult to adapt to the dynamically changing park operation environment. Conventional reinforcement learning algorithms suffer from low sample efficiency and slow single-step reward propagation in high-dimensional action spaces, resulting in insufficient timeliness and accuracy of planning results. They cannot fully balance economy, reliability and environmental protection, and are difficult to meet the needs of refined operation of green energy parks. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a more timely and accurate method for planning the source, grid, load and storage of green energy parks.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A source-grid-load-storage planning method for green energy parks based on deep reinforcement learning, the key of which includes the following steps:

[0007] S1. Based on historical data, construct a planning optimization model for green power parks, including constructing an objective function for optimizing the configuration of power generation, grid, load, and storage in green power parks; then constructing a micro gas turbine model within green power parks; and finally adding constraints to the optimization model.

[0008] S2. Construct a training environment for the optimal allocation of power generation, grid, load and storage in green energy parks, including setting the state space, action space and reward function;

[0009] S3. Based on the green power park planning optimization model, the DDPG algorithm with integrated scenario memory mechanism is used to train and optimize the source-grid-load-storage configuration of the green power park to obtain the optimal configuration; including the following steps:

[0010] (3-1) Initialize the experience pool, Q network parameters, and scenario memory playback table;

[0011] (3-2) Obtain the current environment state st ;

[0012] (3-3) Integrating the contextual memory mechanism, select the current action a t ;

[0013] (3-4) Update the experience pool and update the scenario memory table;

[0014] (3-5) Update the Q network parameters;

[0015] (3-6) Repeat steps (3-2) to (3-5) until the maximum number of training rounds is reached.

[0016] Preferably, the objective function for optimizing the allocation of power generation, grid, load, and storage in the green energy park is constructed in step S1, as follows:

[0017]

[0018] in, For the full life cycle cost of green energy parks, To control scheduling costs, This represents the cost of purchasing electricity from the upper-level power grid for the industrial park, excluding green certificates. This represents the cost of generator units within the park. This represents the cost required to construct green electricity direct supply lines and build self-built photovoltaic sites.

[0019] Preferably, the construction of the micro gas turbine model within the green energy park is based on the power generation capacity of the micro gas turbine, as detailed below:

[0020] (6)

[0021] in This represents the power generation of the micro gas turbine at time t. This represents the power generation efficiency coefficient of the micro gas turbine. This indicates the fuel flow rate.

[0022] Preferably, the constraints of the optimization model include upper and lower limits of gas turbine output and ramping constraints, power constraints of distributed energy unit interconnection lines, power balance constraints, upper and lower limits of voltage and active and reactive power of each node in the green energy park distribution network, and construction economic cost constraints.

[0023] Preferably, the state space in S2 represents the multi-dimensional environmental states that affect the optimal configuration of power generation, grid, load, and energy storage during the operation of the green energy park; specifically as follows:

[0024] (13)

[0025] in For state space, This represents the current time series information, which can be obtained from the current day and the current hour. This represents the load data of the green energy park at that moment. This represents the output data of the distributed energy units received by the green energy park at that moment. This represents the amount of green certificate electricity purchased by the green energy park at that moment;

[0026] The action space refers to the control actions of each energy supply point within the green energy park; specifically as follows:

[0027] (14)

[0028] in For the action space, This is the control signal for the gas turbine's power generation. These represent the electricity purchased from the park's green certificates and distributed energy units, respectively.

[0029] Therefore, in order to maximize profits, a reward function is defined as follows:

[0030] (15)

[0031] in, Revenue generated from the sale of energy in green energy parks The cost incurred for obtaining energy for green energy parks.

[0032] Preferably, step (3-1) initializes the experience pool, Q-network parameters, and scenario memory replay table:

[0033] The control center in the green energy park is regarded as an intelligent agent. The actions of the intelligent agent are randomly initialized and set. The state transition process of the green energy park is recorded. The current state of the green energy park, the corresponding action, the next state, and the reward function are stored in the experience pool until the experience pool is full. At the same time, the weights and the scenario memory replay table of the target Q network are initialized.

[0034] The step (3-2) involves obtaining the current environmental state s. t It saves information about each dimension of the state space at the current time period.

[0035] Preferably, step (3-3) integrates a contextual memory mechanism to select the current action a. t The contextual memory mechanism first filters the highest reward value corresponding to the state-action pair in the current contextual memory table, and combines it with the original TD error, guiding the agent's learning through the historical best strategy corresponding to the current state.

[0036] The mathematical expression for the emergency memory mechanism is:

[0037] (18)

[0038] in, Corresponding to the highest historical reward, The function corresponds to the reward generated by the current state-action pair. For embedded functions;

[0039] As mentioned in (3-4): Update the experience pool and update the context memory table. Specifically, execute the current action a obtained after the improvement of the context memory mechanism. t Obtain reward value r t The state s in the next moment t+1 The state transition process is described in terms of (s) t , a t , r t ,s t+1 The experience pool is stored in the form of a new strategy state-action pair. The newly emerging strategy state-action pair is stored in the context memory table. If the experience pool is full, the oldest experience record is deleted. If the context memory table is full, the least used strategy is discarded.

[0040] Preferably, step (3-5) involves updating the Q network parameters.

[0041] Data is randomly drawn from the experience pool to calculate the target network's predicted value:

[0042] (19)

[0043] in, This represents the target network prediction value for the i-th sample. The attenuation coefficient is... Calculated for the target network Action value function under a given state;

[0044] Combining the contextual memory mechanism, the loss function is minimized as follows:

[0045] (20)

[0046] in, To evaluate the network's calculations Action value function in a state H represents the weight of the emergency memory in the loss function, and H represents the historical best policy state-action pair.

[0047] The beneficial effects of adopting the above technical solution are as follows:

[0048] This invention aims to optimize the economic efficiency of green energy parks throughout their entire lifecycle, taking power balance constraints as conditions and considering constraints such as power generation capacity and the state of charge of energy storage devices. It constructs an optimization model for the configuration of source-grid-load-storage scale, and then, based on a delay-deterministic strategy gradient algorithm architecture, introduces a scenario memory mechanism to solve the problems of low sample efficiency and slow single-step reward propagation in high-dynamic environments and high-dimensional action spaces of traditional algorithms, thereby improving the timeliness and accuracy of the planning method. Attached Figure Description

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0050] Figure 1 This is a flowchart illustrating a green energy park source-grid-load-storage planning method based on deep reinforcement learning proposed in this invention.

[0051] Figure 2 This is a schematic diagram of the DDPG algorithm that integrates contextual memory mechanism proposed in this invention. Detailed Implementation

[0052] To make the above-mentioned objectives, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, this application proposes a source-grid-load-storage planning method for green energy parks based on deep reinforcement learning, including the following steps:

[0054] S1. Construct a planning optimization model for green energy industrial parks;

[0055] S2. Construct a training environment for optimizing the allocation of power generation, grid, load, and energy storage in green energy parks;

[0056] S3. Based on the green power park planning optimization model, the DDPG algorithm with integrated scenario memory mechanism is used to train and optimize the source-grid-load-storage configuration of the green power park to obtain the optimal configuration.

[0057] Based on historical operational data of the green power park, the planning optimization model for the green power park is constructed. This includes constructing an objective function for optimizing the configuration of power generation, grid, load, and storage in the green power park; then constructing a micro gas turbine model within the green power park; and finally adding constraints to the optimization model. This part forms the basis of the planning method. By clarifying the optimization objectives, optimization objects, and constraint boundaries, it provides optimization direction and limiting boundaries for obtaining the optimal configuration in the future.

[0058] First, we construct an objective function for the optimal allocation of power generation, grid, load, and energy storage in green energy parks.

[0059]

[0060] Equation (1) is the objective function of the source-grid-load-storage planning optimization model of the green power park. It takes minimizing the full life cycle cost of the green power park as the core, and comprehensively considers the controllable dispatch cost, grid power purchase cost, generator equipment cost and construction investment cost to achieve multi-objective collaborative optimization.

[0061]

[0062] in (2) It refers to controllable dispatch costs, including the operation and maintenance costs of green electricity and loads. This represents the price at which the park sells electricity to its loads. Represents the quantity of load. This indicates that when considering the cost of green certificates and green electricity, conversion factors such as green certificate premium and international conversion should be included. The conversion factor represents the conversion factor when renewable energy is directly supplied. These represent green electricity provided at time t by different direct supply methods and green certificates. , Representing different forms of green electricity pricing, this section quantifies the dynamic costs of green electricity procurement and load operation and maintenance, through... and By taking into account both the premium of green certificates and actual losses such as direct supply losses, the accuracy of cost calculations can be ensured. This will help distinguish between directly supplied green electricity and green certificate-based green electricity, guiding subsequent intelligent agents to make judgments about low-cost green electricity sources and improve the green electricity consumption rate.

[0063]

[0064] in (3) This represents the cost of the park purchasing electricity from the upper-level power grid, excluding green certificates. This part is a two-way interaction cost. , This represents the price at which the park sells electricity to the main grid and the price at which it purchases electricity at time t. , This represents the amount of electricity that needs to be sold or purchased from the main grid within the park, excluding the green certificate.

[0065]

[0066] In equation (4) This represents the cost of generator units within the park, of which This indicates the cost per unit of natural gas. This indicates the low calorific value of natural gas, used to quantify the fuel consumption cost of micro gas turbines.

[0067]

[0068] In equation (5) This represents the cost of constructing green electricity direct supply lines and building self-owned photovoltaic sites. It is 0 or 1, when 1 represents the green energy park choosing to build its own photovoltaic site, with a cost of ;when A value of 0 indicates that the green energy park has chosen to construct a direct green energy connection line, with a cost of [missing value]. This section weighs the cost difference between two supplementary power solutions: self-built photovoltaic systems and direct green electricity supply lines.

[0069] Furthermore, a micro gas turbine model was constructed within the green energy park. By constructing the output and output models of the micro gas turbine, a simulation of equipment operation within the green energy park was provided for subsequent judgments, bridging control actions and state changes.

[0070] As a backup power source for the industrial park, the micro gas turbine supplements the power supply in the power-grid-load-storage system. The non-linear relationship between its power generation and fuel consumption directly impacts its operating costs. It generates electricity by consuming natural gas. The power generation of the micro gas turbine can be expressed as:

[0071] (6)

[0072] In formula (6) This represents the power generation of the micro gas turbine at time t. This represents the power generation efficiency coefficient of the micro gas turbine. In actual use, the coefficient can be obtained from the equipment manual and historical experience data. This indicates the fuel flow rate.

[0073] Finally, optimization model constraints are added, including upper and lower limits of gas turbine output and ramping constraints, power constraints of distributed energy unit interconnection lines, power balance constraints, upper and lower limits of voltage and active and reactive power of each node in the green energy park distribution network, and construction economic cost constraints. These constraints are used to limit the range of action of subsequent intelligent agents, ensuring both the safety boundary of system operation and the practical feasibility of the scheme, thus guaranteeing the applicability of this application.

[0074] Among them, the upper and lower limits of gas turbine output and the ramping constraint are:

[0075]

[0076] (8)

[0077] In formula (7) This represents the maximum power generation of the gas turbine at time t, which mainly ensures the feasibility of the scheme and avoids unexpected events that could affect the overall cost due to equipment damage. At the same time, in equation (8) This represents the maximum ramp-up gradient for the gas turbine's power generation. The maximum descent gradient of the gas turbine's power generation is consistent with the physical characteristics of the equipment.

[0078] Distributed energy unit tie-line power constraints:

[0079]

[0080] In equation (9) This represents the grid-connected power of the i-th distributed energy unit. This represents the minimum grid-connected power allowed for the i-th distributed energy unit. This represents the maximum grid-connected power allowed for the i-th distributed energy unit.

[0081] Power balance constraints:

[0082]

[0083] in, This represents the energy provided by distributed energy units. This indicates the amount of electricity provided by the green certificate. This represents the electricity traded between the park and the higher-level power grid, excluding the purchase of green certificates. Representing the electrical energy consumed by the load within the park, ensuring a real-time balance between energy supply and demand, and avoiding power outages or energy waste are the core constraints for guaranteeing the reliability of power supply in the park.

[0084] The upper and lower limits of voltage and active and reactive power at each node in the green energy park distribution network are as follows:

[0085] (11)

[0086] in, This represents the active power of node i in the park's power distribution network at time t. This represents the upper and lower limits of the active power of node i in the park's distribution network at time t; similarly, This represents the reactive power of node i in the park's power distribution network at time t. This represents the upper and lower limits of the reactive power of distribution network node i at time t; This represents the voltage magnitude of distribution network node i at time t. This represents the upper and lower limits of the reactive power of node i in the park's distribution network at time t.

[0087] The maximum acceptable construction cost is represented by the following economic cost constraints:

[0088]

[0089] Furthermore, in step (2), the steps for constructing the reinforcement learning environment required for learning and training the source-grid-load-storage regulation strategy in the green power park include setting the state space, action space, and reward function to achieve closed-loop regulation from control to state transition; specifically as follows:

[0090] The control center of the green power park is considered as an intelligent agent. The data dimensions of the green power park that the intelligent agent can observe are taken as the state space, which can be represented as the multi-dimensional environmental state affecting the optimal configuration of power generation, grid, load and storage during the operation of the green power park, as shown below:

[0091] (13)

[0092] in, This represents the current time series information, derived from the current day and the current hour. This represents the load size of the green energy park at that moment. This represents the amount of distributed energy units being supplied to the green energy park at that moment. The state space represents the amount of green certificate electricity purchased by the green energy park at that moment. The design of the state space directly affects the learning effect of the agent. Too little information will prevent the agent from judging the environmental situation, while too much information will increase the computational complexity of the network and reduce the training efficiency.

[0093] The action space represents the agent's control over the output of each unit within the green energy park, as shown below:

[0094] (14)

[0095] In the formula, This is the control signal for the gas turbine's power generation. These represent the electricity purchased by the park's green certificates and distributed energy units, respectively. Similarly, the park's core control variables should be selected to avoid redundant actions and improve learning efficiency.

[0096] Therefore, in order to maximize the benefits of the green energy park, a reward function expression was designed:

[0097] (15)

[0098] In the formula, Revenue generated from selling energy in a green energy park The cost incurred for obtaining energy for a certain green energy park, and the reward function defines the optimization trend of the entire system.

[0099] The revenue generated from energy sales in green energy parks consists of two parts: one part comes from selling electricity to the upper-level power grid, and the other part comes from selling electricity to users within the park.

[0100] (16)

[0101] In equation (16), This represents the number of users within the park. This represents the load of the m-th user at time t. This represents the price at which the green energy park sells electricity to the higher-level power grid at this time. This represents the amount of electricity that the green energy park sells to the upstream power grid at this time. The cost of energy acquisition for the park comes not only from the cost of purchasing electricity from the upstream power grid using green certificates, but also from the cost of direct green electricity generation from renewable energy suppliers, as shown in (17):

[0102] (17)

[0103] For the cost of electricity generation, Let be the power generation of the micro gas turbine at time t. This represents the price at which the green energy park purchases electricity from the higher-level power grid at this time. This represents the amount of electricity that the green energy park purchases from the upper-level power grid, excluding the amount purchased through green certificates. Represents the green certificate electricity purchase price. and These represent the electricity purchase prices for direct green electricity supply and self-built photovoltaic base stations, respectively.

[0104] like Figure 2 Based on the green power park planning optimization model, the DDPG algorithm, which incorporates a context memory mechanism, is used to train and optimize the source-grid-load-storage configuration of the green power park to obtain the optimal configuration; the process includes the following steps:

[0105] (3-1) Initialize the experience pool, Q network parameters, and scenario memory playback table;

[0106] (3-2) Obtain the current environment state s t ;

[0107] (3-3) Integrating the contextual memory mechanism, select the current action a t ;

[0108] (3-4) Update the experience pool and update the scenario memory table;

[0109] (3-5) Update the Q network parameters;

[0110] (3-6) Repeat steps (3-2) to (3-5) until the maximum number of training rounds is reached.

[0111] The process of initializing the experience pool begins by randomly initializing the agent's actions, recording the state transition process of the green energy park observed by the agent, and storing the agent's actions, the corresponding state of the green energy park at this moment, the state at the next moment, and the reward function into the experience pool until the experience pool is full. At the same time, the weights of the Q network and the context memory revisit table are initialized.

[0112] Get the current environment state s t By acquiring the output of distributed energy units in the green energy park, the amount of green certificate electricity, the load size of the green energy park, and time-series information within the current time period, i.e., the corresponding data in the state space of the current green energy park, the intelligent agent can obtain the observable environmental state s. t ;

[0113] Furthermore, by integrating episodic memory mechanisms and using deep learning algorithms, the current action 'a' is selected. t；

[0114] The contextual memory mechanism first filters the highest reward value corresponding to the state-action pair in the current contextual memory table and combines it with the original TD error, i.e., the sequential difference error, which is used to measure the difference between the current estimated action value and the better estimate. During the agent's learning process, it evaluates the difference between the actual reward and the predicted reward brought by the action taken, and uses it to update the value function to improve the accuracy of the policy.

[0115] By guiding the agent's learning through the historical best policy corresponding to the current state, the optimal policy can be preserved.

[0116] The mathematical expression for the emergency memory mechanism is:

[0117] (18)

[0118] in, Corresponding to the highest historical reward, The function corresponds to the reward generated by the current state-action pair. This is an embedded function used for agent state projection, which facilitates searching the context memory table.

[0119] This will then update the experience pool and the context memory table.

[0120] Execute the current action a obtained after improvement based on the episodic memory mechanism. t Obtain reward value r t The state s in the next moment t+1The state transition process is described in terms of (s) t , a t , r t , s t+1 The strategy state-action pair is stored in the experience pool in the form of ) and the newly emerging strategy state-action pair is stored in the context memory table. If the experience pool is full, the oldest experience record is deleted. If the context memory table is full, the least used strategy is discarded.

[0121] Update Q network parameters: Randomly sample data from the experience pool and calculate the target network's predicted values.

[0122] (19)

[0123] In the formula, This represents the target network prediction value for the i-th sample. The attenuation coefficient is... Calculated for the target network The action value function under a given state.

[0124] Combining the contextual memory mechanism, the loss function is minimized as follows:

[0125] (20)

[0126] In equation (20), To evaluate the network's calculations Action value function in a state H represents the weight of the emergency memory in the loss function, and H represents the historical best policy state-action pair.

[0127] Finally, continue repeating steps (3-2) to (3-5) until the maximum number of training rounds is reached.

[0128] By selecting the policy network parameters corresponding to Lenz with the highest average reward value and the lowest loss value as the optimal configuration policy, specific configuration schemes such as the corresponding gas turbine power curve can be output.

[0129] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A source-grid-load-storage planning method for green energy parks based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Based on historical data, construct a planning optimization model for green power parks, including constructing an objective function for optimizing the configuration of power generation, grid, load, and storage in green power parks; then constructing a micro gas turbine model within green power parks; and finally adding constraints to the optimization model. S2. Construct a training environment for the optimal allocation of power generation, grid, load and storage in green energy parks, including setting the state space, action space and reward function; S3. Based on the green power park planning optimization model, the DDPG algorithm with integrated scenario memory mechanism is used to train and optimize the source-grid-load-storage configuration of the green power park to obtain the optimal configuration; including the following steps: (3-1) Initialize the experience pool, Q network parameters, and scenario memory playback table; (3-2) Obtain the current environment state s t ; (3-3) Integrating the contextual memory mechanism, select the current action a t ; (3-4) Update the experience pool and update the scenario memory table; (3-5) Update the Q network parameters; (3-6) Repeat steps (3-2) to (3-5) until the maximum number of training rounds is reached.

2. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, The objective function for optimizing the allocation of power generation, grid, load, and energy storage in green energy parks is constructed in S1, as follows: ； in, For the full life cycle cost of green energy parks, To control scheduling costs, This represents the cost of purchasing electricity from the upper-level power grid for the industrial park, excluding green certificates. This represents the cost of generator units within the park. This represents the cost required to construct green electricity direct supply lines and build self-built photovoltaic sites.

3. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, The construction of the micro gas turbine model within the green energy park is based on the power generation capacity of the micro gas turbine, as detailed below: (6) in This represents the power generation of the micro gas turbine at time t. This represents the power generation efficiency coefficient of the micro gas turbine. This indicates the fuel flow rate.

4. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, The constraints of the optimization model include upper and lower limits of gas turbine output and ramping constraints, power constraints of distributed energy unit interconnection lines, power balance constraints, upper and lower limits of voltage and active and reactive power of each node in the green energy park distribution network, and construction economic cost constraints.

5. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, The state space in S2 represents the multi-dimensional environmental states that affect the optimal configuration of power generation, grid, load, and energy storage during the operation of the green energy park; specifically as follows: (13) in For state space, This represents the current time series information, which can be obtained from the current day and the current hour. This represents the load data of the green energy park at that moment. This represents the output data of the distributed energy units received by the green energy park at that moment. This represents the amount of green certificate electricity purchased by the green energy park at that moment; The action space refers to the control actions of each energy supply point within the green energy park; specifically as follows: (14) in For the action space, This is the control signal for the gas turbine's power generation. These represent the electricity purchased from the park's green certificates and distributed energy units, respectively. Therefore, in order to maximize profits, a reward function is defined as follows: (15) in, Revenue generated from the sale of energy in green energy parks The cost incurred for obtaining energy for green energy parks.

6. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, The initialization of the experience pool, Q network parameters, and scenario memory replay table in (3-1) is as follows: The control center in the green energy park is regarded as an intelligent agent. The actions of the intelligent agent are randomly initialized and set. The state transition process of the green energy park is recorded. The current state of the green energy park, the corresponding action, the next state, and the reward function are stored in the experience pool until the experience pool is full. At the same time, the weights and the scenario memory replay table of the target Q network are initialized. The step (3-2) involves obtaining the current environmental state s. t It saves information about each dimension of the state space at the current time period.

7. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, The above (3-3) integrates the contextual memory mechanism, selecting the current action a t The contextual memory mechanism first filters the highest reward value corresponding to the state-action pair in the current contextual memory table, and combines it with the original TD error, guiding the agent's learning through the historical best strategy corresponding to the current state. The mathematical expression for the emergency memory mechanism is: (18) in, Corresponding to the highest historical reward, The function corresponds to the reward generated by the current state-action pair. For embedded functions; As mentioned in (3-4): Update the experience pool and update the context memory table. Specifically, execute the current action a obtained after the improvement of the context memory mechanism. t Obtain reward value r t The state s in the next moment t+1 The state transition process is described in terms of (s) t , a t , r t , s t+1 The experience pool is stored in the form of a new strategy state-action pair. The newly emerging strategy state-action pair is stored in the context memory table. If the experience pool is full, the oldest experience record is deleted. If the context memory table is full, the least used strategy is discarded.

8. The method for source-grid-load-storage planning in green energy parks based on deep reinforcement learning according to claim 1, characterized in that, Step (3-5): Update the Q network parameters: Data is randomly drawn from the experience pool to calculate the target network's predicted value: (19) in, This represents the target network prediction value for the i-th sample. The attenuation coefficient is... Calculated for the target network Action value function under a given state; Combining the contextual memory mechanism, the loss function is minimized as follows: (20) in, To evaluate the network's calculations Action value function in a state H represents the weight of the emergency memory in the loss function, and H represents the historical best policy state-action pair.

Images