Micro-grid energy management reinforcement learning method based on diffusion model

Abstract

The invention provides a micro-grid energy management reinforcement learning method based on a diffusion model. The micro-grid energy management reinforcement learning method comprises the steps of constructing a multi-component state space; a diffusion model is adopted to generate a continuous action strategy, and the diffusion model gradually adds noise through a forward diffusion process and reconstructs an optimal action through a reverse denoising process; according to a soft actor criticality framework, through optimization decision of a maximized weighted Q value and strategy entropy, a probability exploration enhanced bounded heavy parameterization method is adopted to calculate a strategy gradient; a priority experience playback mechanism is adopted, importance sampling experience is selected according to a time difference error, and the training stability is maintained through soft target network updating; the action safety is ensured through selective noise injection and action clipping, temperature parameters are dynamically adjusted to balance exploration and development capabilities, and a microgrid energy management model is obtained; and performing energy management on the micro-grid by adopting the micro-grid energy management model. According to the invention, an efficient and reliable solution is provided for micro-grid energy management.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of microgrid energy management, and in particular to a microgrid energy management reinforcement learning method and device based on a diffusion model. Background Art

[0002] As the global energy transition progresses, microgrids, as core infrastructure for achieving a low-carbon economy, face multiple challenges in operational optimization. Traditional optimization methods such as mathematical programming and heuristic algorithms have significant limitations when dealing with high-dimensional constraints in microgrids (such as energy storage SOC limits and equipment ramp rate constraints): the computational complexity of linear / nonlinear programming increases exponentially, making it difficult to meet the needs of real-time dynamic adjustment; while metaheuristic methods such as genetic algorithms can obtain approximate solutions, they lack global optimality guarantees and have poor dynamic adaptability. The introduction of multi-agent reinforcement learning provides a new paradigm for microgrid scheduling, but its distributed decision-making mechanism exposes problems such as difficulty in policy coordination and dimensional explosion of the action space in large-scale systems. The decisions generated by the independent policy networks of each agent lack consistency, the complex communication mechanism increases system overhead, and the exploration of the high-dimensional action space is inefficient, resulting in insufficient utilization of training samples.

[0003] In recent years, generative diffusion models have demonstrated unique advantages in the field of reinforcement learning. However, their application in microgrids still suffers from a lack of constraint embedding and an imbalance between exploration and exploitation. Existing diffusion frameworks do not explicitly address physical constraints, their offline strategies rely on expert datasets, their online learning is prone to local optima, and they lack a dynamically adaptable exploration strategy. Therefore, developing a diffusion-based reinforcement learning method and device for microgrid energy management that can effectively overcome these shortcomings has become a pressing technical challenge in the industry. Summary of the Invention

[0004] In response to the above-mentioned problems existing in the prior art, an embodiment of the present invention provides a microgrid energy management reinforcement learning method and device based on a diffusion model.

[0005] In the first aspect, an embodiment of the present invention provides a microgrid energy management reinforcement learning method based on a diffusion model, including: constructing a multi-component state space, including power state, energy storage state, load state, time state and reward state; using a diffusion model to generate a continuous action strategy, the diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process; according to the soft actor criticism framework, by maximizing the weighted Q value and policy entropy optimization decision, a probability exploration enhanced bounded reparameterization method is used to calculate the policy gradient; using a priority experience replay mechanism, importance sampling experience is selected according to the time difference error, and training stability is maintained through soft target network updates; ensuring action safety through selective noise injection and action clipping, dynamically adjusting the temperature parameter to balance the exploration and development capabilities, and obtaining a microgrid energy management model; using the microgrid energy management model to perform energy management on the microgrid.

[0006] Based on the content of the above method embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention constructs a multi-component state space, including power state, energy storage state, load state, time state and reward state, including: the power state includes the solar and wind power generation power of the previous time step and the current time step; the energy storage state includes the charging state, grid exchange power and charging power of the previous time step and the current time step; the load state includes commercial and residential load information; the time state includes the current time and electricity price information; the reward state includes the instantaneous reward value.

[0007] Based on the content of the above method embodiment, the embodiment of the present invention provides a microgrid energy management reinforcement learning method based on a diffusion model. The diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process, including:

[0008]

[0009] Among them, x0 is the initial strategy; x t For noisy strategies; is the cumulative variance; is the dynamic variance; ε θ is the learned noise prediction network; t is the duration; s is the spatiotemporal state.

[0010] Based on the content of the above method embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention adopts the priority experience replay mechanism and selects the importance sampling experience according to the time difference error, including:

[0011]

[0012] Among them, w iis the importance sampling weight; k is the sample index in the experience replay buffer; δ k is the absolute value of the time difference error of the kth sample; δ i is the time difference error of state transfer; ε is a positive constant to ensure non-zero sampling probability; prior α It is the first control priority parameter; β is the second control priority parameter; N is the experience replay cache; Q φ is the Q function defined by the parameter φ; s i is the state represented by the current sample, that is, the state of the agent at time step i; a i is the state s represented by the current sample i Action taken next; r i The state s represented by the current sample in state i Take action a i The reward obtained after γ is the discount factor, and its value range is [0,1]. Min is the minimum value symbol. is the target Q network; α is the prior distribution priority control parameter; β is the importance sampling correction adjustment parameter.

[0013] Based on the content of the above method embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, wherein the maintenance of training stability through soft target network update, includes:

[0014] φ i '←τφ i +(1- τ )φ i '

[0015] θ'←τθ+(1-τ)θ'

[0016] τ =0.005

[0017] Among them, φ i is the online critic network parameter; φ i ' is the target critic network parameter; τ is the soft update coefficient; θ is the online actor network parameter; θ' is the target actor network parameter; ← is the soft target network parameter update symbol.

[0018] Based on the content of the above method embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, which ensures action safety through selective noise injection and action tailoring, includes:

[0019]

[0020] Among them, a tis an action selection mechanism with selective noise injection; ψ(μ t ) is a bounded action; clip is a clipping function; η is Gaussian noise; p is the optimized energy scheduling strategy; p noise is the noise injection probability threshold.

[0021] Based on the content of the above method embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, wherein the dynamic adjustment of the temperature parameter balances the exploration and development capabilities, includes:

[0022]

[0023] Among them, θ e+1 is the parameter at the iteration step e+1; θ e is the parameter in the iterative step e; η a is the learning rate; To represent the e The state s represented by the current sample sampled in i Average, that is, the subsequent gradient term is the average gradient calculated over a batch of experience; B e For the experience batch; is θ e The gradient of ; H is the policy entropy; is the state s represented by the current sample i The parameter θ e The output of the defined strategy is the probability distribution of each action; T is the matrix transpose symbol; By parameter φ e Define the Q function.

[0024] In the second aspect, an embodiment of the present invention provides a microgrid energy management reinforcement learning device based on a diffusion model, including: a first main module for realizing the construction of a multi-component state space, including power state, energy storage state, load state, time state and reward state; a second main module for realizing the generation of a continuous action strategy using a diffusion model, wherein the diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process; a third main module for realizing the calculation of the policy gradient using a bounded reparameterization method enhanced by probabilistic exploration according to a soft actor criticism framework, by maximizing the optimization decision of weighted Q value and policy entropy; a fourth main module for realizing the use of a priority experience replay mechanism, selecting importance sampling experience according to the time difference error, and maintaining training stability through soft target network updates; a fifth main module for realizing the safety of actions through selective noise injection and action clipping, dynamically adjusting the temperature parameter to balance the exploration and development capabilities, and obtaining a microgrid energy management model; a sixth main module for realizing the energy management of the microgrid using the microgrid energy management model.

[0025] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0026] At least one processor, at least one memory and a communication interface; wherein,

[0027] The processor, memory and communication interface communicate with each other;

[0028] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the microgrid energy management reinforcement learning method based on the diffusion model provided by any one of the various implementation methods of the first aspect.

[0029] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute the microgrid energy management reinforcement learning method based on the diffusion model provided by any one of the various implementation methods of the first aspect.

[0030] The diffusion model-based reinforcement learning method and device for microgrid energy management provided by the embodiments of the present invention directly model high-dimensional action space through the diffusion process, and combine the bounded reparameterization mechanism enhanced by probabilistic exploration to achieve efficient coordination and stable control of microgrid resources, breaking through the bottlenecks of existing technologies in dynamic response, constraint processing and strategy smoothness, and significantly outperforming related methods in convergence speed, final performance and training stability, providing an efficient and reliable solution for microgrid energy management. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flow chart of a microgrid energy management reinforcement learning method based on a diffusion model provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the structure of a microgrid energy management reinforcement learning device based on a diffusion model provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention. If there are step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0036] The embodiment of the present invention provides a microgrid energy management reinforcement learning method based on a diffusion model. Figure 1 The method includes: constructing a multi-component state space, including power state, energy storage state, load state, time state and reward state; using a diffusion model to generate a continuous action strategy, the diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process; according to the soft actor-critic framework, through the optimization decision of maximizing the weighted Q value and policy entropy, the policy gradient is calculated using a bounded reparameterization method enhanced by probabilistic exploration; using a priority experience replay mechanism, importance sampling experience is selected according to the time difference error, and training stability is maintained through soft target network update; ensuring action safety through selective noise injection and action clipping, dynamically adjusting the temperature parameter to balance the exploration and development capabilities, and obtaining a microgrid energy management model; using the microgrid energy management model to manage the microgrid energy.

[0037] Based on the content of the above method embodiment, as an optional embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, the multi-component state space is constructed, including power state, energy storage state, load state, time state and reward state, including: the power state includes the solar energy and wind power generation power of the previous time step and the current time step; the energy storage state includes the charging state, grid exchange power and charging power of the previous time step and the current time step; the load state includes commercial and residential load information; the time state includes the current time and electricity price information; the reward state includes the instantaneous reward value.

[0038] Based on the content of the above method embodiment, as an optional embodiment, the embodiment of the present invention provides a microgrid energy management reinforcement learning method based on a diffusion model, wherein the diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process, including:

[0039]

[0040] Among them, x0 is the initial strategy; x t For noisy strategies; is the cumulative variance; is the dynamic variance; ε θ is the learned noise prediction network; t is the duration; s is the spatiotemporal state.

[0041] The optimal solution generated through the diffusion chain effectively avoids various uncertainties and significantly enhances the smoothness and simplicity of the generation strategy. This method not only ensures the quality of the generated power allocation solution, but also improves the stability and efficiency of the overall learning process.

[0042] Based on the content of the above method embodiment, as an optional embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention adopts a priority experience replay mechanism and selects importance sampling experience according to the time difference error, including:

[0043]

[0044]

[0045] Among them, w i is the importance sampling weight; k is the sample index in the experience replay buffer; δ k is the absolute value of the time difference error of the kth sample; δ i is the time difference error of state transfer; ε is a positive constant to ensure non-zero sampling probability; prior α It is the first control priority parameter; β is the second control priority parameter; N is the experience replay cache; Q φ is the Q function defined by the parameter φ; S i is the state represented by the current sample, that is, the state of the agent at time step i; a i is the state S represented by the current sample i Action taken next; r i The state S represented by the current sample in state i Take action a i The reward obtained after γ is the discount factor, and its value range is [0, 1]. Min is the minimum value symbol. is the target Q network; α is the prior distribution priority control parameter; β is the importance sampling correction adjustment parameter.

[0046] PERDASAC Policy Network Based on a Diffusion Model: This network adopts a diffusion model structure to efficiently encode the input states and , representing commercial and residential loads, respectively; and , representing the number of solar and wind turbines, as conditions, better modeling the dependencies between the observation and action spaces during the diffusion process. This fundamentally differs from traditional multi-layer perceptrons (MLPs) and significantly improves the expressive power and learning efficiency of the policy network. Experience Prioritized Buffer Replay: To address the challenge of delayed reward feedback, PERDASAC implements a prioritized experience replay mechanism that selectively samples transitions based on temporal difference (TD) errors. This approach ensures that experiences with greater learning potential receive higher priority during sampling, thereby improving convergence efficiency. The degree of priority, previously adjusted by the importance sampling correction, is a small positive constant that ensures non-zero sampling probability, and is the size of the experience replay buffer. Importance sampling weights compensate for the bias introduced by non-uniform sampling, thereby preserving the expected value of learning updates while benefiting from prioritized experience selection.

[0047] Based on the content of the above method embodiment, as an optional embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, wherein the maintenance of training stability through soft target network update includes:

[0048] φ i ←τφ i +(1-τ)φ′ i

[0049] θ′←τθ+(1-τ)θ′

[0050] τ=0.005

[0051] Among them, φ i is the online critic network parameter; φ i are the target critic network parameters; τ is the soft update coefficient; θ is the online actor network parameter; θ′ is the target actor network parameter; ← is the soft target network parameter update symbol.

[0052] Soft target network update: For both the critic and actor networks, the target network is updated using a soft update coefficient via exponential moving average, which ensures the stability of training and prevents drastic fluctuations in the estimation. For the critic and actor networks, the soft target network update mechanism is as follows: i ′ and θ′.

[0053] Based on the content of the above method embodiment, as an optional embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, wherein the selective noise injection and action tailoring are used to ensure action safety, includes:

[0054]

[0055] Among them, a t is an action selection mechanism with selective noise injection; ψ(μ t ) is a bounded action; clip is a clipping function; η is Gaussian noise; p is the optimized energy scheduling strategy; p noise is the noise injection probability threshold.

[0056] Prior to reparameterization, given that diffusion models assume data to be distributed over relatively uniform scales, microgrid systems face the challenge of significantly varying action spaces across agents. To address this, the different action dimensions are first normalized to via a linear mapping and then concatenated into a target composite action shape. This fine-tuned normalization improves the accuracy of diffusion modeling, consistent with the core principle of progressive noise addition and denoising in diffusion models. A subsequent hyperbolic tangent transformation ensures action safety by constraining outputs within a certain range. During model execution, exploratory noise is selectively injected and hard-clipping is performed, which enhances the model's generalization while preserving action boundaries.

[0057] Based on the content of the above method embodiment, as an optional embodiment, the microgrid energy management reinforcement learning method based on the diffusion model provided in the embodiment of the present invention, wherein the dynamic adjustment of the temperature parameter balances the exploration and development capabilities, includes:

[0058]

[0059] Among them, θ e+1 is the parameter at the iteration step e+1; θ e is the parameter in the iterative step e; η a is the learning rate; To represent the e The state S represented by the current sample in i Average, that is, the subsequent gradient term is the average gradient calculated over a batch of experience; B e For the experience batch; is θ e The gradient of ; H is the policy entropy; is the state S represented by the current sample i The parameter θ eThe output of the defined strategy is the probability distribution of each action; T is the matrix transpose symbol; By parameter φ e The Q function is defined. The parameter α is initialized to 0.2 and adaptively adjusted during training. The Adam optimizer is used to implement policy gradient updates with entropy regularization. GDM-based policy generation: In the absence of expert datasets, the generative diffusion model (GDM) is used to generate an optimized energy scheduling policy P, which is then optimized by analyzing the network Qu.

[0060] The diffusion model-based reinforcement learning method for microgrid energy management provided by the embodiment of the present invention directly models the high-dimensional action space through the diffusion process, and combines the bounded reparameterization mechanism enhanced by probabilistic exploration to achieve efficient coordination and stable control of microgrid resources, breaking through the bottlenecks of existing technologies in dynamic response, constraint processing and strategy smoothness, and significantly outperforming related methods in convergence speed, final performance and training stability, providing an efficient and reliable solution for microgrid energy management.

[0061] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a microgrid energy management reinforcement learning device based on a diffusion model, which is used to execute the microgrid energy management reinforcement learning method based on a diffusion model in the above method embodiment. Figure 2 The device includes: a first main module, which is used to construct a multi-component state space, including power state, energy storage state, load state, time state and reward state; a second main module, which is used to generate a continuous action strategy using a diffusion model, wherein the diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process; a third main module, which is used to calculate the policy gradient by using a bounded reparameterization method enhanced by probabilistic exploration based on a soft actor-critic framework, by maximizing the optimization decision of weighted Q value and policy entropy; a fourth main module, which is used to adopt a priority experience replay mechanism, select importance sampling experience according to the time difference error, and maintain training stability through soft target network update; a fifth main module, which is used to ensure action safety through selective noise injection and action clipping, dynamically adjust temperature parameters to balance exploration and development capabilities, and obtain a microgrid energy management model; and a sixth main module, which is used to implement energy management of the microgrid using the microgrid energy management model.

[0062] The embodiment of the present invention provides a microgrid energy management reinforcement learning device based on a diffusion model, which adopts Figure 2Several modules in it directly model high-dimensional action space through the diffusion process, and combine the bounded reparameterization mechanism enhanced by probabilistic exploration to achieve efficient coordination and stable control of microgrid resources, breaking through the bottlenecks of existing technologies in dynamic response, constraint processing and strategy smoothness. It is significantly better than related methods in convergence speed, final performance and training stability, providing an efficient and reliable solution for microgrid energy management.

[0063] It should be noted that the device in the device embodiment provided by the present invention can be used to implement the method in the above-mentioned method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above-mentioned device embodiment provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, and ensure the practicality of the technical solutions, they can improve the device in the above-mentioned device embodiment to obtain the corresponding device class embodiment, thereby obtaining the corresponding device class embodiment for implementing the methods in other method class embodiments. For example:

[0064] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the microgrid energy management reinforcement learning device based on the diffusion model provided in the embodiment of the present invention further includes: a first submodule, used to realize the construction of the multi-component state space, including power state, energy storage state, load state, time state and reward state, including: the power state includes the solar energy and wind power generation power of the previous time step and the current time step; the energy storage state includes the charging state, grid exchange power and charging power of the previous time step and the current time step; the load state includes commercial and residential load information; the time state includes the current time and electricity price information; the reward state includes the instantaneous reward value.

[0065] Based on the content of the above device embodiment, as an optional embodiment, the microgrid energy management reinforcement learning device based on the diffusion model provided in the embodiment of the present invention further includes: a second submodule, which is used to implement the diffusion model to gradually add noise through a forward diffusion process and reconstruct the optimal action through a reverse denoising process, including:

[0066]

[0067] Among them, x0 is the initial strategy; xt is the noisy strategy; is the cumulative variance; is the dynamic variance; ε θ is the learned noise prediction network; t is the duration; s is the spatiotemporal state.

[0068] Based on the content of the above device embodiment, as an optional embodiment, the microgrid energy management reinforcement learning device based on the diffusion model provided in the embodiment of the present invention further includes: a third submodule for implementing the priority experience replay mechanism and selecting importance sampling experience according to the time difference error, including:

[0069]

[0070] Among them, w i is the importance sampling weight; k is the sample index in the experience replay buffer; δ k is the absolute value of the time difference error of the kth sample; δ i is the time difference error of state transfer; ε is a positive constant to ensure non-zero sampling probability; prior α It is the first control priority parameter; β is the second control priority parameter; N is the experience replay cache; Q φ is the Q function defined by the parameter φ; s i is the state represented by the current sample, that is, the state of the agent at time step i; ai is the state s represented by the current sample i Action taken next; r i The state s represented by the current sample in state i Take action a i The reward obtained after γ is the discount factor, and its value range is [0,1]. Min is the minimum value symbol. is the target Q network; α is the prior distribution priority control parameter; β is the importance sampling correction adjustment parameter.

[0071] Based on the content of the above device embodiment, as an optional embodiment, the microgrid energy management reinforcement learning device based on the diffusion model provided in the embodiment of the present invention further includes: a fourth submodule for implementing the maintenance of training stability through soft target network update, including:

[0072] φ i '←τφ i +(1- τ )φ i '

[0073] θ'←τθ+(1-τ)θ'

[0074] τ =0.005

[0075] Among them, φ i is the online critic network parameter; φ i' is the target critic network parameter; τ is the soft update coefficient; θ is the online actor network parameter; θ' is the target actor network parameter; ← is the soft target network parameter update symbol.

[0076] Based on the content of the above device embodiment, as an optional embodiment, the microgrid energy management reinforcement learning device based on the diffusion model provided in the embodiment of the present invention further includes: a fifth submodule for ensuring action safety through selective noise injection and action tailoring, including:

[0077]

[0078] Among them, a t is an action selection mechanism with selective noise injection; ψ(μ t ) is a bounded action; clip is a clipping function; η is Gaussian noise; p is the optimized energy scheduling strategy; p noise is the noise injection probability threshold.

[0079] Based on the content of the above device embodiment, as an optional embodiment, the microgrid energy management reinforcement learning device based on the diffusion model provided in the embodiment of the present invention further includes: a sixth submodule, which is used to realize the dynamic adjustment of the temperature parameter to balance the exploration and development capabilities, including:

[0080]

[0081] Among them, θ e+1 is the parameter at the iteration step e+1; θ e is the parameter in the iterative step e; η a is the learning rate; To represent the e The state s represented by the current sample sampled in i Average, that is, the subsequent gradient term is the average gradient calculated over a batch of experience; B e For the experience batch; is θ e The gradient of ; H is the policy entropy; is the state s represented by the current sample i The parameter θ e The output of the defined strategy is the probability distribution of each action; T is the matrix transpose symbol; By parameter φ e Define the Q function.

[0082] The method of the embodiment of the present invention is implemented by electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 3As shown, the electronic device includes: at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor can call logic instructions in the at least one memory to execute all or part of the steps of the methods provided in the aforementioned method embodiments.

[0083] In addition, the logic instructions in the at least one memory mentioned above can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0086] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] It should be noted that the terms "include", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements. Any "predetermined threshold", "preset threshold" or similar expressions that do not indicate a specific value can be determined by a person of ordinary skill in the art through simple experiments or corresponding debugging.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A reinforcement learning method for microgrid energy management based on diffusion model, characterized in that: include: Construct a multi-component state space, including power state, energy storage state, load state, time state, and reward state; A continuous action policy is generated using a diffusion model that gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process. Based on a soft actor-critic framework, the policy gradient is calculated using a bounded reparameterization method enhanced by probabilistic exploration by optimizing the decision-making process by maximizing the weighted Q-value and policy entropy. A prioritized experience replay mechanism is used to select importance sampling experiences based on the temporal error, and training stability is maintained through soft target network updates. Selective noise injection and action tailoring are used to ensure action safety, and temperature parameters are dynamically adjusted to balance exploration and exploitation capabilities, resulting in a microgrid energy management model. The microgrid energy management model is used to manage the energy of the microgrid.

2. The microgrid energy management reinforcement learning method based on diffusion model according to claim 1 is characterized in that The multi-component state space is constructed, including power state, energy storage state, load state, time state and reward state, including: the power state includes the solar energy and wind power generation power of the previous time step and the current time step; the energy storage state includes the charging state, grid exchange power and charging power of the previous time step and the current time step; the load state includes commercial and residential load information; the time state includes the current time and electricity price information; the reward state includes the instantaneous reward value.

3. The microgrid energy management reinforcement learning method based on diffusion model according to claim 2 is characterized in that: The diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process, including: Among them, x0 is the initial strategy; x t For noisy strategies; is the cumulative variance; is the dynamic variance; ε θ is the learned noise prediction network; t is the duration; s is the spatiotemporal state.

4. The microgrid energy management reinforcement learning method based on the diffusion model according to claim 3 is characterized in that: The priority experience replay mechanism is used to select importance sampling experience according to the time difference error, including: Among them, w i is the importance sampling weight; k is the sample index in the experience replay buffer; δ k is the absolute value of the time difference error of the kth sample; δ i is the time difference error of state transfer; ε is a positive constant to ensure non-zero sampling probability; prior α It is the first control priority parameter; β is the second control priority parameter; N is the experience replay cache; Q φ is the Q function defined by the parameter φ; s i is the state represented by the current sample, that is, the state of the agent at time step i; a i is the state s represented by the current sample i Action taken next; r i The state s represented by the current sample in state i Take action a i The reward obtained after γ is the discount factor, and its value range is [0,1]. Min is the minimum value symbol. is the target Q network; α is the prior distribution priority control parameter; β is the importance sampling correction adjustment parameter.

5. The microgrid energy management reinforcement learning method based on diffusion model according to claim 4 is characterized in that: Maintaining training stability through soft target network updates includes: f' i ←tf i +(1-t)φ' i θ'←τθ+(1-τ)θ' τ ＝0.005 Among them, φ i is the online critic network parameter; φ' i is the target critic network parameter; τ is the soft update coefficient; θ is the online actor network parameter; θ' is the target actor network parameter; ← is the soft target network parameter update symbol.

6. The microgrid energy management reinforcement learning method based on diffusion model according to claim 5 is characterized in that: The method of ensuring motion safety through selective noise injection and motion tailoring includes: Among them, a t is an action selection mechanism with selective noise injection; ψ(μ t ) is a bounded action; clip is a clipping function; η is Gaussian noise; p is the optimized energy scheduling strategy; p noise is the noise injection probability threshold.

7. The microgrid energy management reinforcement learning method based on diffusion model according to claim 6 is characterized in that: The dynamic adjustment of temperature parameters balances exploration and development capabilities, including: Among them, θ e+1 is the parameter at the iteration step e+1; θ e is the parameter in the iterative step e; η a is the learning rate; To represent the e The state s represented by the current sample sampled in i Average, that is, the subsequent gradient term is the average gradient calculated over a batch of experience; B e For the experience batch; is θ e The gradient of ; H is the policy entropy; is the state s represented by the current sample i The parameter θ e The output of the defined strategy is the probability distribution of each action; T is the matrix transpose symbol; By parameter φ e Define the Q function.

8. A microgrid energy management reinforcement learning device based on a diffusion model, characterized in that: include: The first main module is used to realize the construction of multi-component state space, including power state, energy storage state, load state, time state and reward state; The second main module is used to implement the generation of continuous action policies using a diffusion model. The diffusion model gradually adds noise through a forward diffusion process and reconstructs the optimal action through a reverse denoising process. The third main module is used to implement the optimization decision by maximizing the weighted Q value and policy entropy according to the soft actor-critic framework, and calculate the policy gradient using the bounded reparameterization method enhanced by probabilistic exploration. The fourth main module is used to implement the priority experience replay mechanism, select the importance sampling experience according to the time difference error, and maintain training stability through soft target network updates. The fifth main module is used to ensure action safety through selective noise injection and action tailoring, dynamically adjust temperature parameters to balance exploration and development capabilities, and obtain a microgrid energy management model; the sixth main module is used to implement energy management of the microgrid using the microgrid energy management model.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which cause a computer to execute the method of any one of claims 1 to 7.

Images