Optical storage integrated device joint operation strategy generation method and system, and medium

Abstract

The invention relates to the field of new energy, in particular to a light-storage integrated device combined operation strategy generation method and system and a medium. According to the method, firstly, quantitative modeling is carried out on a single optical storage integrated micro-grid system through input / output power, a plurality of optical storage integrated micro-grid systems form an integral optical storage micro-grid system, internal energy of the integral optical storage micro-grid system is mutually complementary, and the operation cost minimization of the integral optical storage micro-grid system is taken as an optimization target. According to the method, the problem of operation cost minimization of an overall optical storage micro-grid system is converted into the problem of excitation maximization of reinforcement learning, a reinforcement learning network is constructed based on an AC framework, the reinforcement learning network comprises Actor and Critic, and an action sequence with the maximum reward is obtained through iterative calculation of multiple periods of Actor and Critic. According to the invention, energy mutual aid and energy exchange in the micro-grid formed by the multiple optical storage all-in-one machines are realized, and energy mutual aid and high-efficiency use of equipment in the micro-grid are realized.

Description

Technical Field

[0001] The present invention relates to the field of new energy, and in particular to a method, system and medium for generating a joint operation strategy for an integrated photovoltaic and storage device. Background Art

[0002] Solar energy is a widely available renewable energy source with broad application prospects. However, problems such as power generation efficiency, instability and reliability of solar power generation systems have always troubled people. In order to solve these problems, researchers began to integrate solar power generation systems with energy storage systems to form a photovoltaic and energy storage integrated machine. The photovoltaic and energy storage integrated machine is a new type of equipment that integrates solar power generation systems and energy storage systems. It converts solar energy into electrical energy through a photovoltaic power generation system and stores the electrical energy in a battery energy storage system, thereby providing power supply when needed. The photovoltaic and energy storage integrated machine integrates photovoltaic power generation and energy storage functions, which can reduce the burden on the power grid, improve the utilization rate of electricity, and has broad application prospects. In terms of the control strategy of the photovoltaic and energy storage integrated machine, researchers usually use control strategies such as PID control, fuzzy control, neural network control, and maximum power point tracking control.

[0003] However, in the scenario of a local microgrid formed by the joint operation of multiple photovoltaic and energy storage devices, considering that the capacity of photovoltaic and energy storage is different in a substation area, the joint control of multiple photovoltaic and energy storage devices can achieve internal balance, thereby reducing losses and reducing the inclusion of external power grids. There is still a lack of relevant research on how to control these photovoltaic and energy storage devices. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, system and medium for generating a joint operation strategy for an integrated photovoltaic and storage device. Aiming at the scenario of joint operation of multiple integrated photovoltaic and storage devices, a method for generating a joint operation strategy for multiple integrated photovoltaic and storage devices is designed based on a multi-agent reinforcement learning method to achieve energy mutual assistance and energy exchange within a microgrid composed of multiple integrated photovoltaic and storage devices, and to achieve energy mutual assistance and efficient use of equipment within the microgrid.

[0005] In order to solve the technical problem, the technical solution adopted by the present invention is: a method for generating a joint operation strategy of an integrated photovoltaic and storage device, comprising the following steps: S01. Quantitatively model a single photovoltaic-storage integrated microgrid system through input / output power, and regard each photovoltaic-storage integrated microgrid system as an intelligent entity. The input / output power of a single intelligent entity is the sum of its corresponding photovoltaic power generation power, photovoltaic storage power, and load consumption power; S02. Multiple integrated photovoltaic and storage microgrid systems form an overall photovoltaic and storage microgrid system. The internal energy of the overall photovoltaic and storage microgrid system is mutually beneficial, and the optimization goal is to minimize the operating cost of the overall photovoltaic and storage microgrid system; S03. Convert the problem of minimizing the operating cost of the overall photovoltaic storage microgrid system into the problem of maximizing the incentive of reinforcement learning, and define the incentive of reinforcement learning as: , Where F represents the overall photovoltaic and energy storage microgrid system operating cost; The action-value function of reinforcement learning is defined as: , in represents the action taken at time t, represents the environmental information at time t, Represents the reward of environmental information and environmental feedback, Indicates that action a is determined by strategy Generated, Represents a given strategy The expected value of the next random variable, represents the reward at time t; Defining actions The environmental information is defined as the photovoltaic power and energy storage power of each photovoltaic-storage integrated microgrid system at time t. Including electricity price information, operating cost information, and load information at time t; S04. Construct a reinforcement learning network based on the AC framework. Actor represents the learning strategy. The input is environmental information and the output is action value. Its function is to output actions under different environmental information. Critic is the learning value function. The input is environmental information and work. The output is action value Q, which is used to evaluate the value of actions under corresponding environmental information. Through iterative calculation of multiple cycles of Actor and Critic, the action sequence with maximum reward is obtained.

[0006] Furthermore, step S04) is specifically as follows: S41. Define the action function of Actor and the Critic's value network , s represents environmental information, a represents action, is the network parameter of Actor, is the network parameter of Critic; A deep neural network is used to form the Actor network and the Critic network. The Actor network inputs the environment information and outputs the action value. The Critic network inputs the environment information and action, and outputs the action value Q. S42, define the iteration cycle as M, the number of execution steps in a single cycle as T, and each time step as , construct an independent exploration strategy to undertake the exploration problem of the agent, namely: in is noise, used to add noise to the action. Execute an action , explore strategies and interact with the environment to get rewards Environmental Information , Deposit into experience pool , as feedback for interaction with the environment; S43. In the experience pool Randomly select a number of data Z from the , let: , in represents the reward of the ith agent, Represents the discount factor, which is a fixed value and is used to control the update speed of the algorithm. represents the target network of the Critic network, Represents the target network of the Actor network; Update Critic Network , Where L represents the loss function, Updating the Actor Network , Indicates the partial derivative calculation of the subscript variable. It is a form of partial derivative calculation, and the partial derivative is calculated when the lower right parameter is this value; It indicates the expected return; By soft updating the target network parameters: in, and is a preset value, Finally, the target strategy is formed , deployed as a strategy for the joint operation of multiple optical storage integrated machines.

[0007] Furthermore, in step S42, a strategy experience pool is preset to divide the working modes of a single photovoltaic-storage integrated microgrid system into the following five types: Mode 1: When there is excess photovoltaic energy, the energy storage system is charged; Mode 2: When the PV system has excess energy, it feeds electricity to the internal microgrid or grid; Mode 3: When photovoltaic power cannot meet the load demand, photovoltaic power and energy storage together supply power to the load; Mode 4: If neither photovoltaic power generation nor energy storage can meet the demand, the grid will directly supply power to the load; Mode 5: The grid charges the energy storage system; Based on the working mode of the integrated photovoltaic and storage microgrid system, Based on the single photovoltaic storage integrated microgrid system working mode above, it will be deployed in the reinforcement learning training environment to interact and generate a series of experience pool data , which is in the form of Each working mode corresponds to multiple sets of data. As an early experience pool, it provides experience for strategy learning and randomly selects data in subsequent iterations of reinforcement learning algorithms. And the data obtained through the reinforcement learning algorithm is used to update the data.

[0008] Furthermore, during the iteration process, environmental information includes load information, local electricity price information, and the energy storage capacity and maximum operating efficiency of each photovoltaic-storage integrated microgrid system.

[0009] Furthermore, in step S02, the optimization goal of the overall photovoltaic energy storage microgrid system is defined as: , in , F represents the operating cost of the overall photovoltaic and energy storage microgrid system. express The interaction cost of the entire photovoltaic storage microgrid system to the large power grid at any time is calculated by subtracting the power sales cost from the power purchase cost. represents the maintenance cost of the energy storage system at time t, represents the operation and maintenance cost of the photovoltaic system at time t, and They represent the purchased power and sold power at time t respectively; express The electricity purchase price at the time; express The electricity price at the moment.

[0010] Furthermore, in step S01, the input / output power of a single agent It is expressed as: ; in Represents photovoltaic power generation power, , Indicates the maximum power of the photovoltaic inverter, which is also the maximum value of photovoltaic power generation. It is the power output or input of the energy storage system. The positive value is the energy storage output power, and the negative value is the energy storage input power. , represents the maximum power of the energy storage converter, and Controlled by battery capacity, , represents the current battery capacity of the photovoltaic and energy storage integrated microgrid system itself, t is the time, It is the load consumption of the photovoltaic and energy storage integrated microgrid system. The default value is negative. is the real-time power of the photovoltaic and energy storage integrated microgrid system. If it is a negative value, it means that other photovoltaic and energy storage integrated microgrid systems or the large power grid need to provide it with power. If it is a positive value, it means that it transfers energy to other photovoltaic and energy storage integrated microgrid systems. N integrated photovoltaic and storage microgrid systems are combined into an overall photovoltaic and storage microgrid system. The internal energy of the overall photovoltaic and storage microgrid system is mutually beneficial. The real-time energy of the overall photovoltaic and storage microgrid system is: ; when When , it means that the energy of the entire photovoltaic storage microgrid system is negative, and it is necessary to purchase electricity from the grid. Indicates that if , it means that the overall photovoltaic and storage microgrid system has excess energy, and sells electricity to the grid. express.

[0011] Furthermore, is an OU random function.

[0012] Furthermore, the number of execution steps in a single cycle is one day, and each time step is one hour.

[0013] The present invention also discloses a system for generating a joint operation strategy for an integrated optical-storage device, comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the above-mentioned method for generating a joint operation strategy for an integrated optical-storage device when running the program instructions.

[0014] The present invention further discloses a storage medium storing program instructions, wherein when the program instructions are run, the method for generating a joint operation strategy for an integrated optical-storage device as described above is executed.

[0015] The beneficial effects of the present invention are as follows: The present invention takes into account the scenario of multiple integrated photovoltaic and storage devices operating in conjunction, and realizes the reproduction of the problem by performing power quantitative modeling on the integrated photovoltaic and storage devices. At the same time, based on the deep reinforcement learning method of policy gradient, a preset design basic strategy method is proposed based on the general operating conditions of the integrated photovoltaic and storage devices, thereby reducing the learning curve and providing a solution for multiple integrated photovoltaic and storage device operating strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the overall photovoltaic storage microgrid system. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings and specific implementations.

[0018] Example 1 The integrated photovoltaic and energy storage microgrid system is a comprehensive device that integrates photovoltaic power generation and energy storage technology. On the one hand, it integrates the energy storage system to realize the energy storage function, and on the other hand, it is connected to the photovoltaic system to regulate, store and utilize the power output of the photovoltaic power generation system, thereby improving the stability and reliability of the photovoltaic power generation system. The present invention considers the scenario where multiple integrated photovoltaic and energy storage microgrid systems form a microgrid, and is used to provide a macro control strategy for each integrated photovoltaic and energy storage microgrid system at the system level. Figure 1 As shown in the figure, the overall photovoltaic and energy storage microgrid system is composed of four photovoltaic and energy storage integrated microgrid systems. In each photovoltaic and energy storage integrated microgrid system, the photovoltaic system, energy storage system and microgrid load in the photovoltaic and energy storage integrated machine are included. These elements constitute an photovoltaic and energy storage integrated microgrid system. Multiple photovoltaic and energy storage integrated microgrid systems are connected together through the AC end to form an overall photovoltaic and energy storage microgrid system, which interacts with the power grid to obtain or sell electricity.

[0019] In order to achieve energy mutual assistance within the overall photovoltaic storage microgrid system, this embodiment discloses a method for generating a joint operation strategy for photovoltaic storage integrated devices. This method first models a single photovoltaic storage integrated microgrid system. In order to adapt the model to more photovoltaic storage integrated devices, generalization processing is performed. The design and structural differences of the photovoltaic storage integrated devices are not considered here, but quantitative modeling is performed through input and output power, etc., and each photovoltaic storage integrated microgrid system is regarded as an intelligent entity. , Indicates The input and output power of the integrated photovoltaic and storage microgrid system (internalized as one system) is : in is the photovoltaic power generation power, Indicates that the photovoltaic conversion power cannot exceed the maximum power of the photovoltaic inverter. This limited power is also the maximum power of the photovoltaic itself; It is the power output or input of the energy storage system. The positive value is the energy storage output power, and the negative value is the energy storage input power. Indicates that the maximum conversion power of the energy storage cannot exceed the power of the energy storage converter. At the same time, the energy storage charging and discharging should be performed when the battery conditions are met. The current battery capacity of the photovoltaic storage device itself is , that is, the charging and discharging power of the energy storage system is controlled by the battery capacity, and the energy storage charging and discharging should be carried out when the battery capacity is met; It is the load consumption of the integrated photovoltaic and energy storage microgrid system, which is a negative value by default. For this reason, the real-time power of the integrated photovoltaic and energy storage microgrid system, if it is a negative value, it means that other systems or the large power grid are needed to provide power. If it is a positive value, it means that energy is transmitted to other integrated photovoltaic and energy storage microgrid systems within the overall photovoltaic and energy storage microgrid system.

[0020] Will The photovoltaic and storage integrated microgrid systems are combined into an overall photovoltaic and storage microgrid system. The internal energy of the overall photovoltaic and storage microgrid system is mutually beneficial. The real-time energy of the overall photovoltaic and storage microgrid system is: when When , it means that the energy of the entire photovoltaic storage microgrid system is negative, and it is necessary to purchase electricity from the grid. If , it means that the overall photovoltaic storage microgrid system has excess energy and can return energy to the grid, that is, sell electricity. express.

[0021] The optimization goal of the overall photovoltaic storage microgrid system is defined as: , in , F represents the operating cost of the overall photovoltaic and energy storage microgrid system. express The interaction cost of the entire photovoltaic storage microgrid system to the large power grid at any time is calculated by subtracting the power sales cost from the power purchase cost. represents the maintenance cost of the energy storage system at time t, represents the operation and maintenance cost of the photovoltaic system at time t, and They represent the purchased power and sold power at time t respectively; express The electricity purchase price at the time; express The electricity selling price at the moment, these parameters are configured according to the actual situation of the area where the microgrid is located.

[0022] The scenario of designing the joint application of multiple integrated photovoltaic and energy storage microgrid systems is mainly based on the consideration that the capacities of photovoltaic and energy storage are different within a substation area. Through the joint control of multiple integrated photovoltaic and energy storage microgrid systems, internal balance can be achieved, thereby reducing losses and reducing the consumption of the external power grid.

[0023] Reinforcement learning is an important field of machine learning, and the purpose of its algorithm design is to find an action sequence to maximize the cumulative benefits. It has good adaptability to the problems involved in the present invention, especially the microgrid operation benefits within a cycle. Therefore, this embodiment designs a strategy generation method based on reinforcement learning. This embodiment transforms the problem of minimizing the operating cost of the overall photovoltaic storage microgrid system into the problem of maximizing the incentive of reinforcement learning, and defines the incentive of reinforcement learning as: , Where F represents the overall photovoltaic and energy storage microgrid system operating cost; The action-value function of reinforcement learning is defined as: , The action value function is used to describe the expected reward after taking an action in the current state. represents the action taken at time t, represents the environmental information at time t, Represents the reward of environmental information and environmental feedback, Indicates that action a is determined by strategy Generated, Represents a given strategy The expected value of the next random variable. The random variable is the one in the brackets. , represents the reward at time t.

[0024] Defining actions is the photovoltaic power and energy storage power of each photovoltaic-storage integrated microgrid system at time t, that is, .

[0025] Defining environment information It includes the electricity price information, operation cost information, and load information at time t, that is, .

[0026] In terms of network structure design, a reinforcement learning network is constructed based on the AC framework. The reinforcement learning network includes actors and critics. Actor represents the learning strategy, and its input is environmental information and its output is action value. Its function is to output actions under different environmental information. Critic is the learning value function, and its input is environmental information and work, and its output is action value Q, which is used to evaluate the value of actions under corresponding environmental information. Actor directly interacts with the output action and the environment, and obtains the evaluation index update strategy from Critic. Critic modifies the evaluation of the benefits that can be obtained from the action according to the information of the environment, and can update the value function through TD error. Through iterative calculation of multiple cycles of Actor and Critic, the action sequence with maximum reward is obtained.

[0027] In the specific learning process, at the level of Actor output action, parameterize an action function , and its network parameters are In terms of critic value function evaluation, the target network of the critic network , and its network parameters are In terms of network design, considering the deep neural network, the Actor network input is environmental information and the output is the action value, the Critic network input is environmental information and action, and the output is the action value Q.

[0028] Define the iteration cycle as , is the cycle test of algorithm iteration. The number of execution steps of a single cycle is defined as , at each time step In this embodiment, each cycle is defined as one day, and considering the electricity price income of one day, one step is executed in one hour.

[0029] Construct an independent exploration strategy to undertake the exploration problem of the agent, namely: in Specifically, the OU random function can be used to add noise to the action.

[0030] Execute an action , the microgrid strategy interacts with the environment to obtain and , Deposit into experience pool , as feedback for interaction with the environment.

[0031] In the original algorithm, the information interacting with the environment is stored in the experience pool, and the maximum number of experience pools is set, and the old data is updated with new data. However, in the problem and environment designed by the present invention, the learning curvature is too high, so it is easy to fall into the problem of local optimality. Based on the photovoltaic and storage integrated operation mode of the microgrid, the present invention proposes a method of presetting the strategy experience pool.

[0032] In the normal operation mode of a single photovoltaic storage integrated microgrid system, it can be divided into the following five working modes: Mode 1: When there is excess photovoltaic energy, the energy storage system is charged; Mode 2: When the PV system has excess energy, it feeds power to the internal microgrid or grid, making effective use of PV power generation; Mode 3: When photovoltaic power cannot meet the load demand, photovoltaic power and energy storage together supply power to the load; Mode 4: If neither photovoltaic power generation nor energy storage can meet the demand, the grid will directly supply power to the load; Mode 5: The grid charges the energy storage system.

[0033] Based on the above single optical storage integrated machine operation mode, it will be deployed in the reinforcement learning training environment to interact and generate a series of experience pool data , which is in the form of an array Each working mode corresponds to multiple sets of data. As an early experience pool, it provides experience for strategy learning. In the subsequent iteration of the reinforcement learning algorithm, data is randomly selected at the same time. And the data obtained through the reinforcement learning algorithm is used to update the data.

[0034] During the algorithm iteration process, the action environment for reinforcement learning algorithm iteration is established based on the actual usage environment, including the load information involved in the microgrid, local electricity price information, and the energy storage capacity and maximum operating power of each photovoltaic storage device, so that the action input under given environmental information can be as consistent as possible with the actual environment.

[0035] After establishing the environmental feedback mechanism, the algorithm is iterated and learned. Randomly select a number of data Z from the , let: , in represents the reward of the ith agent, Represents the discount factor, which is a fixed value and is used to control the update speed of the algorithm. represents the target network of the Critic network, Represents the target network of the Actor network; Update Critic Network , Where L represents the loss function, This is an intermediate calculation, so the meaning is not explained; Updating the Actor Network , Indicates the partial derivative calculation of the subscript variable. It is a form of partial derivative calculation, and the partial derivative is calculated when the lower right parameter is this value; It indicates the expected return; By soft updating the target network parameters: in, and is a preset value, Finally, the target strategy is formed , deployed as a strategy for the joint operation of multiple optical storage integrated machines.

[0036] During the actual operation process, real-time monitoring software is used to obtain information such as photovoltaic conversion power, load power, energy storage capacity, etc., and the information is summarized into the system through communication methods, which serves as the basis for the system to change the operating status of each photovoltaic storage system.

[0037] Example 2 The embodiment of the present disclosure provides a system for generating a joint operation strategy for an integrated optical storage device, including a processor and a memory. Optionally, the device may also include a communication interface and a bus. The processor, the communication interface, and the memory may communicate with each other through the bus. The communication interface may be used for information transmission. The processor may call the logic instructions in the memory to execute the method for generating a joint operation strategy for an integrated optical storage device of the above embodiment.

[0038] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0039] The memory, as a computer-readable storage medium, can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes the function application and data processing by running the program instructions / modules stored in the memory, that is, the method for generating a joint operation strategy for an integrated optical storage device in the above embodiment is implemented.

[0040] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory.

[0041] Example 3 An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for generating a joint operation strategy for an integrated optical-storage device.

[0042] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0043] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0044] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terms used in this application are only used to describe the embodiments and are not used to limit the scope of protection.

Claims

1. A method for generating a joint operation strategy for an integrated photovoltaic and storage device, characterized in that: The following steps are involved: S01. Quantitatively model a single photovoltaic-storage integrated microgrid system through input / output power, and regard each photovoltaic-storage integrated microgrid system as an intelligent entity. The input / output power of a single intelligent entity is the sum of its corresponding photovoltaic power generation power, photovoltaic storage power, and load consumption power; S02. Multiple integrated photovoltaic and storage microgrid systems form an overall photovoltaic and storage microgrid system. The internal energy of the overall photovoltaic and storage microgrid system is mutually beneficial, and the optimization goal is to minimize the operating cost of the overall photovoltaic and storage microgrid system; S03. Convert the problem of minimizing the operating cost of the overall photovoltaic storage microgrid system into the problem of maximizing the incentive of reinforcement learning, and define the incentive of reinforcement learning as: ， Where F represents the overall photovoltaic and energy storage microgrid system operating cost; The action-value function of reinforcement learning is defined as: ， in represents the action taken at time t, represents the environmental information at time t, Represents a given strategy The expected value of the next random variable, represents the reward at time t; Defining actions The environmental information is defined as the photovoltaic power and energy storage power of each photovoltaic-storage integrated microgrid system at time t. Including electricity price information, operating cost information, and load information at time t; S04. Construct a reinforcement learning network based on the AC framework. The reinforcement learning network includes Actor and Critic. Actor represents the learning strategy. Its input is environmental information and its output is action value. Its function is to output actions under different environmental information. Critic is the learning value function. Its input is environmental information and work. Its output is action value Q, which is used to evaluate the value of actions under corresponding environmental information. Through iterative calculation of multiple cycles of Actor and Critic, the action sequence with maximum reward is obtained.

2. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 1, characterized in that: Step S04) is specifically: S41. Define the action function of Actor and the Critic's value network , s represents environmental information, a represents action, is the network parameter of Actor, is the network parameter of Critic; A deep neural network is used to form the Actor network and the Critic network. The Actor network inputs the environment information and outputs the action value. The Critic network inputs the environment information and action, and outputs the action value Q. S42, define the iteration cycle as M, the number of execution steps in a single cycle as T, and each time step as , construct an independent exploration strategy to undertake the exploration problem of the agent, namely: in is noise, used to add noise to the action. Execute an action , explore strategies and interact with the environment to get rewards Environmental Information , Deposit into experience pool , as feedback for interaction with the environment; S43. In the experience pool Randomly select a number of data Z from the , let: ， in represents the reward of the ith agent, Represents the discount factor, which is a fixed value and is used to control the update speed of the algorithm. represents the target network of the Critic network, Represents the target network of the Actor network; Update Critic Network , Where L represents the loss function, Updating the Actor Network , Indicates the partial derivative calculation of the subscript variable. It is a form of partial derivative calculation, and the partial derivative is calculated when the lower right parameter is this value; It indicates the expected return; By soft updating the target network parameters: in, and is a preset value, Finally, the target strategy is formed , deployed as a strategy for the joint operation of multiple optical storage integrated machines.

3. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 2, characterized in that: In step S42, a strategy experience pool is preset to divide the working modes of a single photovoltaic-storage integrated microgrid system into the following five types: Mode 1: When there is excess photovoltaic energy, the energy storage system is charged; Mode 2: When the PV system has excess energy, it feeds electricity to the internal microgrid or grid; Mode 3: When photovoltaic power cannot meet the load demand, photovoltaic power and energy storage together supply power to the load; Mode 4: If neither photovoltaic power generation nor energy storage can meet the demand, the grid will directly supply power to the load; Mode 5: The grid charges the energy storage system; Based on the working mode of the integrated photovoltaic and storage microgrid system, Based on the single photovoltaic storage integrated microgrid system working mode above, it will be deployed in the reinforcement learning training environment to interact and generate a series of experience pool data , which is in the form of Each working mode corresponds to multiple sets of data. As an early experience pool, it provides experience for strategy learning and randomly selects data in subsequent iterations of reinforcement learning algorithms. And the data obtained through the reinforcement learning algorithm is used to update the data.

4. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 2, characterized in that: During the iteration process, environmental information includes load information, local electricity price information, and the energy storage capacity and maximum operating efficiency of each photovoltaic and energy storage integrated microgrid system.

5. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 1, characterized in that: In step S02, the optimization goal of the overall photovoltaic energy storage microgrid system is defined as: ， in , F represents the operating cost of the overall photovoltaic and energy storage microgrid system. express The interaction cost of the entire photovoltaic storage microgrid system to the large power grid at any time is calculated by subtracting the power sales cost from the power purchase cost. represents the maintenance cost of the energy storage system at time t, represents the operation and maintenance cost of the photovoltaic system at time t, and They represent the purchased power and sold power at time t respectively; express The electricity purchase price at the time; express The electricity price at the moment.

6. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 1, characterized in that: In step S01, the input / output power of a single agent It is expressed as: ； in represents photovoltaic power generation power, , Indicates the maximum power of the photovoltaic inverter, which is also the maximum value of photovoltaic power generation. It is the power output or input of the energy storage system. The positive value is the energy storage output power, and the negative value is the energy storage input power. , represents the maximum power of the energy storage converter, and Controlled by battery capacity, , represents the current battery capacity of the photovoltaic and energy storage integrated microgrid system itself, t is the time, It is the load consumption of the photovoltaic and energy storage integrated microgrid system. The default value is negative. is the real-time power of the photovoltaic-storage integrated microgrid system. If it is a negative value, it means that other photovoltaic-storage integrated microgrid systems or the large power grid need to provide it with power. If it is a positive value, it means that it transfers energy to other photovoltaic-storage integrated microgrid systems. N integrated photovoltaic and storage microgrid systems are combined into an overall photovoltaic and storage microgrid system. The internal energy of the overall photovoltaic and storage microgrid system is mutually beneficial. The real-time energy of the overall photovoltaic and storage microgrid system is: ； when When , it means that the energy of the entire photovoltaic storage microgrid system is negative, and it is necessary to purchase electricity from the grid. Indicates that if , it means that the overall photovoltaic and storage microgrid system has excess energy, and sells electricity to the grid. express.

7. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 2, characterized in that: is an OU random function.

8. The method for generating a joint operation strategy for an integrated photovoltaic and storage device according to claim 2, characterized in that: The execution steps of a single cycle are one day, and each time step is one hour.

9. A system for generating a joint operation strategy for an integrated optical storage device, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the method for generating a joint operation strategy for an integrated optical-storage device as described in any one of claims 1 to 8 when running the program instructions.

10. A storage medium storing program instructions, characterized in that: When the program instructions are executed, the method for generating a joint operation strategy for an integrated optical-storage device as described in any one of claims 1 to 8 is executed.

Images