Charging and discharging station control method and device, storage medium and electronic device

Abstract

The invention provides a charging and discharging station control method and device, a storage medium and an electronic device, and the method comprises the steps: obtaining a state parameter set of a charging and discharging station, and enabling state parameters in the state parameter set to be used for representing the state of the charging and discharging station and the state of related equipment of the charging and discharging station; the state parameter set is input into a strategy model, the strategy model is used for analyzing the state parameter set, an action parameter set of the charging and discharging station is obtained, the strategy model is obtained through training of state parameter set samples, and action parameters in the action parameter set are used for representing the power of the charging and discharging station and the power of associated equipment; determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set; and according to the control strategy, the charging and discharging station is controlled, so that the problem of low efficiency of performing energy control on the charging and discharging station is solved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and intelligent control, and in particular to a control method, device, storage medium and electronic device for a charging and discharging station. Background Art

[0002] At present, traditional electric vehicles participate in grid regulation, usually through vehicle-to-grid (V2G) technology to control the charging time and charging and discharging power of electric vehicles, which requires real-time communication and control between electric vehicles, charging piles and grids, and has problems such as high technical complexity and inconsistent standards. In addition, when a large number of vehicles participate in V2G at the same time, grid management will become more complicated, more difficult to implement, more costly, and weak active regulation capabilities, resulting in low efficiency in energy control of charging and discharging stations.

[0003] Therefore, there is currently no effective solution to the above-mentioned problem of low efficiency in energy control of charging and discharging stations. Summary of the invention

[0004] Embodiments of the present invention provide a control method, device, storage medium and electronic device for a charging and discharging station, so as to at least solve the problem of low efficiency in energy control of the charging and discharging station.

[0005] According to an embodiment of the present invention, a control method for a charging and discharging station is provided. The method may include: obtaining a state parameter set of the charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station; inputting the state parameter set into a strategy model, and using the strategy model to analyze the state parameter set to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using state parameter set samples, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; based on the state parameter set and the action parameter set, determining a control strategy of the charging and discharging station, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for the energy storage system in the charging and discharging station to store energy; and controlling the charging and discharging station according to the control strategy.

[0006] Optionally, the state parameter set is analyzed by using the strategy model to obtain the action parameter set of the charging and discharging station, including: calling the strategy model and determining the action parameter set matching the state parameter set according to the target step size.

[0007] Optionally, the method further includes: updating the state parameter set to obtain an updated state parameter set, and adjusting the action parameter set to obtain an adjusted action parameter set.

[0008] Optionally, based on the state parameter set and the action parameter set, a control strategy of the charging and discharging station is determined, including: determining the control strategy based on the state parameter set, the updated state parameter set, the adjusted action parameter set, and the reward function of the charging and discharging station, wherein the reward function is obtained by weighted summation of multiple sub-reward functions, and the sub-reward function is used to characterize the adjustment target of the charging and discharging station.

[0009] Optionally, adjusting the action parameter set to obtain an adjusted action parameter set includes: adding noise data to the action parameter set to obtain the adjusted action parameter set.

[0010] Optionally, the method further includes: training an initial value model based on state parameter set samples and action parameter set samples to obtain a value model; obtaining value parameters output by the value model, wherein the value parameters are used to characterize a scoring level of the control strategy.

[0011] Optionally, the method further includes: obtaining a policy gradient parameter of the policy model; adjusting the value parameter based on the policy gradient parameter to obtain a target value parameter, wherein the target value parameter is used to characterize whether the score level of the control strategy reaches a level threshold.

[0012] Optionally, the method further includes: updating the policy model based on the policy gradient parameters, and updating the value model based on the loss function of the value model.

[0013] Optionally, the method further includes: correcting the state parameter set to obtain a corrected state parameter set, and correcting the action parameter set to obtain a corrected action parameter set; and correcting the control strategy using the corrected state parameter set and the corrected action parameter set.

[0014] According to another embodiment of the present invention, a control device for a charging and discharging station is provided. The device may include: an acquisition unit, used to acquire a state parameter set of the charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station; an analysis unit, used to input the state parameter set into a strategy model, and analyze the state parameter set using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using state parameter set samples, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; a determination unit, used to determine a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for the energy storage system in the charging and discharging station to store energy; a control unit, used to control the charging and discharging station according to the control strategy.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0016] According to another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to another embodiment of the present invention, a computer program product is provided, which may include a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0018] According to another embodiment of the present invention, a computer program product is provided, which may include a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and the computer program is configured to execute the steps of any of the above method embodiments when executed.

[0019] According to another embodiment of the present invention, a computer program is provided, which is configured to execute the steps in any of the above method embodiments when running.

[0020] Through the embodiment of the present invention, a state parameter set of a charging and discharging station is obtained, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station; the state parameter set is input into a strategy model, and the state parameter set is analyzed by using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; based on the state parameter set and the action parameter set, a control strategy of the charging and discharging station is determined, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for energy storage of the energy storage system in the charging and discharging station; according to the control strategy, the charging and discharging station is controlled, thereby solving the technical problem of low efficiency of energy control of the charging and discharging station and achieving the technical effect of improving the efficiency of energy control of the charging and discharging station. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1It is a hardware structure block diagram of a mobile terminal of a control method for a charging and discharging station according to an embodiment of the present invention;

[0023] Figure 2 is a flow chart of a control method of a charging and discharging station according to an embodiment of the present invention;

[0024] Figure 3 is a flow chart of an energy control strategy of an S2G ​​charging and discharging station based on a DDPG algorithm according to an embodiment of the present invention;

[0025] Figure 4 is a basic structural schematic diagram of a photovoltaic S2G charging and discharging station according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of energy information exchange based on an S2G ​​charging and discharging station according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of a DDPG algorithm according to an embodiment of the present invention;

[0028] Figure 7 4 is a structural block diagram of a control device for a charging and discharging station according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0031] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a control method for a charging and discharging station according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal 10 may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the control method of the charging and discharging station in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] In this embodiment, a control method for a charging and discharging station running on the above mobile terminal is provided. Figure 2 is a flow chart of a control method of a charging and discharging station according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0035] Step S202: acquiring a state parameter set of the charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station.

[0036] In the technical solution provided in step S202 of the present invention, a state parameter set of the charging and discharging station can be obtained. The state parameter set can be a state space, which can be expressed as S t The state parameter set may include multiple state parameters, which may be used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station. The associated equipment of the charging and discharging station may include at least photovoltaic equipment and vehicles, etc. This is only an example and does not specifically limit the type of associated equipment.

[0037] Optionally, according to the structural characteristics of the charging and discharging station, the charging and discharging station is roughly divided into four parts, namely, the energy storage system, photovoltaic power generation in the station and photovoltaic power generation near the station, real-time charging power of electric vehicles, and other loads in the station. In order to facilitate the real-time monitoring and control of the energy from the charging and discharging station to the grid (station-to-grid, referred to as S2G), the charging piles in the station can be divided into several sub-modules for management. The state space parameters of each part in the S2G charging and discharging station, that is, the state parameter set, can at least include the battery status of the energy storage system in the charging and discharging station, the photovoltaic power generation power, the real-time charging power of electric vehicles, etc. Since other loads in the charging and discharging station are relatively fixed and fluctuate slightly, real-time changes are not considered for the time being.

[0038] Optionally, the battery state of the energy storage system can be the percentage of the current remaining power of the energy storage system battery relative to the total battery capacity, which can be represented by SOC(t). The photovoltaic power generation power can be the active power output generated by photovoltaic power generation at the current moment, which can be represented by P(t). pv The real-time charging power of electric vehicles can be the charging power of electric vehicles connected to the charging and discharging station at the current moment. The real-time power of each module can be monitored according to the modular partitioning situation, which can be expressed by P Li (t) is represented. Therefore, the above state parameter set (i.e., state space) can be represented as S t =[SOC(t),P(t) pv ,P Li (t)].

[0039] Step S204, inputting the state parameter set into the strategy model, and using the strategy model to analyze the state parameter set to obtain the action parameter set of the charging and discharging station, wherein the strategy model is trained using the state parameter set sample, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment.

[0040] In the technical solution provided in the above step S204 of the present invention, after obtaining the state parameter set of the charging and discharging station, the state parameter set can be input into the strategy model, and the state parameter set can be analyzed by using the strategy model to obtain the action parameter set of the charging and discharging station. The strategy model can be obtained by training using the state parameter set sample, and can also be called a strategy network, which can be represented by μ(s|θ μ ). The action parameter set can be an action space, which can be represented by A t The action parameter set may include multiple action parameters, which may be used to characterize the power of the charging and discharging station and the power of associated equipment.

[0041] It should be noted that the action space describes the actions that the agent can take at each time step (which can be represented by t). The main objects to be adjusted in the S2G charging and discharging station are the real-time charging and discharging power of the energy storage system and the real-time charging power of the charging pile. The charging power, discharging power of the energy storage system, and the charging power of the charging pile are controlled according to different environments. The idea of ​​adjusting the above parameters in the actual S2G charging and discharging station is that when the load of the power grid is stable, when the load of the electric vehicle is small, the photovoltaic power mainly discharges to the power grid and the energy storage system charges; when the load of the electric vehicle is large, the photovoltaic power mainly charges the electric vehicle, the energy storage system discharges, and the charging power of the charging pile is appropriately adjusted; when the load of the power grid is at the bottom, the energy storage system is charged, and the charging pile charges the electric vehicle at the maximum charging power; when the load of the power grid is at a peak, the energy storage system discharges to the power grid and reduces the charging power of the charging pile. In more serious cases, the charging pile discharges to the power grid through the battery of the electric vehicle.

[0042] Optionally, the action parameter set may include at least the charging power of the energy storage system, the discharging power of the energy storage system and the charging power of the charging pile. t c , P c (t). The discharge power of the energy storage system can be expressed as P t d , P d (t). The charging power of the charging pile can be expressed by P Lc (t) is indicated.

[0043] Optionally, the charging power P t c It refers to the active power absorbed by the energy storage system from the grid or photovoltaic system at time step t, P t c The value range of is: Among them, P t c It can be used to represent the charging power at the current time step t, It can be used to indicate the maximum charging power limit of the energy storage system. Discharge power P t d It refers to the active power output by the energy storage system to the load or grid at time step t, P t d The value range is: Among them, P t d It can be used to represent the discharge power at the current time step t, It can be used to indicate the maximum discharge power limit of the energy storage system.

[0044] It should be noted that during the charging and discharging process of the energy storage system, in order to prevent over-discharge or over-charge, the battery state SOC(t) of the energy storage system should be constrained and adjusted:

[0045]

[0046] SOC min ≤SOC(t)≤SOC max

[0047] Among them, η c , η d , It can be used to represent the charging efficiency, discharging efficiency and self-discharge rate of the energy storage system, S E Can be used to represent the total capacity of the energy storage system, SOC min , SOC max They can be used to indicate the lower and upper limits of the state of charge of the energy storage system respectively.

[0048] Optionally, P Lc (t) is the charging power of the charging pile corresponding to the current time step t, which is adjusted in real time according to the number of electric vehicles currently connected to the station. Lc The value range of (t) is: in, It can be used to indicate the maximum value of the power that an electric vehicle sends back to the grid in extreme cases. Under normal circumstances, it should be 0, representing the minimum charging power of the module charging pile. It can be used to indicate the maximum limit of the charging power of the charging pile.

[0049] Optionally, the policy network μ(s|θ μ ) is the input of the energy storage system battery status, photovoltaic real-time power generation, charging load, etc. in the state space, and the output is a continuous action A t , such as the charging power and discharging power of energy storage, and the real-time power of charging piles. The main output of this strategy is to dynamically allocate the charging and discharging of the energy storage system based on the real-time photovoltaic power generation and the power of the electric vehicle being charged, and to adjust the power of the charging pile to charge the electric vehicle in real time. μ is the strategic network parameter set, which mainly includes the control strategy of the S2G charging and discharging station, such as the charging and discharging power P of the energy storage c (t), P d (t) and charging power P of the charging pile Lc (t).

[0050] Step S206, based on the state parameter set and the action parameter set, determine the control strategy of the charging and discharging station, wherein the control strategy is used to at least characterize the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for energy storage of the energy storage system in the charging and discharging station.

[0051] In the technical solution provided in the above step S206 of the present invention, after obtaining the action parameter set of the charging and discharging station, the control strategy of the charging and discharging station is determined based on the state parameter set and the action parameter set. The control strategy can be used to at least characterize the rules for the associated device to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated device, and the rules for the energy storage of the energy storage system in the charging and discharging station, and can also be called the S2G charging and discharging station energy control strategy.

[0052] Optionally, after obtaining the state parameter set and action parameter set of the charging and discharging station, at least the state parameter set and the action parameter set can be combined to obtain empirical data, which can at least be used to train the strategy model. The strategy model can be further updated until the strategy converges, thereby obtaining a control strategy.

[0053] Step S208: controlling the charging and discharging station according to the control strategy.

[0054] In the technical solution provided in the above step S208 of the present invention, after the control strategy of the charging and discharging station is determined based on the state parameter set and the action parameter set, the charging and discharging station can be controlled according to the determined control strategy.

[0055] In the above steps S202 to S208 of the present invention, a state parameter set of the charging and discharging station is obtained, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station; the state parameter set is input into the strategy model, and the state parameter set is analyzed by using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; based on the state parameter set and the action parameter set, a control strategy of the charging and discharging station is determined, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for energy storage of the energy storage system in the charging and discharging station; the charging and discharging station is controlled according to the control strategy, thereby solving the technical problem of low efficiency of energy control of the charging and discharging station and achieving the technical effect of improving the efficiency of energy control of the charging and discharging station.

[0056] The above method of this embodiment is further introduced below.

[0057] As an optional embodiment, step S204, using the strategy model to analyze the state parameter set to obtain the action parameter set of the charging and discharging station, includes: calling the strategy model, and determining the action parameter set matching the state parameter set according to the target step size.

[0058] In this embodiment, after obtaining the state parameter set, the policy model may be called to determine the action parameter set matching the state parameter set according to the target step length, wherein the target step length may be represented by t.

[0059] For example, we can divide 24 hours a day into 24 time steps t, and calculate S through the policy network every hour. t The optimal action in state A t , that is, determine the action parameter set that matches the state parameter set.

[0060] As an optional embodiment, the method further includes: updating the state parameter set to obtain an updated state parameter set, and adjusting the action parameter set to obtain an adjusted action parameter set.

[0061] In this embodiment, after obtaining the state parameter set and the action parameter set, the state parameter set can be updated to obtain an updated state parameter set, and the action parameter set can be adjusted to obtain an adjusted action parameter set. The updated state parameter set can be obtained by S t+1 To express.

[0062] Optionally, in order to avoid the strategy from converging to a local optimal solution and enhance the exploration capability, the action parameter set can be adjusted to obtain an adjusted action parameter set. For example, in the original action parameter set A t Add noise N based on t , that is, A t =μ(s t |θ μ )+N t , thus obtaining the adjusted action parameter set.

[0063] As an optional embodiment, step S206, based on the state parameter set and the action parameter set, determines the control strategy of the charging and discharging station, including: determining the control strategy based on the state parameter set, the updated state parameter set, the adjusted action parameter set, and the reward function of the charging and discharging station, wherein the reward function is obtained by weighted summation of multiple sub-reward functions, and the sub-reward function is used to characterize the adjustment target of the charging and discharging station.

[0064] In this embodiment, the control strategy can be determined based on the state parameter set, the updated state parameter set, the adjusted action parameter set, and the reward function of the charging and discharging station. The reward function is obtained by weighted summation of multiple sub-reward functions, which can be expressed as rt The sub-reward function is used to characterize the adjustment target of the charging and discharging station, that is, each sub-reward corresponds to an optimization target.

[0065] Optionally, the reward function r t It usually consists of multiple sub-rewards, each of which corresponds to an optimization goal. Economic benefits are the main goal of S2G charging and discharging stations that take photovoltaics into account. The economic benefits are mainly maximized through low-price charging and high-price discharging. The income reward can be expressed by the following formula:

[0066] r t p =α×(P t sell ×P t d -P t buy ×P t c )

[0067] Among them, r t p Can be used to represent revenue rewards, P t sell , P t buy They can be used to represent the electricity selling price and electricity purchasing price at the current time step t, respectively, and α can be used to represent the weight of economic benefits. In order to extend the battery life of the energy storage system, the reward function should take into account the health status of the battery. The battery health reward can be expressed by the following formula:

[0068]

[0069] Among them, r t h can be used to represent the battery health reward, β can be used to represent the weight of battery health management, P t c It can be used to represent the charging power at the current time step t, It can be used to indicate the maximum charging power limit of the energy storage system, P t d It can be used to represent the discharge power at the current time step t, It can be used to indicate the maximum discharge power limit of the energy storage system. The charging and discharging station must not only meet the load requirements of electric vehicles, but also maintain the stability of the power grid to avoid over-reliance on the power grid or excessive fluctuations. The load balancing reward can be expressed by the following formula:

[0070] r t b =-γ×|∑P Li -(P td -P t c )|

[0071] Among them, r t b can be used to represent the load balancing reward, γ can be used to represent the load balancing weight, P Li It can be used to represent the real-time charging power of electric vehicles. Photovoltaic power generation can improve the system's clean energy utilization efficiency and reduce dependence on the power grid. The photovoltaic power generation utilization reward can be expressed by the following formula:

[0072] r t pv =δ×(P t pv ,∑P Li -P t c +P t d )

[0073] Among them, r t pv can be used to indicate the photovoltaic power generation utilization reward, and δ can be used to indicate the photovoltaic power generation utilization weight. It can be used to represent photovoltaic power generation. The final reward function r t The weighted sum of the above sub-rewards is:

[0074] r t =r t p +r t h +r t b +r t pv

[0075] Among them, the selection of weights needs to be continuously adjusted according to the simulation and actual conditions to find the optimal balance point.

[0076] Optionally, the agent performs action A t , update the state S based on the current state and action t+1 , and returns the immediate reward r t , the empirical data generated by this (S t ,A t ,r t ,S t+1 ) is stored in the experience replay pool (which can be represented by D). A mini-batch (S i ,A i ,r i ,S i+1) is used for training, and the size of the mini-batch is usually set to N. Among them, the experience replay pool stores the experience data generated during the interaction between the agent and the environment, and the format is (S t ,A t ,r t ,S t+1 ).

[0077] As an optional embodiment, adjusting the action parameter set to obtain an adjusted action parameter set includes: adding noise data to the action parameter set to obtain the adjusted action parameter set.

[0078] In this embodiment, noise data may be added on the basis of the action parameter set, thereby obtaining an adjusted action parameter set.

[0079] As an optional embodiment, the method also includes: training the initial value model based on the state parameter set samples and the action parameter set samples to obtain a value model; obtaining value parameters output by the value model, wherein the value parameters are used to characterize the scoring level of the control strategy.

[0080] In this embodiment, the initial value model is trained based on the state parameter set samples and the action parameter set samples to obtain the value model. After the value model is obtained, the value parameters output by the value model can be obtained. The value model can be a value network, which can also be called a Q network, and can be represented by Q(s, a|θ Q The value parameter can be used to characterize the scoring level of the control strategy, and can be represented by a Q value, which can be a reference for evaluating the control strategy.

[0081] Optionally, the value network Q(s,a|θ Q ) includes the state S t and action A t , the output is the Q value of the state-action pair, and the input layer contains the state S t and action A t , the hidden layer uses a fully connected layer, extracts the joint features of state and action through a nonlinear activation function, and the output layer outputs a scalar Q value, which represents the expected cumulative reward after executing the action, mainly reflecting the value fed back to the power grid after adjusting the charging and discharging power of the energy storage system and the charging pile power. Among them, θ Q is the strategic action, including the charging and discharging power P of the energy storage c (t), P d (t) and charging power P of the charging pile Lc (t) The corresponding value after the action.

[0082] Optionally, the target policy network and the target Q network are used to calculate the target Q value, which reflects the expected total reward of the agent in the future:

[0083] y i =r i +λQ'(S i+1 ,μ'(S i+1 |θ μ' )|θ Q' )

[0084] Among them, y i It can be used to represent the target Q value, μ' can be used to represent the target policy network, Q' can be used to represent the target Q network, and λ can be used to represent the discount factor, balancing the immediate reward and the long-term reward. In the day-ahead model, the long-term reward accounts for a larger proportion, and in the intraday regulation, the immediate reward accounts for a larger proportion. Q'(S i+1 ,μ'(S i+1 |θ μ ')|θ Q ') is the target Q network output based on the next state.

[0085] As an optional embodiment, the method also includes: obtaining a policy gradient parameter of the policy model; adjusting the value parameter based on the policy gradient parameter to obtain a target value parameter, wherein the target value parameter is used to characterize whether the score level of the control strategy reaches a level threshold.

[0086] In this embodiment, the policy gradient parameters of the policy model can be obtained. After obtaining the policy gradient parameters, the value parameters are adjusted based on the policy gradient parameters to obtain the target value parameters. Among them, the target value parameters are used to characterize whether the score level of the control strategy reaches the level threshold, which can be represented by the target Q value.

[0087] Optionally, the update goal of the policy network is to maximize the output value of the Q network. The policy gradient is calculated by differentiating the output of the Q network:

[0088]

[0089] The left side of the above equation represents the policy gradient, and the right side is the result of differentiating the output of the Q network. The gradient indicates the parameter θ of the policy network. μ How should it be adjusted so that the output can maximize the Q value in a given state?

[0090] As an optional embodiment, the method further includes: updating the policy model based on the policy gradient parameters, and updating the value model based on the loss function of the value model.

[0091] In this embodiment, in order to optimize the parameters θ of the Q network Q , define the loss function L(θ Q ) is the target Q value and the current Q network estimated value Q(Si ,A i |θ Q ) is the mean square error between:

[0092]

[0093] By minimizing the loss function L(θ Q ) Use gradient descent to update the parameters θ of the Q network Q :

[0094]

[0095] Among them, η Q is the learning rate, which controls the step size of parameter updates.

[0096] Optionally, update the policy network parameters θ using the policy gradient method μ : Among them, η μ is the learning rate of the policy network.

[0097] As an optional embodiment, the method also includes: correcting the state parameter set to obtain a corrected state parameter set, and correcting the action parameter set to obtain a corrected action parameter set; and correcting the control strategy using the corrected state parameter set and the corrected action parameter set.

[0098] In this embodiment, the state parameter set can be modified to obtain a modified state parameter set, and the action parameter set can be modified to obtain a modified action parameter set. The control strategy can be modified by further using the modified state parameter set and the modified action parameter set. The modified state parameter set can be a modified state space. The modified action parameter set can be a modified action space.

[0099] Optionally, the modified state space is set as follows: Among them, ΔP i p It can be used to represent the difference between the short-term predicted power of electric vehicles and the day-ahead predicted power. It can be used to represent the SOC deviation at time t+1 if the energy storage output is according to the day-ahead plan at time step t.

[0100] Optionally, the agent corrects the energy storage power and the charging pile power based on the day-ahead planned value, and the corrected action space is: Among them, ΔP ES (t) can be used to express the correction amount to the energy storage power, ΔP Li (t) can be used to indicate the correction amount for the charging pile power.

[0101] This embodiment obtains a state parameter set of the charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station; the state parameter set is input into a strategy model, and the state parameter set is analyzed by using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; based on the state parameter set and the action parameter set, a control strategy of the charging and discharging station is determined, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for energy storage of the energy storage system in the charging and discharging station; the charging and discharging station is controlled according to the control strategy, thereby solving the technical problem of low efficiency of energy control of the charging and discharging station and achieving the technical effect of improving the efficiency of energy control of the charging and discharging station.

[0102] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0103] At present, with the advancement of carbon peak and carbon neutrality goals, the number of electric vehicles has continued to rise in recent years. Large-scale access to the power grid is prone to load shortages, and the proportion of distributed energy access to the power grid is also increasing year by year. However, distributed photovoltaics are greatly affected by weather factors. How to solve the impact of large-scale electric vehicle charging loads on the power grid and absorb the output of distributed photovoltaics are urgent issues to be solved.

[0104] The safety and stability of the distribution network are closely related to distributed photovoltaics and electric vehicles. Distributed photovoltaic power generation can generate clean electricity near users and reduce dependence on traditional power transmission. However, with a high proportion of distributed photovoltaic access, the average installed capacity per household far exceeds the average household electricity load, resulting in reverse flow and heavy overload in the substation area, causing problems such as voltage exceeding the limit in the substation area and reverse heavy overload in the upper substation. At the same time, electric vehicles are not only the main force of electricity consumption, but also can provide energy storage support for the power grid through bidirectional charging technology, helping to balance the load and improve the absorption capacity of renewable energy. However, the current regulation model of traditional electric vehicles in the process of participating in regulation is not comprehensive enough, and in actual situations, the behavior and charging demand of car owners have random characteristics. The traditional vehicle-to-grid (V2G) research process tends to ignore the spatiotemporal characteristics of electric vehicles, and does not consider the impact of the V2G process on the needs of car owners, making it difficult to reduce the negative impact of excessive discharge of electric vehicles.

[0105] At present, traditional electric vehicles participating in grid regulation usually use V2G technology to control the charging time and charging and discharging power of electric vehicles. However, V2G requires real-time communication and control between electric vehicles, charging piles and power grids. The technology is highly complex and the standards are not unified. It is usually impossible to take into account the interests of both the supply and demand sides. If a large number of vehicles participate in V2G at the same time, grid management will become more complicated, more difficult to implement and more costly. The active regulation capability is weak, which may lead to local grid instability and make it difficult to relieve grid pressure during charging peaks. As a result, there is a problem of low efficiency in energy control of charging and discharging stations.

[0106] In order to solve the above problems, the present invention proposes an energy control strategy for S2G charging and discharging stations based on the Deep Deterministic Policy Gradient (DDPG) algorithm, and defines a charging station that applies the electric vehicle charging station to the power grid interaction technology as an S2G ​​charging and discharging station. The electric vehicle charging station to the power grid interaction technology treats the entire charging station as a whole object, and through the station-level energy management system and the characteristics of the energy storage system, the various charging piles in the station are grouped and modularly designed. The energy storage system is used to balance the load difference between the power grid and a large number of electric vehicle loads, and replaces electric vehicles to participate in the power grid power regulation. Compared with traditional charging stations, electric vehicles participating in the power grid regulation have more advantages.

[0107] The main content of the S2G charging and discharging station energy control method based on the DDPG algorithm described in this embodiment is to take photovoltaics, energy storage systems, and electric vehicle loads as intelligent agents. Based on the DDPG algorithm, on the basis of continuous learning and interaction between the intelligent agents and the environment, with the goal of maximizing economic benefits and smoothing load fluctuations, the charging and discharging station can be managed intelligently and meticulously, resource utilization efficiency can be improved, and the adaptability of the charging and discharging station to a dynamic environment can be enhanced.

[0108] Due to the randomness of electric vehicle charging, there will be multiple charging load peaks within 24 hours a day. The first is the peak caused by the charging of electric taxis on night shift in the early morning, the second is the peak caused by the charging of electric buses and electric taxis at noon, and the third is the peak caused by a large number of residents charging their electric vehicles after returning home in the evening. Based on the time characteristics of the above charging load peaks, combined with the structural characteristics and operation characteristics of the S2G charging and discharging station, the time period is divided according to the peak value of the charging load, and the following three important operation scenarios are defined as the basic strategy for the day-ahead regulation of the S2G charging and discharging station:

[0109] During the time period from 0:00 to 7:00, the main load of the S2G charging and discharging station includes the charging peak of electric taxis at night, the charging load of the energy storage system and other basic loads. During this stage, the operating strategy of the S2G charging and discharging station is to purchase electricity from the power grid to meet the load demand within the station.

[0110] During the time period from 11:00 to 15:00, the main load of the S2G charging and discharging station includes electric buses and electric taxis charging at noon, as well as disorderly charging of some private cars. In summer, the power grid is also accompanied by a peak in residential electricity consumption. The main operating strategy of the S2G charging and discharging station at this stage is: when there is sufficient photovoltaic power, the photovoltaic system is used to charge the charging load first, and when there is a surplus of photovoltaic power, the excessive residential load is discharged to the grid; when there is insufficient photovoltaic power, on the basis of utilizing photovoltaic power, electricity is purchased from the grid appropriately according to the load situation, and the peak load is smoothed through the energy storage system and the reduction of the charging power of each charging pile, thereby reducing the impact of the charging load on the grid.

[0111] During the time period from 18:00 to 22:00, the load structure of the S2G charging and discharging station includes charging of various electric vehicles, which is complex and difficult to adjust. At the same time, the power grid is also accompanied by a peak in residential load. The main operating strategy of the S2G charging and discharging station at this stage is to give priority to using the energy storage system to charge the electric vehicle load, while reducing the charging efficiency of each charging pile, staggering the peak period of residential electricity consumption and the peak period of electric vehicle charging, and after smoothly passing the peak period of the power grid, use the power grid to charge the energy storage system and electric vehicles that have not been fully charged.

[0112] At the same time, if a considerable number of electric vehicles enter or leave the charging and discharging station at the same time, it will cause an instantaneous imbalance in the charging and discharging power of the charging and discharging station, thereby causing voltage fluctuations. In order to reflect the real-time adjustment capability of the S2G charging and discharging station for such scenarios, two working scenarios are defined as the power strategy for real-time adjustment within the day: the charging vehicle enters the charging and discharging station and leaves the charging and discharging station. When a vehicle enters the charging and discharging station, the charging and discharging station reasonably allocates the power in the station based on factors such as the vehicle's charging demand, the allowed charging load in the station, and the capacity of the energy storage battery; when a vehicle leaves the charging and discharging station, the charging and discharging station gives priority to reducing the energy storage output power based on the available power, and then reasonably adjusts the power distribution in the station based on the existing charging power. At the same time, S2G charging and discharging stations should also participate in grid regulation. When the grid needs to reduce the power of the charging and discharging station, the charging power of each charging pile will be reduced in the station, and the energy storage discharge power will be adjusted to meet the load demand. When the grid needs the S2G charging and discharging station to discharge to it, the charging and discharging station will use the energy storage system to discharge to the grid and reduce the charging power of the charging pile. In more serious cases, electric vehicles will be used as energy storage batteries to discharge to the grid through S2G charging and discharging equipment.

[0113] Based on the above power strategy, this embodiment proposes an S2G ​​charging and discharging station energy control strategy based on the DDPG algorithm, which solves the problems of the current traditional V2G technology due to its complex technology and difficulty in handling large-scale electric vehicles participating in grid control at the same time, while taking photovoltaics into account, and optimizes and improves the load quality of electric vehicles and the operability of charging and discharging stations participating in grid regulation.

[0114] Figure 3 is a flow chart of an energy control strategy of an S2G ​​charging and discharging station based on a DDPG algorithm according to an embodiment of the present invention. Figure 3 As shown, the process of the S2G charging and discharging station energy control strategy based on the DDPG algorithm may include at least the following steps:

[0115] Step S301, obtaining status data of the charging and discharging station.

[0116] In the above steps, the status data of the battery status, electric vehicle load demand, photovoltaic power generation power, etc. of the energy storage system of the S2G charging and discharging station are obtained.

[0117] Step S302, define the state space, action space and reward function.

[0118] In the above steps, the acquired data is preprocessed, and the state space, action space and reward function of the S2G charging and discharging station are defined.

[0119] Optionally, a state space is defined based on the battery status of the energy storage system in the charging and discharging station, the load demand of the electric vehicle, and the photovoltaic power generation power; based on the state space data, the action space of the charging and discharging station and photovoltaic is defined; a reward function is defined, and a photovoltaic charging and discharging station reward function is defined based on load matching balance.

[0120] Step S303, initializing the DDPG algorithm.

[0121] In the above steps, the DDPG model is initialized: the policy network, Q network, and target network of the DDPG algorithm are set.

[0122] Optionally, set the step time t, and calculate the optimal action in the state through the policy network every t time; add exploration noise to increase the diversity of actions and ensure that the agent can fully explore the state space in the early stage. After the agent performs an action, the environment feedbacks the new state and immediate reward. At this time, the agent will experience (S t ,A t ,r t ,S t+1 ) into the experience pool, and randomly draw a batch of experience from the experience playback pool (S i ,A i ,r i ,S i+1 ) for network training. Further network updates are performed, including soft updates of the Q network, strategy network, and target network. Repeat the above training until the strategy converges to obtain the energy control strategy of the S2G charging and discharging station.

[0123] The DDPG algorithm is used to train and analyze the original data to obtain the day-ahead energy control strategy of the S2G charging and discharging station taking photovoltaics into account.

[0124] Step S304: Use the DDPG algorithm to perform training analysis on the original data.

[0125] In the above steps, the DDPG algorithm is used to train and analyze the original data to obtain the day-ahead energy control strategy of the S2G charging and discharging station taking photovoltaics into account.

[0126] Step S305, determining whether the DDPG algorithm has reached the termination condition.

[0127] In the above steps, when the DDPG algorithm reaches the termination condition, that is, the DDPG algorithm converges, the next step is entered, otherwise it returns to step S303.

[0128] Step S306: output the control strategy.

[0129] In the above steps, the day-ahead energy control strategy of the S2G charging and discharging station can be output.

[0130] Step S307, adjusting the control strategy to obtain an adjustment result.

[0131] In the above steps, the control strategy can be optimized for intraday regulation and real-time adjustment to achieve intraday scheduling optimization within the S2G station. That is, based on the day-ahead energy control strategy, intraday real-time control optimization is performed.

[0132] Step S308, putting the adjustment result into the experience pool.

[0133] In the above steps, the adjustment results can be put into the day-ahead experience pool after the end of the day, that is, after the end of the day-ahead optimization, the data is updated to the day-ahead energy control strategy experience sample pool.

[0134] Figure 4 is a basic structural diagram of a photovoltaic S2G charging and discharging station according to an embodiment of the present invention, such as Figure 4 As shown in the figure, the high-voltage side voltage of the charging and discharging station is 10 kilovolts (kV), the low-voltage side voltage is 0.4kV, and the alternating current (AC) bus is connected to the S2G charging and discharging station load, AC / direct current (DC) converter, and multiple AC charging piles. The DC bus is connected to a DC / DC converter, and the DC / DC converter is connected to multiple DC charging piles through the DC bus (slow charge) or DC bus (fast charge). In addition, the two DC / DC converters are also connected to the energy storage system and the photovoltaic system.

[0135] Figure 5 is a schematic diagram of energy information exchange based on S2G ​​charging and discharging stations according to an embodiment of the present invention, such as Figure 5As shown, the schematic diagram of energy information exchange based on the S2G charging and discharging station at least includes an S2G ​​charging and discharging station 501, a station-level energy management system 502, an energy storage system 503, a charging device 504, a power grid dispatching 505, a photovoltaic system 506, a power grid 507, and a charging vehicle 508. Among them, the S2G charging and discharging station 501 includes a station-level energy management system 502, an energy storage system 503, and a charging device 504.

[0136] against Figure 5 The station-level energy management system 502 exchanges information with the energy storage system 503, the station-level energy management system 502 issues charging and discharging instructions to the energy storage system 503, and the energy storage system 503 issues information feedback such as battery capacity to the station-level energy management system 502. The station-level energy management system 502 exchanges information with the charging device 504, the station-level energy management system 502 issues an instruction to adjust the charging pile power to the charging device 504, and the charging device 504 issues charging load feedback in the station to the station-level energy management system 502. The station-level energy management system 502 exchanges information with the power grid dispatch 505, the station-level energy management system 502 issues instructions to the power grid dispatch 505, and the power grid dispatch 505 issues S2G station information feedback to the station-level energy management system 502. The station-level energy management system 502 exchanges information with the photovoltaic system 506. The power grid dispatch 5075 exchanges information with the power grid 5075. The charging device 504 exchanges information with the charging vehicle 508.

[0137] against Figure 5 In the energy exchange, the energy storage system 503, the photovoltaic system 506, the power grid 507, the charging device 504 and the charging vehicle 508 exchange energy.

[0138] The S2G charging and discharging station energy control strategy provided in this embodiment is different from the orderly charging and discharging control of a single vehicle and a single pile in a traditional charging station. The charging and discharging station is taken as an overall object, and photovoltaic power generation and energy storage system are used inside the station to smooth the power load. In a short period of time, the charging and discharging station has a charging capacity several times that of the distribution capacity and can reduce the dependence of the charging and discharging station on the power grid. A flexible power distribution control strategy is used outside the station to make electric vehicles a stable and reliable high-quality load. At the same time, when the power grid load is peak, the energy storage system battery and photovoltaic power generation are used to return electricity to the power grid to assist in peak load regulation of the power grid.

[0139] The S2G energy control strategy in this embodiment is different from the uncertainty of traditional charging stations that rely on the participation of user electric vehicles in regulation. It gives priority to using energy storage systems to participate in load regulation, and modularizes the charging piles to adjust the real-time charging power according to different grid states, making energy management more refined and standardized. This embodiment uses the DDPG algorithm in deep reinforcement learning to provide an energy control strategy suitable for S2G charging and discharging stations. On the one hand, it can realize energy control of the day-ahead regulation of the charging and discharging station, and on the other hand, it can perform short-term real-time regulation, smooth load fluctuations, and can adapt to the uncertainty of the electric vehicle load in the charging and discharging station.

[0140] S2G charging and discharging stations must not only consider the load of electric vehicles and their own energy storage systems, but also the involvement of photovoltaic power generation. Therefore, a reasonable state space (including battery status, photovoltaic power generation, load demand, etc.), action space (charging and discharging power) and comprehensive reward function are designed to capture the key dynamics of the system and balance economic benefits, battery health and grid stability.

[0141] First, define the state space. According to the structural characteristics of the charging and discharging station, the charging and discharging station is roughly divided into four parts, namely the energy storage system, photovoltaic power generation in the station and photovoltaic power generation near the station, real-time charging power of electric vehicles, and other loads in the station. In order to facilitate the real-time monitoring and control of the energy of the S2G charging and discharging station, the charging piles in the station are divided into several sub-modules for management. The state space parameters of each part of the S2G charging and discharging station mainly include the following aspects: the energy storage system describes the current remaining power of the battery, which is a percentage of the total battery capacity SOC(t); the photovoltaic power generation power describes the active power output P(t) generated by the photovoltaic power generation at the current moment pv ; The real-time charging power of electric vehicles describes the charging power of electric vehicles connected to the charging and discharging station at the current moment, and monitors the real-time power P of each module according to the modular partitioning situation Li (t). Other loads in the station are relatively fixed and have small fluctuations, so real-time changes are not considered for the time being. The state space of the above variables can be expressed as: S t =[SOC(t),P(t) pv ,P Li (t)]

[0142] Secondly, define the action space. The action space describes the actions that the agent can take at each time step. In the S2G charging and discharging station, the main objects to be regulated are the real-time charging and discharging power of the energy storage system and the real-time charging power of the charging pile. The charging power P of the energy storage system is controlled according to different environments. c (t) and discharge power P d (t) and charging power P of the charging pile Lc(t). The idea of ​​adjusting the above parameters in the actual S2G charging and discharging station is that when the grid load is stable and the electric vehicle load is small, the photovoltaic power mainly discharges to the grid and the energy storage system charges; when the electric vehicle load is large, the photovoltaic power mainly charges the electric vehicle and the energy storage system discharges, and the charging power of the charging pile is appropriately adjusted; when the grid load reaches the bottom, the energy storage system charges, and the charging pile charges the electric vehicle at the maximum charging power; when the grid load reaches a peak, the energy storage system discharges to the grid and reduces the charging power of the charging pile. In more serious cases, the charging pile discharges to the grid through the electric vehicle battery.

[0143] Charging power P t c It refers to the active power absorbed by the energy storage system from the grid or photovoltaic system at time step t. The value range is: P t c is the charging power at the current time step t, Is the maximum charging power limit of the energy storage system. Discharge power P t d It refers to the active power output by the energy storage system to the load or grid at time step t, and its value range is: P t d is the discharge power at the current time step t, It is the maximum discharge power limit of the energy storage system.

[0144] During the charging and discharging process of the energy storage system, in order to prevent over-discharge or over-charge, the SOC(t) should be constrained and adjusted:

[0145]

[0146] SOC min ≤SOC(t)≤SOC max

[0147] Among them, η c , η d , are respectively the charging efficiency, discharging efficiency and self-discharging rate of the energy storage system, S E is the total capacity of the energy storage system, SOC min , SOC max They are the lower and upper limits of the state of charge of the energy storage system respectively.

[0148] P Lc (t) is the charging power of the charging pile corresponding to the current time step t, which is adjusted in real time according to the number of electric vehicles currently connected to the station, and its value range is: in, It is the maximum value of the power that the electric vehicle sends back to the grid under extreme conditions. It should be 0 under normal conditions, representing the minimum charging power of the module charging pile.

[0149] Furthermore, a reward function is designed. The reward function is usually composed of multiple sub-rewards, each of which corresponds to an optimization goal. Economic benefits are the main goal of the S2G charging and discharging station taking into account photovoltaics. The economic benefits are mainly maximized through low-price charging and high-price discharging. The benefit reward can be expressed by the following formula:

[0150] r t p =α×(P t sell ×P t d -P t buy ×P t c )

[0151] Among them, r t p Can be used to represent revenue rewards, P t sell , P t buy are the electricity selling price and electricity purchasing price at the current time step t, respectively, and α is the weight of economic benefits. In order to extend the battery life of the energy storage system, the reward function should take into account the health status of the battery. The battery health reward can be expressed by the following formula:

[0152]

[0153] Among them, r t h It can be used to represent the battery health reward, β is the weight of battery health management, P t c is the charging power at the current time step t, is the maximum charging power limit of the energy storage system, P t d is the discharge power at the current time step t, It is the maximum discharge power limit of the energy storage system. The charging and discharging station must not only meet the load demand of electric vehicles, but also maintain the stability of the power grid to avoid over-reliance on the power grid or excessive fluctuations. The load balancing reward can be expressed by the following formula:

[0154] r t b =-γ×|∑P Li -(P t d -P t c )|

[0155] Among them, r t b can be used to represent the load balancing reward, γ is the load balancing weight, P Li Real-time charging power for electric vehicles. The use of photovoltaic power generation can improve the system's clean energy utilization efficiency and reduce dependence on the power grid. The photovoltaic power generation utilization reward can be expressed by the following formula:

[0156] r t pv =δ×(P t pv ,∑P Li -P t c +P t d )

[0157] Among them, r t pv It can be used to represent the photovoltaic power generation utilization reward, δ is the weight of photovoltaic power generation utilization, It can be used to represent photovoltaic power generation. The final reward function r t The weighted sum of the above sub-rewards is:

[0158] r t =r t p +r t h +r t b +r t pv

[0159] Among them, the selection of weight values ​​needs to be continuously adjusted according to the simulation and combined with the actual situation to find the optimal balance point.

[0160] The idea of ​​using the DDPG algorithm to obtain the energy control strategy of the S2G charging and discharging station taking into account photovoltaics is to train and analyze the state parameters such as energy storage batteries, electric vehicle charging power, photovoltaic power generation power, and action parameters such as the charging and discharging kilometers of energy storage and the real-time charging power of charging piles based on the DDPG algorithm. This step mainly describes the training and learning process of the DDPG algorithm. The DDPG algorithm continuously trains and optimizes the strategy network and value network through the continuous interaction between the intelligent agent and the photovoltaic charging and discharging station system, using experience replay and target network soft update mechanism. The intelligent agent collects experience and learns from it at each time step, and improves the charging and discharging decisions through repeated iterations, and finally achieves the optimal energy management strategy to maximize the overall benefits of the S2G charging and discharging station. This step specifically includes the following steps:

[0161] The initialization of the DDPG algorithm involves the construction of the policy network and the value network, the setting of the target network, and the initialization of the experience replay pool.

[0162] Policy network μ(s|θ μ ) is the battery status of the energy storage system in the state space, the real-time photovoltaic power generation, the charging load, etc. The output is a continuous action, such as the charging power and discharging power of the energy storage and the real-time power of the charging pile. The main way of its strategy output is to dynamically allocate the charging and discharging of the energy storage system based on the real-time photovoltaic power generation and the power of the electric vehicle being charged, and to adjust the power of the user's charging pile to charge the electric gun vehicle in real time. μ is the strategic network parameter set, which mainly includes the control strategy of the S2G charging and discharging station, such as the charging and discharging power P of the energy storage c (t), P d (t) and charging power P of the charging pile Lc (t).

[0163] Value network Q(s,a|θ Q ) includes the state S t and action A t , the output is the Q value of the state-action pair, and the input layer contains the state S t and action A t , the hidden layer uses a fully connected layer, extracts the joint features of state and action through a nonlinear activation function, and the output layer outputs a scalar Q value, which represents the expected cumulative reward after executing the action, mainly reflecting the value fed back to the power grid after adjusting the charging and discharging power of the energy storage system and the charging pile power. Among them, θ Q The strategy action includes the charging and discharging power of energy storage P c (t), P d (t) and charging power P of the charging pile Lc (t) The corresponding value after the action.

[0164] The target network is a copy of the policy network and the value network, and its parameters are updated at a slower rate in order to reduce the fluctuation of Q value and policy during training and enhance the stability of training. The parameters of the target network are adjusted through a soft update strategy:

[0165] θ Q′ ←τθ Q +(1-τ)θ Q′

[0166] θ μ′ ←τθ μ +(1-τ)θ μ′

[0167] Among them, τ is usually a small constant used to control the speed of target network update.

[0168] The experience replay pool stores the experience data generated during the interaction between the agent and the environment. The format is (S t ,A t ,r t ,S t+1 ).

[0169] Figure 6 is a schematic diagram of a DDPG algorithm according to an embodiment of the present invention, such as Figure 6 As shown, the 24 hours of each day are divided into 24 time steps t, and S is calculated by the policy network every 1 hour. t In order to avoid the strategy from converging to the local optimal solution and enhance the exploration ability, in the original A t Add noise N t :

[0170] A t =μ(s t |θ μ )+N t

[0171] The agent performs action A t , update the state S based on the current state and action t+1 And return the immediate reward, and the experience data generated by this (S t ,A t ,r t ,S t+1 ) is stored in the experience replay pool D. A mini-batch (S i ,A i ,r i ,S i+1 ) is used for training, and the size of the mini-batch is usually set to N.

[0172] The target Q-value calculation uses the target policy network μ' and the target Q network Q' to calculate the target Q-value, which reflects the expected total reward of the agent in the future period of time:

[0173] y i =r i +λQ'(S i+1 ,μ'(S i+1 |θ μ' )θ Q' )

[0174] Among them, λ is the discount factor, which balances the immediate reward and the long-term reward. In the day-ahead model, the long-term reward accounts for a larger proportion, while in the intraday regulation, the immediate reward accounts for a larger proportion. Q'(S i+1 ,μ'(S i+1 |θ μ' )θ Q') is the target Q network output based on the next state.

[0175] In order to optimize the parameters θ of the Q network Q , define the loss function L(θ Q ) is the target Q value and the current Q network estimated value Q(S i ,A i |θ Q ) is the mean square error between:

[0176]

[0177] Q network parameters are updated by minimizing the loss function L(θ Q ) Use gradient descent to update the parameters θ of the Q network Q : Among them, η Q is the learning rate, which controls the step size of parameter updates.

[0178] The update goal of the policy network is to maximize the output value of the Q network. The policy gradient is calculated by differentiating the output of the Q network:

[0179]

[0180] This gradient indicates the parameters θ of the policy network. μ How should it be adjusted so that the output can maximize the Q value in a given state? Policy network parameter update, using the policy gradient method to update the policy network parameters θ μ : Among them, η μ is the learning rate of the policy network.

[0181] Soft update of the target network. In order to improve the training stability, the parameters θ of the target network are Q' and θ μ' Gradually approach the parameters of the main network through soft update strategy:

[0182] θ Q′ ←τθ Q +(1-τ)θ Q′

[0183] θ μ′ ←τθ μ +(1-τ)θ μ′

[0184] Among them, τ is the soft update coefficient, and a smaller τ value helps to smooth the update and prevent drastic fluctuations in network parameters.

[0185] The above steps are repeated in each time step, the policy network and Q network of the agent are gradually optimized, and the policy gradually tends to be optimal. When the actions output by the policy network tend to be stable under different states and the overall benefit of the system reaches the expected goal, the training converges and the energy control strategy of the S2G charging and discharging station is obtained.

[0186] Based on the energy control strategy of the S2G charging and discharging station, in order to adapt to the daily fluctuations in photovoltaic and electric vehicle charging power to achieve more refined energy management, the time step is shortened to 5 minutes. The intelligent agent calculates the electric vehicle charging power deviation value and the energy storage charge state deviation based on the ultra-short-term prediction, and determines the correction amount of the energy storage system output and the charging power of the charging pile accordingly.

[0187] The trained strategy is tested in a simulation environment for intraday regulation. The agent needs to smooth out the uncertain power fluctuations generated by photovoltaics and electric vehicles while tracking the day-ahead plan. The state space is set as follows:

[0188]

[0189] Among them, ΔP i p is the difference between the short-term predicted power of the electric vehicle and the day-ahead predicted power, is the SOC deviation at time t+1 if the energy storage output is in accordance with the day-ahead plan at time step t. The agent corrects the energy storage power and the charging pile power based on the day-ahead plan value, and its action space is:

[0190]

[0191] Among them, ΔP ES (t) is the correction value for energy storage power, ΔP Li (t) is the correction value for the charging pile power.

[0192] In order to reduce the load fluctuation of the charging and discharging station, reduce the deviation between the state of charge of the energy storage system and the day-ahead plan, and achieve the intraday control target, the following reward function is defined. The effect of the agent's action tracking the day-ahead plan can be reflected in two aspects. On the one hand, it is the deviation of the energy storage system before and after the action at time t, and on the other hand, it is the deviation of the charging power of the charging pile before and after time t. Its reward function is:

[0193]

[0194] in, If the energy storage at time t is corrected according to the current control action, the SOC deviation at time t+1; SOC d (t) is the energy storage SOC deviation at time t, is the difference between the power of the charging pile after adjustment at time t and the power predicted a day ago, The difference between the short-term predicted power and the day-ahead predicted power. λ1, λ2, λ3, and λ4 are weight coefficients.

[0195] Based on the DDPG algorithm, the energy status of the charging and discharging station is corrected in real time, and the result of each correction is put back into the experience pool.

[0196] After the training and learning of the above three steps, an energy control strategy for S2G charging and discharging stations based on the DDPG algorithm is formed, so as to reasonably and accurately control the energy flow of S2G charging and discharging stations taking into account photovoltaics.

[0197] This embodiment obtains a state parameter set of the charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station; the state parameter set is input into a strategy model, and the state parameter set is analyzed by using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; based on the state parameter set and the action parameter set, a control strategy of the charging and discharging station is determined, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for energy storage of the energy storage system in the charging and discharging station; the charging and discharging station is controlled according to the control strategy, thereby solving the technical problem of low efficiency of energy control of the charging and discharging station and achieving the technical effect of improving the efficiency of energy control of the charging and discharging station.

[0198] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0199] In this embodiment, a control device for a charging and discharging station is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0200] Figure 7 is a structural block diagram of a control device for a charging and discharging station according to an embodiment of the present invention. Figure 7 As shown, the control device 700 of the charging and discharging station may include: an acquisition unit 702 , an analysis unit 704 , a determination unit 706 and a control unit 708 .

[0201] The acquisition unit 702 is used to acquire a state parameter set of the charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station.

[0202] The analysis unit 704 is used to input the state parameter set into the strategy model, and use the strategy model to analyze the state parameter set to obtain the action parameter set of the charging and discharging station, wherein the strategy model is trained using the state parameter set sample, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment.

[0203] The determination unit 706 is used to determine the control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for energy storage of the energy storage system in the charging and discharging station.

[0204] The control unit 708 is used to control the charging and discharging station according to the control strategy.

[0205] As an optional embodiment, the analyzing unit 704 includes: a first determining module, configured to call the strategy model and determine an action parameter set that matches the state parameter set according to the target step size.

[0206] As an optional embodiment, the device further includes: a processing unit, configured to update the state parameter set to obtain an updated state parameter set, and to adjust the action parameter set to obtain an adjusted action parameter set.

[0207] As an optional embodiment, the determination unit 706 includes: a second determination module, used to determine the control strategy based on the state parameter set, the updated state parameter set, the adjusted action parameter set, and the reward function of the charging and discharging station, wherein the reward function is obtained by weighted summation of multiple sub-reward functions, and the sub-reward function is used to characterize the adjustment target of the charging and discharging station.

[0208] As an optional embodiment, the processing unit adjusts the action parameter set to obtain the adjusted action parameter set, including: an adjustment module, used to add noise data to the action parameter set to obtain the adjusted action parameter set.

[0209] As an optional embodiment, the device also includes: a training unit, used to train the initial value model based on state parameter set samples and action parameter set samples to obtain a value model; a first acquisition unit, used to acquire value parameters output by the value model, wherein the value parameters are used to characterize the scoring level of the control strategy.

[0210] As an optional embodiment, the device also includes: a second acquisition unit, used to acquire a policy gradient parameter of a policy model; a first adjustment unit, used to adjust the value parameter based on the policy gradient parameter to obtain a target value parameter, wherein the target value parameter is used to characterize whether the score level of the control strategy reaches a level threshold.

[0211] As an optional embodiment, the device also includes: an updating unit, used to update the policy model based on the policy gradient parameter, and to update the value model based on the loss function of the value model.

[0212] As an optional embodiment, the device also includes: a first correction unit, used to correct the state parameter set to obtain a corrected state parameter set, and to correct the action parameter set to obtain a corrected action parameter set; a second correction unit, used to correct the control strategy using the corrected state parameter set and the corrected action parameter set.

[0213] This embodiment obtains the state parameter set of the charging and discharging station through the acquisition unit 702, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of the associated equipment of the charging and discharging station. The state parameter set is input into the strategy model through the analysis unit 704, and the state parameter set is analyzed by the strategy model to obtain the action parameter set of the charging and discharging station, wherein the strategy model is obtained by training with the state parameter set sample, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment. The control strategy of the charging and discharging station is determined by the determination unit 706 based on the state parameter set and the action parameter set, wherein the control strategy is used to characterize at least the rules for the associated equipment to charge the charging and discharging station, the rules for the charging and discharging station to discharge the associated equipment, and the rules for the energy storage system in the charging and discharging station. The charging and discharging station is controlled according to the control strategy by the control unit 708.

[0214] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0215] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0216] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0217] S1, obtaining a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station;

[0218] S2, inputting the state parameter set into the strategy model, and using the strategy model to analyze the state parameter set to obtain an action parameter set of the charging and discharging station, wherein the strategy model is trained using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment;

[0219] S3, determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize a rule for an associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge an associated device, and a rule for energy storage of an energy storage system in the charging and discharging station;

[0220] S4, controlling the charging and discharging station according to the control strategy.

[0221] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0222] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0223] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0224] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0225] S1, obtaining a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station;

[0226] S2, inputting the state parameter set into the strategy model, and using the strategy model to analyze the state parameter set to obtain an action parameter set of the charging and discharging station, wherein the strategy model is trained using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment;

[0227] S3, determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize a rule for an associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge an associated device, and a rule for energy storage of an energy storage system in the charging and discharging station;

[0228] S4, controlling the charging and discharging station according to the control strategy.

[0229] An embodiment of the present disclosure further provides a computer program product, which may include a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0230] Optionally, in this embodiment, the computer program product may be configured to include a computer program for performing the following steps:

[0231] S1, obtaining a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station;

[0232] S2, inputting the state parameter set into the strategy model, and using the strategy model to analyze the state parameter set to obtain an action parameter set of the charging and discharging station, wherein the strategy model is trained using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment;

[0233] S3, determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize a rule for an associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge an associated device, and a rule for energy storage of an energy storage system in the charging and discharging station;

[0234] S4, controlling the charging and discharging station according to the control strategy.

[0235] An embodiment of the present disclosure further provides a computer program product, which may include a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and the computer program is configured to execute the steps of any of the above method embodiments when running.

[0236] Optionally, in this embodiment, the computer program product may be configured to include a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0237] S1, obtaining a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station;

[0238] S2, inputting the state parameter set into the strategy model, and using the strategy model to analyze the state parameter set to obtain an action parameter set of the charging and discharging station, wherein the strategy model is trained using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment;

[0239] S3, determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize a rule for an associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge an associated device, and a rule for energy storage of an energy storage system in the charging and discharging station;

[0240] S4, controlling the charging and discharging station according to the control strategy.

[0241] An embodiment of the present disclosure further provides a computer program, which is configured to execute the steps of any of the above method embodiments when running.

[0242] Optionally, in this embodiment, the computer program is configured to perform the following steps:

[0243] S1, obtaining a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station;

[0244] S2, inputting the state parameter set into the strategy model, and using the strategy model to analyze the state parameter set to obtain an action parameter set of the charging and discharging station, wherein the strategy model is trained using samples of the state parameter set, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of associated equipment;

[0245] S3, determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize a rule for an associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge an associated device, and a rule for energy storage of an energy storage system in the charging and discharging station;

[0246] S4, controlling the charging and discharging station according to the control strategy.

[0247] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0248] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0249] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A control method for a charging and discharging station, characterized in that: include: Acquire a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station; Inputting the state parameter set into a strategy model, and analyzing the state parameter set using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is trained using state parameter set samples, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; Determine a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is used to at least characterize a rule for the associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge the associated device, and a rule for energy storage of the energy storage system in the charging and discharging station; The charging and discharging station is controlled according to the control strategy.

2. The method according to claim 1, characterized in that The state parameter set is analyzed by using the strategy model to obtain the action parameter set of the charging and discharging station, including: The strategy model is called to determine the action parameter set that matches the state parameter set according to the target step size.

3. The method according to claim 1, characterized in that The method further comprises: The state parameter set is updated to obtain an updated state parameter set, and the action parameter set is adjusted to obtain an adjusted action parameter set.

4. The method according to claim 3, characterized in that Determining a control strategy of the charging and discharging station based on the state parameter set and the action parameter set includes: The control strategy is determined based on the state parameter set, the updated state parameter set, the adjusted action parameter set, and the reward function of the charging and discharging station, wherein the reward function is obtained by weighted summation of multiple sub-reward functions, and the sub-reward function is used to characterize the adjustment target of the charging and discharging station.

5. The method according to claim 3, characterized in that: The action parameter set is adjusted to obtain an adjusted action parameter set, including: Noise data is added to the action parameter set to obtain the adjusted action parameter set.

6. The method according to claim 1, characterized in that The method further comprises: Training the initial value model based on the state parameter set samples and the action parameter set samples to obtain a value model; A value parameter output by the value model is obtained, wherein the value parameter is used to characterize a scoring level of the control strategy.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining a policy gradient parameter of the policy model; Based on the policy gradient parameter, the value parameter is adjusted to obtain a target value parameter, wherein the target value parameter is used to represent that the score level of the control strategy reaches a level threshold.

8. The method according to claim 7, characterized in that The method further comprises: The policy model is updated based on the policy gradient parameter, and the value model is updated based on the loss function of the value model.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Correcting the state parameter set to obtain a corrected state parameter set, and correcting the action parameter set to obtain a corrected action parameter set; The control strategy is modified using the modified state parameter set and the modified action parameter set.

10. A control device for a charging and discharging station, characterized in that: include: An acquisition unit, used to acquire a state parameter set of a charging and discharging station, wherein the state parameters in the state parameter set are used to characterize the state of the charging and discharging station and the state of associated equipment of the charging and discharging station; an analysis unit, configured to input the state parameter set into a strategy model, and analyze the state parameter set using the strategy model to obtain an action parameter set of the charging and discharging station, wherein the strategy model is obtained by training using state parameter set samples, and the action parameters in the action parameter set are used to characterize the power of the charging and discharging station and the power of the associated equipment; a determination unit, configured to determine a control strategy of the charging and discharging station based on the state parameter set and the action parameter set, wherein the control strategy is configured to at least characterize a rule for the associated device to charge the charging and discharging station, a rule for the charging and discharging station to discharge the associated device, and a rule for energy storage by the energy storage system in the charging and discharging station; A control unit is used to control the charging and discharging station according to the control strategy.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 9 when executed.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images