Charging and discharging decision optimization method and device for V2G electric vehicle charging station

Abstract

The invention relates to the field of new energy vehicle charging station operation strategies, and discloses a V2G electric vehicle charging station charging and discharging decision optimization method and device. Constructing a new energy vehicle charging station interaction environment based on a V2G technology; a state variable, an action variable and a reward function of reinforcement learning are set, and the reward function is provided with a plurality of optimization objectives, so that benefits of charging pile operators, power grid operators, users and the like can be balanced, and global optimization is realized. An SD3 algorithm is adopted to construct an electric vehicle charging station charging and discharging decision-making model based on reinforcement learning, an optimal electric vehicle charging station charging and discharging decision-making model is obtained through training, and an optimal charging and discharging scheduling scheme is obtained through calculation according to real-time data and the optimal electric vehicle charging station charging and discharging decision-making model. According to the method, better performance can be obtained under the condition of fewer training samples, and the reliability of intelligent decision making and the stability in the training process can be ensured in a dynamic environment. Decision strategies in a complex environment can be automatically learned, and the burden of manual rule design is relieved.

Description

Technical Field

[0001] The present application relates to the field of new energy vehicle charging station operation strategy, and specifically to a method and device for optimizing charging and discharging decisions of a V2G electric vehicle charging station. Background Art

[0002] With the rapid development of new energy vehicles, the application of on-board energy equipment in the power grid has received more and more attention. Vehicle-to-Grid (V2G) technology allows on-board batteries to participate in the load regulation of the power grid without affecting the user's travel, providing a new solution for realizing the "peak shaving and valley filling" of the power grid. At present, there are some technical problems in the actual operation of new energy vehicle charging stations, mainly including: low energy utilization efficiency due to peak power load, large-scale access to the power grid exacerbating power grid load fluctuations, and difficulty in meeting the personalized charging needs of different users. These problems seriously restrict the economy and user experience of charging infrastructure.

[0003] In order to optimize the charging and discharging decision-making of V2G new energy vehicle charging stations, researchers have proposed some charging and discharging strategy optimization methods, which can be roughly divided into three categories:

[0004] The rule-based charging and discharging decision optimization method makes decisions based on the current environmental status by presetting a series of rules, including fuzzy logic control, deterministic rule control, etc. It is simple to understand and easy to implement. However, this method requires manual design of a large number of if-then rules. As the complexity of the charging environment increases, the rules become more and more complex, and it is difficult to foresee all possible situations. The rules are difficult to fully cover, and decision failure is prone to occur.

[0005] The charging and discharging decision optimization method based on optimization theory models the entire charging system as a mathematical optimization problem, and uses various optimization theories and algorithms to solve and obtain the optimal decision. Commonly used optimization algorithms include genetic algorithms, particle swarm optimization, dynamic programming, etc. This method has high computational complexity, but it can obtain more intelligent and adaptive decision strategies in complex dynamic charging environments, making up for the limitations of rule-based methods. However, this method usually only optimizes a single goal, such as minimizing grid load or maximizing charging station revenue, and cannot take into account the needs of multiple stakeholders such as charging pile operators, grid operators, and users at the same time.

[0006] Learning-based charging and discharging decision optimization methods include those based on deep learning, reinforcement learning, and deep reinforcement learning. Among them, deep reinforcement learning has become a new research hotspot. However, the traditional deep reinforcement learning-based methods have low sample efficiency and require a large number of training samples to converge. At the same time, the traditional deep reinforcement learning methods are prone to unstable target values, which affects the convergence of training. Summary of the invention

[0007] The present application provides a charging and discharging decision optimization method for a V2G electric vehicle charging station to solve the problems in the prior art that the charging and discharging strategy research method for a new energy vehicle charging station is difficult to cope with the complex and changeable charging environment and user behavior, has a single optimization target, cannot take into account both grid flexibility and user experience, and has low decision-making efficiency.

[0008] Correspondingly, the present application also provides a V2G electric vehicle charging station charging and discharging decision optimization device, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0009] In order to solve the above technical problems, the present application discloses a V2G electric vehicle charging station charging and discharging decision optimization method, the method comprising:

[0010] Based on V2G technology, an interactive environment for new energy vehicle charging stations is constructed to describe vehicle-related factors;

[0011] Set the state variables, action variables and reward function of reinforcement learning; the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid;

[0012] The SD3 algorithm is used to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning;

[0013] Use actual historical data to train the charging and discharging decision model of electric vehicle charging stations to obtain the optimal charging and discharging decision model of electric vehicle charging stations;

[0014] The optimal charging and discharging scheduling plan is calculated based on real-time data and the optimal electric vehicle charging station charging and discharging decision model.

[0015] The present application also discloses a V2G electric vehicle charging station charging and discharging decision optimization device, the device comprising:

[0016] The environment construction module is used to build an interactive environment for new energy vehicle charging stations based on V2G technology to describe vehicle-related factors;

[0017] The learning factor setting module sets the state variables, action variables and reward functions of reinforcement learning; the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid;

[0018] The model building module uses the SD3 algorithm to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning;

[0019] The model optimization module uses actual historical data to train the charging and discharging decision model of electric vehicle charging stations to obtain the optimal charging and discharging decision model of electric vehicle charging stations;

[0020] The decision generation module calculates the optimal charging and discharging scheduling plan based on real-time data and the optimal electric vehicle charging station charging and discharging decision model.

[0021] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0022] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0023] In this application, an interactive environment for new energy vehicle charging stations is constructed based on V2G technology to describe vehicle-related factors and realize two-way interaction between electric vehicles and charging stations. Set the state variables, action variables and reward functions of reinforcement learning; the reward function aims to maximize the benefits of charging stations, maximize user satisfaction and maximize the adjustment effect on the power grid, which can balance the interests of charging pile operators, power grid operators and users, and achieve global optimization. The SD3 algorithm is used to construct a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning, and the charging and discharging decision model for electric vehicle charging stations is trained using actual historical data to obtain the optimal charging and discharging decision model for electric vehicle charging stations. The optimal charging and discharging scheduling scheme is calculated based on real-time data and the optimal charging and discharging decision model for electric vehicle charging stations. The SD3 algorithm supports continuous action space, enabling the intelligent agent to explore in a more delicate decision space and obtain better performance with fewer training samples. At the same time, SD3 is more adaptable and robust in a dynamically changing environment, ensuring the reliability of intelligent decision-making and showing better stability during training. The embodiment of the present application combines V2G with the SD3 algorithm, and automatically learns decision-making strategies in complex environments through single-agent modeling, which can reduce the burden of manual rule design and achieve efficient optimization of charging and discharging decisions of charging stations.

[0024] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flow chart of a V2G electric vehicle charging station charging and discharging decision optimization method provided in an embodiment of the present application;

[0027] Figure 2 A specific flow chart of a new energy vehicle charging station charging and discharging decision optimization method based on V2G technology and SD3 algorithm provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of an interactive environment of a new energy vehicle charging station based on V2G technology provided in an embodiment of the present application;

[0029] Figure 4 A framework diagram of a charging and discharging decision optimization method for a new energy vehicle charging station based on V2G technology and SD3 algorithm provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the structure of a V2G electric vehicle charging station charging and discharging decision optimization device provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0033] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0034] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0035] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. With regard to the technical problems existing in the prior art, the V2G electric vehicle charging station charging and discharging decision optimization method and device provided in this application are intended to solve at least one of the technical problems of the prior art.

[0036] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0037] The present application embodiment provides a possible implementation method, such as Figure 1As shown, a flow chart of a method for optimizing charging and discharging decisions of a V2G electric vehicle charging station is provided. The method can be executed by any electronic device, and optionally, can be executed on a server or terminal device.

[0038] like Figure 1 As shown in , the method may include the following steps:

[0039] Step 101, constructing a new energy vehicle charging station interactive environment based on V2G technology to describe vehicle-related factors.

[0040] Among them, vehicle-related factors include vehicle charging demand, electricity prices and grid load.

[0041] Step 102, setting the state variables, action variables and reward function of reinforcement learning; the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid. The setting of the reward function can ensure the effectiveness and feasibility of charging and discharging decisions.

[0042] Step 103, using the SD3 algorithm to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning.

[0043] The SD3 algorithm is an improved deep deterministic policy gradient algorithm that has faster convergence speed and better stability than traditional deep deterministic policy gradient algorithms. Specifically, the SD3 algorithm is an improvement on TD3, aiming to improve the stability and sample efficiency of reinforcement learning. Compared with TD3, SD3 introduces a soft policy update mechanism to reduce volatility during policy updates, thereby enhancing the stability of training. In addition, SD3 enhances exploration capabilities, automatically adjusts the balance between exploration and utilization through entropy regularization, and improves the experience replay mechanism to increase sample utilization. These improvements make SD3 generally perform better than TD3 in complex environments.

[0044] Step 104 , using actual historical data to train the electric vehicle charging station charging and discharging decision model to obtain the optimal electric vehicle charging station charging and discharging decision model.

[0045] Step 105, calculating and obtaining an optimal charging and discharging scheduling plan based on the real-time data and the optimal electric vehicle charging station charging and discharging decision model.

[0046] In a dynamic environment, learning-based algorithms can dynamically adjust optimization strategies through real-time data collection and machine learning model training, thereby more effectively coping with complex optimization problems. This flexibility enables charging stations to make smarter decisions in real-time changing environments. Among all learning-based algorithms, the SD3 algorithm is used in the embodiment of the present application. Compared with other deep reinforcement learning decision-making methods, the charging and discharging decision model of the electric vehicle charging station constructed based on the SD3 algorithm in the embodiment of the present application can effectively solve complex optimization problems in a dynamic environment and improve the stability and sample efficiency of reinforcement learning. It also comprehensively considers multiple factors such as the revenue of the charging station, user satisfaction, and the adjustment effect on the power grid, and achieves a more comprehensive system optimization goal. In addition, the SD3 algorithm has stronger scalability and can better adapt to the dynamic changes of the power grid, charging infrastructure, and vehicle groups. At the same time, the scheme performs well in real-time and robustness, ensuring that the charging station can still make efficient and reliable charging and discharging decisions in a constantly changing environment. These advantages enable this scheme to provide more intelligent support in electric vehicle charging and discharging decisions.

[0047] As a first example, the specific process in the embodiment of the present application is as follows Figure 2 As shown, first of all, the control problem is defined. For electric vehicle charging stations and their interactive environment with the power grid, the embodiments of the present application aim to solve the problem of optimizing charging and discharging decisions.

[0048] The second step is the modeling phase. Figure 3 The interactive environment of the new energy vehicle charging station shown in the figure includes an electric vehicle queue electric vehicle charging station, a power grid system, a price system and a target module. Among them, the electric vehicle queue has n electric vehicles, which are represented as vehicle 1, vehicle 2, ..., vehicle n; the electric vehicle charging station has n charging piles, which are represented as charging pile 1, charging pile 2, ..., charging pile n; the power grid load data can be obtained through the power grid system; the price information can be obtained through the price system, and the price information includes the charging station purchase price, the charging station feed price, the vehicle charging price and the vehicle feed price; the target module includes maximizing charging benefits, maximizing user satisfaction and achieving "peak shaving and valley filling" of the power grid.

[0049] Figure 3 In the figure, the solid arrows represent the information flow between the parts, and the dotted arrows represent the power flow between the parts. Specifically, the electric vehicle queue, the power grid system, the price system, and the target module transmit the vehicle status, the power grid status, the price information, and the optimization target information to the electric vehicle charging station respectively. The power grid system and the electric vehicle charging station can realize the transmission of power, and the vehicles in the electric vehicle queue and the corresponding charging piles in the electric vehicle charging station can realize the transmission of charging / discharging power.

[0050] The interactive environment of the new energy vehicle charging station in the embodiment of the present application realizes two-way power interaction between the charging station and the electric vehicle, so that the charging station can not only provide charging services for the electric vehicle, but also obtain power from the electric vehicle for discharge.

[0051] The third step is to set up the reinforcement learning state variables, action variables, and reward function.

[0052] Optionally, state variables are defined as:

[0053]

[0054] in, They represent the state of charge of the i-th vehicle at time t, the difference between the current battery capacity and the target capacity, and the owner’s profit. t It describes the revenue of the charging station at the tth moment. It represents the total load of the power grid at the tth moment.

[0055] The action variable of the agent refers to making decisions based on the observed state information, such as adjusting the charging power of the charging pile and the power fed by the electric vehicle to the power grid. In the embodiment of the present application, the action variable is designed to be the charging / feeding power of the vehicle, and the constraints on the action variable can be defined as follows:

[0056]

[0057] in, Indicates the charging power, Represents the discharge power, p max_charging Indicates the maximum charging power of a charging pile for ordinary unidirectional charging.

[0058] In the embodiment of the present application, maximizing the revenue of the charging station, maximizing the user satisfaction and maximizing the adjustment effect on the power grid are taken as optimization goals, and the corresponding reward function r(t) is set as follows:

[0059]

[0060] Among them, w1~w3 are weight coefficients of corresponding factors, which can be adjusted according to actual needs; is the revenue of the charging station at time t, which consists of the net revenue when the charging station charges electric vehicles and the net revenue when it feeds electricity to the grid; is the average user satisfaction at the tth moment. When the state of charge (SOC) of the electric vehicle when leaving is greater than or equal to the expected SOC, the user satisfaction is 1, otherwise it is the ratio of the actual SOC to the expected SOC; G t is the adjustment effect of the charging station on the power grid, which consists of the adjustment effect on the peak and valley values ​​of the power grid. Its related expressions are as follows:

[0061]

[0062]

[0063] G t =max(0,peak expect -peak actual )+max(0,valley expect -valley actual )

[0064] Among them, i is the index of all car numbers; j is the index of the car number involved in the discharge process; and They represent the charging power and discharging power of the i-th car at the t-th moment respectively; represents the electricity purchase price of the charging station at time t; represents the charging station power supply price at time t; and They represent the charging and feeding prices of the car at time t respectively; and M represent the actual power, expected power and the total number of users at time t when the car leaves, respectively; peak expect 、peak actual 、valley expect and valley actual It represents the expected peak value, actual peak value, expected valley value and actual valley value of the power grid load respectively.

[0065] The fourth step is to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning, initialize the neural network in the model and set the initial environment state. Figure 2 As shown in the figure, after initializing the neural network in the model and setting the initial environmental state, the model training phase is entered, in which actual historical data (actual charging station operation historical data and grid load historical data) are used as training data. In the process of training the charging and discharging decision model of the electric vehicle charging station, factors such as historical charging demand, load conditions and electricity price fluctuations are analyzed, and the SD3 algorithm is used to continuously adjust the model parameters and optimize the charging and discharging decision, so as to maximize the benefits of the charging station and the owner, as well as the adjustment effect on the grid.

[0066] During the iterative training of the model, the training results are evaluated to determine whether the training results meet the optimization objectives. If so, the optimal electric vehicle charging station charging and discharging decision model is obtained, and the decision is optimized using the optimal electric vehicle charging station charging and discharging decision model. Otherwise, the training model continues to be trained.

[0067] In the decision optimization stage, according to the real-time data and the optimal charging and discharging decision model of the electric vehicle charging station, the optimization plan (optimal charging and discharging power instructions) is output, and the instructions are sent to each charging pile to guide the charging pile to perform real-time charging and discharging control. In order to maintain the intelligence and robustness of the charging and discharging decision, an online learning mechanism is adopted to continuously collect new operating data, and use these data to continuously optimize the optimal charging and discharging decision model of the electric vehicle charging station, adaptively learn the dynamic changes of factors such as charging demand and electricity price, and finally obtain the optimized charging and discharging scheduling plan.

[0068] In the embodiment of the present application, an interactive environment of a new energy vehicle charging station is constructed based on V2G technology to describe vehicle-related factors and realize two-way interaction between electric vehicles and charging stations. The state variables, action variables and reward functions of reinforcement learning are set; wherein the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid, and can balance the interests of charging pile operators, power grid operators and users to achieve global optimization. The SD3 algorithm is used to construct a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning, and the charging and discharging decision model for electric vehicle charging stations is trained using actual historical data to obtain the optimal charging and discharging decision model for electric vehicle charging stations. The optimal charging and discharging scheduling scheme is calculated based on real-time data and the optimal charging and discharging decision model for electric vehicle charging stations. The SD3 algorithm supports continuous action space, enabling the intelligent agent to explore in a more delicate decision space and obtain better performance with fewer training samples. At the same time, SD3 has stronger adaptability and robustness in a dynamically changing environment, ensuring the reliability of intelligent decision-making and showing better stability during training. The embodiment of the present application combines V2G with the SD3 algorithm, and automatically learns decision-making strategies in complex environments through single-agent modeling, which can reduce the burden of manual rule design and achieve efficient optimization of charging and discharging decisions of charging stations.

[0069] In an optional embodiment, the charging and discharging decision model of an electric vehicle charging station includes a policy network and two value networks;

[0070] The policy network is used to make deterministic actions for a given state;

[0071] The value network is used to calculate the expected value of the future cumulative discounted rewards that can be obtained given a state and a deterministic action.

[0072] As a second example, the SD3 algorithm is a single-agent deep reinforcement learning algorithm based on the Actor-Critic framework. The charging and discharging decision model of electric vehicle charging stations built based on the SD3 algorithm is as follows: Figure 4 As shown, it mainly consists of a strategy network μ(s|φ) and two value networks and Each policy network and value network is equipped with a target network with the same structure to improve the stability of the training process. They are policy target network and value target network Among them, the policy network adopts a deterministic strategy, that is, for a given state s t , the control action given by the policy network is a deterministic action a t , not obtained by random sampling. In order to improve the exploratory nature of the deterministic policy network during training, a certain amount of noise is usually added to the control action output by the deterministic policy network, so that the control action selection is random. The value network is an approximation of the action value function, that is, for a given state s t and deterministic control action a t The expected value of the future cumulative discounted return that can be obtained reflects the quality of the action and evaluates the performance of the strategy network according to the size of the expected value.

[0073] The value network and the value target network are input vectors composed of states and actions. After the neural network outputs an action value, it indicates the expected return of performing a specific action under a given state. It is mainly composed of a fully connected layer, a ReLU activation function, and a Tanh activation function. The orthogonal initialization method is selected for the parameter initialization of the fully connected layer:

[0074] Linear rectification function (ReLU activation function):

[0075]

[0076] Tanh activation function:

[0077]

[0078] Among them, x represents the output of the fully connected layer, that is, the result of the input vector after weight and bias calculation.

[0079] In an optional embodiment, the charging and discharging decision model of an electric vehicle charging station is trained using actual historical data to obtain an optimal charging and discharging decision model of an electric vehicle charging station, including:

[0080] Acquire actual historical data based on the interaction between the interactive environment of new energy vehicle charging stations and the strategy network;

[0081] The actual historical data is used to update the policy network and value network to obtain the optimal charging and discharging decision model for electric vehicle charging stations.

[0082] The SD3 algorithm is an off-policy reinforcement learning method. Therefore, in the process of interaction between the policy network and the environment, experience needs to be collected as actual historical data to train the value network and the policy network and guide the update of the policy network and the value network. The actual historical data includes the state s at the current time step. t , the action a at the current time step t , the reward r at the current time step t and the state s of the next time step t+1 , the collected experience is expressed as (s t ,a t ,r t ,s t+1 ).

[0083] In an optional embodiment, the update objective of the policy network is to maximize the expected value of the cumulative discounted return.

[0084] In an optional embodiment, the updating goal of the value network is to make the prediction of the expected value of the cumulative discounted return close to the true value.

[0085] In an optional embodiment, using actual historical data to update the policy network and the value network includes:

[0086] Based on actual historical data, the policy gradient boosting method is used to update the parameters of the policy network;

[0087] The time difference method is used to update the parameters of the value network with the goal of minimizing the time difference loss.

[0088] For example, the update objective of the policy network is mainly to maximize the expected value of the cumulative discounted return:

[0089] softmax.J(f)=E S [Q(S,μ(S|f)|θ)]

[0090] Where J(f) represents the objective function of the policy network; E is the expected value operator; S is the state extracted from the experience replay pool; Q(S, μ(S|f) is the value network; μ(S|f) is the policy network; θ represents the parameters of the policy network.

[0091] It is necessary to update the policy network parameter φ to maximize the cumulative discounted return. In the embodiment of the present application, historical experience (s j ,a j ,r j ,s j+1 ), and use the policy gradient boosting method to update the parameter φ:

[0092]

[0093]

[0094] The update goal of the value network is to make its prediction of the expected value of the cumulative discounted return as close to the true value as possible, so as to correctly guide the update of the policy network. The update method of the value network mainly adopts the time difference method, which updates the parameters by minimizing the time difference loss.

[0095] As a third example, in the embodiment of the present application, it is necessary to use the historical experience (s j ,a j ,r j ,s j+1 ) as actual historical data to update parameters. Specifically, the experience collected during the interaction between the policy network and the interactive environment is put into the experience replay pool D, and the historical experience replay technology is used to randomly sample the minimum batch of historical experience samples from the experience replay pool D to update parameters. Among them, the interaction between the policy network and the interactive environment includes: the current time step state s transmitted by the interactive environment to the policy network t , the action a of the current time step transmitted by the policy network to the interactive environment t And the vehicle's charging and discharging power (charging power: [0,1]×maximum one-way charging power; discharging power: [-1,0]×maximum one-way charging power).

[0096] During the parameter update process, the value network is first used to predict the future cumulative discounted returns, and we can get and Use the policy target network to select the next action Then use the value target network to j+1 and actions Performing discounted return prediction, we can obtain and Based on the above, the time difference target can be calculated and the time difference error δ 1,j , δ 2,j :

[0097]

[0098] Then, the parameter update of the value network is as follows:

[0099]

[0100] Finally, the target network parameters are updated using a soft update method:

[0101]

[0102] In the embodiment of the present application, the SD3 algorithm optimizes the strategy update process by introducing a dual delay mechanism (i.e., the updates of the strategy network and the value network are separate) and a target network, improves the learning stability, and avoids the instability caused by the simultaneous updates of the strategy network and the value network, so that the method can more stably cope with the real-time interaction between electric vehicles, charging stations and power grids. The use of dual Q function estimation (i.e., using two value networks) effectively reduces the over-estimation problem of Q value and improves the accuracy of strategy learning.

[0103] In the embodiment of the present application, the charging power and discharging power of each vehicle at the charging station can be dynamically adjusted through continuous iterative training, thereby maximizing the charging station, user satisfaction and grid adjustment effect while meeting user needs.

[0104] The method in the embodiment of the present application can be extended to the coordinated optimization scheduling of other new energy storage equipment that requires bidirectional charging and discharging control and has complex energy scheduling problems.

[0105] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a V2G electric vehicle charging station charging and discharging decision optimization device, such as Figure 5 As shown, the device comprises:

[0106] An environment construction module 501 is used to construct an interactive environment of a new energy vehicle charging station based on V2G technology to describe vehicle-related factors;

[0107] The learning element setting module 502 sets the state variables, action variables and reward function of the reinforcement learning; the reward function aims to maximize the revenue of the charging station, maximize the user satisfaction and maximize the adjustment effect on the power grid;

[0108] A model building module 503, using the SD3 algorithm to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning;

[0109] The model optimization module 504 uses actual historical data to train the charging and discharging decision model of the electric vehicle charging station to obtain the optimal charging and discharging decision model of the electric vehicle charging station;

[0110] The decision generation module 505 calculates and obtains the optimal charging and discharging scheduling plan based on the real-time data and the optimal electric vehicle charging station charging and discharging decision model.

[0111] In the embodiment of the present application, an interactive environment of a new energy vehicle charging station is constructed based on V2G technology to describe vehicle-related factors and realize two-way interaction between electric vehicles and charging stations. The state variables, action variables and reward functions of reinforcement learning are set; wherein the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid, and can balance the interests of charging pile operators, power grid operators and users to achieve global optimization. The SD3 algorithm is used to construct a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning, and the charging and discharging decision model for electric vehicle charging stations is trained using actual historical data to obtain the optimal charging and discharging decision model for electric vehicle charging stations. The optimal charging and discharging scheduling scheme is calculated based on real-time data and the optimal charging and discharging decision model for electric vehicle charging stations. The SD3 algorithm supports continuous action space, enabling the intelligent agent to explore in a more delicate decision space and obtain better performance with fewer training samples. At the same time, SD3 has stronger adaptability and robustness in a dynamically changing environment, ensuring the reliability of intelligent decision-making and showing better stability during training. The embodiment of the present application combines V2G with the SD3 algorithm, and automatically learns decision-making strategies in complex environments through single-agent modeling, which can reduce the burden of manual rule design and achieve efficient optimization of charging and discharging decisions of charging stations.

[0112] The V2G electric vehicle charging station charging and discharging decision optimization device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0113] The V2G electric vehicle charging station charging and discharging decision optimization device of the embodiment of the present application can execute the V2G electric vehicle charging station charging and discharging decision optimization method provided in the embodiment of the present application, and its implementation principle is similar. The actions performed by each module and unit in the V2G electric vehicle charging station charging and discharging decision optimization device in each embodiment of the present application correspond to the steps in the V2G electric vehicle charging station charging and discharging decision optimization method in each embodiment of the present application. For the detailed functional description of each module of the V2G electric vehicle charging station charging and discharging decision optimization device, please refer to the description in the corresponding V2G electric vehicle charging station charging and discharging decision optimization method shown in the previous text, which will not be repeated here.

[0114] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the V2G electric vehicle charging station charging and discharging decision optimization method shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the V2G electric vehicle charging station charging and discharging decision optimization method provided by the present application builds a new energy vehicle charging station interactive environment based on V2G technology to describe vehicle-related factors and realize two-way interaction between electric vehicles and charging stations. Set the state variables, action variables and reward functions of reinforcement learning; wherein the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid, and can balance the interests of charging pile operators, power grid operators and users to achieve global optimization. The SD3 algorithm is used to construct a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning, and the charging and discharging decision model for electric vehicle charging stations is trained using actual historical data to obtain the optimal charging and discharging decision model for electric vehicle charging stations, and the optimal charging and discharging scheduling scheme is calculated based on real-time data and the optimal charging and discharging decision model for electric vehicle charging stations. The SD3 algorithm supports continuous action space, enabling the agent to explore in a more delicate decision space and achieve better performance with fewer training samples. At the same time, SD3 is more adaptable and robust in a dynamically changing environment, ensuring the reliability of intelligent decision-making and showing better stability during training. The embodiment of the present application combines V2G with the SD3 algorithm, and automatically learns decision-making strategies in complex environments through single-agent modeling, which can reduce the burden of manual rule design and achieve efficient optimization of charging and discharging decisions of charging stations.

[0115] In an optional embodiment, an electronic device is also provided, such as Figure 6 As shown, Figure 6 The electronic device 600 shown may be a server, including: a processor 601 and a memory 603. The processor 601 and the memory 603 are connected, such as through a bus 602. Optionally, the electronic device 600 may further include a transceiver 604. It should be noted that in actual applications, the transceiver 604 is not limited to one, and the structure of the electronic device 600 does not constitute a limitation on the embodiments of the present application.

[0116] Processor 601 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 601 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0117] The bus 602 may include a path to transmit information between the above components. The bus 602 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0118] The memory 603 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0119] The memory 603 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 601. The processor 601 is used to execute the application code stored in the memory 603 to implement the contents shown in the above method embodiment.

[0120] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0122] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0123] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0124] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0125] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0126] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0127] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the V2G electric vehicle charging station charging and discharging decision optimization method and device provided in the above-mentioned various optional implementations.

[0128] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The modules involved in the embodiments of the present application may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the environment construction module may also be described as "an environment construction module for describing vehicle-related factors in constructing an interactive environment for a new energy vehicle charging station based on V2G technology".

[0131] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A charging and discharging decision optimization method for a V2G electric vehicle charging station, characterized in that: The method comprises: Construct an interactive environment for new energy vehicle charging stations based on V2G technology to describe vehicle-related factors; Setting state variables, action variables and reward functions for reinforcement learning; the reward function aims to maximize the revenue of the charging station, maximize user satisfaction and maximize the adjustment effect on the power grid; The SD3 algorithm is used to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning; Using actual historical data to train the electric vehicle charging station charging and discharging decision model to obtain an optimal electric vehicle charging station charging and discharging decision model; The optimal charging and discharging scheduling plan is calculated based on the real-time data and the optimal electric vehicle charging station charging and discharging decision model.

2. The V2G electric vehicle charging station charging and discharging decision optimization method according to claim 1 is characterized in that: The charging and discharging decision model of the electric vehicle charging station includes a strategy network and two value networks; The policy network is used to make deterministic actions for a given state; The value network is used to calculate the expected value of future cumulative discounted rewards that can be obtained given a state and a deterministic action.

3. The V2G electric vehicle charging station charging and discharging decision optimization method according to claim 2 is characterized in that: The method of using actual historical data to train the electric vehicle charging station charging and discharging decision model to obtain the optimal electric vehicle charging station charging and discharging decision model includes: Acquiring actual historical data based on the interaction between the new energy vehicle charging station interactive environment and the strategy network; The actual historical data is used to update the strategy network and the value network to obtain the optimal electric vehicle charging station charging and discharging decision model.

4. The V2G electric vehicle charging station charging and discharging decision optimization method according to claim 3 is characterized in that: The updating goal of the policy network is to maximize the expected value of the cumulative discounted return.

5. The V2G electric vehicle charging station charging and discharging decision optimization method according to claim 3 is characterized in that: The updating goal of the value network is to make the prediction of the expected value of the cumulative discounted return close to the true value.

6. The V2G electric vehicle charging station charging and discharging decision optimization method according to claim 3 is characterized in that: The updating of the policy network and the value network using the actual historical data comprises: Based on the actual historical data, updating the parameters of the policy network using a policy gradient boosting method; The time difference method is adopted to update the parameters of the value network with the goal of minimizing the time difference loss.

7. The V2G electric vehicle charging station charging and discharging decision optimization method according to claim 3 is characterized in that: The actual historical data includes the state of the current time step, the action of the current time step, the reward of the current time step, and the state of the next time step.

8. A V2G electric vehicle charging station charging and discharging decision optimization device, characterized in that: The device comprises: The environment construction module is used to build an interactive environment for new energy vehicle charging stations based on V2G technology to describe vehicle-related factors; A learning element setting module is used to set the state variables, action variables and reward function of reinforcement learning; the reward function aims to maximize the revenue of the charging station, maximize the user satisfaction and maximize the adjustment effect on the power grid; The model building module uses the SD3 algorithm to build a charging and discharging decision model for electric vehicle charging stations based on reinforcement learning; A model optimization module, which uses actual historical data to train the electric vehicle charging station charging and discharging decision model to obtain an optimal electric vehicle charging station charging and discharging decision model; The decision generation module calculates and obtains the optimal charging and discharging scheduling plan based on the real-time data and the optimal electric vehicle charging station charging and discharging decision model.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images