Orderly charging method for electric vehicles in power distribution areas based on load forecasting and deep reinforcement learning

Abstract

The present invention relates to electric vehicle charging technology, and aims to provide a method for orderly charging of electric vehicles in substations based on load forecasting and deep reinforcement learning. It includes: according to the historical charging remaining capacity information of the substation, a fully connected feedforward neural network is used to predict the future remaining charging capacity; according to the flexible electricity price mechanism and the predicted future remaining charging capacity, a DQN-based deep reinforcement learning algorithm is used to adjust the electric vehicle charging strategy and control the number of electric vehicles connected to the charging pile. The present invention introduces the load forecasting method into the orderly charging operation of electric vehicles, and uses the predicted future remaining charging capacity as the basis for the charging scheduling of electric vehicles, which is more real-time and advanced than traditional methods. By introducing deep reinforcement learning into the orderly charging operation of electric vehicles, the predicted future information is used to pre-schedule the orderly charging of electric vehicles, and it has real-time self-learning and updating capabilities, which is more flexible and effective than traditional algorithms, and further improves the economy and efficiency.

Description

Technical Field

[0001] The present invention relates to electric vehicle charging technology, and in particular to a method for orderly charging electric vehicles in a substation based on load forecasting and deep reinforcement learning. Background Art

[0002] As environmental pollution becomes increasingly serious, electric vehicles have developed rapidly in recent years due to their clean and environmentally friendly characteristics, and related content has become more and more a research focus. More and more electric vehicles are connected to the power grid for charging, which undoubtedly increases the difficulty of optimizing the configuration of the power grid. Through orderly charging operations, it will be beneficial to further improve the economy and environmental protection of the power grid.

[0003] As an important part of the economic dispatch of power systems, accurate load forecasting can provide an effective reference for generator output control and load access dispatching, which is beneficial to maintaining the stability of power grid operation and improving the economic efficiency of the power grid. However, most existing orderly charging operations rely on historical charging data, with poor real-time performance and no advanced dispatching capabilities. The present invention introduces the load forecasting method into the orderly charging operation of electric vehicles, uses the predicted future remaining charging capacity as the basis for electric vehicle charging dispatch, and uses deep reinforcement learning to complete electric vehicle charging dispatch, which is more real-time and advanced than traditional methods. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a method for orderly charging of electric vehicles in a substation based on load forecasting and deep reinforcement learning, which is an intelligent scheduling method for orderly charging of electric vehicles with advanced scheduling capability.

[0005] To solve the technical problem, the solution of the present invention is:

[0006] A method for orderly charging of electric vehicles in a substation based on load forecasting and deep reinforcement learning is provided, comprising the following steps:

[0007] (1) Based on the historical remaining charging capacity information of the substation, a fully connected feedforward neural network (FFN) is used to predict the future remaining charging capacity;

[0008] (2) According to the flexible electricity price mechanism and the future remaining charging capacity predicted in step (1), the electric vehicle charging strategy is adjusted using a deep reinforcement learning algorithm based on DQN (DeepQ Network) to control the number of electric vehicles connected to charging piles.

[0009] As a preferred embodiment of the present invention, the step (1) specifically comprises:

[0010] (1.1) Obtain the historical remaining charging capacity data of the substation area and split the data samples to form training and input data, and the output data is the future remaining charging capacity data;

[0011] (1.2) Build an FFN model consisting of one convolutional layer and two linear layers; the output formula of the FFN model used is:

[0012] Out(X)=Tanh(conv(X)W1+b1)W2+b2

[0013] Where: out(X) is the output of the FFN model; Tanh(.) is the activation function; conv(.) is the convolutional network; (W i , b i ) are network parameters; X is network input;

[0014] (1.3) Determine the loss function for training:

[0015]

[0016] In the formula is the loss function; T is the time step; x i and Represent the model prediction value and the true label value respectively;

[0017] (1.4) Use the gradient descent algorithm to train the fully connected feedforward neural network (FFN) model;

[0018] (1.5) The historical remaining charging capacity of the substation area is input into the trained model to obtain the predicted future remaining charging capacity.

[0019] As a preferred embodiment of the present invention, the step (2) specifically comprises:

[0020] (2.1) The future remaining charge capacity obtained in step (1) is used as the state of the DQN model, specifically s t ={E t , E t+1 ,…,E T}, represents the remaining charging capacity from the current time t to the future time T;

[0021] (2.2) Determine the action of the DQN model, expressed as a t ={0, 1, .., N}, i.e., the number of electric vehicles connected to the charging pile at the current time t; where N represents the maximum number of electric vehicles that can be connected;

[0022] (2.3) Assuming that the charging power λ of each electric vehicle is constant and the same, the reward of the DQN model is expressed as:

[0023] rt =-P t a t λ-α(Na t ) 2

[0024] Among them, α is the satisfaction parameter, P t The current electricity price for electric vehicle charging;

[0025] (2.4) Determine the objective function of the DQN model as:

[0026]

[0027] Among them, π represents the action strategy; E π is the expectation under the current action strategy; γ∈[0,1] is the discount factor, which indicates the relative importance of future rewards to current rewards; γ k To consider the impact of the k-th step reward on the current reward; K is the total number of action steps considered; k is the single-step action step length; r t+1 is the reward value that can be obtained by taking the corresponding action at the next moment; s t is the current state; a t is the action at the current moment;

[0028] (2.5) Train the iterative DQN model to obtain the optimal action strategy That is, the number of electric vehicles allowed to connect to the charging pile at any moment.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention introduces the load forecasting method into the orderly charging operation of electric vehicles, and uses the predicted future remaining charging capacity as the basis for electric vehicle charging scheduling, which is more real-time and advanced than traditional methods.

[0031] 2. The present invention introduces deep reinforcement learning into the orderly charging operation of electric vehicles, uses predicted future information to pre-schedule the orderly charging of electric vehicles, and has real-time self-learning and updating capabilities. It is more flexible and effective than traditional algorithms, further improving economy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a structural diagram of the fully connected feedforward neural network (FFN) prediction model;

[0033] Figure 2 This is the flowchart of the orderly charging operation of electric vehicles based on DQN. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0035] The specific implementation scenario of this example is a substation area with electric vehicle charging piles. First, based on the historical remaining charging capacity of the charging piles in the area, a fully connected feedforward neural network (FFN) is used to predict the future remaining charging capacity. Then, based on the predicted future remaining charging capacity and the flexible electricity price mechanism, the DQN-based electric vehicle orderly charging operation is used to adjust the charging strategy of the electric vehicle, that is, the number of electric vehicles connected to the charging piles. When the number of electric vehicles connected to the charging piles in the substation area is too large, resulting in exceeding the maximum capacity of the transformer in the area, the energy storage discharge in the area is adjusted first to alleviate the charging pressure. If the energy storage is also insufficient, the charging pile is shut down through direct load control. The specific steps are as follows:

[0036] Step 1: Build and train a fully connected feedforward neural network (FFN) for predicting future remaining charge capacity, including the following steps:

[0037] (1) Build the FFN model, such as Figure 1 As shown in the figure, the model includes one convolutional layer and two linear layers; the FFN model output formula is:

[0038] Out(X)=Tanh(conv(X)W1+b1)W2+b2

[0039] Where: out(X) is the output of the FFN model; Tanh(.) is the activation function; conv(.) is the convolutional network; (W i , b i ) are network parameters; X is the network input.

[0040] The training and input data samples used for FFN model training come from the segmented data of the historical remaining charging capacity data of the substation area; the output of the FFN model is the future remaining charging capacity data;

[0041] (2) Determine the loss function for training:

[0042]

[0043] In the formula is the loss function; T is the time step; x i and Represent the model prediction value and the true label value respectively;

[0044] (3) Use the gradient descent algorithm to train the model. The specific steps are as follows:

[0045] a. Randomly initialize model parameters, i.e. weights w i and deviation b i ;

[0046] b. Pass the input data into the model and get the output;

[0047] c. Calculate the loss L according to the loss function;

[0048] d. For each neuron that produces an error, adjust the model parameters to reduce the error according to the following formula:

[0049]

[0050] Where L is the loss function; w i and b i are the neuron weights and biases of the model; α is the learning rate;

[0051] e. Repeat steps ab cd until the loss converges.

[0052] Step 2: Build and train the DQN model for orderly charging of electric vehicles. The algorithm flow is as follows: Figure 2 As shown, the following steps are included:

[0053] (1) The future remaining charge capacity obtained in step 1 is used as the state of the DQN model, specifically represented as s t ={E t , E t+1 , ..., E T , that is, the remaining charging capacity from the current time t to the future time T;

[0054] (2) Determine the action of the DQN model, represented by a t ={0, 1, ..., N}, i.e., the number of electric vehicles connected to the charging pile at the current time t; where N represents the maximum number of electric vehicles that can be connected;

[0055] (3) Assuming that the charging power λ of each electric vehicle is constant and the same, the reward of the DQN model is defined as:

[0056] r t =-P t a t λ-α(Na t ) 2

[0057] Among them, α is the satisfaction parameter, P t The electricity price for charging electric vehicles at the current moment;

[0058] The electric vehicle charging electricity price adopts a flexible electricity price mechanism, and its calculation formula is:

[0059]

[0060] Among them, A∈[0.2, 0.6] is the electricity price adjustment coefficient, is the occupancy rate of charging piles;

[0061] (4) Determine the objective function of the DQN model as:

[0062]

[0063] Among them, π represents the action strategy; Eπ is the expectation under the current action strategy; γ∈[0,1] is the discount factor, which indicates the relative importance of future rewards to current rewards; γ k To consider the impact of the k-th step reward on the current reward; K is the total number of action steps considered; k is the single-step action step length; r t+1 is the reward value that can be obtained by taking the corresponding action at the next moment; s t is the current state; a t is the action at the current moment;

[0064] (5) Train the iterative DQN model as follows:

[0065] a. According to the current environment t , use the ε greedy algorithm to select action a in the current state t ;

[0066] b. Calculate the current reward r t ;

[0067] c. Update the objective function Q(s) by the following formula t , a t ):

[0068]

[0069] In the formula, r(s t , a t ) is the current reward, i.e. r t ;max(Q(s t+1 , a t+1 )) is the maximum Q value that can be obtained by taking action at the next moment; θ∈[0,1] is the learning rate of the objective function, which indicates how much prior data is retained when the Q function is updated, that is, weighing the relative importance of the previous Q value to the current Q value;

[0070] d. Repeat steps a, b, c until |Q (σ) -Q (σ-1) |≤τ, where σ is the current step and τ is the threshold.

[0071] (6) Obtaining the optimal action strategy That is, the number of electric vehicles allowed to connect to the charging pile at any moment.

[0072] Step 3: Deploy the trained FFN model and DQN model to the user end, read the historical remaining charging capacity of the substation area in real time, and input the data into the FFN model to predict the future remaining charging capacity within a certain period of time after the current moment.

[0073] Step 4: The future remaining charging capacity predicted in step 3 is used as the state of the trained DQN model, and the DQN-based deep reinforcement learning algorithm is used to implement orderly charging operations to adjust the number of electric vehicles connected at each moment in the future. In this way, the electric vehicle charging retrieval strategy with a fixed time step within a certain period of time after the current moment is obtained in real time, that is, the number of electric vehicles connected to the charging pile at each moment.

[0074] Step 5: When the number of electric vehicles connected to the charging pile in the substation area is too large, resulting in exceeding the maximum capacity of the area transformer, the energy storage discharge in the area is adjusted first until the charging pile load is less than the sum of the maximum capacity of the transformer and the released energy storage capacity to alleviate the charging pressure; if the energy storage is also insufficient, the charging pile is shut down through direct load control.

Claims

1. A method for orderly charging of electric vehicles in a substation based on load forecasting and deep reinforcement learning, characterized in that: The following steps are involved: (1) Based on the historical remaining charging capacity information of the substation, a fully connected feedforward neural network is used to predict the future remaining charging capacity. This step specifically includes: (1.1) Obtain the historical remaining charging capacity data of the substation area and split the data samples to form training and input data, and the output data is the future remaining charging capacity data; (1.2) Build an FFN model consisting of one convolutional layer and two linear layers; the output calculation formula of the FFN model used is: Out(X)=Tanh(conv(X)W1+b1)W2+b2 Where: out(X) is the output of the FFN model; Tanh(.) is the activation function; conv(.) is the convolutional network; (W i ,b i ) are network parameters; X is network input; (1.3) Determine the loss function for training: In the formula is the loss function, abbreviated as L; T is the time step; x i and Represent the model prediction value and the true label value respectively; (1.4) Use the gradient descent algorithm to train the fully connected feedforward neural network model; (1.5) Input the historical remaining charging capacity of the substation area into the trained model to obtain the predicted future remaining charging capacity; (2) According to the flexible electricity price mechanism and the future remaining charging capacity predicted in step (1), the DQN-based deep reinforcement learning algorithm is used to adjust the electric vehicle charging strategy and control the number of electric vehicles connected to the charging piles; this step specifically includes: (2.1) The future remaining charge capacity obtained in step (1) is used as the state of the DQN model, specifically s t ={E t ,E t+1 ,...,E T }, represents the remaining charging capacity from the current time t to the future time T; (2.2) Determine the action of the DQN model, expressed as a t ={0,1,...,N}, i.e., the number of electric vehicles connected to the charging pile at the current time t; where N represents the maximum number of electric vehicles that can be connected; (2.3) Assuming that the charging power λ of each electric vehicle is constant and the same, the reward of the DQN model is expressed as: r t =-P t and t λ-α1(Na t ) 2 Among them, α1 is the satisfaction parameter; P t The electricity price for charging electric vehicles at the current moment; (2.4) Determine the objective function of the DQN model as: Among them, π represents the action strategy; E π is the expectation under the current action strategy; γ∈[0,1] is the discount factor, which indicates the relative importance of future rewards to current rewards; γ k To consider the impact of the k-th step reward on the current reward; K is the total number of action steps considered; k is the single-step action step length; r t+1 is the reward value that can be obtained by taking the corresponding action at the next moment; s t is the current state; a t is the action at the current moment; (2.5) Train the iterative DQN model to obtain the optimal action strategy That is, the number of electric vehicles allowed to connect to the charging pile at any moment.

2. The method according to claim 1, characterized in that: The specific steps of step (1.4) using the gradient descent algorithm for training include: (a) Randomly initialize model parameters, i.e. weights w i and deviation b i ; (b) Pass input data into the model and get output; (c) Calculate the loss L according to the loss function; (d) For each neuron that produces an error, adjust the model parameters to reduce the error according to the following formula: Where L is the loss function; w i and b i are the neuron weights and biases of the model; α2 is the learning rate; (e) Repeat steps (a)-(d) until the loss converges.

3. The method according to claim 1, characterized in that: In the step (2.3), P t When the flexible electricity price mechanism is adopted, the formula is: Among them, A∈[0.2,0.6] is the electricity price adjustment coefficient; is the occupancy rate of charging piles.

4. The method according to claim 1, characterized in that In the step (2.5), the specific steps of training the iterative DQN model include: (a) According to the current state s t , use the ε greedy algorithm to select action a in the current state t ; (b) Calculate the current reward r t ; (c) Update the objective function Q(s) by the following formula t ,a t ): In the formula, r(s t ,a t ) is the current reward, i.e. r t ;max(Q(s t+1 ,a t+1 )) is the maximum Q value that can be obtained by taking action at the next moment; θ∈[0,1] is the learning rate of the objective function, which indicates how much prior data is retained when the Q function is updated, that is, weighing the relative importance of the previous Q value to the current Q value; (d) Repeat steps (a)-(c) until |Q (σ) -Q (σ-1) |≤τ, where σ is the current step and τ is the threshold.

5. The method according to claim 1, characterized in that: When there are too many electric vehicles connected to the charging piles in the substation area, resulting in exceeding the maximum capacity of the transformer in the area: priority is given to adjusting the discharge of the energy storage in the area until the load on the charging pile is less than the sum of the maximum capacity of the transformer and the released energy storage capacity, thereby alleviating the charging pressure; if the energy storage is also insufficient, the charging pile is shut down through direct load control.

Images