A Deep Reinforcement Learning-Based Ordered Charging Method for Large-Scale Charging Stations Based on Disjunctive Graphs

Abstract

This invention discloses a deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs, comprising the following steps: 1) Discretizing the charging tasks of electric vehicles to obtain the number of discrete charging tasks and the expected parking time requirement; 3) Assigning electricity price identifiers to electric vehicles; 4) Constructing a charging scheduling disjunctive graph based on the number of discrete charging tasks and the electricity price identifiers; 5) Extracting features from the disjunctive graph using a graph attention mechanism to obtain global features; 6) Constructing a value network and a policy network; 7) Inputting the global features into the value network and the policy network, and using a multilayer perceptron to predict the score of each action to obtain the probability of each charging action; 8) Selecting the charging action with the highest probability as the electric vehicle charging strategy. This invention proposes an ordered charging scheduling strategy for electric vehicles based on a charging priority model and a graph neural network reinforcement learning method, which significantly optimizes the operating efficiency and resource allocation of charging stations.

Description

Technical Field

[0001] This invention relates to the field of electric vehicles, specifically a deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs. Background Technology

[0002] Electric vehicles, as a clean mode of transportation that reduces greenhouse gas emissions and dependence on fossil fuels, have experienced rapid development in recent years. By the end of 2023, China's new energy vehicle ownership had exceeded 20.41 million, continuing to grow at an annual rate of over 30%. However, the rapid popularization of electric vehicles has also placed higher demands on the scientific deployment and scheduling of charging infrastructure, as unreasonable charging behavior may pose challenges to the stable operation of the power grid. Therefore, research on orderly charging strategies is of great significance for the stability of the power grid and the development of low-carbon transportation.

[0003] Although ordered charging strategies are widely considered an effective way to address grid load fluctuations, existing technologies still have several shortcomings in practical applications. First, due to the highly random arrival times of electric vehicles, charging demand exhibits significant dynamic fluctuations. Existing optimization methods often assume complete and fixed environmental information, making it difficult to cope with this uncertainty. This assumption leads to low applicability of traditional optimization algorithms in real-world scenarios, and charging scheduling results often fail to meet expectations. Second, many current optimization methods rely on complex objective functions and large-scale iterative calculations, such as Q-learning in reinforcement learning. While these methods can theoretically achieve a global optimum, the dimensionality of the state and action spaces increases significantly with the number of electric vehicles, exacerbating the curse of dimensionality. Especially in real-time charging scheduling, high computational complexity and large solution delays make these algorithms difficult to meet real-time requirements, thus affecting the execution efficiency of the scheduling strategy. Furthermore, most existing methods focus on a single vehicle or vehicle cluster, training each vehicle independently, neglecting the balance between group characteristics and global optimization, further increasing system instability. Under the dual pressure of dynamic grid load changes and high-frequency electric vehicle access, existing technologies still have considerable room for improvement in balancing efficiency and reliability. These shortcomings make designing efficient and real-time scheduling optimization algorithms under the uncertainties and dynamics of the charging process a significant challenge in the research of orderly charging scheduling for electric vehicles. Summary of the Invention

[0004] The purpose of this invention is to provide a deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs, comprising the following steps:

[0005] 1) Obtain charging station data, including electricity price data and charging data;

[0006] 2) Based on charging data, the charging tasks of electric vehicles are discretized to obtain the number of discrete charging tasks and the expected parking time requirements.

[0007] 3) Based on the expected parking time demand and electricity price data, assign an electricity price identifier to electric vehicles;

[0008] 4) Construct a charging scheduling disjunction graph based on the number of discrete charging tasks and the electricity price identifier;

[0009] 5) Employ graph attention mechanism to extract features from the disjunctive graph to obtain global features;

[0010] 6) Construct value networks and policy networks;

[0011] 7) Input the global features into the value network and policy network, and use the multilayer perceptron to predict the score of each action to obtain the probability of each charging action;

[0012] 8) Select the charging action with the highest probability as the electric vehicle charging strategy.

[0013] Furthermore, the charging station data includes electricity price data and charging data.

[0014] Furthermore, the electricity price data refers to the electricity price in the past time period t;

[0015] The charging data includes charging demand data and estimated departure time data;

[0016] The charging demand data refers to the time when an electric vehicle arrives at an electric vehicle charging station and begins charging.

[0017] The estimated departure time data refers to the time when the electric vehicle charging will end.

[0018] Furthermore, the number of discrete charging tasks is shown below:

[0019]

[0020] in, I represents the charging demand of the i-th electric vehicle at time t. t Let Δ represent the set of electric vehicles charging at the charging station at time t. t Indicates the time interval between each scheduled task; P i This represents the number of discrete charging tasks for the i-th electric vehicle.

[0021] Furthermore, the expected parking time demand is as follows:

[0022]

[0023] In the formula, d iD represents the estimated parking time requirement of the i-th electric vehicle in a scheduling task; i Let represent the total estimated parking time requirement for the i-th electric vehicle.

[0024] Furthermore, the electricity price identifier is shown below:

[0025]

[0026] Among them, c t T represents the current electricity price, and T represents the current time. The task of charging at time t; Let t be the estimated parking time requirement.

[0027] Furthermore, in the charging scheduling parsing graph, electric vehicles with an electricity price identifier of 1 have a higher charging priority than electric vehicles with an electricity price identifier of 0.

[0028] When the electricity price identifier is the same, electric vehicles with fewer discrete charging tasks have higher charging priority than electric vehicles with more discrete charging tasks.

[0029] Furthermore, global feature h pool As shown below:

[0030]

[0031] In the formula, w j For weights; h j Let N be the feature of the j-th node in the charging scheduling disjunction graph; N is the number of nodes in the charging scheduling disjunction graph.

[0032] Furthermore, the value network is shown below:

[0033] Q(s,a)=f(h pool (5)

[0034] Where Q(s,a) represents the future reward for state S and action a. f(·) represents the prediction function of the neural network.

[0035] Furthermore, the policy network P(a i As shown below:

[0036]

[0037] The technical effects of this invention are undeniable. This invention proposes an orderly charging scheduling strategy for electric vehicles based on a charging priority model and a graph neural network reinforcement learning method, which significantly optimizes the operating efficiency and resource allocation of charging stations.

[0038] The beneficial effects of this invention are as follows:

[0039] 1) By accurately quantifying the key characteristics of each vehicle (such as estimated charging amount, remaining battery power, and parking time), and combining this with an efficient data statistics and processing workflow, the overall charging demand is transformed into multiple manageable sub-tasks. Utilizing a charging priority calculation model, combined with historical electricity price data and real-time load information, task priorities are scientifically calculated, and an innovative charging task sorting mechanism is proposed. This prioritizes time-sensitive and urgent tasks, ensuring the fairness and efficiency of scheduling.

[0040] 2) By introducing Graph Neural Networks (GNNs), the charging scheduling problem is transformed into a graph problem through the construction of a disjunctive graph structure, significantly simplifying the dimensions of the state and action spaces. Combined with Graph Attention Networks (GATs), features are extracted from nodes in the graph to capture the relationships between electric vehicles, thereby optimizing the accuracy and coordination of task allocation. Simultaneously, this method employs a pulse charging approach, transforming the charging process into multiple discrete tasks, avoiding the complexity and shortcomings of centralized training for all vehicles in traditional methods, effectively reducing computational complexity and training costs.

[0041] 3) By combining priority with graph neural networks, an efficient and flexible scheduling model was constructed, which not only improved the operating efficiency of electric vehicle charging stations, but also provided technical support for the future expansion of charging networks. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention;

[0043] Figure 2 This is the interactive display of scheduling information in this invention;

[0044] Figure 3 This is the disjunction graph structure of the present invention;

[0045] Figure 4 This is a schematic diagram of the unified multimodal mechanism of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0047] Example 1:

[0048] See Figures 1 to 4 A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs includes the following steps:

[0049] 1) Obtain charging station data, including electricity price data and charging data;

[0050] 2) Based on charging data, the charging tasks of electric vehicles are discretized to obtain the number of discrete charging tasks and the expected parking time requirements.

[0051] 3) Based on the expected parking time demand and electricity price data, assign an electricity price identifier to electric vehicles;

[0052] 4) Construct a charging scheduling disjunction graph based on the number of discrete charging tasks and the electricity price identifier;

[0053] 5) Employ graph attention mechanism to extract features from the disjunctive graph to obtain global features;

[0054] 6) Construct value networks and policy networks;

[0055] 7) Input the global features into the value network and policy network, and use the multilayer perceptron to predict the score of each action to obtain the probability of each charging action;

[0056] 8) Select the charging action with the highest probability as the electric vehicle charging strategy.

[0057] The charging station data includes electricity price data and charging data.

[0058] The electricity price data refers to the electricity price in the past time period t;

[0059] The charging data includes charging demand data and estimated departure time data;

[0060] The charging demand data refers to the time when an electric vehicle arrives at an electric vehicle charging station and begins charging.

[0061] The estimated departure time data refers to the time when the electric vehicle charging will end.

[0062] The number of discrete charging tasks is shown below:

[0063]

[0064] in, I represents the charging demand of the i-th electric vehicle at time t. t Let Δ represent the set of electric vehicles charging at the charging station at time t. t Indicates the time interval between each scheduled task; P i This represents the number of discrete charging tasks for the i-th electric vehicle.

[0065] The estimated parking time demand is as follows:

[0066]

[0067] In the formula, d iD represents the estimated parking time requirement of the i-th electric vehicle in a scheduling task; i Let represent the total estimated parking time requirement for the i-th electric vehicle.

[0068] Electricity price identifiers are shown below:

[0069]

[0070] Among them, c t T represents the current electricity price, and T represents the current time. The task of charging at time t; Let t be the estimated parking time requirement.

[0071] In the charging scheduling parsing diagram, electric vehicles with an electricity price identifier of 1 have a higher charging priority than electric vehicles with an electricity price identifier of 0.

[0072] When the electricity price identifier is the same, electric vehicles with fewer discrete charging tasks have higher charging priority than electric vehicles with more discrete charging tasks.

[0073] Global feature h pool As shown below:

[0074]

[0075] In the formula, w j For weights; h j Let N be the feature of the j-th node in the charging scheduling disjunction graph; N is the number of nodes in the charging scheduling disjunction graph.

[0076] The value network is shown below:

[0077] Q(s,a)=f(h pool (5)

[0078] Where Q(s,a) represents the future reward for state S and action a. f(·) represents the prediction function of the neural network.

[0079] Policy network P(a i As shown below:

[0080]

[0081] Example 2:

[0082] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs includes the following steps:

[0083] 1) Obtain charging station data, including electricity price data and charging data;

[0084] 2) Based on charging data, the charging tasks of electric vehicles are discretized to obtain the number of discrete charging tasks and the expected parking time requirements.

[0085] 3) Based on the expected parking time demand and electricity price data, assign an electricity price identifier to electric vehicles;

[0086] 4) Construct a charging scheduling disjunction graph based on the number of discrete charging tasks and the electricity price identifier;

[0087] 5) Employ graph attention mechanism to extract features from the disjunctive graph to obtain global features;

[0088] 6) Construct value networks and policy networks;

[0089] 7) Input the global features into the value network and policy network, and use the multilayer perceptron to predict the score of each action to obtain the probability of each charging action;

[0090] 8) Select the charging action with the highest probability as the electric vehicle charging strategy.

[0091] Example 3:

[0092] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs is provided. The technical content is the same as in Embodiment 2. Furthermore, the charging station data includes electricity price data and charging data.

[0093] Example 4:

[0094] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs, with the same technical content as any one of embodiments 2-3, further wherein the electricity price data is the electricity price in the past t time period;

[0095] The charging data includes charging demand data and estimated departure time data;

[0096] The charging demand data refers to the time when an electric vehicle arrives at an electric vehicle charging station and begins charging.

[0097] The estimated departure time data refers to the time when the electric vehicle charging will end.

[0098] Example 5:

[0099] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs, with the same technical content as any one of embodiments 2-4, further wherein the number of discrete charging tasks is as follows:

[0100]

[0101] in, I represents the charging demand of the i-th electric vehicle at time t. tLet Δ represent the set of electric vehicles charging at the charging station at time t. t Indicates the time interval between each scheduled task; P i This represents the number of discrete charging tasks for the i-th electric vehicle.

[0102] Example 6:

[0103] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs is proposed. The technical content is the same as any one of embodiments 2-5. Furthermore, the expected parking time requirement is as follows:

[0104]

[0105] In the formula, d i D represents the estimated parking time requirement of the i-th electric vehicle in a scheduling task; i Let represent the total estimated parking time requirement for the i-th electric vehicle.

[0106] Example 7:

[0107] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs is proposed, with the technical content being the same as any one of embodiments 2-6. Further, the electricity price identifier is as follows:

[0108]

[0109] Among them, c t T represents the current electricity price, and T represents the current time.

[0110] Example 8:

[0111] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs, with the same technical content as any one of embodiments 2-7. Further, in the charging scheduling disjunctive graph, electric vehicles with an electricity price identifier of 1 have a higher charging priority than electric vehicles with an electricity price identifier of 0.

[0112] When the electricity price identifier is the same, electric vehicles with fewer discrete charging tasks have higher charging priority than electric vehicles with more discrete charging tasks.

[0113] Example 9:

[0114] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs is proposed. The technical content is the same as any one of embodiments 2-8, further comprising the global feature h. pool As shown below:

[0115]

[0116] In the formula, w j For weights; h jLet N be the feature of the j-th node in the charging scheduling disjunction graph; N is the number of nodes in the charging scheduling disjunction graph.

[0117] Example 10:

[0118] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs is proposed. The technical content is the same as any one of embodiments 2-9. Furthermore, the value network is shown below:

[0119] Q(s,a)=f(h pool (5)

[0120] Where Q(s,a) represents the future reward for state S and action a. f(·) represents the prediction function of the neural network.

[0121] Example 11:

[0122] A method for ordered charging of large-scale charging stations based on disjunctive graphs using deep reinforcement learning, with technical content identical to any one of embodiments 2-10, further comprising a policy network P(a i As shown below:

[0123]

[0124] Example 12:

[0125] A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs is proposed, which employs a data input module, a priority determination module, a scheduling disjunctive graph construction module, and a policy and value network construction module.

[0126] The specific implementation steps are as follows:

[0127] Step 1: Data Input

[0128] The model receives data in different modalities, including:

[0129] Electricity price data: Electricity prices for the past 24 hours

[0130] Charging data:

[0131] Charging demand data: Electric vehicles inform the aggregator of their charging information when they arrive at an electric vehicle charging station.

[0132] Estimated departure time data: Electric vehicles inform the aggregator of their charging information when they arrive at the electric vehicle charging station.

[0133] Step Two:

[0134] For the charging demand of the i-th electric vehicle, we first transform the continuous charging task into multiple subtasks, discretizing the continuous problem. The current charging task contains P subtasks. i It can be obtained from formula (1):

[0135]

[0136] in, I represents the charging demand of the i-th electric vehicle at time t. t Let Δ represent the set of electric vehicles charging at the charging station at time t. t Indicates the time interval between each scheduling task.

[0137] The estimated parking time requirement for the i-th electric vehicle can also be quantified using the above approach:

[0138]

[0139] The next step is to determine the identifier. This determination is based on our initial assumption that the electric vehicle will always operate at its maximum power output. According to the modeling above, we know that P is required. i The time allotted for each sub-task can be used to complete charging as quickly as possible. Then, find the fastest charging completion time to reach the estimated departure time to complete task d. i If, during this period, there are times when the electricity price is lower than the current price, then we assign an identifier to that vehicle.

[0140]

[0141] Among them, c t T represents the current electricity price, and T represents the current time.

[0142] Step 3: Construction of the charging scheduling parsing graph

[0143] like Figure 3 As shown, to enhance the coupling relationships during the electric vehicle charging process, a disjunctive graph structure is introduced, transforming the charging scheduling problem into a graph problem. The charging tasks for each electric vehicle are divided into P... i We consider the subtasks and focus on the priority relationship between the first subtasks of each electric vehicle. Using the identifiers generated in step two, we believe that electric vehicles without identifiers should be charged with priority to electric vehicles with identifiers, and electric vehicles with identifiers should be charged with priority to electric vehicles with more charging subtasks.

[0144] Step 4: Strategy and Value Network Construction Module

[0145] like Figure 4 Specifically, we extract features from the disjunctive graph constructed in step three. We use a graph attention mechanism to extract features from the disjunctive graph to achieve comprehensive fusion of information from different nodes. Specifically, we use an attention mechanism to extract features from each node, and finally pool the features extracted from each node to obtain the global features.

[0146] Obtain global features

[0147]

[0148] Among them, N is the number of nodes, and F is the number of features for each node. The output of this layer is a new set of node features h′.

[0149]

[0150] in F′ is the new feature number. We then perform weighted pooling on these newly obtained node features.

[0151]

[0152] Weighted pooling works by applying weighted pooling to each vector h. i Apply weight W i The features of all nodes are weighted and summed to obtain a global vector h. pool Weight W i Depending on the specific task, if a node represents the first task of charging an electric vehicle, then we will assign higher weight to the characteristics of that node. In addition, we will also assign higher weight to nodes without an identifier for electric vehicle charging tasks.

[0153] Constructing value networks

[0154] Q(s,a)=f(h pool (7)

[0155] Where Q(s,a) represents the future reward for state S and action a. f(·) represents the prediction function of the neural network.

[0156] Constructing policy networks

[0157]

[0158] The algorithm uses a multilayer perceptron (MLP) to predict the score for each action, and finally converts all possible scores into the probability of each action using a softmax function.

Claims

1. A deep reinforcement learning-based ordered charging method for large-scale charging stations based on disjunctive graphs, characterized in that, Includes the following steps: Step 1. Obtain charging station data, including electricity price data and charging data; Step 2. Based on the charging data, discretize the charging tasks of electric vehicles to obtain the number of discrete charging tasks and the expected parking time requirements; Step 3. Based on the expected parking time demand and electricity price data, assign an electricity price identifier to the electric vehicle; Step 4. Construct a charging scheduling disjunction graph based on the number of discrete charging tasks and the electricity price identifier; Step 5. Use graph attention mechanism to extract features from the disjunctive graph to obtain global features; Step 6. Construct the value network and policy network; Step 7. Input the global features into the value network and policy network, and use the multilayer perceptron to predict the score of each action to obtain the probability of each charging action; Step 8. Select the charging action with the highest probability as the electric vehicle charging strategy; The number of discrete charging tasks is shown below: （1） in, Let represent the charging demand of the i-th electric vehicle at time t. Let represent the set of electric vehicles charging at the charging station at time t. This indicates the time interval between each scheduled task; This represents the number of discrete charging tasks for the i-th electric vehicle; The estimated parking time demand is as follows: （2） In the formula, This represents the estimated parking time requirement of the i-th electric vehicle during a charging task; This represents the total estimated parking time requirement for the i-th electric vehicle; Electricity Price Identifier As shown below: （3） in, This indicates the current electricity price. Indicates the current time; The task of charging at time t; Let t be the estimated parking time requirement.

2. The method for ordered charging of large-scale charging stations based on disjunctive graphs according to claim 1, characterized in that, The charging station data includes electricity price data and charging data.

3. The method for ordered charging of large-scale charging stations based on disjunctive graphs according to claim 1, characterized in that, The electricity price data refers to the electricity price in the past time period t; The charging data includes charging demand data and estimated departure time data; The charging demand data refers to the time when an electric vehicle arrives at an electric vehicle charging station and begins charging. The estimated departure time data refers to the time when the electric vehicle charging will end.

4. The method for ordered charging of large-scale charging stations based on disjunctive graphs according to claim 1, characterized in that, In the charging scheduling parsing diagram, electric vehicles with an electricity price identifier of 1 have a higher charging priority than electric vehicles with an electricity price identifier of 0. When the electricity price identifier is the same, electric vehicles with fewer discrete charging tasks have higher charging priority than electric vehicles with more discrete charging tasks.

5. The method for ordered charging of large-scale charging stations based on disjunctive graphs according to claim 1, characterized in that, Global features As shown below: （4） In the formula, As weight; Let N be the feature of the j-th node in the charging scheduling disjunction graph; N is the number of nodes in the charging scheduling disjunction graph.

6. The method for ordered charging of large-scale charging stations based on disjunctive graphs according to claim 1, characterized in that, The value network is shown below: （5） in This represents the future reward of state S and action a; This represents the prediction function of the neural network.

7. The method for ordered charging of large-scale charging stations based on disjunctive graphs according to claim 1, characterized in that, Policy Network As shown below: （6） In the formula, For policy networks.

Images