Highway charging station intelligent guiding method based on multi-agent reinforcement learning

Abstract

The invention belongs to the technical field of electric vehicle charging, and discloses a highway charging station intelligent guiding method based on multi-agent reinforcement learning, and the method comprises the steps: S1, collecting the data of a highway network, the geographic position and capacity information of a charging station, and the dynamic information of a vehicle from real order data; s2, constructing a scheduling system model, wherein the scheduling system model comprises an environment modeling and structure sensing module, a multi-agent decision module and a joint scheduling module; and S3, inputting the information acquired in the step S1 into a scheduling system model to generate a specific charging distribution scheme. The method has the following advantages that the charging congestion problem in the peak period of holidays and festivals is effectively relieved, and the average waiting time of vehicles is remarkably shortened; the overall resource utilization rate of the site is improved, and load balancing in a global range is realized; the adaptability of the scheduling strategy to the dynamic change of traffic flow is enhanced, and the charging completion rate and the system throughput are improved; and the method has high engineering feasibility and popularization price.

Description

Technical Field

[0001] This invention belongs to the field of electric vehicle charging technology, specifically relating to an intelligent guidance method for highway charging stations based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous advancement of the new energy strategy, electric vehicles (EVs) are rapidly gaining popularity across the country. By the end of 2024, China's EV ownership had exceeded 20 million, a significant portion of which were used for medium- and long-distance travel. As the core road network for long-distance commuting and logistics, highways rely heavily on their charging infrastructure, which is a crucial component of EV operation. Especially during peak travel periods such as the Spring Festival, May Day, and National Day holidays, the demand for EV charging in highway service areas experiences explosive growth, creating a typical spatial-temporal imbalance between supply and demand. Some popular service areas (such as entrance / exit hubs and intersections of popular tourist routes) have consistently high charging pile loads, resulting in severe congestion with vehicles waiting for charging stations. On the other hand, some remote service areas, even with well-equipped facilities, suffer from extremely low utilization rates due to a lack of guidance, leading to resource waste.

[0003] Electric vehicle drivers face several challenges while driving: First, they cannot keep track of the queue status and available charging stations at various service areas along the route in real time, often relying on navigation software to choose the nearest service area, which easily leads to vehicle congestion. Second, even if in-vehicle navigation or mobile apps provide charging status information for some service areas, the information is delayed, and decisions are made based on this delayed data, which can easily lead to misjudgments. Third, due to the non-cooperative and competitive nature of driver behavior, when several vehicles simultaneously change their destination stations, secondary congestion often occurs at the new stations, resulting in queue fluctuations or even empty runs. Fourth, drivers' anxiety about remaining battery power and range further amplifies the impact of congestion; some vehicles, even if the detour is only a few kilometers, dare not change their predetermined routes due to uncertainty, leading to low utilization efficiency of charging infrastructure and failure to maximize service capacity.

[0004] Highway environments are characterized by high traffic volume, sparse charging station distribution, and significant driver range anxiety, making traditional urban charging guidance strategies difficult to apply directly to highway scenarios. Research remains lacking in real-time guidance strategies for highway environments, particularly when considering charging demand triggered by vehicle state of charge (SOC). While traditional urban charging guidance strategies have achieved some success in urban environments, they still suffer from poor adaptability, insufficient real-time performance, incomplete driver behavior modeling, limited multi-agent collaborative optimization capabilities, and a lack of predictive ability for future charging demand.

[0005] From the perspective of highway operation and maintenance management units, the most critical operational challenge lies in how to achieve load balancing and minimize user waiting time across the entire road network service areas through effective scheduling and guidance, while ensuring the safe range of electric vehicles. Traditional rule-based or centralized scheduling solutions struggle to adapt to the complexity and nonlinear dynamic changes in traffic behavior, resulting in slow responses to high-frequency charging requests and sudden surges in traffic volume, and a lack of flexibility. Therefore, how to build a highly responsive, highly collaborative, and deployable intelligent guidance system without introducing additional high-cost V2X infrastructure or relying on vehicle-side heavy inference computing has become a key technical challenge in the field of transportation and energy integration. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an intelligent guidance method for highway charging stations based on multi-agent reinforcement learning. This method aims to solve problems such as poor adaptability of existing charging guidance strategies in highway environments, insufficient real-time performance, incomplete driver behavior modeling, limited multi-agent collaborative optimization capabilities, and a lack of ability to predict future charging demand. This invention recommends optimal charging solutions for electric vehicle users through intelligent algorithms, aiming to reduce charging queuing time, improve the utilization efficiency of charging infrastructure, alleviate the difficulty of charging on highways, and enhance user travel experience and charging success rate.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for intelligent guidance at highway charging stations based on multi-agent reinforcement learning includes the following steps:

[0009] S1. Collect data on the highway network, the geographical location and capacity information of charging stations, and dynamic information on vehicles based on real order data;

[0010] S2. Construct a scheduling system model, which includes an environment modeling and structure perception module, a multi-agent decision-making module, and a joint scheduling module;

[0011] Environment Modeling and Structure Awareness Module: Responsible for capturing the topological features and real-time status information of highway networks and charging infrastructure, and encoding complex relationships through graph structure embedding;

[0012] Multi-agent decision-making module: Based on the environmental information obtained by the environmental modeling and structure perception module, it uses a multi-agent deep reinforcement learning algorithm to learn policies and generate policies for each local decision under a centralized training and decentralized execution framework.

[0013] Joint scheduling module: integrates the actions output by each agent, combines system constraints and objectives to perform global allocation and optimization, so as to ensure that scheduling decisions are feasible and efficient;

[0014] S3. Input the information collected in step S1 into the scheduling system model constructed in step S2 to generate a specific charging allocation scheme.

[0015] Furthermore, the charging station capacity information includes the current load of each charging station, the number of remaining charging piles, and the available power.

[0016] Furthermore, the dynamic information of the vehicles, derived from real order data, includes the location, driving trajectory, destination arrival time, and SOC status of each electric vehicle.

[0017] Furthermore, in step S2, the design of the environment modeling and structure perception module is as follows:

[0018] I. Introduce structure perception to form a graph G = (V, E) consisting of road network and charging stations;

[0019] Wherein, node V includes charging stations and vehicle units; edge E represents road connectivity or accessibility between vehicles and stations; the feature vector of charging station nodes includes real-time queue length, charging rate limit, and historical demand statistics; the feature vector of vehicle nodes includes current SOC, remaining driving range, destination, and highway congestion information.

[0020] II. Graph neural networks (GNNs) are used to embed the graph G = (V, E) to encode node features and neighborhood structure in a comprehensive manner in order to obtain a perception of global relationships.

[0021] III. A graph convolutional network (GCN) model is adopted to update node representations using an iterative message-passing method;

[0022] For node i, the embedding at layer l+1 is represented as:

[0023]

[0024] In the formula, N(i) represents the set of neighbors connected to node i; W,b are the learnable weights and biases; σ is the activation function; The embedding of neighbor node j at layer l;

[0025] IV. Output an environmental state representation for use by subsequent modules;

[0026] The environment state representation includes the global state vector s t and the local observations required by each agent. i (t).

[0027] Furthermore, in step S2, the design of the multi-agent decision-making module is as follows:

[0028] Policy learning is performed using the reinforcement learning algorithm of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) to achieve intelligent decision-making for charging scheduling.

[0029] I. Define states and actions;

[0030] Assuming there are N agents in the system, given a time step t, what action a is performed by agent i? i (t) is defined as the decision on the charging strategy; the combination of actions forms a joint action a(t) = (a1(t), ..., a N (t)), the environment updates its state based on the joint action;

[0031] II. Design the reward function r based on the scheduling objective. i (t);

[0032] The scheduling objectives include minimizing charging wait time, minimizing the total cost of electric vehicle driving and charging, and balancing the load of each station;

[0033] r i (t) = -(α × travel time + β × charging cost + γ × waiting time)

[0034] In the formula, α, β, and γ represent the weights of driving time, charging cost, and waiting time, respectively.

[0035] The overall optimization objective is to maximize the cumulative reward for all agents or minimize the cumulative cost, i.e., to obtain the expected discount benefit. maximum;

[0036] Multi-agent deep deterministic policy gradient (MADDPG) belongs to the centralized training and distributed execution (CTDE) paradigm: during training, the Critic network can access the global state s of all agents. t And the joint action a(t), while the Actor network for each agent only depends on the local observations o of the corresponding agent. i (t); the policy of the i-th agent is determined by the Actor network μ. i (o i |θ i The value is indicated by the Critic network Q. i (s,a|φ i Evaluation; training utilizes experience replay and target network techniques: maintaining the Critic network parameters φ i and the corresponding target network φ′ i and the Actor network parameters θ i With the target network θ′ i ;

[0037] For a given empirical sample (s) t ,a(t),ri (t),s t+1 The target Q-value of the Critic network is calculated as follows:

[0038] y i (t)=r i (t)+γ′Q i′ (s t+1 ,μ 1′ (o1(t+1)),...,μ N′ (o N (t+1)))

[0039] Among them, y i (t) represents the target value of the i-th agent at time step t; Q i′ (t) represents the Q-value calculated by the target Critic network for the i-th agent; μ N′ Represents the target policy network; γ′ is the discount factor;

[0040] The training objective of Critic is to minimize the mean squared error loss:

[0041] L(φ i )=E[(Q i (s t ,a(t)|φ i )-y i (t)) 2 ]

[0042] In the formula, Q i (s t ,a(t)|φ i Indicates the global state s t When the action a(t) is taken, the parameter φ i The expected return of the decision; E[...] represents the expectation, which is the average over all possible outcomes;

[0043] When calculating gradient updates, the chain rule is used to output the corresponding action from the Actor, which drives the Critic to output a higher Q value.

[0044] The update objective of the Actor network is to maximize the Critic evaluation value, that is, to increase the expected return through the policy gradient. Its update direction is written as:

[0045]

[0046] Where J(θ) i Let represent the policy performance index function of the i-th agent; Es represent the expectation of state s; D represent the data distribution sampled from the experience pool.

[0047] Adjust θ using a negative gradient, either ascending or descending.i ;

[0048] In the algorithm implementation, all agents share a unified experience replay pool and perform gradient updates in batches.

[0049] Furthermore, in step S2, the joint scheduling module is designed as follows:

[0050] W=(∑P(x)*H(x)) / H(D)

[0051] The system takes the decision intentions of all agents at the current time step and the real-time status of the charging stations as inputs, and performs load aggregation and constraint judgment on each charging station.

[0052] For charging station j, calculate the set Ij of vehicles planning to travel to charging station j and the corresponding total demand ∑i∈Ija. i Meanwhile, the total number of charging piles C at charging station j is also considered. j and the maximum charging rate;

[0053] If the total demand ∑i∈Ija i If the supply does not exceed the maximum feasible supply, it is allocated directly; otherwise, conflict resolution is required.

[0054] In the event of a conflict, minimize the additional waiting time for all vehicles or balance the load differences between stations to determine the final vehicle station allocation and charging start sequence;

[0055] The following decision variables and constraints are introduced for formalization:

[0056] Let x be a binary variable i,j =1 indicates that vehicle i is eventually dispatched to station j, then the goal is:

[0057]

[0058] Among them, c i,j The cost of vehicle i traveling to station j includes travel time and queuing time; constraints include... and This means that each vehicle is assigned only one station. This indicates the capacity constraint of the charging piles at the site.

[0059] Beneficial effects:

[0060] This invention provides an intelligent guidance method for highway charging stations based on multi-agent reinforcement learning. In highway electric vehicle charging scenarios, by constructing a multi-agent joint scheduling model driven by real orders, it achieves full-process charging guidance based on station state perception, structural relationship modeling, and strategy optimization. Compared to traditional rule-based or single-point optimization-based scheduling methods, the proposed solution has the following significant advantages: First, it effectively alleviates charging congestion during peak holiday periods, significantly reducing the average vehicle waiting time; second, it improves the overall resource utilization of stations, achieving global load balancing; third, it enhances the adaptability of the scheduling strategy to dynamic changes in traffic flow, improving charging completion rate and system throughput; fourth, the model is built and optimized entirely based on real data, possessing strong engineering feasibility and promotional value. Overall, this solution demonstrates significant effectiveness in ensuring user charging experience and improving charging station operational efficiency.

[0061] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0062] Figure 1 This is an architecture diagram of the Multi-Agent Deep Deterministic Policy Gradient (MADCG). Detailed Implementation

[0063] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0064] This invention provides an intelligent guidance method for highway charging stations based on multi-agent reinforcement learning, comprising the following steps:

[0065] S1. Collect data on the highway network, the geographical location and capacity information of charging stations, and dynamic information on vehicles based on real order data;

[0066] The charging station capacity information includes the current load of each charging station, the number of remaining charging piles, and the available power.

[0067] The vehicle's dynamic information, derived from real order data, includes the location, driving trajectory, destination arrival time, and SOC status of each electric vehicle.

[0068] S2. Construct a scheduling system model;

[0069] The scheduling system model mainly comprises three modules: an environment modeling and structure perception module, a multi-agent decision-making module, and a joint scheduling module. These three modules work together to achieve efficient guidance and optimization of the charging process. The environment modeling module is responsible for capturing the topological features and real-time state information of the highway network and charging infrastructure, and encoding complex relationships through graph embedding. Based on the above environmental information, the multi-agent decision-making module uses the Multi-Agent Deep Reinforcement Learning (MADDPG) algorithm for policy learning, generating policies for each agent's local decisions within a centralized training and distributed execution (CTDE) framework. The joint scheduling module integrates the actions output by each agent, combining system constraints and objectives for global allocation and optimization to ensure feasible and efficient scheduling decisions. The following sections provide detailed explanations of the structure, data sources, working mechanisms, inputs and outputs, core formulas, state and action definitions, reward functions, and optimization objectives of each module, and present the key algorithms.

[0070] Environment Modeling and Structure Awareness Module

[0071] The Environment Modeling and Structure Awareness module aims to accurately describe the environmental state and spatial structure of the highway charging scheduling problem. In the highway scenario, the environment can be formalized as a graph model based on the road network and charging stations: nodes include charging stations and vehicles in motion, and edges represent road connectivity or reachability between vehicles and stations. The inputs to this module mainly include highway network data, charging station geographical location and capacity information, and dynamic information derived from real order data (such as vehicle travel trajectories, arrival times, and SOC status). Specifically, the original data source for the environmental state can be constructed using actual operational data of highway service areas and charging stations, as well as vehicle travel order data. The environmental state includes two parts of information: first, the local state of charging stations, such as the current load of each station, the number of remaining charging piles, and the available power; second, the global state of the vehicle group, such as the location, remaining battery power (SOC), and destination of each electric vehicle.

[0072] Because highway networks have obvious spatial structure characteristics, structural perception is introduced to form a graph G = (V, E) consisting of road networks and charging stations. Here, node V includes charging stations and on-board units; edge E represents road connectivity or accessibility between vehicles and stations; the feature vector of charging station nodes includes real-time queue length, charging rate limit, and historical demand statistics; the feature vector of vehicle nodes includes current SOC, remaining driving range, destination, and highway congestion information.

[0073] By embedding a graph G = (V, E) using a graph neural network (GNN), node features and neighborhood structure are comprehensively encoded to obtain a perception of global relationships.

[0074] A graph convolutional network (GCN) model is used to update node representations using an iterative message-passing method;

[0075] For node i, the embedding at layer l+1 is represented as:

[0076]

[0077] In the formula, N(i) represents the set of neighbors connected to node i; W,b are the learnable weights and biases; σ is the activation function; The embedding of neighbor node j at layer l is defined. Through multi-layer propagation, local constraints and global topological information between charging stations and vehicles in the graph can be captured. In this way, the environmental state, after being encoded by the GNN, can reflect the road network structure (station distribution, proximity relationships) and the coupling interaction between vehicles and stations.

[0078] The output of this module is an environmental state representation for use by subsequent decision-making modules, such as the global state vector s. t and the local observations required by each agent. i (t). Under the CTDE framework, the global state s t Typically, it contains complete information about all agents, for use by a centralized Critic network; while the local observations o of each agent i... i (t) contains only information about the agent itself and its local environment (e.g., the vehicle's SOC, current location, and simplified information about nearby charging stations). Through the embedding obtained by the structure perception module, the environmental state representation includes both individual node features and topological neighborhood information, ensuring that subsequent decisions can consider spatial relationships and charging resource distribution. In summary, the environment modeling and structure perception module provides a dynamic environmental representation for the system: the input is the highway map structure and order demand data, and the output is a high-dimensional embedded state, providing an accurate state description for the multi-agent decision-making module.

[0079] Multi-agent decision-making module

[0080] The multi-agent decision-making module is the core of the system. It uses multi-agent deep reinforcement learning to learn policies and achieve intelligent decision-making for charging scheduling. We assume there are N agents in the system. Typically, each electric vehicle needing charging can be considered an agent, or the charging station can be considered an agent, depending on the specific scenario. In this invention, we follow a common approach: treating each electric vehicle as an agent. In this case, the local observation o of the i-th agent... i(t) includes information such as the vehicle's current SOC, remaining driving range, current location, required charging amount, and possible environmental prompts (such as destination direction or traffic congestion at the nearest station). Global State s t It integrates information from all vehicles and charging stations, including the status of all vehicles and the load of all stations.

[0081] State and Action Definition: At a given time step t, the action a of agent i is defined. i (t) is defined as the decision regarding the charging strategy, such as choosing which charging station to go to, the expected charging power, or the charging time. The combination of these actions forms a joint action a(t) = (a1(t), ..., a...). N (t)), the environment updates its state based on the joint action.

[0082] Design a reward function r based on the scheduling objective. i (t). Scheduling objectives include minimizing charging wait time, minimizing the total cost of electric vehicle driving and charging, and balancing the load at each station, etc. Therefore, the immediate reward for each agent can be designed as a negative cost, for example:

[0083] r i (t) = -(α × travel time + β × charging cost + γ × waiting time)

[0084] In the formula, α, β, and γ represent the weights of driving time, charging cost, and waiting time, respectively.

[0085] Or it could be based on satisfaction levels related to charging completion. The overall optimization objective is to maximize the cumulative reward for all agents or minimize the cumulative cost, i.e., to obtain the expected discount benefit. maximum;

[0086] Multi-agent deep deterministic policy gradient (MADDPG) belongs to the centralized training and distributed execution (CTDE) paradigm: during training, the Critic network can access the global state s of all agents. t And the joint action a(t), while the Actor network for each agent only depends on the local observations o of the corresponding agent. i (t); the policy of the i-th agent is determined by the Actor network μ. i (o i |θ i The value is indicated by the Critic network Q. i (s,a|φ i Evaluation; training utilizes experience replay and target network techniques: maintaining the Critic network parameters φ i and the corresponding target network φ′ i and the Actor network parameters θ i With the target network θ′i ;

[0087] The key formulas in the training process include: for a given empirical sample (s) t ,a(t),r i (t),s t+1 The target Q-value of the Critic network is calculated as follows:

[0088] y i (t)=r i (t)+γ′Q i′ (s t+1 ,μ 1′ (o1(t+1)),...,μ N′ (o N (t+1)))

[0089] Among them, y i (t) represents the target value of the i-th agent at time step t; Q i′ (t) represents the Q-value calculated by the target Critic network for the i-th agent; μ N′ Let represent the target policy network; γ′ is the discount factor; these networks calculate the next action from each agent's next observation o1(t+1), which is used as the input to the target Q network.

[0090] The training objective of Critic is to minimize the mean squared error loss:

[0091] L(φ i )=E[(Q i (s t ,a(t)|φ i )-y i (t)) 2 ]

[0092] In the formula, Q i (s t ,a(t)|φ i Indicates the global state s t When the action a(t) is taken, the parameter φ i The expected return of the decision; E[...] represents the expectation, which is the average over all possible outcomes;

[0093] When calculating gradient updates, the chain rule is used to output the corresponding action from the Actor, which drives the Critic to output a higher Q value.

[0094] The update objective of the Actor network is to maximize the Critic evaluation value, that is, to increase the expected return through the policy gradient. Its update direction is written as:

[0095]

[0096] Where J(θ) i ) represents the policy performance index function of the i-th agent, with the goal of maximizing this value, i.e. maximizing the long-term benefits brought by the policy; Es represents the expectation of state s; D represents the data distribution sampled from the experience pool, which is the distribution under the experience replay mechanism.

[0097] Adjust θ using a negative gradient, either ascending or descending. i It is important to note that Critic uses centralized input, which can utilize global information to mitigate non-stationarity, while Actor executes based only on local observations, facilitating the decentralized deployment of the system.

[0098] In the algorithm implementation, all agents share a unified experience replay pool and perform gradient updates in batches. The training algorithm for this decision module (Algorithm 1) is given below, demonstrating the core training process and update mechanism of the multi-agent MADDPG.

[0099]

[0100]

[0101] In the above algorithm, "centralized training and distributed execution" is reflected in Critic's use of the global state s t and the actions of all agents (a1(t),...,a) N (t) is used for training, while the Actor only needs to consider its own observations during execution. i This approach can significantly mitigate the environmental non-stationarity and dimensionality issues in multi-agent systems. Through this training process, the agents ultimately learn reasonable charging scheduling strategies for different environmental states, tending towards optimizing the overall objective (e.g., minimizing total charging time and cost).

[0102] Joint scheduling module

[0103] The joint scheduling module is responsible for integrating and executing the decisions output by each agent at the global level to generate a specific charging allocation scheme. After the multi-agent decision-making module outputs its actions, this module coordinates and optimizes the scheme based on system constraints and scheduling objectives to ensure that the final scheme is feasible and efficient. Its inputs include the decision intentions of all agents at the current time step and the real-time status of the charging stations. The module first collects the demand and target site information of each agent, and then performs load aggregation and constraint judgment on each charging station. For charging station j, it calculates the set Ij of vehicles planning to travel to charging station j and the corresponding total demand ∑i∈Ija. i Meanwhile, the total number of charging piles C at charging station j is also considered. j And the upper limit of charging rate.

[0104] If the total demand ∑i∈Ija i If the supply does not exceed the maximum feasible supply, it is allocated directly; otherwise, conflict resolution is required.

[0105] In the event of a conflict, minimize the additional waiting time for all vehicles or balance the load differences between stations to determine the final vehicle station allocation and charging start sequence;

[0106] The following decision variables and constraints are introduced for formalization:

[0107] Let x be a binary variable i,j =1 indicates that vehicle i is eventually dispatched to station j, then the goal is:

[0108]

[0109] Among them, c i,j The cost of vehicle i traveling to station j includes travel time and queuing time; constraints include... and This means that each vehicle is assigned only one station. This represents the charging station capacity constraint. Such integer programming problems can be solved approximately using heuristics or optimization algorithms. The joint scheduling module, upon output, simultaneously generates charging station allocation and start-up time schemes for each vehicle, and updates the status of both vehicles and stations.

[0110] The following is the algorithm (Algorithm 2) of the joint scheduling module, demonstrating its basic scheduling logic and constraint handling steps:

[0111]

[0112] Through the joint scheduling module, the suggestions from the aforementioned agents are integrated into the final charging scheduling scheme, achieving system-level consistency and constraint satisfaction. This module not only ensures the executability of multi-agent decisions but also further improves overall efficiency through optimization and adjustments. After the joint operation is completed, the system will enter the next time step, where the environment modeling module updates the state (including vehicle battery depletion and changes in charging station queues) and continues the new iteration.

[0113] S3. Input the information collected in step S1 into the scheduling system model constructed in step S2 to generate a specific charging allocation scheme.

[0114] like Figure 1 As shown, this invention addresses the scenario of electric vehicle charging scheduling on highways and proposes a collaborative decision-making architecture that integrates graph neural networks (GCN) and multi-agent reinforcement learning (MADDPG). The overall process is divided into five levels, achieving closed-loop control of "perception → decision → execution → evaluation" layer by layer.

[0115] (1) Data input layer: The system integrates the road network topology, the real-time operating status of charging stations (queueing, SOC level, charging power, etc.), and the historical charging data of electric vehicles, and constructs a graph structure between stations as multi-source heterogeneous input data to provide basic support for subsequent modeling.

[0116] (2) Environment perception layer: The nodes are embedded through a three-layer graph convolutional neural network (GCN) to extract features such as site location, load level and power status, so as to realize the fusion representation of structural information and attribute information and enhance the robustness of downstream decision-making.

[0117] (3) Decision Core Layer: The MADDPG algorithm is introduced to model the distributed optimal charging decision strategy for multiple vehicles under multi-site conditions in the form of Actor-Critic dual-network collaborative learning. The system constructs a joint reward function, which comprehensively considers queuing time (T), resource utilization (C), and guidance distance (W) to achieve multi-objective joint optimization.

[0118] (4) Scheduling output layer: The system outputs guidance schemes based on the strategy network and generates site load balancing heat maps and guidance path maps, which are used to schedule vehicles to the optimal charging station in real time and improve the overall operating efficiency of the charging station.

[0119] I. Experimental Setup

[0120] Dataset

[0121] The dataset used in this invention's experiments comes from real highway charging order logs, covering electric vehicle charging behavior records across multiple stations and time periods. The raw data includes key fields such as charging order number, station, order initiation time, queuing time, charging start / end time, and charging amount.

[0122] To establish structured analysis indicators, we standardized the data and parsed the fields, extracting the following statistical feature fields for modeling:

[0123] record_id: Unique identifier for the order

[0124] station_id: Charging station number

[0125] prepared_time: Time when vehicles begin queuing

[0126] start_time / end_time: Charging start and end times

[0127] charged_power: Actual charging capacity (unit: kWh)

[0128] From the valid orders after cleaning, we extracted the following key statistical indicators:

[0129] Table 1 Statistical Indicators of Dataset

[0130]

[0131]

[0132] Based on this order set, we further constructed input variables such as the spatial graph structure between stations, the queuing state sequence of each station, and the historical order behavior embedding vector, to support the multi-agent collaborative scheduling modeling of subsequent models.

[0133] Contrast Model

[0134] To verify the performance advantages of the proposed multi-agent reinforcement learning scheduling model (MADCG), we designed the following comparative model:

[0135] 1. Rule-Distance: Guides vehicles to the nearest station, regardless of queuing or battery level;

[0136] 2. Rule-Load: Select the site with the shortest current waiting queue;

[0137] 3. DDPG: Single-agent deep deterministic policy gradient algorithm, which optimizes single-vehicle scheduling based on global state;

[0138] 4. MADDPG: A multi-agent version of DDPG, where each agent makes distributed decisions based on its own and its neighborhood's state;

[0139] 5. AMAAC: An improved multi-agent attention-based Actor-Critic network, optimized solely based on site load;

[0140] 6. The MADCG model of this invention: jointly considers factors such as charging pile availability, future load prediction, and historical behavior modeling, and dynamically adjusts vehicle flow through a joint strategy network and scheduling center.

[0141] Model settings

[0142] All reinforcement learning models employ a centralized training, distributed execution (CTDE) architecture. We use the Adam optimizer for training, with the following hyperparameters:

[0143] Table 2 Hyperparameters of the MADCG model

[0144]

[0145] Evaluation indicators

[0146] We used the following metrics to evaluate the model:

[0147] 1. Average Waiting Time

[0148] Definition: The average time taken for an electric vehicle to go from arriving at a charging station to actually starting to charge, measured in minutes.

[0149] Calculation method:

[0150]

[0151] Where N is the number of valid orders. This represents the arrival time of the i-th order. This indicates the actual start time of charging.

[0152] Significance: This metric measures user experience. The smaller the value, the better the scheduling strategy can reduce queuing and improve service efficiency.

[0153] 2. Average Charging Completion Rate

[0154] Definition: The percentage of orders that have actually completed charging out of all orders submitted with the intention to charge, expressed as a percentage.

[0155] Calculation method:

[0156]

[0157] Where, N finish ed N is the number of orders that have been completed for charging. total It is the total number of all orders (including those that failed to complete due to queuing failures, insufficient resources, etc.).

[0158] Significance: Reflects the system's ability to cope with resource shortages during peak periods and holidays. A higher value indicates a more reliable scheduling strategy and a more stable system.

[0159] 3. System Throughput Capacity

[0160] Definition: The total number of charging orders completed by the system per unit of time, usually measured in "orders per day".

[0161] Calculation method:

[0162]

[0163] Significance: This indicator measures the overall processing capacity of the system. The higher the value, the higher the scheduling efficiency and the stronger the service capability of the scheduling system under the existing resources.

[0164] 4. Average Resource Utilization Rate

[0165] Definition: The average utilization efficiency of all charging piles in the system, reflecting the proportion of actual usage time to available time, expressed as a percentage.

[0166] Calculation method:

[0167]

[0168] Where M is the total number of charging stations. It is the actual charging time of the j-th pile. This is the total available service time.

[0169] Significance: It reflects the efficiency of resource scheduling. The higher the value, the more fully the system schedules charging equipment and the less wasted resources.

[0170] II. Analysis of Experimental Results

[0171] To verify the effectiveness of the proposed joint multi-agent charging scheduling strategy based on a centralized training-distributed execution framework, this paper compares it with five representative existing methods: rule-based nearest distance, rule-cost guided charging, single-agent reinforcement learning (DDPG), multi-agent reinforcement learning (MADDPG), and attention-based multi-agent policy (AMAAC). Evaluation metrics include average waiting time, average charging completion rate, system throughput, and resource utilization.

[0172] As can be seen from the table, the joint scheduling MARL strategy proposed in this invention achieves optimal or near-optimal results across all metrics. Specifically:

[0173] (1) In terms of average waiting time, this method is only 8.75 minutes, which is 27.8% lower than MADDPG (12.12 minutes) and more than 56% lower than the traditional Rule-Distance strategy (20.35 minutes), significantly alleviating the charging queuing problem during holidays and peak periods.

[0174] (2) The charging completion rate reached 93.6%, which is higher than AMAAC (91.2%) and DDPG (87.6%), indicating that it has a stronger scheduling adaptability in the face of order congestion and resource competition.

[0175] (3) The overall system throughput is 4785 units / day, which is about 16.4% higher than the baseline method.

[0176] (4) In terms of resource utilization, this scheme achieves 82.1%, which is significantly higher than the 75.5% of the single agent DDPG, fully demonstrating the advantages of the joint strategy in load balancing and dynamic resource adjustment.

[0177] This experiment fully verifies the comprehensive performance advantages of the present invention under multiple objectives. The improvement is mainly attributed to the joint modeling of the spatiotemporal states of each charging station, the introduction of a global graph structure, and the construction of a multi-agent collaborative scheduling mechanism. Furthermore, the generalization ability of the policy is further enhanced by mastering global information during the centralized training phase.

[0178] Table 3 Performance comparison of MADCG and baseline methods on the dataset.

[0179]

[0180] III. Effectiveness Analysis of Method Components

[0181] To further verify the contribution of each module design to the overall performance of this invention, four ablation experiments were designed, in which key components of the system were removed sequentially for comparative evaluation:

[0182] Remove the joint scheduling module (MADCG-noCoordination): Multiple agents are optimized independently without global scheduling;

[0183] Remove graph embedding (MADCG-noGraphEmb): Do not use graph structure information between charging stations;

[0184] (MADCG-noRewardOpt): Uses a simple negative queue length as the reward function, ignoring global completion rate / resource utilization optimization objectives.

[0185] Table 4 shows the ablation experimental results of the MADCG model on the dataset.

[0186]

[0187] The experimental results show that:

[0188] After removing the joint scheduling module, the average waiting time increased to 12.44 minutes, the completion rate decreased to 88.5%, and the throughput decreased by 14.7%, indicating that the global coordination mechanism is the key to performance improvement.

[0189] After removing the graph embedding, the waiting time and resource utilization decreased by 16.4% and 9.2% respectively, indicating that the topology can effectively characterize the traffic interference relationship between sites.

[0190] Without optimizing the reward function, the multi-agent objective deviates from the overall objective, resulting in deterioration across all metrics and a 11.3% decrease in overall system efficiency.

[0191] In summary, each module in this invention plays a crucial role in performance improvement. The joint scheduling module ensures global coordination, graph embedding and order demand features enhance state awareness, and the optimized reward design achieves a dynamic balance between system resource utilization and service quality. Ablation experiments further demonstrate that there is a synergistic gain effect among the modules, jointly constructing an efficient and robust charging guidance strategy system.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A method for intelligent guidance of highway charging stations based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Collect data on the highway network, the geographical location and capacity information of charging stations, and dynamic information on vehicles based on real order data; S2. Construct a scheduling system model, which includes an environment modeling and structure perception module, a multi-agent decision-making module, and a joint scheduling module; Environment Modeling and Structure Awareness Module: Responsible for capturing the topological features and real-time status information of highway networks and charging infrastructure, and encoding complex relationships through graph structure embedding; Multi-agent decision-making module: Based on the environmental information obtained by the environmental modeling and structure perception module, it uses a multi-agent deep reinforcement learning algorithm to learn policies and generate policies for each local decision under a centralized training and decentralized execution framework. Joint scheduling module: integrates the actions output by each agent, combines system constraints and objectives to perform global allocation and optimization, so as to ensure that scheduling decisions are feasible and efficient; S3. Input the information collected in step S1 into the scheduling system model constructed in step S2 to generate a specific charging allocation scheme.

2. The intelligent guidance method for highway charging stations based on multi-agent reinforcement learning according to claim 1, characterized in that: The charging station capacity information includes the current load of each charging station, the number of remaining charging piles, and the available power.

3. The intelligent guidance method for highway charging stations based on multi-agent reinforcement learning according to claim 2, characterized in that: The dynamic information of the vehicles, derived from real order data, includes the location, driving trajectory, destination arrival time, and SOC status of each electric vehicle.

4. The intelligent guidance method for highway charging stations based on multi-agent reinforcement learning according to claim 3, characterized in that, In step S2, the design of the environment modeling and structure perception module is as follows: I. Introduce structure perception to form a graph G = (V, E) consisting of road network and charging stations; Wherein, node V includes charging stations and vehicle units; edge E represents road connectivity or accessibility between vehicles and stations; the feature vector of charging station nodes includes real-time queue length, charging rate limit, and historical demand statistics; the feature vector of vehicle nodes includes current SOC, remaining driving range, destination, and highway congestion information. II. Graph neural networks (GNNs) are used to embed the graph G = (V, E) to encode node features and neighborhood structure in a comprehensive manner in order to obtain a perception of global relationships. III. A graph convolutional network (GCN) model is adopted to update node representations using an iterative message-passing method; For node i, the embedding at layer l+1 is represented as: In the formula, N(i) represents the set of neighbors connected to node i; W,b are the learnable weights and biases; σ is the activation function; The embedding of neighbor node j at layer l; IV. Output an environmental state representation for use by subsequent modules; The environment state representation includes the global state vector s t and the local observations required by each agent. i (t).

5. The intelligent guidance method for highway charging stations based on multi-agent reinforcement learning according to claim 4, characterized in that, In step S2, the multi-agent decision-making module is designed as follows: Policy learning is performed using the reinforcement learning algorithm of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) to achieve intelligent decision-making for charging scheduling. I. Define states and actions; Assuming there are N agents in the system, given a time step t, what action a is performed by agent i? i (t) is defined as the decision on the charging strategy; the combination of actions forms a joint action a(t) = (a1(t), ..., a N (t)), the environment updates its state based on the joint action; II. Design the reward function r based on the scheduling objective. i (t); The scheduling objectives include minimizing charging wait time, minimizing the total cost of electric vehicle driving and charging, and balancing the load of each station; r i (t) = -(α × travel time + β × charging cost + γ × waiting time) In the formula, α, β, and γ represent the weights of driving time, charging cost, and waiting time, respectively. The overall optimization objective is to maximize the cumulative reward for all agents or minimize the cumulative cost, i.e., to obtain the expected discount benefit. maximum; Multi-agent deep deterministic policy gradient (MADDPG) belongs to the centralized training and distributed execution (CTDE) paradigm: during training, the Critic network can access the global state s of all agents. t And the joint action a(t), while the Actor network for each agent only depends on the local observations o of the corresponding agent. i (t); the policy of the i-th agent is determined by the Actor network μ. i (o i |θ i The value is indicated by the Critic network Q. i (s,a|φ i Evaluation; training utilizes experience replay and target network techniques: maintaining the Critic network parameters φ i and the corresponding target network φ′ i and the Actor network parameters θ i With the target network θ′ i ; For a given empirical sample (s) t ,a(t),r i (t),s t+1 The target Q-value of the Critic network is calculated as follows: y i (t)=r i (t)+γ′Q i′ (s t+1 ,μ 1′ (o1(t+1)),...,μ N′ (o N (t+1))) Among them, y i (t) represents the target value of the i-th agent at time step t; Q i′ (t) represents the Q-value calculated by the target Critic network for the i-th agent; μ N′ Represents the target policy network; γ′ is the discount factor; The training objective of Critic is to minimize the mean squared error loss: L(φ i )=E[(Q i (s t ,a(t)|φ i )-y i (t)) 2 ] In the formula, Q i (s t ,a(t)|φ i Indicates the global state s t When the action a(t) is taken, the parameter φ i The expected return of the decision; E[...] represents the expectation, which is the average over all possible outcomes; When calculating gradient updates, the chain rule is used to output the corresponding action from the Actor, which drives the Critic to output a higher Q value. The update objective of the Actor network is to maximize the Critic evaluation value, that is, to increase the expected return through the policy gradient. Its update direction is written as: Wherein, J(θ) i Let represent the policy performance index function of the i-th agent; Es represent the expectation of state s; D represent the data distribution sampled from the experience pool. Adjust θ using a negative gradient, either ascending or descending. i ; In the algorithm implementation, all agents share a unified experience replay pool and perform gradient updates in batches.

6. The intelligent guidance method for highway charging stations based on multi-agent reinforcement learning according to claim 5, characterized in that, In step S2, the joint scheduling module is designed as follows: The system takes the decision intentions of all agents at the current time step and the real-time status of the charging stations as inputs, and performs load aggregation and constraint judgment on each charging station. For charging station j, calculate the set Ij of vehicles planning to travel to charging station j and the corresponding total demand ∑i∈Ija. i Meanwhile, the total number of charging piles C at charging station j is also considered. j and the maximum charging rate; If the total demand ∑i∈Ija i If the supply does not exceed the maximum feasible supply, it is allocated directly; otherwise, conflict resolution is required. In the event of a conflict, minimize the additional waiting time for all vehicles or balance the load differences between stations to determine the final vehicle station allocation and charging start sequence; The following decision variables and constraints are introduced for formalization: Let x be a binary variable i,j =1 indicates that vehicle i is eventually dispatched to station j, then the goal is: Among them, c i,j The cost of vehicle i traveling to station j includes travel time and queuing time; constraints include... and This means that each vehicle is assigned only one station. This indicates the capacity constraint of the charging piles at the site.

Images