Electric vehicle charging scheduling method based on average field multi-agent reinforcement learning

Abstract

The invention discloses an electric vehicle charging scheduling method based on mean field multi-agent reinforcement learning. Charging scheduling is carried out on a mobile charging station and an electric vehicle in a large-scale mobile charging scene at the same time. A charging scheduling problem is modeled as a partially observable Markov decision process, a mobile charging station and an electric vehicle serve as two different intelligent agents, a reinforcement learning scheduling network is trained respectively to recommend a scheduling position, and meanwhile, the average field theory is utilized to simplify the mutual influence between the intelligent agents; the trained scheduling network is distributed to the mobile charging station and the electric vehicle, so that the mobile charging station and the electric vehicle can quickly make a scheduling decision only by obtaining local observation, and complete distributed charging scheduling is realized. According to the scheduling method, the dynamically changing charging demand is met, the charging income of the mobile charging station is obviously increased, and the successful charging proportion of the electric vehicle is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of the Internet of Things for electric vehicles, and relates to an electric vehicle charging scheduling method based on mean-field multi-agent reinforcement learning. Background Art

[0002] At present, fixed charging stations are the main infrastructure for electric vehicle charging. However, problems such as uneven distribution of fixed charging stations and long charging queue times result in some electric vehicles being unable to charge in time. Mobile charging stations can dynamically optimize the distribution of charging resources by adopting reasonable scheduling strategies and provide more flexible and convenient charging services for electric vehicles.

[0003] Currently, the scheduling methods for mobile charging stations mainly face two major challenges: one is to balance charging supply and demand, and the other is to reasonably plan the scheduling routes of mobile charging stations. However, traditional scheduling methods usually cannot take both of the above problems into account, and most methods perform centralized scheduling, making it difficult to cope with dynamically changing charging demands and complex resource allocation problems. Multi-agent deep reinforcement learning has been widely applied in multiple fields due to its flexibility and high scalability in complex tasks and environments, and is also very suitable for large-scale mobile charging scheduling scenarios.

[0004] The present invention focuses on idle mobile charging stations and electric vehicles with remaining battery power insufficient to reach the travel destination and needing to increase the cruising range through charging. According to the different remaining battery powers, such electric vehicles can be divided into two categories: when the remaining battery power is higher than the low battery threshold, they are called quasi-charging electric vehicles; when the remaining battery power is lower than the low battery threshold, they are called to-be-charged electric vehicles. Through the scheduling method, idle mobile charging stations are made to actively track quasi-charging electric vehicles. Once a quasi-charging electric vehicle turns into a to-be-charged electric vehicle and issues a charging request, charging services can be provided in a timely manner. In addition, suitable routes should be recommended for to-be-charged electric vehicles to guide them to find busy mobile charging stations and wait for possible future charging opportunities. Summary of the Invention

[0005] The present invention uses a mean-field based multi-agent reinforcement learning method to perform fully distributed charging scheduling for mobile charging stations and electric vehicles, aiming to increase the charging revenue of mobile charging stations and improve the successful charging ratio of electric vehicles.

[0006] The present invention first models the charging scheduling problems of mobile charging stations and electric vehicles and designs the communication and scheduling processes of mobile charging stations and electric vehicles; then, mobile charging stations and electric vehicles are regarded as different agents, and the mean-field theory is used to handle the complex competition and cooperation relationships between agents; the agent scheduling network is trained through a multi-agent reinforcement learning framework; finally, the trained scheduling network is distributed to mobile charging stations and electric vehicles, enabling them to quickly make scheduling decisions only by obtaining local observations.

[0007] To achieve the above object, the technical solution of the present invention is as follows: An electric vehicle charging scheduling method based on mean-field multi-agent reinforcement learning, the method comprising the following steps,

[0008] Step 1: Demand modeling, clarifying that the objective function is to maximize the average charging revenue of mobile charging stations and the proportion of successfully charged electric vehicles,

[0009] Step 2: Apply mean-field theory to handle the cooperation and competition relationships between mobile charging stations and electric vehicles, and design action-observation vectors and reward functions,

[0010] Step 3: Construct a multi-agent reinforcement learning model, and train the scheduling networks for two types of agents, namely mobile charging stations and electric vehicles, respectively,

[0011] Step 4: Design the communication and scheduling process, deploy the trained scheduling networks to mobile charging stations and electric vehicles, and achieve fully distributed scheduling.

[0012] Among them, Step 1: Demand modeling, clarifying that the objective function is to maximize the average charging revenue of mobile charging stations and the proportion of successfully charged electric vehicles, specifically as follows,

[0013] The state changes of mobile charging stations and electric vehicles include the process from a quasi-charging electric vehicle detecting a low battery state until the completion of the charging task, and the process from an idle mobile charging station discovering a to-be-charged electric vehicle to the completion of the charging task. In addition, the arrowheads between the two state transition lines show the interaction process of an idle mobile charging station and a to-be-charged electric vehicle reaching a charging agreement. In the scheduling problem, it is assumed that each mobile charging station can only charge one electric vehicle at a time.

[0014] The main scheduling objective of the present invention is to improve the average charging revenue of mobile charging stations and the proportion of successfully charged electric vehicles. The expression of the objective function is as follows:

[0015]

[0016] where φ j is a mobile charging station, is the set of mobile charging stations, N c and N e represent the number of successfully charged electric vehicles and the total number of electric vehicles respectively. The above formula means to maximize the sum of the charging revenues Rev(φ j ) of each mobile charging station from time t = 0 to T, and to maximize the proportion of successfully charged electric vehicles.

[0017] This scheduling problem can be constructed as a distributed partially observable Markov decision process, represented as a seven-tuple containing agents state Local Observation Action State Transition Probability Function Reward Function and the discount factor γ,

[0018] (1) Agent In the mobile charging scenario, the agent includes all mobile charging stations and all electric vehicles. The sets of electric vehicles and mobile charging stations are represented as {v1, …, v n} and {φ1, …, φ M}, where n and M represent the total numbers of electric vehicles and mobile charging stations respectively,

[0019] (2) State The state at the t-th time slot is represented by which is composed of real-time states such as the road network structure, the locations and charging tasks of mobile charging stations, the locations and remaining battery levels of electric vehicles,

[0020] (3) Local Observation Agent g k The local observation of agent g at the t-th time slot is represented by which is a subset of s (t) . The local observation range of each agent is equal to the communication range R c .

[0021] (4) Action At the t-th time slot, the action selected by each agent (e.g., g k ) is represented by The action space (possible actions) of agent g k is denoted as

[0022] (5) State Transition Probability Function At the t-th time slot, each agent selects an action in state s (t) , which causes the environment to transition from state s (t) to a new state s (t+1) . This transition follows the state transition probability function

[0023] (6) Reward Function At the end of the t-th time slot, agent g k will obtain a reward The goal of each agent is to maximize the cumulative discounted reward where T represents the number of time slots in one round of training.

[0024] Multi-agent deep reinforcement learning is adopted to solve the scheduling problem of mobile charging stations and electric vehicles, and the scheduling process is constructed as a sequential decision-making process, that is, a scheduling route is generated through multi-step decisions (each scheduling route consists of the optimal scheduling points in the multi-step decisions). It effectively addresses the problem that the vehicle-to-vehicle communication range is limited and each agent can only obtain the information in its local observation (within its communication range).

[0025] Among them, step 2: Apply the mean field theory to handle the cooperation and competition relationship between mobile charging stations and electric vehicles, and design the action-observation vector and reward function, specifically as follows.

[0026] Each agent obtaining the joint observation can enable the agent to have a more comprehensive and accurate understanding of the environment and make more reasonable action decisions. For agent g k , its optimal Q function based on the joint observation o (t) and the joint action a (t) is expressed as:[[]]END]]

[0027]

[0028] Due to the model-free property, the calculation of the expectation in Equation (2) is a challenging problem. Therefore, can be approximated as a Q network Q k (o (t) , a (t) |θ), where θ represents the network parameters.

[0029] Based on the mean field multi-agent deep reinforcement learning, the use of the local interaction between each agent and its neighboring agents approximates the effect of global interaction, reducing the computational complexity. It can be deduced from the mean field theory that:

[0030]

[0031] Therefore, we designed the observation-action pair to implement the mean field-based multi-agent deep reinforcement learning method. For idle mobile charging stations and electric vehicles to be charged, there are different feature representations of the observation-action pair. For the electric vehicle v i , the set consisting of the available scheduling points within the communication range has:

[0032]

[0033] d(v i , a) is the distance to the scheduling point of v i , and fe M (v i ) represents the characteristics of the busy mobile charging stations near v i , and feN (v i ) represents v i the characteristics of the electric vehicle to be charged nearby. Therefore, the agent g k the set of observation-action pairs at the t-th time slot For the idle mobile charging station φ j , there is:

[0034]

[0035] d(φ j , a) is the distance to the scheduling point, fm M (φ j ) is the characteristic of the idle mobile charging stations near φ j , fe N (φ j ) represents the characteristics of the electric vehicles about to be charged near φ j .

[0036] Therefore

[0037] To make the method achieve better scheduling results, the reward function in multi-agent reinforcement learning directly guides the idle mobile charging stations to actively track the electric vehicles about to be charged, and the electric vehicles to be charged to actively approach the busy mobile charging stations. For the two types of different agents, the reward function is set as follows. When the agent g k is an electric vehicle v i to be charged:

[0038] s

[0039] That is, when the remaining power of φ k near the agent g j is more and the distance to the charging position of φ j is closer, or when the distance between g k and v i′ with less power shortage is farther, a greater reward will be obtained; when the agent g k is an idle mobile charging station φ j :

[0040]

[0041] It means that if the idle mobile charging station is scheduled to a position where the electric vehicles about to be charged around have a large power shortage and a large number, and the busy mobile charging stations have less remaining power and a small number, a greater reward can be obtained.

[0042] The mean - field - based method significantly reduces the training complexity by obtaining the action intentions of neighboring agents and calculating the "average" influence of neighboring agents at available scheduling points. The design of the reward function can directly guide the idle mobile charging stations to actively track the quasi - charging electric vehicles, and the electric vehicles waiting to be charged to actively approach the busy mobile charging stations. This method will effectively improve the successful charging ratio of electric vehicles and the average charging revenue of mobile charging stations. Among them, step 3: Construct a multi - agent reinforcement learning model. Two types of agents, namely mobile charging stations and electric vehicles, train the scheduling network respectively, as follows,

[0043] The training model of the present invention is improved based on the DQN algorithm to adapt to the multi - agent environment. The multi - agent reinforcement learning model is as follows. Two types of agents, namely mobile charging stations and electric vehicles, train the Q - networks Q Φ (; θ Φ ) and Q Ψ (; θ Ψ ). To enhance the training stability of Q k (o (t) , a (t) |θ), a Q - target network is added, and the parameters of the target network (denoted by θ′) are copied from θ regularly (every κ time slots). The two target networks are denoted as Q′ Φ (; θ′ Φ ) and Q′ Ψ (; θ′ Ψ ),

[0044] At the t - th time slot, each agent g k will obtain the set of observation - action pairs Then, the action is determined by the ε - greedy policy:

[0045]

[0046] where ε represents the probability of taking a random action. A random action means randomly selecting a position from the available scheduling points, and 1 - ε represents the probability that the selected action maximizes .

[0047] When the action is executed, the environment will be updated, that is, the future set of observation - action pairs and the reward will be returned by the environment at the (t + 1) - th time slot. The target Q - value is expressed as:

[0048]

[0049] where θ′ Φ is the target Q - network Q′ Φparameters. The loss function is defined as:

[0050]

[0051] Using the temporal difference method, at each time slot, the Q-network can be updated according to Equation (10) so that the current Q value approaches the target Q value.

[0052] In the mobile charging scenario, both mobile charging stations and electric vehicles on the road network are regarded as agents. Since the number of agents on the road network is usually large and may change frequently, sharing the same Q-network parameters by the same type of agents reduces the computational overhead. In addition, by adding a communication mechanism based on the DQN algorithm, agents can obtain the state information of other surrounding agents to adapt to the multi-agent environment and make more reasonable action decisions.

[0053] Among them, Step 4: Design the communication and scheduling process, deploy the trained scheduling network to mobile charging stations and electric vehicles to achieve fully distributed scheduling, specifically as follows:

[0054] A. Scheduling of electric vehicles to be charged

[0055] Phase A.1: Each electric vehicle to be charged (e.g., v i ) broadcasts a charging request request_msg containing the next scheduling point and the remaining power shortage of v c within the communication range R i in each time slot,

[0056] Phase A.2.1: After receiving the request message of v i , each nearby idle mobile charging station (e.g., φ j ) will reply an approval_msg to v i , and the message contains the next scheduling position and the remaining power of φ j . Similarly, each busy mobile charging station will reply a status_msg, and the message contains the location of the ongoing charging task and the remaining power after the task is completed.

[0057] Phase A.2.2: Suppose v i receives the approval_msg from nearby idle mobile charging stations, and then v i selects the optimal idle mobile charging station and charging location through the shortest detour distance, requiring that the remaining power of v i is sufficient to support it to travel to the charging location, and the remaining power of the idle mobile charging station is sufficient to support it to travel to the charging location and complete the charging task of v i , and then vi Send an info_msg containing the charging location and detour distance to the selected idle mobile charging station and wait for the confirmation message ack_msg from the selected idle mobile charging station.

[0058] Phase A.2.3: If v i does not receive the ack_msg from any idle mobile charging station, which means that v i will still be a tram waiting to be charged during the t-th time slot, and a new scheduling point should be determined for v i by the dispatching network of electric vehicles;

[0059] B. Scheduling of idle mobile charging stations

[0060] Phase B.1: For each idle mobile charging station (assuming φ j ), if φ j has received the info_msg from one or more nearby trams waiting to be charged, then it selects the tram waiting to be charged that can bring the maximum charging benefit. Then, φ j sends an ack_msg to the selected tram waiting to be charged to reach a charging agreement, and then both parties move towards the charging location and start charging.

[0061] Phase B.2: For the idle mobile charging station φ j , if it does not receive any info_msg from nearby trams waiting to be charged, which means that φ j will still be an idle mobile charging station during the t-th time slot, and a new scheduling point should be determined for φ j by the dispatching network of mobile charging stations.

[0062] The message interaction sequence of the scheduling method is as follows. The quasi-charging tram v1 broadcasts the demand_msg, and the two trams v2 and v3 waiting to be charged broadcast the request_msg. When v2 receives two approval_msgs from the idle mobile charging stations φ1 and φ2, v2 selects φ2 and reaches a charging agreement with φ2. Another idle mobile charging station φ1 (which does not reach any charging agreement during the current time slot) determines a new scheduling point through SA-MARL. For the tram v3 waiting to be charged, it does not receive any approval_msg after broadcasting the request_msg and will also use SA-MARL to determine a new scheduling point. In addition, the busy mobile charging station φ3 will reply with a status_msg after receiving the request_msg from v3.

[0063] The two types of Q - networks trained by the cloud server are respectively distributed to the mobile charging station and the electric vehicle for their use. After that, the mobile charging station and the electric vehicle can communicate with each other and make scheduling decisions independently and quickly, realizing a fully distributed scheduling.

[0064] An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it implements the electric vehicle charging scheduling method based on mean - field multi - agent reinforcement learning as described above.

[0065] A computer - readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, they implement the electric vehicle charging scheduling method based on mean - field multi - agent reinforcement learning as described above.

[0066] Compared with the prior art, the advantages of the present invention are as follows. By using the method based on mean - field multi - agent deep reinforcement learning, the agent can obtain the state information and action intentions of other agents within the communication range, enabling the agent to achieve global interaction within the limit of the V2V communication range. The communication data will be further processed as the basis for the scheduling decision network, effectively coordinating the cooperation and competition relationships among the agents. The multi - agent reinforcement learning scheduling model trained in the offline simulation environment is deployed to the mobile charging station and the electric vehicle, and the communication and charging matching process between the mobile charging station and the electric vehicle is designed, enabling the mobile charging station and the electric vehicle to make independent scheduling decisions, realizing a fully distributed scheduling, improving the scheduling efficiency, and making the scheduling method have good scalability. Description of the Drawings

[0067] Figure 1 It is a schematic diagram of the state change of the mobile charging station and the electric vehicle.

[0068] Figure 2 It is a schematic diagram of the multi - agent reinforcement learning model.

[0069] Figure 3 It is a schematic diagram of the information interaction between the mobile charging station and the electric vehicle. Detailed Embodiments

[0070] To deepen the understanding of the present invention, the electric vehicle charging scheduling method based on mean - field multi - agent reinforcement learning is described in detail as follows:

[0071] Example: The idle mobile charging station is set to actively track the quasi-charging electric vehicle, while the electric vehicle to be charged actively approaches the busy mobile charging station to wait for future charging opportunities. The main problem we focus on is how to schedule the idle mobile charging station and the electric vehicle to be charged using the mean-field multi-agent reinforcement learning algorithm. During the scheduling process, the following conditions need to be met: (1) The scheduling strategy should be able to respond quickly and adopt a fully distributed scheduling mode, that is, each agent can independently make decisions based on local information to avoid the computational bottleneck problem of the central node. (2) Although agents make independent decisions in the fully distributed scheduling mode, they still need to cooperate with other nearby agents to handle the cooperation and competition relationships between agents. (3) The scheduling strategy should not only fully consider the travel experience of electric vehicles, ensure the charging success rate of vehicles, shorten the charging waiting time, reduce the detour distance of electric vehicles for charging, but also improve the operating income of mobile charging stations.

[0072] The electric vehicle charging scheduling method based on mean-field multi-agent reinforcement learning includes the following steps:

[0073] 1. Demand modeling

[0074] The state changes of mobile charging stations and electric vehicles are as Figure 1 shown, which includes the process from the quasi-charging electric vehicle detecting a low battery state until the completion of the charging task, and the process from the idle mobile charging station discovering the electric vehicle to be charged until the completion of the charging task. In addition, the arrowheads between the two state transition lines show the interaction process of the idle mobile charging station and the electric vehicle to be charged reaching a charging agreement. In the scheduling problem, it is assumed that each mobile charging station can only charge one electric vehicle at a time.

[0075] The main scheduling objective of the present invention is to improve the average charging income of mobile charging stations and the proportion of successful charging of electric vehicles. The expression of the objective function is as follows:

[0076]

[0077] where φ j is the mobile charging station, is the set of mobile charging stations, N c and N e represent the number of successfully charged electric vehicles and the total number of electric vehicles respectively. The above formula means to maximize the sum of the charging incomes Rev(φ j ) of each mobile charging station from time t = 0 to T, and to maximize the proportion of successful charging of electric vehicles.

[0078] This scheduling problem can be constructed as a distributed partially observable Markov decision process, represented as a seven-tuple containing agents states Local Observation Action State Transition Probability Function Reward Function and the discount factor γ.

[0079] (1) Agent In the mobile charging scenario, the agent includes all mobile charging stations and all electric vehicles. The sets of electric vehicles and mobile charging stations are denoted as {v1, …, v n} and {φ1, …, φ M}, where n and M represent the total numbers of electric vehicles and mobile charging stations respectively.

[0080] (2) State The state at the t-th time slot is represented by and is composed of real-time states such as the road network structure, the positions and charging tasks of mobile charging stations, the positions and remaining battery levels of electric vehicles.

[0081] (3) Local Observation Agent g k The local observation of agent g at the t-th time slot is represented by which is a subset of s (t) The local observation range of each agent is equal to the communication range R c .

[0082] (4) Action At the t-th time slot, the action selected by each agent (e.g., g k ) is represented by The action space (possible actions) of agent g k is denoted as

[0083] (5) State Transition Probability Function At the t-th time slot, each agent selects an action in state s (t) which causes the environment to transition from state s (t) to a new state s (t+1) This transition follows the state transition probability function

[0084] (6) Reward Function (R): At the end of the t-th time slot, agent g k will receive a reward The goal of each agent is to maximize the cumulative discounted reward where T represents the number of time slots in an episode of training.

[0085] 2. Applying the mean field theory to handle the cooperation and competition relationships between mobile charging stations and electric vehicles

[0086] Each agent obtaining joint observations enables the agent to have a more comprehensive and accurate understanding of the environment and make more reasonable action decisions. For agent g k , based on the joint observation o (t) and the joint action a (t) , the optimal Q-function is expressed as:

[0087]

[0088] Due to the model-free property, the calculation of the expectation in Equation (2) is a challenging problem. Therefore, can be approximated by a Q-network Q k (o (t) , a (t) |θ), where θ represents the network parameters.

[0089] Based on mean-field multi-agent deep reinforcement learning, using the local interaction between each agent and its neighboring agents to approximate the effect of global interaction reduces the computational complexity. It can be deduced from mean-field theory that:

[0090]

[0091] Therefore, we designed the observation-action pair to implement the mean-field based multi-agent deep reinforcement learning method. There are different observation-action pair feature representations for the idle mobile charging station and the electric vehicle to be charged. For the electric vehicle v i , the set of available scheduling points within the communication range is [[ID=3,6]]There is:

[0092]

[0093] d(v i , a) is the distance to the scheduling point for v i , and fe M (v i ) represents the characteristics of the busy mobile charging stations near v i , and fe N (v i ) represents the characteristics of the electric vehicles to be charged near v i . Therefore, the observation-action pair set of agent g k at the t-th time slot is For the idle mobile charging station φ j , there is:

[0094]

[0095] d(φ j , a) is the distance to the scheduling point, and fmM (φ j ) is the characteristic of the nearby idle mobile charging station, fe j N (φ j ) represents φ j the characteristic of the nearby quasi-charging electric vehicle. Therefore

[0096] In order to make the method achieve better scheduling effect, the reward function in multi-agent reinforcement learning directly guides the idle mobile charging station to actively track the quasi-charging electric vehicle, and the electric vehicle to be charged actively approaches the busy mobile charging station. For the two types of different agents, the reward function is set as follows. When the agent g k is an electric vehicle v to be charged i :

[0097]

[0098] That is, when the agent g k has more remaining power nearby φ j and is closer to the φ charging position, or when g j is farther from v k with less power shortage, it will obtain a greater reward. When the agent g i′ is an idle mobile charging station φ k : j

[0099]

[0100] It means that if the idle mobile charging station is scheduled to a position where the nearby quasi-charging electric vehicles have large power shortage and large quantity, and the busy mobile charging stations have less remaining power and small quantity, a greater reward can be obtained.

[0101] 3. Construct a multi-agent reinforcement learning model

[0102] The training model of the present invention is improved on the basis of the DQN algorithm to adapt to the multi-agent environment. It should be noted that in the mobile charging scenario, both the mobile charging stations and electric vehicles on the road network are regarded as agents. Since the number of agents on the road network is usually large and may change frequently, sharing the same Q-network parameters by the same type of agents can reduce the computational overhead. In addition, by adding a communication mechanism on the basis of the DQN algorithm, the agents can obtain the state information and action intentions of other surrounding agents to adapt to the multi-agent environment and make more reasonable action decisions.

[0103] The multi-agent reinforcement learning model is as Figure 2 shown. The two types of agents, the mobile charging station and the electric vehicle, respectively train the Q-network Q Φ ​​(; θ Φ ) and Q Ψ (; θ Ψ ). To enhance the training stability of Q k (o (t) , a (t) |θ), a Q target network is added, and the parameters of the target network (denoted by θ′) are periodically copied from θ (every κ time slots), and the two target networks are respectively denoted as Q′ Φ (; θ′ Φ ) and Q′ Ψ (; θ′ Ψ ).

[0104] At the t-th time slot, each agent g k will obtain the set of observation-action pairs Then, the action is determined by the ε-greedy policy:

[0105]

[0106] where ε represents the probability of taking a random action, and a random action means randomly selecting a position from the available scheduling points, while 1 - ε represents the probability that the selected action makes maximized.

[0107] When the action is executed, the environment will be updated accordingly, that is, the set of future observation-action pairs and the reward will be returned by the environment at the t + 1-th time slot. Therefore, the target Q value can be expressed as:

[0108]

[0109] where θ′ Φ is the parameter of the target Q network Q′ Φ . The loss function can be defined as:

[0110]

[0111] Using the temporal difference method, the Q network can be updated according to Equation (3.10) at each time slot to make the current Q value approach the target Q value.

[0112] 4. Design the communication and scheduling process

[0113] A. Scheduling of the electric vehicles to be charged

[0114] Stage A.1: Each electric vehicle to be charged (e.g., v i ) will be within the communication range R at each time slotc The in - broadcast charging request request_msg contains v i 's next scheduling point and the remaining power shortage situation.

[0115] Phase A.2.1: After receiving the request message of v i , each nearby idle mobile charging station (e.g., φ j ) will send an approval_msg to v i . This message contains φ j 's next scheduling location and remaining power. Similarly, each busy mobile charging station will reply with a status_msg, which contains the location of the ongoing charging task and the remaining power after the task is completed.

[0116] Phase A.2.2: Assume that v i receives an approval_msg from a nearby idle mobile charging station. Then v i selects the optimal idle mobile charging station and charging location through the shortest detour distance, requiring that the remaining power of v i is sufficient to support it to travel to the charging location, and the remaining power of the idle mobile charging station is sufficient to support it to travel to the charging location and complete the charging task for v i . Then v i will send an info_msg containing the charging location and detour distance to the selected idle mobile charging station and wait for the confirmation message ack_msg from the selected idle mobile charging station.

[0117] Phase A.2.3: If v i does not receive an ack_msg from any idle mobile charging station, this means that v i will still be a tram to be charged in the t - th time slot, and a new scheduling point should be determined for v i by the scheduling network of electric vehicles.

[0118] B. Scheduling of idle mobile charging stations

[0119] Phase B.1: For each idle mobile charging station (assuming φ j ), if φ j has received info_msg from one or more nearby trams to be charged, then it will select the tram to be charged that can bring the greatest charging benefit. Then, φ j will send an ack_msg to the selected tram to be charged to reach a charging agreement, and then both parties will move towards the charging location and start charging.

[0120] Phase B.2: For the idle mobile charging station φj For φ, if it does not receive any info_msg from nearby electric vehicles to be charged, it means that in the t-th time slot j φ will still be an idle mobile charging station, and at this time, the scheduling network of the mobile charging station should determine a new scheduling point for φ. j Determine a new scheduling point.

[0121] The message interaction sequence diagram of the scheduling method in the present invention is as follows Figure 3 shown. Among them, the quasi-charging electric vehicle v1 broadcasts demand_msg, and the two electric vehicles v2 and v3 to be charged broadcast request_msg. When v2 receives two approval_msg from the idle mobile charging stations φ1 and φ2, v2 selects φ2 and reaches a charging agreement with φ2. Another idle mobile charging station φ1 (which does not reach any charging agreement in the current time slot) determines a new scheduling point through SA-MARL. For the electric vehicle v3 to be charged, after broadcasting request_msg, it does not receive any approval_msg and will also use SA-MARL to determine a new scheduling point. In addition, the busy mobile charging station φ3 will reply with a status_msg after receiving the request_msg of v3.

[0122] It should be noted that the above is only a preferred embodiment of the present invention and is not used to limit the protection scope of the present invention. For those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made on the basis of the above embodiments, and these improvements and refinements all fall within the protection scope of the claims of the present invention.

Claims

1. A method for electric vehicle charging scheduling based on mean - field multi - agent reinforcement learning, characterized in that, The method includes the following steps: Step 1: Requirement modeling, where the objective functions are to maximize the average charging revenue of mobile charging stations and the successful charging ratio of electric vehicles. Step 2: Apply the mean field theory to handle the cooperation and competition relationships between mobile charging stations and electric vehicles, and design the action-observation vector and reward function. Step 3: Construct a multi-agent reinforcement learning model, and the two types of agents, namely mobile charging stations and electric vehicles, respectively train the scheduling network. Step 4: Design the communication and scheduling process, deploy the trained scheduling network to mobile charging stations and electric vehicles, and achieve fully distributed scheduling.

2. A method for electric vehicle charging scheduling based on mean field multi-agent reinforcement learning according to claim 1, characterized in that: Step 1: Requirement modeling, where the objective functions are to maximize the average charging revenue of mobile charging stations and the successful charging ratio of electric vehicles, specifically as follows: The state changes of mobile charging stations and electric vehicles include the process from a quasi-charging electric vehicle detecting a low battery state until the charging task is completed, and the process from an idle mobile charging station discovering a to-be-charged electric vehicle to the completion of the charging task. In the scheduling problem, it is assumed that each mobile charging station can only charge one electric vehicle at a time. The scheduling goal is to improve the average charging revenue of mobile charging stations and the successful charging ratio of electric vehicles. The expression of the objective function is as follows: where φ j is a mobile charging station, is a set of mobile charging stations, N c and N e represent the number of electric vehicles successfully charged and the total number of electric vehicles respectively. The above formula means to maximize the sum of the charging revenues Rev(φ j ) of each mobile charging station from time t = 0 to T, and to maximize the proportion of electric vehicles successfully charged. The scheduling problem can be formulated as a distributed partially observable Markov decision process, represented as a seven-tuple containing agents states local observations actions a state transition probability function a reward function and a discount factor γ (1) Agent In the mobile charging scenario, the agent includes all mobile charging stations and all electric vehicles. The sets of electric vehicles and mobile charging stations are respectively denoted as {v1, …, v n} and {φ1, …, φ M}, where n and M respectively represent the total numbers of electric vehicles and mobile charging stations, (2) Status The status of the t-th time slot is represented by which is composed of real-time statuses such as road network structure, mobile charging station locations and charging tasks, electric vehicle locations and remaining battery levels. (3) Local Observation Agent g k The local observation at the t-th time slot is denoted by which is a subset of s (t) The local observation range of each agent is equal to the communication range R c , (4) Action At the t-th time slot, the action selected by each agent is denoted as Agent g k The action space of is denoted as (5) State transition probability function At the t-th time slot, each agent selects an action in state s (t) which causes the environment to transition from state s (t) to a new state s (t+1) , and this transition follows the state transition probability function (6) Reward function At the end of the t-th time slot, agent g k will receive a reward The goal of each agent is to maximize the cumulative discounted reward where T represents the number of time slots in one round of training.

3. A method for electric vehicle charging scheduling based on mean field multi-agent reinforcement learning according to claim 1, characterized in that: Step 2: Apply the mean field theory to handle the cooperation and competition relationships between mobile charging stations and electric vehicles, and design the action-observation vector and reward function, specifically as follows: For agent g k , its optimal Q-function based on the joint observation o (t) and the joint action a (t) is expressed as: Due to the model-free property, the calculation of the expectation in Equation (2) is a challenging problem, and is approximated as a Q-network Q k (o (t) ,a (t) |θ), where θ represents the network parameters, Based on mean field multi-agent deep reinforcement learning, the local interaction between each agent and neighboring agents is used to approximate the effect of global interaction, reducing the computational complexity. It can be deduced from the mean field theory that: Design observation-action pairs To implement the mean-field based multi-agent deep reinforcement learning method, different observation-action pair feature representations are used for idle mobile charging stations and electric vehicles to be charged. For the electric vehicle v i , the set of available scheduling points within the communication range is formed There are: d(v i , a) is the distance to the v i dispatching point, and fe M (v i ) represents the characteristics of the busy mobile charging stations near v i , and fe N (v i ) represents the characteristics of the electric trams waiting to be charged near v. Therefore, the agent g i in the set of observation-action pairs at the t-th time slot k . For the idle mobile charging station φ , there is: j , there is: d(φ j , a) is the distance to the scheduling point, fm M (φ j ) is the characteristic of the idle mobile charging station near φ j , fe N (φ j ) represents the characteristic of the quasi-charging tram near φ j ​ Therefore To make the method achieve better scheduling effects, the reward function in multi-agent reinforcement learning directly guides the idle mobile charging stations to actively track the quasi-charging electric vehicles, and the electric vehicles to be charged actively approach the busy mobile charging stations. For the two types of different agents, the reward function is set as follows. When agent g k is an electric vehicle v to be charged i : That is, when the agent g k near φ j has more remaining power and is closer to the charging location of φ j , or when g k is farther from v with a smaller power shortage i′ , a greater reward will be obtained; when the agent g k is an idle mobile charging station φ j : It means that if an idle mobile charging station is scheduled to a location where the surrounding quasi-charging electric vehicles have a large shortage of power and a large number, and the remaining power of busy mobile charging stations is small and the number is small, a greater reward can be obtained.

4. A method for electric vehicle charging scheduling based on mean-field multi-agent reinforcement learning according to claim 1, wherein Step 3: Construct a multi-agent reinforcement learning model, and the two types of agents, namely mobile charging stations and electric vehicles, respectively train the scheduling network, specifically as follows: The multi-agent reinforcement learning model is as follows. Two types of agents, the mobile charging station and the electric vehicle, respectively train Q networks Q Φ (; θ Φ ) and Q Ψ (; θ Ψ ). To enhance the training stability of Q k (o (t) , a (t) |θ), a Q target network is added, and the parameters of the target network (denoted by θ′) are periodically copied from θ (every κ time slots). The two target networks are respectively denoted as Q′ Φ (; θ′ Φ ) and Q′ Ψ (; θ′ Ψ ). At the t-th time slot, each agent g k obtains the set of observation-action pairs Then, the action is determined by the ε-greedy policy: Among them, ε represents the probability of taking a random action, where a random action means randomly selecting a position from the available scheduling points, and 1 - ε represents the probability that the selected action maximizes. After the action is executed, the environment will be updated accordingly, i.e., the future observation-action pair set and the reward will be returned by the environment at the (t + 1)-th time slot. The target Q-value is expressed as: where θ′ Φ are the parameters of the target Q-network Q′ Φ , and the loss function is defined as: Using the temporal difference method, the Q network can be updated according to Equation (10) at each time slot to make the current Q value approach the target Q value.

5. The electric vehicle charging scheduling method based on mean - field multi - agent reinforcement learning according to claim 1, characterized in that, Step 4: Design the communication and scheduling process, deploy the trained scheduling network to mobile charging stations and electric vehicles, and achieve fully distributed scheduling, specifically as follows: A. Scheduling of to-be-charged electric vehicles Phase A.1: Each electric vehicle to be charged broadcasts a charging request request_msg containing the next scheduling point of v c and the remaining power shortage situation within the communication range R in each time slot, i ​ Phase A.2.1: After receiving the request message of v i , each nearby idle mobile charging station will reply to v i with an approval_msg that contains the next scheduling position of φ j and the remaining power. Similarly, each busy mobile charging station will reply with a status_msg that contains the location of the ongoing charging task and the remaining power after the task is completed. Phase A.2.2: Set v i Received approval_msg from a nearby idle mobile charging station, then v i Select the optimal idle mobile charging station and charging location through the shortest detour distance, requiring that v i has enough remaining power to support its journey to the charging location, and the idle mobile charging station has enough remaining power to support its journey to the charging location and complete the charging task for v i , then v i will send an info_msg containing the charging location and detour distance to the selected idle mobile charging station and wait for the confirmation message ack_msg from the selected idle mobile charging station Phase A.2.3: If v i does not receive an ack_msg from any idle mobile charging station, which means that v i will still be a tram to be charged in the t-th time slot and a new scheduling point should be determined for v i by the scheduling network of electric vehicles; B. Scheduling of idle mobile charging stations Phase B.1: For each idle mobile charging station (let φ j ), if φ j has received an info_msg from one or more nearby electric vehicles to be charged, then it will select the electric vehicle to be charged that can bring the maximum charging benefit. Then, φ j will send an ack_msg to the selected electric vehicle to be charged to reach a charging agreement, and then both parties will move towards the charging location and start charging. Phase B.2: For the idle mobile charging station φ j if it does not receive any info_msg from nearby trams waiting to be charged, this means that φ j will still be an idle mobile charging station during the t-th time slot, and at this time, the scheduling network of the mobile charging station should determine j a new scheduling point for φ The message interaction sequence of the scheduling method is as follows. Among them, the quasi-charging electric vehicle v1 broadcasts demand_msg, and the two electric vehicles to be charged, v2 and v3, broadcast request_msg. When v2 receives two approval_msg from the idle mobile charging stations φ1 and φ2, v2 selects φ2 and reaches a charging agreement with φ2. The other idle mobile charging station φ1 (which does not reach any charging agreement in the current time slot) determines a new scheduling point through SA-MARL. For the electric vehicle v3 to be charged, it does not receive any approval_msg after broadcasting request_msg and will also use SA-MARL to determine a new scheduling point. In addition, the busy mobile charging station φ3 will reply with a status_msg after receiving the request_msg of v3. The two types of Q networks trained by the cloud server are respectively distributed to the mobile charging stations and electric vehicles for their use. After that, the mobile charging stations and electric vehicles can communicate with each other and make scheduling decisions independently and quickly, realizing a fully distributed scheduling.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the electric vehicle charging scheduling method based on mean field multi-agent reinforcement learning as described in any one of claims 1 to 5 above.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the computer instruction is executed by the processor, it implements the electric vehicle charging scheduling method based on mean field multi-agent reinforcement learning as described in any one of claims 1-5.

Images