Distributed charging station scheduling method and device based on multi-agent reinforcement learning

Abstract

The invention discloses a distributed charging station scheduling method and device based on multi-agent reinforcement learning, and the method comprises the steps: distributing a charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; and a recommended charging station corresponding to the charging request is determined based on all the received charging scores, and the recommended charging station is fed back to the electric vehicle. According to the method, the charging stations deployed with the reinforcement learning strategy network serve as the distributed nodes, the charging scores of completing the charging request are determined by deploying the reinforcement learning strategy network, and the electric vehicles are scheduled based on the charging scores, so that charging demand fluctuation in different regions and time periods can be flexibly coped with, and the charging efficiency is improved. The risk of overload of a power grid is reduced, and the overall charging efficiency is improved. And meanwhile, the deployment of the reinforcement learning strategy network can be finely adjusted based on scheduling sample data formed by historical charging requests, so that the model performance of the deployment of the reinforcement learning strategy network is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of electric vehicle charging, and in particular to a distributed charging station scheduling method and equipment based on multi-agent reinforcement learning. Background Art

[0002] The widespread popularity of electric vehicles has led to a rapid increase in demand for charging infrastructure, but charging demand has become diversified. Statistics show that the average number of charging times on weekdays is more than 1.5 times higher than on holidays. There are significant differences in the distribution of charging hotspots during the day and at night, and the charging demand of different types of passenger cars is concentrated in different periods. However, the distribution of fixed charging stations is limited and fixed, and cannot flexibly respond to charging demand during peak hours or in specific areas. Therefore, how to effectively dispatch electric vehicle charging stations has become a key issue to improve charging efficiency and alleviate grid load.

[0003] The traditional charging scheduling method for electric vehicles mainly relies on the car owners to independently select charging stations, usually based on map information to go to the nearest or most idle charging station for charging. It is difficult to foresee the impact of future traffic conditions, the length of queues at charging stations, etc., resulting in a large number of electric vehicles concentrated near certain charging stations during peak hours, putting pressure on the stability of the power grid. On the one hand, this will affect the charging efficiency, and on the other hand, it may also increase the charging costs of car owners.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by this application is to provide a distributed charging station scheduling method and equipment based on multi-agent reinforcement learning in response to the shortcomings of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the distributed charging station scheduling method based on multi-agent reinforcement learning specifically includes:

[0007] receiving a charging request from an electric vehicle, wherein the charging request includes charging request data;

[0008] Distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request;

[0009] Determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed back the recommended charging station to the electric vehicle;

[0010] The charging score determination process specifically includes:

[0011] Acquire its own charging station information through the charging station, and splice the charging station information with the charging request data to obtain the intelligent agent observation;

[0012] The intelligent agent observation is input into the reinforcement learning strategy network deployed by the charging station, and the charging score corresponding to the charging station is output through the reinforcement learning strategy network.

[0013] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the distributing the charging request to at least two charging stations is specifically:

[0014] At least two charging stations are selected for the charging request according to the principle of from near to far, and the charging request is distributed to each of the selected charging stations.

[0015] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein after determining the recommended charging station corresponding to the charging request based on all received charging scores and feeding back the recommended charging station to the electric vehicle, the method further includes:

[0016] receiving a charging result of the charging request, wherein the charging result includes charging time, charging cost, and charging load;

[0017] Scheduling sample data is formed based on the charging results, the agent observations of each charging station, and the joint decision of the recommended charging station, and the scheduling sample data is stored in a scheduling sample data set, wherein the scheduling sample data set is used to fine-tune the reinforcement learning strategy network.

[0018] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the method further includes: fine-tuning the reinforcement learning strategy network, and deploying the fine-tuned reinforcement learning strategy network at each charging station, wherein the fine-tuning process specifically includes:

[0019] Determine several target indicators corresponding to the reinforcement learning strategy network, and obtain a first training network for each target indicator, wherein the first training network includes a Q-function network, a target Q-function network, and an entropy parameter;

[0020] For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network of the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is a target Q-function network in the first training network after single-target training;

[0021] Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-objective training is performed on the reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network, wherein the first training network includes a Q-function network, a target Q-function network and an entropy parameter.

[0022] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the charging results include charging time, charging cost and charging load; the target indicators include at least two of the charging time indicator, charging cost indicator and charging load indicator.

[0023] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein for each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network of the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, including:

[0024] For each target indicator, reading the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set;

[0025] Based on the reinforcement learning strategy network and the target Q-function network and entropy parameters in the first training network, modifying the indicator result by a multi-step time difference method to obtain a first modified indicator result;

[0026] Based on the first modification index result, the first training network is updated to obtain a first Q-function network and a first target Q-function network corresponding to the first training network;

[0027] Based on the first Q-function network, the reinforcement learning strategy network is updated to obtain an intermediate reinforcement learning strategy network, and based on the intermediate reinforcement learning strategy network, the entropy parameter in the first training network is updated to obtain a first entropy parameter corresponding to the first training network;

[0028] The first target Q-function network is used as the guidance network corresponding to the target indicator.

[0029] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the multi-objective training of the reinforcement learning strategy network based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator is performed to obtain the optimized reinforcement learning strategy network, specifically including:

[0030] Reading the indicator result corresponding to each target indicator in the scheduling sample data of the scheduling sample data set;

[0031] Based on the reinforcement learning strategy network and the target Q-function and entropy parameters in the second training network of each target indicator, the indicator result of each target indicator is modified by a multi-step time difference method to obtain a second modified indicator result of each target indicator;

[0032] Based on the second modified indicator result of each target indicator, the second training network of each target indicator is updated to obtain a second Q-function network and a second target Q-function network corresponding to the training network;

[0033] Using the second Q-function network and the guidance network of each target indicator, the optimization weight of each target indicator is calculated;

[0034] The reinforcement learning strategy network is updated based on the optimization weight of each target indicator and the second Q-function network of each target indicator, and the entropy parameter in the second training network is updated based on the updated reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network.

[0035] The second aspect of the present application provides a distributed charging station scheduling system based on multi-agent reinforcement learning, wherein the distributed charging station scheduling system based on multi-agent reinforcement learning includes a central server and a number of charging stations deployed with a reinforcement learning strategy network;

[0036] The central server is used to receive a charging request from an electric vehicle, distribute the charging request to at least two charging stations, and receive a charging score formed by each charging station based on the charging request; determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed back the recommended charging station to the electric vehicle, wherein the charging request includes charging request data;

[0037] The charging station is used to obtain its own charging station information, and splice the charging station information with the charging request data to obtain the intelligent agent observation, and input the intelligent agent observation into the reinforcement learning strategy network deployed by itself, and output the charging score corresponding to the charging station through the reinforcement learning strategy network.

[0038] The third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the above-mentioned distributed charging station scheduling methods based on multi-agent reinforcement learning.

[0039] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0040] The memory stores a computer-readable program executable by the processor;

[0041] When the processor executes the computer-readable program, the steps in any of the above-described distributed charging station scheduling methods based on multi-agent reinforcement learning are implemented.

[0042] Beneficial effects: Compared with the prior art, the present application provides a distributed charging station scheduling method and device based on multi-agent reinforcement learning, the method comprising receiving a charging request from an electric vehicle, wherein the charging request comprises charging request data; distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle. The present application deploys a reinforcement learning strategy network as a distributed node at each charging station, so that each charging station can determine the charging score corresponding to the charging request, and then schedules the electric vehicle based on the charging score through a central server, so that it can flexibly respond to fluctuations in charging demand in different regions and time periods, reduce the risk of overload on the power grid, improve overall charging efficiency, and reduce charging costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A block diagram of the principles of a distributed charging station scheduling system based on multi-agent reinforcement learning provided in an embodiment of the present application.

[0045] Figure 2 A flowchart of a distributed charging station scheduling method based on multi-agent reinforcement learning provided in an embodiment of the present application.

[0046] Figure 3 A flow chart of a distributed charging station scheduling method based on multi-agent reinforcement learning provided in an embodiment of the present application.

[0047] Figure 4 The figure is a flowchart of the pre-training process.

[0048] Figure 5 Schematic diagram of the fine-tuning process.

[0049] Figure 6 Schematic diagram of the training process during fine-tuning.

[0050] Figure 7 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiment of the present application provides a distributed charging station scheduling method and device based on multi-agent reinforcement learning. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0052] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0054] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0055] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0056] This embodiment provides a distributed charging station scheduling method based on multi-agent reinforcement learning, which is applied to a distributed charging station scheduling system. Figure 1As shown, the distributed charging station scheduling system includes a central server 100 and several charging stations 200, each of which is a charging station agent, deployed with a reinforcement learning strategy network, and each charging station agent is a distributed node of the distributed charging station scheduling system. The central server 100 is used to receive the charging request of the electric vehicle, distribute the charging request to the charging station to which it is connected, and determine the recommended charging station corresponding to the charging request based on all the charging scores received, and feed the recommended charging station back to the electric vehicle. Each charging station 200 is used to obtain its own charging station information, and splice the charging station information with the charging request data to obtain agent observation; and input the agent observation into the reinforcement learning strategy network deployed by itself, and output the charging score corresponding to the charging station through the reinforcement learning strategy network, wherein the reinforcement learning strategy network is pre-trained on the central server. That is to say, the central server is also used to pre-train the reinforcement learning strategy network, and deploy the pre-trained reinforcement learning strategy network to each charging station agent.

[0057] Specifically, the central server 100 may include an electric vehicle scheduling and charging simulation module (referred to as the simulation module) and a multi-agent reinforcement learning central controller (referred to as the central controller), wherein the simulation module generates a simulation program corresponding to the controllable charging station in the road network through historical scheduling data, road network data and charging station data, and the simulation program supports real-time provision of observation data of each charging station, and after the simulation is completed, automatically calculates the effect of the charging station on the scheduling and charging of electric vehicles during the simulation. The central controller includes three single-objective training subcomponents and one multi-objective training component, the network in the single-objective training subcomponent includes a reinforcement learning strategy network and a first training network, the network in the multi-objective training subcomponent includes a reinforcement learning strategy network, a second training network and a guidance network, the single-objective training subcomponent is used to perform single-objective training on the reinforcement learning strategy network, and the multi-objective training subcomponent is used to perform multi-objective training on the reinforcement learning strategy network.

[0058] The following is an explanation of the distributed charging station scheduling method based on multi-agent reinforcement learning provided by the embodiment of the present application, taking the central server as the execution subject. It should be noted that in practical applications, other hardware devices can also be used to execute the distributed charging station scheduling method based on multi-agent reinforcement learning provided by the embodiment of the present application, for example, the cloud, etc., and no specific restrictions are made here.

[0059] like Figure 2 and Figure 3 As shown, the distributed charging station scheduling method based on multi-agent reinforcement learning provided in the embodiment of the present application specifically includes:

[0060] S10. Receive a charging request from an electric vehicle, wherein the charging request includes charging request data.

[0061] Specifically, the charging request is charging request information submitted by the electric vehicle, which carries charging request data, wherein the charging request data may include one or more of the request time, request coordinates, vehicle battery information, and expected charging target. Of course, in actual applications, the charging request may not carry the charging request data, but after receiving the charging request, the central server requests the charging request data from the electric vehicle based on the charging request.

[0062] S20: Distribute the charging request to at least two charging stations, and receive a charging score formed by each charging station based on the charging request.

[0063] Specifically, each of the at least two charging stations communicates with the central server and is a distributed node in the distributed charging station scheduling system. That is, the central server selects at least two charging stations from all the charging stations it manages as candidate charging stations corresponding to the charging request, and then distributes the charging request to the at least two charging stations. Among them, the at least two charging stations can be randomly selected by the central server from all the charging stations it is connected to, or can be selected according to the distance between the charging station and the electric vehicle corresponding to the charging request, etc.

[0064] Exemplarily, the distributing the charging request to at least two charging stations specifically includes:

[0065] At least two charging stations are selected for the charging request according to the principle of from near to far, and the charging request is distributed to each of the selected charging stations.

[0066] Specifically, the principle of following from near to far refers to the principle of following from near to far distance, wherein the distance may be the driving distance between the electric vehicle and the charging station, or the straight-line distance between the electric vehicle and the charging station, etc. That is, when selecting at least two charging stations for a charging request, the distance from the electric vehicle to each charging station connected to the central server is first calculated, and then at least two charging stations are selected for the electric vehicle in order of distance from small to large, so as to obtain at least two charging stations corresponding to the charging request.

[0067] Furthermore, the charging score is obtained by evaluating the charging request by the charging station, and is used to reflect the charging effect of charging the electric vehicle through the charging station. That is to say, after obtaining the charging request, the charging station will evaluate the charging request to obtain the charging score corresponding to the charging request completed at the charging station, wherein the charging score can be determined based on the charging request data carried by the charging request and the charging station information of the charging station itself. Based on this, the charging score determination process specifically includes:

[0068] Acquire its own charging station information through the charging station, and splice the charging station information with the charging request data to obtain the intelligent agent observation;

[0069] The intelligent agent observation is input into the reinforcement learning strategy network deployed by the charging station, and the charging score corresponding to the charging station is output through the reinforcement learning strategy network.

[0070] Specifically, the charging station information may include one or more of the charging station location information, charging pile configuration data, charging pile usage data, queuing conditions and charging prices. The intelligent agent observation is obtained by splicing the charging station information and the charging request data, which includes all the information in the charging station information and the charging request data. In addition, before splicing the charging station information and the charging request data, the shortest driving time and driving distance of the electric vehicle corresponding to the charging request to reach the charging station can also be calculated, and then the shortest driving time and driving distance are added to the charging request data, and the charging request data is spliced ​​with the charging station information to obtain the intelligent agent observation. Of course, in actual applications, it is also possible to calculate the shortest driving time and driving distance of the electric vehicle to the charging station through the central server, add the shortest driving time and driving distance to the charging request data, and then distribute the charging request carrying the charging request data to the charging station, so that the charging station directly splices the charging station information with the charging request data after obtaining the charging station information.

[0071] Furthermore, before concatenating the charging station information and the charging request data, the charging station information and the charging request data may be normalized, and then the normalized charging station information and the charging request data may be concatenated to obtain the intelligent agent observation, wherein the normalization process of the charging request data may be performed by the charging station to reduce the computational requirements of the central server. Of course, in practical applications, it may also be performed on the central server. For example, the normalized charging request data is The normalized charging station information is Then the agent observes

[0072]

[0073] The reinforcement learning policy network is a policy network deployed in the charging station, which is used to make decisions on charging requests to obtain the charging score corresponding to the charging request completed at the charging station. The reinforcement learning policy network is pre-trained, and its input items are the agent observations. The output item is the charging score For this purpose, charging stations The process of determining the charging score by the reinforcement learning policy network deployed in θ is the network parameter set of the reinforcement learning policy network, Indicates the set of distributed charging stations. The number of charging stations included is denoted as M, π(·|·) represents the conditional probability distribution of the strategy in reinforcement learning, It means that the jth agent, under the tth charging request, according to the current observation value By probability Select Action The conditional probability distribution of . Indicates that the jth agent has observed Choose a probability distribution over all possible actions.

[0074] Among them, the strategy in the policy network is realized by the conversion of the neural network (denoted as f θ ), ε i is a random noise factor. The output of the neural network is a spherical Gaussian distribution. The specific strategy is obtained by sampling the noise factor.

[0075] Specifically, Figure 4 As shown, the pre-training process of the reinforcement learning strategy network can be:

[0076] Constructing an electric vehicle charging simulation scenario and an initial strategy network, wherein the initial strategy network is configured with randomly generated initial network parameters;

[0077] Deploy the initial strategy network to each simulated charging station agent in the electric vehicle charging simulation scenario;

[0078] Performing simulated charging station scheduling through the electric vehicle charging simulation scenario, and obtaining simulated charging results of the simulated charging station scheduling, wherein the simulated charging results include one or more of average charging time, cost, and grid load;

[0079] If the simulation charging result meets the preset requirements, a pre-trained reinforcement learning strategy network is obtained;

[0080] If the simulation charging result is not satisfactory, reinforcement learning is performed on the initial strategy network until the simulation result meets the preset requirements to obtain a pre-trained reinforcement learning strategy network.

[0081] Specifically, the electric vehicle charging simulation scene can be generated by a simulation module deployed in a central server, and the simulation module obtains historical charging data (i.e., pre-collected scheduling sample data), road network data, and charging station information, and then generates an electric vehicle charging simulation scene based on the historical charging data, road network data, and charging station information. Among them, the electric vehicle charging simulation scene includes a simulation central server and a plurality of simulation charging station intelligent agents, and each of the plurality of simulation charging station intelligent agents is connected to the simulation central server to form a simulation distributed scheduling system. The simulation charging station scheduling is to perform the distributed charging station scheduling method based on multi-agent reinforcement learning provided in the embodiment of the present application through the simulation distributed scheduling system to obtain a simulated charging result. In addition, the reinforcement learning process is the same as the subsequent fine-tuning process, and the embodiment of the present application is described in detail in the subsequent fine-tuning process.

[0082] Furthermore, after obtaining the charging score, the charging station will feed back the charging score to the central server. In addition, in order to facilitate the central server to fine-tune the reinforcement learning strategy network, the intelligent agent observation can also be fed back to the central server while feeding back the charging score to the central server. j After the charging request is decided by the reinforcement learning policy network, it will be j Agent observations that record charging requests locally and charging score Among them, charging station c j The process of making decisions on charging requests through the reinforcement learning policy network can be expressed as:

[0083]

[0084] in, Indicates the charging score, represents the agent observation, f θ represents the conversion function used for policy conversion in the reinforcement learning policy network, ε i is the random noise factor. θ The output is a spherical Gaussian distribution, and the charging score is obtained by sampling the noise factor.

[0085] The reinforcement learning policy network observes After that, a strategy that satisfies Gaussian distribution is formed. (including the probability distribution of different specific strategies), where: It can be expressed as:

[0086]

[0087] in, represents the standard deviation of the Gaussian distribution, Represents the variance of the Gaussian distribution, representing that the charging station gives this charging request a strategy, then passes the gradient through re-parameter sampling, and generates an action It can be expressed as:

[0088]

[0089] Then, a noise factor that satisfies the normal distribution is sampled to obtain a specific strategy from a strategy distribution. After sampling, it is the output of the strategy network, so f θ To distinguish. When the training is completed, the variance will approach 0. This means that when faced with an observation, it will converge to a strategy.

[0090] S30: Determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed back the recommended charging station to the electric vehicle.

[0091] Specifically, after receiving the charging scores of all the charging stations distributed to the charging request, all the charging scores will be aggregated to form a joint decision, wherein the process of aggregating all the charging scores to form a joint decision may be to select the charging station with the highest charging score as the recommended charging station, or to select a preset number of charging stations as a candidate charging station set according to the charging score, and feed the candidate charging station set back to the electric vehicle, receive the target charging station determined by the electric vehicle based on the candidate charging station set, and then use the target charging station as the recommended charging station, or obtain the charging scores of all the charging stations distributed to the charging request, read the charging preference of the electric vehicle, and then modify the charging score of the charging station based on the charging preference. The charging station corresponding to the maximum charging score after modification is selected as the recommended charging station, etc. In addition, after forming the joint decision, the charging score set composed of the charging scores of each charging station is used as the joint decision for recommending the charging station.

[0092] Furthermore, after determining the recommended charging station, the central server returns navigation information to the electric vehicle, guiding the electric vehicle to go to the charging station for charging. At the same time, the central server also receives the charging results fed back by the charging station to form scheduling sample data. The specific formation process of the scheduling sample data can be:

[0093] receiving a charging result of the charging request, wherein the charging result includes charging time, charging cost, and charging load;

[0094] Scheduling sample data is formed based on the charging results, the agent observations of each charging station, and the joint decision of the recommended charging station, and the scheduling sample data is stored in a scheduling sample data set, wherein the scheduling sample data set is used to fine-tune the reinforcement learning strategy network.

[0095] Specifically, when the electric vehicle is finished charging, the charging station records the charging result r of this charging request. t Return to the central server, where the charging result r t Including charging time Charging cost and grid load After obtaining the charging result, the central server collects the distributed charging stations. The global observation s is composed of the agent observations of each charging station t , joint decision-making t And charging results t The scheduling sample data is stored in a scheduling sample data set in the central server to fine-tune the reinforcement learning policy network based on the scheduling sample data set.

[0096] In one implementation, during the use of the reinforcement learning strategy network, the reinforcement learning strategy network can also be fine-tuned based on the scheduling sample data set stored in the central server to continuously optimize the reinforcement learning strategy network deployed at the charging station. That is, the method also includes: fine-tuning the reinforcement learning strategy network, and deploying the fine-tuned reinforcement learning strategy network at each charging station. In other words, Figure 5 As shown, the method also includes triggering fine-tuning, fine-tuning the reinforcement learning policy network based on the scheduling sample data set through the central controller, and when the reinforcement learning policy network meets the training requirements, the fine-tuned reinforcement learning policy network is tested through the simulation model, and when the test requirements are met, the reinforcement learning policy network is deployed at each charging station. Among them, the fine-tuning process of the reinforcement learning policy network can be triggered according to the received fine-tuning instruction, or it can be automatically triggered based on preset conditions, for example, when a preset number (such as 1000, 1500, etc.) of new experience data is collected from the previous fine-tuning, or the time from the previous fine-tuning reaches a preset time (such as 3 hours, 5 hours, etc.) or an emergency (such as the completion score of the request is too low and needs to be modified to adapt to sudden changes, etc.). Among them, when fine-tuning the reinforcement learning policy network based on the scheduling sample data set, a small batch of scheduling sample data can be randomly extracted from the experience database for training, and whether the result of this training meets the training requirements is evaluated. If not, the training is repeated until the training requirements are met to obtain the fine-tuned reinforcement learning policy network.

[0097] Further, if Figure 6 As shown, the fine-tuning process specifically includes:

[0098] H10, determine several target indicators corresponding to the reinforcement learning strategy network, and obtain the first training network for each target indicator;

[0099] H20. For each target indicator, single-target training is performed based on a preset scheduling sample data set, the first training network of the target indicator, and the reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training;

[0100] H30. Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-objective training is performed on the reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network.

[0101] Specifically, in step H10, several target indicators are determined according to the charging results in the scheduling sample data, and each target instruction in the several target indicators is included in the charging results, that is, the target indicator may include some result indicators in the charging results, or may include all result indicators in the charging results. In the implementation of this application, the charging results include charging time, charging cost, and charging load. Accordingly, the target indicator includes at least two of the charging time indicator, charging cost indicator, and charging load indicator. Typically, the target indicator includes the charging time indicator, the charging cost indicator, and the charging load indicator.

[0102] The first training network is used as an auxiliary network of the reinforcement learning strategy network, wherein the first training network includes a Q-function network, a target Q-function network and an entropy parameter. That is to say, the first training network for each target indicator includes a Q-function network, a target Q-function network and an entropy parameter. Moreover, each first training network may include one or more Q-function networks and one or more target Q-function networks, and the number of Q-function networks is the same as the number of target Q-function networks. In the embodiment of the present application, the example that the number of Q-function networks and the number of target Q-function networks are both 2 is used for illustration, that is, in the embodiment of the present application, the first training network for each target indicator includes two Q-function networks, two target Q-function networks and an entropy parameter, and the two Q-function networks are recorded as The two target Q-function networks are denoted as The entropy parameter is denoted as α, where the network parameters of the two Q-function networks are and The network parameters of the two target Q-function networks are and It should be noted that when fine-tuning the reinforcement learning strategy network, each fine-tuning can use the first training network after the previous fine-tuning training as the first training network to be fine-tuned in the current fine-tuning.

[0103] Further, in step H20, single-target training refers to taking each target indicator as the optimization target, using the first training network of each target indicator, the reinforcement learning strategy network and the scheduling sample data set to train the training network and the reinforcement learning strategy network to determine the guidance network for each target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training.

[0104] Exemplarily, for each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network of the target indicator, and a reinforcement learning strategy network to obtain a guidance network body corresponding to the target indicator, including:

[0105] H21. For each target indicator, read the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set;

[0106] H22. Based on the reinforcement learning strategy network and the target Q-function network and entropy parameters in the first training network, modify the indicator result by a multi-step time difference method to obtain a first modified indicator result;

[0107] H23. Update the first training network based on the first modification index result to obtain a first Q-function network and a first target Q-function network corresponding to the first training network;

[0108] H24. Based on the first Q-function network, the reinforcement learning strategy network is updated to obtain an intermediate reinforcement learning strategy network, and based on the intermediate reinforcement learning strategy network, the entropy parameter in the first training network is updated to obtain a first entropy parameter corresponding to the first training network;

[0109] H25. Use the first target Q-function network as the guidance network corresponding to the target indicator.

[0110] Specifically, in step H21, the indicator result of the target indicator includes the charging result of the scheduling sample data. When the indicator result corresponding to the target indicator is read in the scheduling sample data in the preset scheduling sample data set, a training batch is first selected from the scheduling sample data set, and then for each scheduling sample data in the training batch, the indicator result of the target indicator is selected from the charging result in the scheduling sample data. For example, the target indicator includes the charging time indicator, the charging cost indicator, and the charging load indicator; then, the charging time is selected as the indicator result of the charging time indicator, the charging cost is selected as the indicator result of the charging cost indicator, and the charging load is selected as the indicator result of the charging load indicator in the charging result.

[0111] Further, in step H22, after obtaining the indicator result of each target indicator, the indicator result is modified by a multi-step time difference method, wherein the number of time difference steps is denoted as λ, and the modified first modified indicator result can be expressed as:

[0112]

[0113] in, represents the first modification index result; t represents the charging request of the t-th electric vehicle; r p represents the target reward, P = {charging time index time, charging cost index cost, charging load index load} represents the target index set; s t+λ represents the global observation of the charging request of the t+λth electric vehicle; u t+λ represents the joint decision of the charging request of the t+λth electric vehicle, represents the charging score of the charging request of the t+λth electric vehicle; represents the intelligent agent observation of the charging request of the t+λth electric vehicle; γ is the discount factor, such as 0.99, 0.98, etc.; α represents the entropy parameter; represents the set of distributed charging stations for the t+λth electric vehicle; represents the target reward for the charging request of the t+λth EV.

[0114] Further, in step H23, after obtaining the first modified indicator result, the reinforcement learning goal of the target indicator is determined based on the first modified indicator result, wherein the reinforcement learning goal of each target indicator is to maximize the cumulative reward r p And make the joint strategy more random. The reinforcement learning objective of the target indicator can be expressed as:

[0115]

[0116] Among them, H(π(·|s t)) represents action entropy, which is used to characterize the randomness of the joint strategy; s t represents the global observation, u t represents joint decision making, π(·|·) represents the reinforcement learning policy network, α represents the entropy parameter, and π * represents the optimal strategy, represents the reward of the target indicator p of the t-th electric vehicle, Indicates the charging score, represents the agent observation, represents the set of distributed charging stations, c j represents the jth charging station.

[0117] Based on the reinforcement learning objective, the target indicator result y′ can be obtained, and then the gradients of the network parameters of the two Q-function networks are calculated based on the target indicator result, and the network parameters are updated based on the gradient to obtain two first Q-function networks, wherein the calculation method of the network parameters of the two Q-function networks and the update method of the network parameters can be respectively:

[0118]

[0119] in, represents Q-function network, j=1,2; represents the gradient of the network parameters of the Q-function network, and y′ represents the target indicator result.

[0120] After obtaining the two first Q-function networks, the network parameters of the two target Q-function networks are soft-updated by the network parameters of the two first Q-function networks to obtain the two first target Q-function networks, wherein the soft update can be expressed as:

[0121]

[0122] in, represents the network parameter set of the target Q-function network, τ represents the adjustment parameter, and its value can be 0.005, etc. Represents the set of network parameters of the Q-function network.

[0123] Further, in step H24, after obtaining the two first Q-function networks, the gradient of the network parameters of the reinforcement learning strategy network is calculated based on the two target Q-function networks, the reinforcement learning strategy network and the entropy parameters in the first training network, and the network parameters are updated using the gradient, wherein the calculation method of the gradient of the network parameters of the reinforcement learning strategy network and the updating method of the network parameters can be respectively:

[0124]

[0125] in, represents the gradient of the reinforcement learning policy network; Represents the gradient of the resampling process of the reinforcement learning policy network.

[0126] After updating the network parameters of the reinforcement learning policy network, the updated reinforcement learning policy network is recorded as the intermediate reinforcement learning policy network, and then the entropy parameter α is updated based on the reinforcement learning policy network. Among them, the loss function and update process of the entropy parameter α can be expressed as:

[0127]

[0128] H0=M·(-dim(A))

[0129] Among them, H0 represents the target entropy, A represents the action space of the charging station, and M represents the number of charging stations.

[0130] Further, in step H25, when the loss values ​​of the network gradients of the first Q-function network, the first target Q-function network, and the intermediate reinforcement learning strategy network converge, the first target Q-function network of each target indicator is used as the guidance network corresponding to the target indicator.

[0131] Further, in step H30, after obtaining the guidance network of each target indicator, multi-target training is performed on the reinforcement learning strategy network based on the scheduling sample data set, the first training network of each target indicator and the guidance network of each target indicator, wherein the guidance network of each target indicator is used to inform the reinforcement learning strategy network how to adjust the coefficients between multiple targets.

[0132] Exemplarily, the multi-objective training of the reinforcement learning strategy network based on the scheduling sample data set, the second training network of each target indicator, and the guidance network of each target indicator to obtain the optimized reinforcement learning strategy network specifically includes:

[0133] Reading the indicator result corresponding to each target indicator in the scheduling sample data of the scheduling sample data set;

[0134] Based on the reinforcement learning strategy network and the target Q-function and entropy parameters in the second training network of each target indicator, the indicator result of each target indicator is modified by a multi-step time difference method to obtain a second modified indicator result of each target indicator;

[0135] Based on the second modified indicator result of each target indicator, the second training network of each target indicator is updated to obtain a second Q-function network and a second target Q-function network corresponding to the training network;

[0136] Using the second Q-function network and the guidance network of each target indicator, the optimization weight of each target indicator is calculated;

[0137] The reinforcement learning strategy network is updated based on the optimization weight of each target indicator and the second Q-function network of each target indicator, and the entropy parameter in the second training network is updated based on the updated reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network.

[0138] Specifically, the index result acquisition process, the second modified index result acquisition process, and the second Q-function network and the second target Q-function network update process are the same as the above-mentioned monocular training process, and will not be repeated here. Only the difference between multi-target training and single-target training is explained here.

[0139] First, in multi-objective training, the reinforcement learning goal of multi-objective training is:

[0140]

[0141] Among them, P = {charging time indicator time, charging cost indicator cost, charging load indicator load} represents the target indicator set.

[0142] Secondly, the optimization weight of each target indicator is calculated, where the expression of the optimization weight can be:

[0143]

[0144] in, represents the guidance network, represents the optimization weight, Represents the Q-function network in the second training network.

[0145] After obtaining the optimization weight, the optimization weight can be normalized and used as the optimization weight of each target indicator. The normalization process can be expressed as:

[0146]

[0147] in, Represents the normalized optimization weight.

[0148] Finally, the gradient of the network parameters of the multi-objective reinforcement learning policy network can be expressed as:

[0149]

[0150] in, Represents the gradients of the network parameters of the reinforcement learning policy network.

[0151] Of course, it should be noted that during the multi-objective training process, multiple trainings will be performed until the loss values ​​of each network gradient converge.

[0152] Further, if Figure 5 As shown, after the optimized reinforcement learning strategy network, the optimized reinforcement learning strategy network is first tested. When the test passes, the optimized reinforcement learning strategy network is deployed at each charging station. When the test fails, the above fine-tuning process is repeated until the optimized reinforcement learning strategy network is tested. Among them, the test process may include: generating an electric vehicle scheduling charging simulation scenario through a simulation module, and deploying the optimized reinforcement learning strategy network at each charging station in the electric vehicle scheduling charging simulation scenario; simulating charging station scheduling through the electric vehicle scheduling charging simulation scenario, and obtaining the simulated charging results of the simulated charging station scheduling; if the simulated charging results meet the preset requirements, it is determined that the optimized reinforcement learning strategy network test passes; if the simulated charging results do not meet the preset requirements, the above fine-tuning process is performed on the optimized reinforcement learning strategy network until the simulated charging results meet the preset requirements.

[0153] It should be noted that the electric vehicle scheduling charging simulation scenario is generated based on the scheduling sample data, road network data and charging station data in the scheduling sample data set, and the generation process of the electric vehicle scheduling charging simulation scenario is the same as the generation process of the aforementioned pre-trained electric vehicle scheduling charging simulation scenario. In addition, the simulation scheduling process in the aforementioned pre-training and testing process is the same as the process of the aforementioned steps S10-S30, and the reinforcement learning process in the pre-training process is the same as the aforementioned fine-tuning process.

[0154] Of course, in practical applications, after the optimized reinforcement learning policy network, the optimized reinforcement learning policy network can also be directly deployed at each charging station as a fine-tuned optimized reinforcement learning policy network.

[0155] In summary, this embodiment provides a distributed charging station scheduling method and device based on multi-agent reinforcement learning, the method comprising receiving a charging request from an electric vehicle, wherein the charging request includes charging request data; distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle. This application deploys a reinforcement learning strategy network as a distributed node at each charging station, so that each charging station can determine the charging score corresponding to the charging request, and then schedules the electric vehicle based on the charging score through a central server, so that it can flexibly respond to charging demand fluctuations in different regions and time periods, reduce the risk of overload on the power grid, and improve the overall charging efficiency. At the same time, this application adopts a training method that combines single-objective training and multi-objective training to obtain a decision network that can provide service efficiency, reduce owner costs and balance power grid load.

[0156] Based on the above-mentioned distributed charging station scheduling method based on multi-agent reinforcement learning, this embodiment provides a distributed charging station scheduling system based on multi-agent reinforcement learning, such as Figure 1 As shown, the distributed charging station scheduling system based on multi-agent reinforcement learning includes a central server 100 and a number of charging stations 200 deployed with reinforcement learning strategy networks;

[0157] The central server 100 is used to receive a charging request from an electric vehicle, distribute the charging request to at least two charging stations, and receive a charging score formed by each charging station based on the charging request; determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed back the recommended charging station to the electric vehicle;

[0158] The charging station 200 is used to obtain its own charging station information, and splice the charging station information with the charging request data to obtain the intelligent agent observation, and input the intelligent agent observation into the reinforcement learning strategy network deployed by itself, and output the charging score corresponding to the charging station through the reinforcement learning strategy network.

[0159] Based on the above-mentioned distributed charging station scheduling method based on multi-agent reinforcement learning, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the distributed charging station scheduling method based on multi-agent reinforcement learning as described in the above-mentioned embodiment.

[0160] Based on the above-mentioned distributed charging station scheduling method based on multi-agent reinforcement learning, the present application also provides a terminal device, such as Figure 7 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0161] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0162] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0163] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0164] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed charging station scheduling method based on multi-agent reinforcement learning, characterized in that: The distributed charging station scheduling method based on multi-agent reinforcement learning specifically includes: receiving a charging request from an electric vehicle, wherein the charging request includes charging request data; Distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; Determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed back the recommended charging station to the electric vehicle; The charging score determination process specifically includes: Acquire its own charging station information through the charging station, and splice the charging station information with the charging request data to obtain the intelligent agent observation; The intelligent agent observation is input into the reinforcement learning strategy network deployed by the charging station, and the charging score corresponding to the charging station is output through the reinforcement learning strategy network.

2. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The distributing the charging request to at least two charging stations specifically includes: At least two charging stations are selected for the charging request according to the principle of from near to far, and the charging request is distributed to each of the selected charging stations.

3. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: After determining the recommended charging station corresponding to the charging request based on all received charging scores and feeding back the recommended charging station to the electric vehicle, the method further includes: receiving a charging result of the charging request, wherein the charging result includes charging time, charging cost, and charging load; Scheduling sample data is formed based on the charging results, the agent observations of each charging station, and the joint decision of the recommended charging station, and the scheduling sample data is stored in a scheduling sample data set, wherein the scheduling sample data set is used to fine-tune the reinforcement learning strategy network.

4. The distributed charging station scheduling method based on multi-agent reinforcement learning according to any one of claims 1 to 3, characterized in that: The method further includes: fine-tuning the reinforcement learning strategy network, and deploying the fine-tuned reinforcement learning strategy network at each charging station, wherein the fine-tuning process specifically includes: Determine several target indicators corresponding to the reinforcement learning strategy network, and obtain a first training network for each target indicator, wherein the first training network includes a Q-function network, a target Q-function network, and an entropy parameter; For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network of the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is a target Q-function network in the first training network after single-target training; Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-target training is performed on the reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network, wherein the first training network includes a Q-funct ion network, a target Q-funct ion network and an entropy parameter.

5. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 4 is characterized in that: The target index includes at least two of a charging time index, a charging cost index and a charging load index.

6. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 4 is characterized in that: For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network of the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, including: For each target indicator, reading the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set; Based on the reinforcement learning strategy network and the target Q-function network and entropy parameters in the first training network, modifying the indicator result by a multi-step time difference method to obtain a first modified indicator result; Based on the first modification index result, the first training network is updated to obtain a first Q-function network and a first target Q-function network corresponding to the first training network; Based on the first Q-function network, the reinforcement learning policy network is updated to obtain an intermediate reinforcement learning policy network, and based on the intermediate reinforcement learning policy network, the entropy parameter in the first training network is updated to obtain a first entropy parameter corresponding to the first training network; The first target Q-function network is used as the guidance network corresponding to the target indicator.

7. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 4 is characterized in that: The multi-objective training of the reinforcement learning strategy network based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator to obtain the optimized reinforcement learning strategy network specifically includes: Reading the indicator result corresponding to each target indicator in the scheduling sample data of the scheduling sample data set; Based on the reinforcement learning strategy network and the target Q-function and entropy parameters in the second training network of each target indicator, the indicator result of each target indicator is modified by a multi-step time difference method to obtain a second modified indicator result of each target indicator; Based on the second modified indicator result of each target indicator, the second training network of each target indicator is updated to obtain a second Q-function network and a second target Q-function network corresponding to the training network; Using the second Q-function network and the guidance network of each target indicator, the optimization weight of each target indicator is calculated; The reinforcement learning strategy network is updated based on the optimization weight of each target indicator and the second Q-function network of each target indicator, and the entropy parameter in the second training network is updated based on the updated reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network.

8. A distributed charging station scheduling system based on multi-agent reinforcement learning, characterized in that: The distributed charging station scheduling system based on multi-agent reinforcement learning includes a central server and a number of charging stations deployed with reinforcement learning strategy networks; The central server is used to receive a charging request from an electric vehicle, distribute the charging request to at least two charging stations, and receive a charging score formed by each charging station based on the charging request; determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed back the recommended charging station to the electric vehicle, wherein the charging request includes charging request data; The charging station is used to obtain its own charging station information, and splice the charging station information with the charging request data to obtain the intelligent agent observation, and input the intelligent agent observation into the reinforcement learning strategy network deployed by itself, and output the charging score corresponding to the charging station through the reinforcement learning strategy network.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the distributed charging station scheduling method based on multi-agent reinforcement learning as described in any one of claims 1-7.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the distributed charging station scheduling method based on multi-agent reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images