Distributed charging station scheduling method and equipment based on multi-agent reinforcement learning

Abstract

The present application discloses a distributed charging station scheduling method and device based on multi-agent reinforcement learning, the method comprising distributing a charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle. The present application uses the charging station where the reinforcement learning strategy network is deployed as a distributed node, determines the charging score for completing the charging request by itself through the deployment of the reinforcement learning strategy network, and schedules the electric vehicle based on each charging score. This can flexibly respond to fluctuations in charging demand in different regions and time periods, reduce the risk of grid overload, and improve overall charging efficiency. At the same time, the deployment of the reinforcement learning strategy network will be fine-tuned based on the scheduling sample data formed by historical charging requests to ensure the model performance of the deployment of the reinforcement learning strategy network.

Description

Technical Field

[0001] The present application relates to the field of electric vehicle charging technology, and in particular to a distributed charging station scheduling method and equipment based on multi-agent reinforcement learning. Background Art

[0002] The widespread adoption of electric vehicles has led to a rapid increase in demand for charging infrastructure, but this demand is also diverse. Statistics show that charging times on weekdays are, on average, over 1.5 times higher than on holidays. There are significant differences in the distribution of charging hotspots between daytime and nighttime, and different types of passenger vehicles have different peak charging times. However, the limited and fixed distribution of fixed charging stations makes it difficult to flexibly respond to peak charging demand or specific areas. Therefore, effectively scheduling electric vehicle charging stations is a key issue in improving charging efficiency and alleviating grid load.

[0003] Traditional electric vehicle charging scheduling methods mainly rely on car owners to independently select charging stations. They usually go to the nearest or least idle charging station for charging based on map information. It is difficult to foresee the impact of future traffic conditions, the length of queues at charging stations, etc., resulting in a large number of electric vehicles concentrating near certain charging stations during peak hours, putting pressure on the stability of the power grid. On the one hand, this will affect charging efficiency, and on the other hand, it may also increase the charging costs of car owners.

[0004] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a distributed charging station scheduling method and equipment based on multi-agent reinforcement learning in response to the shortcomings of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the distributed charging station scheduling method based on multi-agent reinforcement learning specifically includes:

[0007] receiving a charging request from an electric vehicle, wherein the charging request includes charging request data;

[0008] Distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request;

[0009] determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle;

[0010] The charging score determination process specifically includes:

[0011] Obtaining its own charging station information through the charging station, and splicing the charging station information with the charging request data to obtain the intelligent agent observation;

[0012] The intelligent agent observation is input into the reinforcement learning strategy network deployed by the charging station, and the charging score corresponding to the charging station is output through the reinforcement learning strategy network.

[0013] In the distributed charging station scheduling method based on multi-agent reinforcement learning, the step of distributing the charging request to at least two charging stations is as follows:

[0014] At least two charging stations are selected for the charging request according to the principle of from near to far, and the charging request is distributed to each of the selected charging stations.

[0015] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein after determining the recommended charging station corresponding to the charging request based on all received charging scores and feeding the recommended charging station back to the electric vehicle, the method further includes:

[0016] receiving a charging result of the charging request, wherein the charging result includes charging time, charging cost, and charging load;

[0017] Scheduling sample data is formed based on the charging results, the intelligent agent observations of each charging station, and the joint decision of the recommended charging station, and the scheduling sample data is stored in a scheduling sample data set, wherein the scheduling sample data set is used to fine-tune the reinforcement learning policy network.

[0018] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the method further includes: fine-tuning the reinforcement learning strategy network and deploying the fine-tuned reinforcement learning strategy network at each charging station, wherein the fine-tuning process specifically includes:

[0019] Determine several target indicators corresponding to the reinforcement learning strategy network, and obtain a first training network for each target indicator, wherein the first training network includes a Q-function network, a target Q-function network, and an entropy parameter;

[0020] For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network for the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training;

[0021] Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-objective training is performed on the reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network, wherein the first training network includes a Q-function network, a target Q-function network and an entropy parameter.

[0022] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the charging results include charging time, charging cost and charging load; the target indicators include at least two of the charging time indicator, charging cost indicator and charging load indicator.

[0023] In the distributed charging station scheduling method based on multi-agent reinforcement learning, for each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network for the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, including:

[0024] For each target indicator, read the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set;

[0025] Based on the reinforcement learning strategy network and the target Q-function network and entropy parameters in the first training network, modifying the indicator result by a multi-step time difference method to obtain a first modified indicator result;

[0026] Based on the first modification index result, the first training network is updated to obtain a first Q-function network and a first target Q-function network corresponding to the first training network;

[0027] Updating the reinforcement learning policy network based on the first Q-function network to obtain an intermediate reinforcement learning policy network, and updating the entropy parameter in the first training network based on the intermediate reinforcement learning policy network to obtain a first entropy parameter corresponding to the first training network;

[0028] The first target Q-function network is used as the guidance network corresponding to the target indicator.

[0029] The distributed charging station scheduling method based on multi-agent reinforcement learning, wherein the multi-objective training of the reinforcement learning strategy network based on the scheduling sample data set, the second training network of each target indicator, and the guidance network of each target indicator is performed to obtain the optimized reinforcement learning strategy network, specifically including:

[0030] Reading the indicator result corresponding to each target indicator in the scheduling sample data of the scheduling sample data set;

[0031] Based on the reinforcement learning strategy network and the target Q-function and entropy parameters in the second training network of each target indicator, the indicator result of each target indicator is modified by a multi-step time difference method to obtain a second modified indicator result of each target indicator;

[0032] Based on the second modified indicator result of each target indicator, the second training network of each target indicator is updated to obtain a second Q-function network and a second target Q-function network corresponding to the training network;

[0033] Utilizing the second Q-function network and the guidance network for each target indicator, the optimization weight of each target indicator is calculated;

[0034] The reinforcement learning policy network is updated based on the optimization weight of each target indicator and the second Q-function network of each target indicator, and the entropy parameter in the second training network is updated based on the updated reinforcement learning policy network to obtain an optimized reinforcement learning policy network.

[0035] A second aspect of the present application provides a distributed charging station scheduling system based on multi-agent reinforcement learning, wherein the distributed charging station scheduling system based on multi-agent reinforcement learning includes a central server and a plurality of charging stations deployed with a reinforcement learning strategy network;

[0036] The central server is configured to receive a charging request from an electric vehicle, distribute the charging request to at least two charging stations, and receive a charging score generated by each charging station based on the charging request; determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed the recommended charging station back to the electric vehicle, wherein the charging request includes charging request data;

[0037] The charging station is used to obtain its own charging station information, and splice the charging station information with the charging request data to obtain the intelligent agent observation, and input the intelligent agent observation into the reinforcement learning strategy network deployed by itself, and output the charging score corresponding to the charging station through the reinforcement learning strategy network.

[0038] The third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the steps in the distributed charging station scheduling method based on multi-agent reinforcement learning as described above.

[0039] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0040] The memory stores a computer-readable program executable by the processor;

[0041] When the processor executes the computer-readable program, the steps in any of the above-described distributed charging station scheduling methods based on multi-agent reinforcement learning are implemented.

[0042] Beneficial effects: Compared with the prior art, the present application provides a distributed charging station scheduling method and device based on multi-agent reinforcement learning, the method comprising receiving a charging request from an electric vehicle, wherein the charging request includes charging request data; distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle. The present application deploys a reinforcement learning strategy network as a distributed node at each charging station, so that each charging station can determine the charging score corresponding to the charging request, and then schedules the electric vehicle based on the charging score through a central server. This can flexibly respond to fluctuations in charging demand in different regions and time periods, reduce the risk of grid overload, improve overall charging efficiency, and reduce charging costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a block diagram of the principles of a distributed charging station scheduling system based on multi-agent reinforcement learning provided in an embodiment of the present application.

[0045] Figure 2 Flowchart of a distributed charging station scheduling method based on multi-agent reinforcement learning provided in an embodiment of the present application.

[0046] Figure 3 A flow chart of a distributed charging station scheduling method based on multi-agent reinforcement learning provided in an embodiment of the present application.

[0047] Figure 4 Schematic diagram of the pre-training process.

[0048] Figure 5 Schematic diagram of the fine-tuning process.

[0049] Figure 6 Flowchart of the training process during fine-tuning.

[0050] Figure 7 This is a block diagram of the principles of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The present application provides a distributed charging station scheduling method and device based on multi-agent reinforcement learning. To clarify the purpose, technical solution, and effects of this application, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to explain this application and are not intended to limit this application.

[0052] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0054] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0055] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0056] This embodiment provides a distributed charging station scheduling method based on multi-agent reinforcement learning, which is applied to a distributed charging station scheduling system. Figure 1As shown, the distributed charging station scheduling system includes a central server 100 and several charging stations 200. Each of the charging stations 200 is a charging station agent deployed with a reinforcement learning policy network. Each charging station agent serves as a distributed node in the distributed charging station scheduling system. The central server 100 is configured to receive charging requests from electric vehicles, distribute the requests to the connected charging stations, determine a recommended charging station corresponding to the request based on all received charging scores, and provide the recommended charging station to the electric vehicle. Each charging station 200 is configured to obtain its own charging station information, concatenate this information with the charging request data to generate agent observations, input these agent observations into its deployed reinforcement learning policy network, and output a charging score corresponding to the charging station through the reinforcement learning policy network. The reinforcement learning policy network is pre-trained on the central server. In other words, the central server is also configured to pre-train the reinforcement learning policy network and deploy the pre-trained reinforcement learning policy network on each charging station agent.

[0057] Specifically, the central server 100 may include an electric vehicle scheduling and charging simulation module (referred to as the simulation module) and a multi-agent reinforcement learning central controller (referred to as the central controller). The simulation module generates a simulation program for each controllable charging station in the road network based on historical scheduling data, road network data, and charging station data. The simulation program supports real-time observation data for each charging station and, after the simulation is complete, automatically calculates the effect of the charging station on electric vehicle scheduling and charging during the simulation. The central controller includes three single-objective training subcomponents and one multi-objective training component. The network within the single-objective training subcomponent includes a reinforcement learning policy network and a first training network, while the network within the multi-objective training subcomponent includes a reinforcement learning policy network, a second training network, and a guidance network. The single-objective training subcomponent is used to perform single-objective training on the reinforcement learning policy network, while the multi-objective training subcomponent is used to perform multi-objective training on the reinforcement learning policy network.

[0058] The following describes the distributed charging station scheduling method based on multi-agent reinforcement learning provided by the embodiments of this application, using a central server as the execution entity. It should be noted that in actual applications, other hardware devices can also be used to execute the distributed charging station scheduling method based on multi-agent reinforcement learning provided by the embodiments of this application, such as the cloud, and no specific limitations are imposed here.

[0059] like Figure 2 and Figure 3 As shown, the distributed charging station scheduling method based on multi-agent reinforcement learning provided in the embodiment of the present application specifically includes:

[0060] S10. Receive a charging request from an electric vehicle, wherein the charging request includes charging request data.

[0061] Specifically, a charging request is a charging request message submitted by an electric vehicle and carries charging request data. The charging request data may include one or more of the request time, request coordinates, vehicle battery information, and the expected charging destination. Of course, in actual applications, a charging request may not carry charging request data. Instead, upon receiving the charging request, the central server requests charging request data from the electric vehicle based on the charging request.

[0062] S20: Distribute the charging request to at least two charging stations, and receive a charging score formed by each charging station based on the charging request.

[0063] Specifically, each of the at least two charging stations communicates with a central server and serves as a distributed node in the distributed charging station scheduling system. In other words, the central server selects at least two charging stations from all the charging stations it manages as candidate charging stations for a charging request, and then distributes the charging request to the at least two charging stations. The at least two charging stations can be randomly selected by the central server from all connected charging stations, or they can be selected based on the distance between the charging station and the electric vehicle corresponding to the charging request.

[0064] Exemplarily, distributing the charging request to at least two charging stations specifically includes:

[0065] At least two charging stations are selected for the charging request according to the principle of from near to far, and the charging request is distributed to each of the selected charging stations.

[0066] Specifically, the principle of following the principle of "from near to far" refers to the principle of following the principle of "from near to far" in terms of distance, where the distance can be the driving distance between the electric vehicle and the charging station, or the straight-line distance between the electric vehicle and the charging station, etc. That is, when selecting at least two charging stations for a charging request, the distance from the electric vehicle to each charging station connected to the central server is first calculated, and then at least two charging stations are selected for the electric vehicle in ascending order of distance, to obtain the at least two charging stations corresponding to the charging request.

[0067] Furthermore, the charging score is obtained by the charging station evaluating the charging request, which is used to reflect the charging effect of charging the electric vehicle through the charging station. In other words, after obtaining the charging request, the charging station will evaluate the charging request to obtain the charging score corresponding to the charging request completed at the charging station. The charging score can be determined based on the charging request data carried by the charging request and the charging station information of the charging station itself. Based on this, the charging score determination process specifically includes:

[0068] Obtaining its own charging station information through the charging station, and splicing the charging station information with the charging request data to obtain the intelligent agent observation;

[0069] The intelligent agent observation is input into the reinforcement learning strategy network deployed by the charging station, and the charging score corresponding to the charging station is output through the reinforcement learning strategy network.

[0070] Specifically, charging station information may include one or more of charging station location information, charging pile configuration data, charging pile usage data, queuing conditions, and charging prices. Agent observations are obtained by concatenating charging station information and charging request data, and include all information in the charging station information and charging request data. Furthermore, before concatenating the charging station information and charging request data, the shortest travel time and distance for the electric vehicle corresponding to the charging request to reach the charging station may be calculated, and then the shortest travel time and distance may be added to the charging request data. The charging request data and the charging station information may then be concatenated to obtain agent observations. Of course, in practical applications, a central server may also be used to calculate the shortest travel time and distance for the electric vehicle to reach the charging station, add the shortest travel time and distance to the charging request data, and then distribute the charging request carrying the charging request data to the charging station. This allows the charging station to directly concatenate the charging station information with the charging request data after obtaining the charging station information.

[0071] Furthermore, before concatenating the charging station information and the charging request data, the charging station information and the charging request data can be normalized, and then the normalized charging station information and the charging request data can be concatenated to obtain the intelligent agent observation. The normalization process of the charging request data can be performed by the charging station to reduce the computational requirements of the central server. Of course, in practical applications, it can also be performed on the central server. For example, the normalized charging request data is The normalized charging station information is Then the agent observes

[0072] The reinforcement learning policy network is a policy network deployed in the charging station, which is used to make decisions on charging requests to obtain the charging score corresponding to the charging request completed at the charging station. The reinforcement learning policy network is pre-trained, and its input is the agent observation The output item is charging score For this purpose, charging stations The process of determining charging scores by the reinforcement learning policy network deployed within the network can be expressed as θ is the network parameter set of the reinforcement learning policy network, Represents the collection of distributed charging stations, The number of charging stations included is denoted as M, π(·|·) represents the conditional probability distribution of the strategy in reinforcement learning, Indicates that the jth agent responds to the tth charging request based on the current observation value By probability Select Action The conditional probability distribution of . Indicates that the jth agent is in the observation value Choose a probability distribution over all possible actions.

[0073] Among them, the strategy in the strategy network is realized through the transformation of the neural network (denoted as f θ ), ε i Is a random noise factor. The output of the neural network is a spherical Gaussian distribution. The specific strategy is obtained by sampling the noise factor.

[0074] Specifically, if Figure 4 As shown, the pre-training process of the reinforcement learning policy network can be:

[0075] Constructing an electric vehicle charging simulation scenario and an initial policy network, wherein the initial policy network is configured with randomly generated initial network parameters;

[0076] Deploying the initial strategy network to each simulated charging station agent in the electric vehicle charging simulation scenario;

[0077] Performing simulated charging station scheduling through the electric vehicle charging simulation scenario, and obtaining simulated charging results of the simulated charging station scheduling, wherein the simulated charging results include one or more of average charging time, cost, and grid load;

[0078] If the simulation charging results meet the preset requirements, a pre-trained reinforcement learning strategy network is obtained;

[0079] If the simulation charging result does not meet the requirements, reinforcement learning is performed on the initial strategy network until the simulation result meets the preset requirements to obtain a pre-trained reinforcement learning strategy network.

[0080] Specifically, the electric vehicle charging simulation scene can be generated by a simulation module deployed in a central server, and the simulation module obtains historical charging data (i.e., pre-collected scheduling sample data), road network data, and charging station information, and then generates an electric vehicle charging simulation scene based on the historical charging data, road network data, and charging station information. Wherein, the electric vehicle charging simulation scene includes a simulation central server and a plurality of simulation charging station intelligent agents, and each of the plurality of simulation charging station intelligent agents is connected to the simulation central server to form a simulation distributed scheduling system. The simulation charging station scheduling is to perform the distributed charging station scheduling method based on multi-agent reinforcement learning provided in the embodiment of the present application through the simulation distributed scheduling system to obtain a simulated charging result. In addition, the reinforcement learning process is the same as the subsequent fine-tuning process, and the embodiment of the present application is described in detail in the subsequent fine-tuning process.

[0081] Furthermore, after obtaining the charging score, the charging station will feed the charging score back to the central server. In addition, in order to facilitate the central server to fine-tune the reinforcement learning strategy network, the intelligent agent observation can also be fed back to the central server at the same time as the charging score is fed back to the central server. j After the decision on charging request is made by reinforcement learning policy network, the charging station c j Agent observations that locally record charging requests and charging score Among them, charging station c j The process of making decisions on charging requests through the reinforcement learning policy network can be expressed as:

[0082]

[0083] in, Indicates the charging score, represents the agent observation, f θ represents the conversion function used for policy conversion in the reinforcement learning policy network, ε i is the random noise factor. f θ The output is a spherical Gaussian distribution, and the charging score is obtained by sampling the noise factor.

[0084] The reinforcement learning policy network observes After that, a strategy that satisfies Gaussian distribution is formed (including the probability distribution of different specific strategies), where: It can be expressed as:

[0085]

[0086] in, represents the standard deviation of the Gaussian distribution, Represents the variance of the Gaussian distribution, representing the charging station giving this charging request a strategy, then passing the gradient through re-parameter sampling and generating an action It can be expressed as:

[0087]

[0088] Then, a noise factor that satisfies the normal distribution is sampled to obtain a specific strategy from a strategy distribution. After sampling, it is the output of the strategy network, so f is used. θ To distinguish. After training is complete, the variance will approach 0, which means that it will converge to a policy when faced with an observation.

[0089] S30: Determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed the recommended charging station back to the electric vehicle.

[0090] Specifically, after receiving the charging scores of all charging stations assigned to a charging request, all charging scores are aggregated to form a joint decision. The process of aggregating all charging scores to form a joint decision can include selecting the charging station with the highest charging score as the recommended charging station, selecting a preset number of charging stations based on their charging scores as a set of candidate charging stations, feeding the candidate charging station set back to the electric vehicle, receiving a target charging station determined by the electric vehicle based on the set of candidate charging stations, and then recommending the target charging station. Alternatively, after obtaining the charging scores of all charging stations assigned to a charging request, reading the electric vehicle's charging preferences, and then revising the charging scores of the charging stations and the joint decision of recommended charging stations based on the charging preferences, and then selecting the charging station corresponding to the highest revised charging score as the recommended charging station. Furthermore, after forming the joint decision, the charging score set consisting of the charging scores of each charging station is used as the joint decision of recommended charging stations.

[0091] Furthermore, after determining the recommended charging station, the central server returns navigation information to the electric vehicle, guiding the electric vehicle to the charging station for charging. At the same time, the central server also receives the charging results fed back by the charging station to form scheduling sample data. The specific process of forming the scheduling sample data can be:

[0092] receiving a charging result of the charging request, wherein the charging result includes charging time, charging cost, and charging load;

[0093] Scheduling sample data is formed based on the charging results, the intelligent agent observations of each charging station, and the joint decision of the recommended charging station, and the scheduling sample data is stored in a scheduling sample data set, wherein the scheduling sample data set is used to fine-tune the reinforcement learning policy network.

[0094] Specifically, when the electric vehicle is finished charging, the charging station records the charging result r of this charging request. t Return to the central server, where the charging result r t Including charging time Charging costs and grid load After obtaining the charging result, the central server collects the distributed charging stations. The global observation s is composed of the agent observations of each charging station t , joint decision-making t and charging results t The scheduling sample data is stored in a scheduling sample data set in the central server to fine-tune the reinforcement learning policy network based on the scheduling sample data set.

[0095] In one implementation, during the use of the reinforcement learning policy network, the reinforcement learning policy network can also be fine-tuned based on the scheduling sample data set stored in the central server to continuously optimize the reinforcement learning policy network deployed at the charging station. That is, the method further includes: fine-tuning the reinforcement learning policy network and deploying the fine-tuned reinforcement learning policy network at each charging station. In other words, if Figure 5 As shown, the method also includes triggering fine-tuning, fine-tuning the reinforcement learning policy network based on the scheduling sample data set by the central controller, and when the reinforcement learning policy network meets the training requirements, testing the fine-tuned reinforcement learning policy network through the simulation model. When the test requirements are met, the reinforcement learning policy network is deployed at each charging station. The fine-tuning process of the reinforcement learning policy network can be triggered by a received fine-tuning instruction, or it can be automatically triggered based on preset conditions, for example, when a preset amount (e.g., 1000, 1500, etc.) of new experience data is collected since the previous fine-tuning, or when the time from the previous fine-tuning reaches a preset length (e.g., 3 hours, 5 hours, etc.), or an emergency (e.g., the completion score of the request is too low and needs to be modified to adapt to sudden changes, etc.). When fine-tuning the reinforcement learning policy network based on the scheduling sample data set, a small batch of scheduling sample data can be randomly extracted from the experience database for training, and the results of this training can be evaluated to see whether they meet the training requirements. If not, the training is repeated until the training requirements are met to obtain the fine-tuned reinforcement learning policy network.

[0096] Further, if Figure 6 As shown, the fine-tuning process specifically includes:

[0097] H10. Determine several target indicators corresponding to the reinforcement learning strategy network and obtain the first training network for each target indicator;

[0098] H20. For each target indicator, single-target training is performed based on a preset scheduling sample data set, the first training network for the target indicator, and the reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training;

[0099] H30. Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-objective training is performed on the reinforcement learning strategy network to obtain an optimized reinforcement learning strategy network.

[0100] Specifically, in step H10, several target indicators are determined based on the charging results in the scheduling sample data. Each target instruction in the several target indicators is included in the charging results. That is, the target indicator may include some result indicators in the charging results, or may include all result indicators in the charging results. In the implementation of this application, the charging results include charging time, charging cost, and charging load. Accordingly, the target indicator includes at least two of the charging time indicator, charging cost indicator, and charging load indicator. Typically, the target indicator includes the charging time indicator, charging cost indicator, and charging load indicator.

[0101] The first training network is used as an auxiliary network of the reinforcement learning strategy network, wherein the first training network includes a Q-function network, a target Q-function network and an entropy parameter. That is to say, the first training network of each target indicator includes a Q-function network, a target Q-function network and an entropy parameter. Moreover, each first training network may include one or more Q-function networks, and one or more target Q-function networks, and the number of Q-function networks is the same as the number of target Q-function networks. In the embodiment of the present application, the example in which the number of Q-function networks and the number of target Q-function networks are both 2 is used for illustration, that is, in the embodiment of the present application, the first training network of each target indicator includes two Q-function networks, two target Q-function networks and an entropy parameter, and the two Q-function networks are recorded as The two target Q-function networks are denoted as The entropy parameter is denoted as α, where the network parameters of the two Q-function networks are and The network parameters of the two target Q-function networks are and It should be noted that when fine-tuning the reinforcement learning strategy network, each fine-tuning can use the first training network after the previous fine-tuning training as the first training network to be fine-tuned in the current fine-tuning.

[0102] Further, in step H20, single-target training refers to taking each target indicator as the optimization target, using the first training network of each target indicator, the reinforcement learning strategy network and the scheduling sample data set to train the training network and the reinforcement learning strategy network to determine the guidance network for each target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training.

[0103] Exemplarily, for each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network for the target indicator, and a reinforcement learning strategy network to obtain a guidance network body corresponding to the target indicator, including:

[0104] H21. For each target indicator, read the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set;

[0105] H22. Based on the reinforcement learning strategy network and the target Q-function network and entropy parameters in the first training network, modify the indicator result by a multi-step time difference method to obtain a first modified indicator result;

[0106] H23. Update the first training network based on the first modification indicator result to obtain a first Q-function network and a first target Q-function network corresponding to the first training network;

[0107] H24. Update the reinforcement learning policy network based on the first Q-function network to obtain an intermediate reinforcement learning policy network, and update the entropy parameter in the first training network based on the intermediate reinforcement learning policy network to obtain a first entropy parameter corresponding to the first training network;

[0108] H25. Use the first target Q-function network as the guidance network corresponding to the target indicator.

[0109] Specifically, in step H21, the indicator result of the target indicator includes the charging results of the scheduling sample data. When reading the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set, a training batch is first selected from the scheduling sample data set. Then, for each scheduling sample data in the training batch, the indicator result of the target indicator is selected from the charging results in the scheduling sample data. For example, if the target indicator includes a charging time indicator, a charging cost indicator, and a charging load indicator, then the charging time is selected from the charging results as the indicator result for the charging time indicator, the charging cost is selected as the indicator result for the charging cost indicator, and the charging load is selected as the indicator result for the charging load indicator.

[0110] Furthermore, in step H22, after obtaining the indicator result of each target indicator, the indicator result is modified by a multi-step time difference method, wherein the number of time difference steps is denoted as λ. The modified first modified indicator result can be expressed as:

[0111]

[0112] in, represents the first modified indicator result; t represents the charging request of the t-th electric vehicle; r p represents the target reward, P = {charging time indicator time, charging cost indicator cost, charging load indicator load} represents the target indicator set; s t+λ represents the global observation of the charging request of the t+λth electric vehicle; u t+λ represents the joint decision of the charging request of the t+λth electric vehicle, represents the charging score of the charging request of the t+λth electric vehicle; represents the agent observation of the charging request of the t+λth electric vehicle; γ is the discount factor, such as 0.99, 0.98, etc.; α represents the entropy parameter; represents the set of distributed charging stations for the t+λth electric vehicle; represents the target reward for the charging request of the t+λth EV.

[0113] Further, in step H23, after obtaining the first modified indicator result, the reinforcement learning goal of the target indicator is determined based on the first modified indicator result, wherein the reinforcement learning goal of each target indicator is to maximize the cumulative reward r p And make the joint strategy more random. The reinforcement learning objective of the target indicator can be expressed as:

[0114]

[0115] Among them, H(π(·|s t)) represents action entropy, which is used to characterize the randomness of the joint strategy; s t represents the global observation, u t represents joint decision making, π(·|·) represents the reinforcement learning policy network, α represents the entropy parameter, π * represents the optimal strategy, represents the reward of the target indicator p of the t-th electric vehicle, Indicates the charging score, represents the agent observation, represents the set of distributed charging stations, c j represents the jth charging station.

[0116] Based on the reinforcement learning objective, a target indicator result y′ can be obtained, and then the gradients of the network parameters of the two Q-function networks are calculated based on the target indicator result, and the network parameters are updated based on the gradients to obtain two first Q-function networks, wherein the calculation method of the network parameters of the two Q-function networks and the update method of the network parameters can be respectively:

[0117]

[0118] in, represents the Q-function network, j = 1, 2; represents the gradient of the network parameters of the Q-function network, and y′ represents the target indicator result.

[0119] After obtaining the two first Q-function networks, the network parameters of the two target Q-function networks are soft-updated using the network parameters of the two first Q-function networks to obtain the two first target Q-function networks. The soft update can be expressed as:

[0120]

[0121] in, represents the network parameter set of the target Q-function network, τ represents the adjustment parameter, and its value can be 0.005, etc. Represents the network parameter set of the Q-function network.

[0122] Further, in step H24, after obtaining the two first Q-function networks, the gradient of the network parameters of the reinforcement learning policy network is calculated based on the two target Q-function networks, the reinforcement learning policy network, and the entropy parameters in the first training network, and the network parameters are updated using the gradient. The calculation method of the gradient of the network parameters of the reinforcement learning policy network and the updating method of the network parameters can be respectively:

[0123]

[0124] in, represents the gradient of the reinforcement learning policy network; Represents the gradient of the resampling process of the reinforcement learning policy network.

[0125] After updating the network parameters of the reinforcement learning policy network, the updated reinforcement learning policy network is recorded as the intermediate reinforcement learning policy network, and then the entropy parameter α is updated based on the reinforcement learning policy network. The loss function and update process of the entropy parameter α can be expressed as:

[0126]

[0127] H0=M·(-dim(A))

[0128] Among them, H0 represents the target entropy, A represents the action space of the charging station, and M represents the number of charging stations.

[0129] Furthermore, in step H25, when the loss values of the network gradients of the first Q-function network, the first target Q-function network, and the intermediate reinforcement learning policy network converge, the first target Q-function network of each target indicator is used as the guidance network corresponding to the target indicator.

[0130] Furthermore, in step H30, after obtaining the guidance network for each target indicator, multi-target training is performed on the reinforcement learning strategy network based on the scheduling sample data set, the first training network for each target indicator, and the guidance network for each target indicator, wherein the guidance network for each target indicator is used to inform the reinforcement learning strategy network how to adjust the coefficients between multiple targets.

[0131] Exemplarily, the multi-objective training of the reinforcement learning strategy network based on the scheduling sample data set, the second training network of each target indicator, and the guidance network of each target indicator to obtain the optimized reinforcement learning strategy network specifically includes:

[0132] Reading the indicator result corresponding to each target indicator in the scheduling sample data of the scheduling sample data set;

[0133] Based on the reinforcement learning strategy network and the target Q-function and entropy parameters in the second training network of each target indicator, the indicator result of each target indicator is modified by a multi-step time difference method to obtain a second modified indicator result of each target indicator;

[0134] Based on the second modified indicator result of each target indicator, the second training network of each target indicator is updated to obtain a second Q-function network and a second target Q-function network corresponding to the training network;

[0135] Utilizing the second Q-function network and the guidance network for each target indicator, the optimization weight of each target indicator is calculated;

[0136] The reinforcement learning policy network is updated based on the optimization weight of each target indicator and the second Q-function network of each target indicator, and the entropy parameter in the second training network is updated based on the updated reinforcement learning policy network to obtain an optimized reinforcement learning policy network.

[0137] Specifically, the process of obtaining the index result, obtaining the second modified index result, and updating the second Q-function network and the second target Q-function network are the same as the monocular training process described above, and will not be repeated here. Only the differences between multi-target training and single-target training are explained here.

[0138] First, in multi-objective training, the reinforcement learning goal of multi-objective training is:

[0139]

[0140] Wherein, P={charging time indicator time, charging cost indicator cost, charging load indicator load} represents the target indicator set.

[0141] Secondly, the optimization weight of each target indicator is calculated, where the expression of the optimization weight can be:

[0142]

[0143] in, represents the guidance network, represents the optimization weight, Represents the Q-function network in the second training network.

[0144] After obtaining the optimization weight, the optimization weight can be normalized and used as the optimization weight of each target indicator. The normalization process can be expressed as:

[0145]

[0146] in, Represents the normalized optimization weight.

[0147] Finally, the gradient of the network parameters of the reinforcement learning policy network under multiple objectives can be expressed as:

[0148]

[0149] in, Represents the gradient of the network parameters of the reinforcement learning policy network.

[0150] Of course, it should be noted that during the multi-objective training process, multiple training sessions will be performed until the loss values of each network gradient converge.

[0151] Further, if Figure 5 As shown, after the optimized reinforcement learning policy network, the optimized reinforcement learning policy network is first tested. When the test passes, the optimized reinforcement learning policy network is deployed at each charging station. When the test fails, the above fine-tuning process is repeated until the optimized reinforcement learning policy network passes the test. The testing process may include: generating an electric vehicle scheduling and charging simulation scenario through a simulation module, and deploying the optimized reinforcement learning policy network at each charging station in the electric vehicle scheduling and charging simulation scenario; simulating charging station scheduling through the electric vehicle scheduling and charging simulation scenario, and obtaining simulated charging results of the simulated charging station scheduling; if the simulated charging results meet the preset requirements, then determining that the optimized reinforcement learning policy network has passed the test; if the simulated charging results do not meet the preset requirements, then performing the above fine-tuning process on the optimized reinforcement learning policy network until the simulated charging results meet the preset requirements.

[0152] It should be noted that the EV scheduling and charging simulation scenario is generated based on the scheduling sample data, road network data, and charging station data in the scheduling sample dataset, and the generation process of this EV scheduling and charging simulation scenario is the same as the generation process of the pre-trained EV scheduling and charging simulation scenario. In addition, the simulation scheduling process during the pre-training and testing process is the same as the process of steps S10-S30 above, and the reinforcement learning process during the pre-training process is the same as the fine-tuning process mentioned above.

[0153] Of course, in practical applications, after the optimized reinforcement learning policy network is deployed, the optimized reinforcement learning policy network can be directly deployed at each charging station as a fine-tuned optimized reinforcement learning policy network.

[0154] In summary, this embodiment provides a distributed charging station scheduling method and device based on multi-agent reinforcement learning, the method comprising receiving a charging request from an electric vehicle, wherein the charging request includes charging request data; distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle. This application deploys a reinforcement learning strategy network as a distributed node at each charging station, so that each charging station can determine the charging score corresponding to the charging request, and then schedules the electric vehicle based on the charging score through a central server. This can flexibly respond to fluctuations in charging demand in different regions and time periods, reduce the risk of grid overload, and improve overall charging efficiency. At the same time, this application adopts a training method that combines single-objective training and multi-objective training to obtain a decision network that can provide service efficiency, reduce owner costs, and balance grid load.

[0155] Based on the above-mentioned distributed charging station scheduling method based on multi-agent reinforcement learning, this embodiment provides a distributed charging station scheduling system based on multi-agent reinforcement learning, such as Figure 1 As shown, the distributed charging station scheduling system based on multi-agent reinforcement learning includes a central server 100 and several charging stations 200 deployed with reinforcement learning strategy networks;

[0156] The central server 100 is configured to receive a charging request from an electric vehicle, distribute the charging request to at least two charging stations, and receive a charging score generated by each charging station based on the charging request; determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed the recommended charging station back to the electric vehicle;

[0157] The charging station 200 is used to obtain its own charging station information, and splice the charging station information with the charging request data to obtain the intelligent agent observation, and input the intelligent agent observation into the reinforcement learning strategy network deployed by itself, and output the charging score corresponding to the charging station through the reinforcement learning strategy network.

[0158] Based on the above-mentioned distributed charging station scheduling method based on multi-agent reinforcement learning, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the distributed charging station scheduling method based on multi-agent reinforcement learning as described in the above-mentioned embodiment.

[0159] Based on the above-mentioned distributed charging station scheduling method based on multi-agent reinforcement learning, this application also provides a terminal device, such as Figure 7 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0160] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0161] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0162] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0163] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed charging station scheduling method based on multi-agent reinforcement learning, characterized in that: The distributed charging station scheduling method based on multi-agent reinforcement learning specifically includes: receiving a charging request from an electric vehicle, wherein the charging request includes charging request data; Distributing the charging request to at least two charging stations, and receiving a charging score formed by each charging station based on the charging request; determining a recommended charging station corresponding to the charging request based on all received charging scores, and feeding back the recommended charging station to the electric vehicle; The charging score determination process specifically includes: Obtaining its own charging station information through the charging station, and splicing the charging station information with the charging request data to obtain the intelligent agent observation; The charging station inputs the agent observation into its own deployed reinforcement learning strategy network, and the reinforcement learning strategy network outputs the charging score corresponding to the charging station; The method further includes: fine-tuning the reinforcement learning strategy network and deploying the fine-tuned reinforcement learning strategy network at each charging station, wherein the fine-tuning process specifically includes: Determine several target indicators corresponding to the reinforcement learning strategy network, and obtain a first training network for each target indicator, wherein the first training network includes a Q-function network, a target Q-function network, and an entropy parameter; For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network for the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training; Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-objective training is performed on the reinforcement learning policy network to obtain an optimized reinforcement learning policy network, wherein the second training network includes a Q-function network, a target Q-function network and an entropy parameter; Among them, the gradient of the network parameters of the reinforcement learning strategy network under multiple objectives is expressed as: in, represents the gradient of the network parameters of the reinforcement learning policy network, Indicates the charging score, represents the agent observation, represents a strategy that satisfies the Gaussian distribution, represents the set of distributed charging stations for the t+λth electric vehicle, represents the gradient of the resampling process of the reinforcement learning policy network, represents the normalized optimization weight, represents the Q-function network, s t represents the global observation, u t represents joint decision making, f θ Represents the conversion function used for policy conversion in the reinforcement learning policy network, ∈ t represents the random noise factor, α represents the entropy parameter, θ represents the network parameter of the reinforcement learning policy network, and P represents the target indicator set.

2. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The distributing the charging request to at least two charging stations specifically includes: At least two charging stations are selected for the charging request according to the principle of from near to far, and the charging request is distributed to each of the selected charging stations.

3. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: After determining a recommended charging station corresponding to the charging request based on all received charging scores and feeding the recommended charging station back to the electric vehicle, the method further includes: receiving a charging result of the charging request, wherein the charging result includes charging time, charging cost, and charging load; Scheduling sample data is formed based on the charging results, the intelligent agent observations of each charging station, and the joint decision of the recommended charging station, and the scheduling sample data is stored in a scheduling sample data set, wherein the scheduling sample data set is used to fine-tune the reinforcement learning policy network.

4. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The target indicators include at least two of a charging time indicator, a charging cost indicator, and a charging load indicator.

5. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network for the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, including: For each target indicator, read the indicator result corresponding to the target indicator from the scheduling sample data in the preset scheduling sample data set; Based on the reinforcement learning strategy network and the target Q-function network and entropy parameters in the first training network, modifying the indicator result by a multi-step time difference method to obtain a first modified indicator result; Based on the first modification index result, the first training network is updated to obtain a first Q-function network and a first target Q-function network corresponding to the first training network; Updating the reinforcement learning policy network based on the first Q-function network to obtain an intermediate reinforcement learning policy network, and updating the entropy parameter in the first training network based on the intermediate reinforcement learning policy network to obtain a first entropy parameter corresponding to the first training network; The first target Q-function network is used as the guidance network corresponding to the target indicator.

6. The distributed charging station scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The multi-objective training of the reinforcement learning strategy network based on the scheduling sample data set, the second training network of each target indicator, and the guidance network of each target indicator to obtain the optimized reinforcement learning strategy network specifically includes: Reading the indicator result corresponding to each target indicator in the scheduling sample data of the scheduling sample data set; Based on the reinforcement learning strategy network and the target Q-function and entropy parameters in the second training network of each target indicator, the indicator result of each target indicator is modified by a multi-step time difference method to obtain a second modified indicator result of each target indicator; Based on the second modified indicator result of each target indicator, the second training network of each target indicator is updated to obtain a second Q-function network and a second target Q-function network corresponding to the training network; Utilizing the second Q-function network and the guidance network for each target indicator, the optimization weight of each target indicator is calculated; The reinforcement learning policy network is updated based on the optimization weight of each target indicator and the second Q-function network of each target indicator, and the entropy parameter in the second training network is updated based on the updated reinforcement learning policy network to obtain an optimized reinforcement learning policy network.

7. A distributed charging station scheduling system based on multi-agent reinforcement learning, characterized in that: The distributed charging station scheduling system based on multi-agent reinforcement learning includes a central server and several charging stations deployed with reinforcement learning strategy networks; The central server is configured to receive a charging request from an electric vehicle, distribute the charging request to at least two charging stations, and receive a charging score generated by each charging station based on the charging request; determine a recommended charging station corresponding to the charging request based on all received charging scores, and feed the recommended charging station back to the electric vehicle, wherein the charging request includes charging request data; The charging station is configured to obtain its own charging station information, concatenate the charging station information with the charging request data to obtain an agent observation, and input the agent observation into its own deployed reinforcement learning strategy network, and output a charging score corresponding to the charging station through the reinforcement learning strategy network; The reinforcement learning strategy network is deployed at each charging station. The fine-tuning process of the reinforcement learning strategy network specifically includes: Determine several target indicators corresponding to the reinforcement learning strategy network, and obtain a first training network for each target indicator, wherein the first training network includes a Q-function network, a target Q-function network, and an entropy parameter; For each target indicator, single-target training is performed based on a preset scheduling sample data set, a first training network for the target indicator, and a reinforcement learning strategy network to obtain a guidance network corresponding to the target indicator, wherein the guidance network is the target Q-function network in the first training network after single-target training; Based on the scheduling sample data set, the second training network of each target indicator and the guidance network of each target indicator, multi-objective training is performed on the reinforcement learning policy network to obtain an optimized reinforcement learning policy network, wherein the second training network includes a Q-function network, a target Q-function network and an entropy parameter; Among them, the gradient of the network parameters of the reinforcement learning strategy network under multiple objectives is expressed as: in, represents the gradient of the network parameters of the reinforcement learning policy network, Indicates the charging score, represents the agent observation, represents a strategy that satisfies the Gaussian distribution, represents the set of distributed charging stations for the t+λth electric vehicle, represents the gradient of the resampling process of the reinforcement learning policy network, represents the normalized optimization weight, represents the Q-function network, s t represents the global observation, u t represents joint decision making, f θ Represents the conversion function used for policy conversion in the reinforcement learning policy network, ∈ t represents the random noise factor, α represents the entropy parameter, θ represents the network parameter of the reinforcement learning policy network, and P represents the target indicator set.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the distributed charging station scheduling method based on multi-agent reinforcement learning as described in any one of claims 1-6.

9. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps of the distributed charging station scheduling method based on multi-agent reinforcement learning as described in any one of claims 1 to 6 are implemented.

Images