Method for jointly determining a taxi dispatching strategy and a charging station pricing strategy

Abstract

The application discloses a method for jointly determining a taxi scheduling strategy and a charging station pricing strategy, and constructs a double-layer multi-agent reinforcement learning model; state information of a current moment of interaction between a taxi agent and / or a charging station agent and an environment is acquired; a scheduling action of the taxi agent is calculated and generated by a policy network based on the state information of the current moment, and a scheduling strategy of the taxi agent is determined; a charging action of the charging station agent is calculated and generated by an improved double-delay deep deterministic policy model based on the state information of the current moment and historical state information, and a charging strategy of the charging station agent is determined. The application realizes the collaborative optimization of the scheduling strategy and the charging pricing strategy of the electric taxi.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things, and in particular to a method for jointly determining a taxi dispatching strategy and a charging station pricing strategy. BACKGROUND

[0002] With the popularity of electric vehicles, electric taxis, as a green and environmentally friendly mode of transportation, have gradually become an important part of urban transportation. However, in the process of fleet operation, unique challenges are also faced. How to balance battery power and charging demand while ensuring service quality, and how to dynamically allocate orders and reasonably arrange charging resources, have become key problems to be solved.

[0003] In related technical solutions, order dispatching and charging pricing are usually split into two independent sub-problems to address the above technical problems. Order dispatching solutions usually use operations research methods and modern reinforcement learning methods, but operations research methods often lack flexibility when faced with high uncertainty and dynamic changes in actual urban environments. Modern reinforcement learning methods usually use two main processing paradigms: online methods and methods based on fixed time windows. Online methods make dispatching decisions immediately when each order arrives. A key limitation of this method is that it often overlooks the coordination opportunities between orders arriving in adjacent time periods, resulting in low dispatching efficiency. Methods based on fixed time windows group orders into batches within fixed time windows and make unified dispatching decisions, thereby alleviating the near-sighted allocation problem.

[0004] In related technical solutions, charging pricing decision-making solutions usually use dynamic programming, genetic algorithms, and game theory. However, these model-based methods either require accurate environmental models or result in high computational overhead, limiting their applicability in real-time systems.

[0005] In actual operation, the order dispatching and charging pricing of taxis are not independent tasks, but are closely coupled. Order dispatching decisions directly affect the spatiotemporal distribution of charging demand, while charging pricing strategies in turn affect the availability of taxis, thereby affecting dispatching efficiency. The dispatching and pricing methods in related technical solutions are usually independent, resulting in a lack of sufficient adaptability to dynamic changes in urban environments and an inability to respond quickly in complex environments. SUMMARY

[0006] The purpose of the present application is to provide a method for jointly determining a taxi dispatching strategy and a charging station pricing strategy, which combines the dispatching strategy and charging pricing strategy of electric taxis to achieve the coordinated optimization of the two.

[0007] In a first aspect, the present application provides a method for jointly determining a taxi dispatching strategy and a charging station pricing strategy, the method comprising:

[0008] constructing a double-layer multi-agent reinforcement learning model comprising a first multi-agent reinforcement learning model and a second multi-agent reinforcement learning model, each agent in the first multi-agent reinforcement learning model being a taxi agent, each agent in the second multi-agent reinforcement learning model being a charging station agent, each taxi agent being provided with a corresponding policy network, and each charging station agent being provided with a corresponding improved double-delay deep deterministic policy model, the improved double-delay deep deterministic policy model being a fusion model of a Transformer encoder and a double-delay deep deterministic policy gradient algorithm model;

[0009] obtaining state information of the taxi agents and / or the charging station agents interacting with the environment at a current time, and inputting the state information at the current time into the double-layer multi-agent reinforcement learning model, wherein the state information comprises global vehicle order information, state information of the taxi agents, and state information of the charging station agents;

[0010] the policy network calculates and generates a dispatch action of the taxi agent based on the state information at the current time, and determines a dispatch strategy of the taxi agent based on the dispatch action, wherein the dispatch strategy comprises a taxi agent that is not assigned an order and a taxi agent that is not fully loaded and is assigned to a target order;

[0011] the improved double-delay deep deterministic policy model calculates and generates a charging action of the charging station agent based on the state information at the current time and historical state information, and determines a charging strategy of the charging station agent based on the charging action, wherein the charging strategy comprises a charging price of the charging station agent and a charging power allocated to each taxi.

[0012] Further, constructing a double-layer multi-agent reinforcement learning model comprising a first multi-agent reinforcement learning model and a second multi-agent reinforcement learning model, comprising:

[0013] establishing a Markov decision process, setting an agent set, a state space, a joint action space, a reward function, a state transition function, and a discount factor of the Markov decision process, wherein,

[0014] The state space is a set of state information, and the state information includes global vehicle order information, state information of each taxi agent, and state information of a charging station agent. The global vehicle order information includes order information to be processed and order information being processed of all taxi agents. The state information of the taxi agent includes a current position, a remaining seat capacity, a cumulative reward value, a battery level, and an accepted order. The state information of the charging station agent includes a number of vehicles waiting for charging, a number of vehicles being charged, a charging price of each charging station, and a charging demand of each taxi.

[0015] The joint action space includes a dispatch action of a taxi agent and a charging action of a charging station agent. The dispatch action of the taxi agent includes that the taxi agent is not assigned an order and an unfull taxi agent is assigned to a target order. The charging action of the charging agent includes a charging price of the charging station agent and a charging power assigned to each taxi agent.

[0016] The reward function includes a reward function of a taxi agent and a reward function of a charging station agent. The reward function of the charging station agent is determined based on a difference between a charging cost and a power grid cost.

[0017] Further, the policy network outputs a dispatch action of a taxi agent based on state information of the current time, and the dispatch action includes:

[0018] The state information of the current time is input into the policy network, and the policy network calculates and generates a dispatch action of a taxi agent.

[0019] The dispatch action is run in an environment to interact with a double-layer multi-agent reinforcement learning model environment.

[0020] State information of a next time is obtained based on a result of the interaction, and a reward value of the current time is determined based on a reward function of the taxi agent by executing the dispatch action.

[0021] An advantage function is determined based on the reward value of the current time, a reward mean value of the current time, a reward standard deviation of the current time, and a carpooling timeout penalty. An optimization objective function of the policy network is determined based on the advantage function.

[0022] A policy gradient is calculated based on the advantage function, and parameters of the policy network are updated according to the policy gradient, so that the optimization objective function is optimized in a preset direction.

[0023] Further, an advantage function is determined based on the reward value, the reward mean value, the reward standard deviation, and the carpooling timeout penalty. An optimization objective function of the policy network is determined based on the advantage function, and the optimization objective function includes:

[0024] Advantage function is expressed as:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] wherein, represents a reward value obtained by a taxi agent i in a state under a charging action , represents an average reward at the current time, represents a standard deviation of the reward at the current time, represents the total number of taxi agents at the current time, represents an abbreviation of the standard deviation, represents a carpooling timeout penalty of the taxi agent i, represents a carpooling combination of the taxi agent i, represents an average additional time caused by carpooling of the taxi agent i, represents a maximum tolerance value of the average additional time caused by carpooling, represents a hyperparameter; Optimization objective function

[0030] is expressed as:

[0031] ;

[0032] wherein, represents an expectation, represents a probability of a policy network outputting a scheduling action, represents a regularization coefficient, represents a parameter before and after updating the mean square error.

[0033] Further, the double-delay deep deterministic policy gradient algorithm model includes a current policy network, a target policy network, a current critic network and a target critic network, the current critic network includes two Q networks, the target critic network includes two target Q networks, and the improved double-delay deep deterministic policy model outputs a charging action of a charging station agent based on state information at the current time and historical state information, including:

[0034] obtain historical state information of a plurality of historical time steps before the current time;​

[0035] inputting the historical state information into the Transformer encoder for feature extraction to obtain historical feature information;

[0036] concatenating the historical feature information and the state information at the current time to obtain enhanced state information, and inputting the enhanced state information into the current policy network;

[0037] The current policy network calculates and generates a corresponding charging action based on the enhanced state information;

[0038] The charging action is executed to interact with the environment of the double-layer multi-agent reinforcement learning model;

[0039] determining the state information at the next time based on the result of the interaction, and determining a reward value obtained by executing the charging action based on a reward function of the charging station agent;

[0040] The state information at the current time, the charging action, the state information at the next time, and the reward value constitute an experience sample stored in an experience pool.

[0041] Further, the improved double-delay deep deterministic policy model further includes an adversarial policy generator, and the method further includes training the improved double-delay deep deterministic policy model, and the training includes:

[0042] randomly selecting an experience sample from the experience pool, assuming that the experience sample is identified as (state information s t , current charging action a t , current reward value r t , state information s t+1 at the next time);

[0043] inputting the historical state information into the Transformer encoder for feature extraction to obtain historical feature information, and concatenating the historical feature information and the state information s t at the current time to obtain enhanced state information;

[0044] The charging action a t output by the current policy network based on the enhanced state information;

[0045] The adversarial policy generator adds an adversarial disturbance to the charging action a t to generate a worst action, and determines an action after confrontation based on the charging action a t and the worst action;

[0046] The two Q networks respectively calculate estimated Q values of the post-adversarial action, and the minimum value among the calculated estimated Q values is taken as a minimum estimated Q value;

[0047] The target policy network calculates a target Q value of the post-adversarial action based on the state information s t+1 The target policy network calculates a target Q value of the post-adversarial action based on the state information s t+1 ;

[0048] The two target Q networks respectively calculate target Q values of the post-adversarial action based on the state information s t+1 , and the minimum value among the calculated target Q values is taken as a minimum target Q value, and the current reward value r t and the minimum target Q value are used to determine a target value;

[0049] The gradient descent algorithm is used to minimize the mean square error between the target value and the minimum estimated Q value, so as to update the parameters of the current critic network.

[0050] Further, the training further comprises:

[0051] After updating the parameters of the current critic network for several times, the parameters of the current policy network are updated, comprising:

[0052] The current policy network outputs a charging action a1 based on the state information s1;

[0053] The adversarial policy generator adds adversarial disturbance to the charging action a1 to generate a worst action, and determines a post-adversarial action based on the charging action a1 and the worst action;

[0054] The two Q networks respectively calculate estimated Q values of the post-adversarial action, and the minimum value among the calculated estimated Q values is taken as a minimum estimated Q value;

[0055] The policy gradient is determined based on the gradient of the minimum estimated Q value and the gradient of the parameters of the current policy network, and the parameters of the current policy network are updated in the direction of gradient ascent based on the policy gradient.

[0056] Further, the adversarial policy generator adds adversarial disturbance to the charging action a t , and outputs a post-adversarial action, comprising:

[0057] The post-adversarial action is expressed as:

[0058] ;

[0059] Wherein, represents a charging action output by a current policy network of an i-th charging station agent based on state information, represents a disturbance intensity, denote parameters of the Transformer encoder, denote action value output by the current critic network based on the action after the confrontation, denote derivation.

[0060] Further, the training comprises:

[0061] Derivations of the parameters of the Transformer encoder and the parameters of the current critic network are performed using the chain rule, and the parameters of the Transformer encoder and the parameters of the current critic network are updated simultaneously using a gradient descent algorithm.

[0062] In a second aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for jointly determining a taxi scheduling strategy and a charging station pricing strategy when executing the computer program.

[0063] The present application adopts a double-layer multi-agent reinforcement learning model for vehicle scheduling and charging pricing, optimizes the decision-making strategies of agents on two levels of vehicle scheduling and charging pricing based on a multi-agent framework of reinforcement learning, so that the vehicle scheduling and charging pricing can gradually optimize their respective strategies while sharing the global environmental state, realize the collaborative optimization of the two, and can adjust the scheduling and pricing strategies in real time according to the dynamically changing urban environment and power grid load conditions, having strong adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The first flowchart of the method for jointly determining a taxi scheduling strategy and a charging station pricing strategy provided by the embodiments of the present application;

[0065] Figure 2 The second flowchart of the method for jointly determining a taxi scheduling strategy and a charging station pricing strategy provided by the embodiments of the present application;

[0066] Figure 3 The third flowchart of the method for jointly determining a taxi scheduling strategy and a charging station pricing strategy provided by the embodiments of the present application;

[0067] Figure 4 The system block diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0068] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings, but these embodiments do not limit the present application, and the structural, method, or functional changes made by those of ordinary skill in the art based on these embodiments are included in the protection scope of the present application.

[0069] Please refer to Figure 1 The embodiment of the application provides a method for jointly determining a taxi scheduling strategy and a charging station pricing strategy, the method comprising:

[0070] S101, constructing a double-layer multi-agent reinforcement learning model comprising a first multi-agent reinforcement learning model and a second multi-agent reinforcement learning model;

[0071] S102, obtaining state information of a current time at which a taxi agent and / or a charging station agent interacts with an environment, and inputting the state information of the current time into the double-layer multi-agent reinforcement learning model;

[0072] S103, calculating and generating a scheduling action of the taxi agent based on the state information of the current time, and determining a scheduling strategy of the taxi agent based on the scheduling action;

[0073] S104, calculating and generating a charging action of the charging station agent based on the state information of the current time and historical state information, and determining a charging strategy of the charging station agent based on the charging action.

[0074] Exemplarily, each agent in the first multi-agent reinforcement learning model is a taxi agent, each agent in the second multi-agent reinforcement learning model is a charging station agent, each taxi agent is provided with a corresponding policy network, and each charging station agent is provided with a corresponding improved double-delay deep deterministic policy model, the improved double-delay deep deterministic policy model being a fusion model of a Transformer encoder and a double-delay deep deterministic policy gradient algorithm model.

[0075] Exemplarily, the state information comprises global vehicle order information, state information of the taxi agent, and state information of the charging station agent.

[0076] Exemplarily, the scheduling strategy comprises a taxi agent that is not assigned an order and a taxi agent that is not fully loaded being assigned to a target order.

[0077] Exemplarily, the charging strategy comprises a charging price of the charging station agent and a charging power allocated to each taxi.

[0078] In the embodiment, a double-layer multi-agent reinforcement learning model is adopted for vehicle scheduling and charging pricing, a multi-agent framework based on reinforcement learning is adopted, and the decision-making strategies of the agents are optimized on two levels of vehicle scheduling and charging pricing, so that the vehicle scheduling and the charging pricing can gradually optimize their respective strategies while sharing the global environmental state, realize the collaborative optimization of the two, and can adjust the scheduling and pricing strategies in real time according to the dynamically changing urban environment and power grid load conditions, and have strong adaptability.

[0079] In one embodiment of the present application, a double-layer multi-agent reinforcement learning model including a first multi-agent reinforcement learning model and a second multi-agent reinforcement learning model is constructed, which includes:

[0080] A Markov decision process is established, and an agent set, a state space, a joint action space, a reward function, a state transition function and a discount factor of the Markov decision process are set, wherein,

[0081] The state space is a set of state information, and the state information includes global vehicle order information, state information of each taxi agent and state information of a charging station agent, the global vehicle order information includes order information to be processed and order information being processed of all taxi agents, the state information of the taxi agent includes a current position, a remaining seat capacity, a cumulative reward value, a battery power and an accepted order, and the state information of the charging station agent includes a number of vehicles waiting for charging, a number of vehicles being charged, a charging price of each charging station and a charging demand of each taxi.

[0082] The joint action space includes a dispatch action of the taxi agent and a charging action of the charging station agent, the dispatch action of the taxi agent includes that the taxi agent is not assigned an order and a taxi agent not full is assigned to a target order, and the charging action of the charging station agent includes a charging price of the charging station agent and a charging power assigned to each taxi agent.

[0083] The reward function includes a reward function of the taxi agent and a reward function of the charging station agent, and the reward function of the charging station agent is determined based on a difference between a charging fee obtained and a power grid cost.

[0084] Exemplarily, the Markov decision process uses six elements to represent , respectively representing the agent set, the state space, the joint action space, the state transition function, the reward function and the discount factor. Each subsequent training step is a Markov decision process.

[0085] Exemplarily, the agent set includes two different agent groups, wherein the taxi agent group , and the charging station agent group . The state space is state information observed by the agent interacting with the environment, and the state space S t at time t is represented as , represents global vehicle order information, and the global vehicle order information includes order information to be processed and order information being processed of all taxi agents. The order information can be understood as an order issued by a passenger who needs to take a car, including arrival time, starting point and ending point, etc. For available taxi agents, represents the state information of the i-th taxi agent, the state information including the current position, the remaining seat capacity, the cumulative reward value, the battery level and the accepted order. The current position can be understood as the position of the taxi when participating in order taking, and the remaining seat capacity can be understood as the carrying capacity of the taxi - the number of passengers actually on board. The battery level can be understood as the battery level of the taxi. The accepted order can be understood as indicating the taxi assigned to the passenger's request to ride a taxi. represents the state information of the charging station agent, the state information including the number of vehicles waiting for charging , the number of vehicles being charged , the charging price of each charging station and the charging demand of each taxi .

[0086] Exemplarily, the joint action space is used to reflect the combination of actions that the agent can take. The joint action space is represented as . represents the dispatching action of the taxi agent, which is a binary decision when represents that the taxi agent is not assigned an order, represents that the unfull taxi agent is assigned to the target order j. represents the charging action of the charging station agent, including the charging price of the charging station agent and the charging power allocated to each taxi agent .

[0087] Exemplarily, the reward function refers to obtaining the corresponding reward based on the state of the executed action at each time step. The reward function includes the reward function of the taxi agent and the reward function of the charging station agent. The reward function of the taxi agent is represented as:

[0088] .

[0089] The reward function of the charging station agent is represented as:

[0090] ;

[0091] wherein, represents the number of charging station agents, represents the charging price of the taxi agent, represents the charging demand of the taxi agent, represents the node marginal price, i.e. the cost price of the power of the power grid, represents the number of taxi agents being charged, represents the charging power.

[0092] Exemplarily, the state transition function represents that, given the current state and the joint action, the environment evolves to the next state according to the state transition function. The function implicitly models the complex dynamics of travel, energy flow and market interaction, capturing the coupled evolution of the transportation and power systems. The state transition function is represented as:

[0093] .

[0094] discount factor represents a discount factor of a future reward, .

[0095] One embodiment of the present application, as shown in Figure 2 The policy network outputs the dispatch action of the taxi agent based on the state information at the current time, including:

[0096] S201, inputting the state information at the current time into the policy network, and the policy network calculating and generating the dispatch action of the taxi agent;

[0097] S202, running the dispatch action in the environment to interact with the double-layer multi-agent reinforcement learning model environment;

[0098] S203, obtaining the state information at the next time based on the result of the interaction, and determining the reward value at the current time based on the reward function of the taxi agent for executing the dispatch action;

[0099] S204, determining the advantage function based on the reward value at the current time, the reward mean value at the current time, the reward standard deviation at the current time and the carpooling timeout penalty, and determining the optimization objective function of the policy network based on the advantage function;

[0100] S205, calculating the policy gradient based on the advantage function, and updating the parameters of the policy network according to the policy gradient, so that the optimization objective function is optimized in a preset direction.

[0101] In this embodiment, the vehicle dispatching is implemented by using the first multi-agent reinforcement learning model. By improving the framework of the proximal policy optimization, the taxi agent in the first multi-agent reinforcement learning model adopts the architecture of the policy network, and the critic network is not used. The architecture of the policy network not only simplifies, but also can effectively adapt to the environment of the taxi agent, avoiding explicit Q value or V value estimation.

[0102] In view of the homogeneity of taxis and their common goal of maximizing income, a benchmark mechanism based on group average reward is adopted in the present example, which greatly improves the computational efficiency by updating the policy using the average reward of the entire taxi agent group instead of using the value function of a single agent. At each time step, the reward mean and standard deviation of the active vehicle group are calculated.

[0103] Exemplarily, the reward mean at the current time is represented as:

[0104] .

[0105] wherein represents the total number of taxi agents at the current time, represents the reward value obtained by taxi agent i scheduling action in state .

[0106] The reward standard deviation at the current time is represented as:

[0107] .

[0108] wherein represents the abbreviation of standard deviation.

[0109] The advantage function is used to measure the goodness of the action relative to the average level under the current policy, to guide the policy to update in a better direction. It is a comparison quantity, if the action is better than the average level, the policy should increase the probability of the action, if the action is worse than the average level, the policy should reduce the probability of the action. The advantage function is used for the update of the policy gradient, and the parameters of the policy network are updated by the policy gradient, so that the objective function is optimized in the preset direction, i.e. maximizing the expected return.

[0110] In the actual vehicle scheduling long scenario, the immediate income of the taxi will also be limited by the delay and detour of carpooling, therefore, in the present embodiment, the carpooling overtime penalty is quantitatively processed, and the carpooling overtime penalty is defined as the overtime ratio of the vehicle assigned to the carpooling group. Exemplarily, the carpooling overtime penalty of taxi agent i is represented as:

[0111] .

[0112] wherein represents the carpooling combination of taxi agent i, represents the average additional time caused by carpooling of taxi agent i, ​Maximal tolerated value of average extra time caused by carpooling. When the carpooling overtime penalty exceeds the maximal tolerated value , , the advantage is penalized.

[0113] The reward value at the current moment, the reward mean value at the current moment, the reward standard deviation at the current moment and the carpooling overtime penalty are considered in the construction of the advantage function. Exemplarily, the advantage function is expressed as:

[0114] ;

[0115] wherein, denotes a hyperparameter, and denotes a hyperparameter balancing the income normalization and the service quality execution through time penalty.

[0116] In this embodiment, by improving the traditional advantage function, the single-step reward benchmark is used to approximate the value term, avoiding the complexity of long-term value estimation. The calculation method of the above advantage function ensures that the policy optimization not only suppresses the variance in the batch reward, but also balances the profitability and service quality by dynamically constraining the service delay.

[0117] Exemplarily, the optimization objective function is expressed as:

[0118] ;

[0119] wherein, denotes expectation, denotes the probability of the policy network output scheduling action, denotes a regularization coefficient, denotes the mean square error before and after the parameter update.

[0120] In this embodiment, by adopting a single policy network, the need for time difference learning or target network is eliminated, while ensuring efficient scheduling, the calculation of the value function of each agent is avoided, thereby significantly reducing sample correlation and calculation delay, greatly improving the calculation efficiency, so that the policy can quickly adapt during high order density and remain responsive during sparsity, thereby realizing robust convergence and efficient scheduling, effectively solving the scheduling efficiency problem caused by the homogeneity of taxis.

[0121] In an embodiment of the present application, the double-delay deep deterministic policy gradient algorithm model comprises a current policy network, a target policy network, a current critic network and a target critic network, the current critic network comprises two parallel Q networks, and the target critic network comprises two parallel target Q networks. The current policy network and the target policy network have the same structure. The Q network and the target Q network have the same structure, and the target policy network has a slower update speed to ensure the stability of the training process. By calculating the Q value in parallel and selecting the minimum value, the overestimation bias of the Q value can be reduced, and the stability of the training can be improved.

[0122] In the embodiment, when constructing the second multi-agent reinforcement learning model of the charging station level, the double-delay deep deterministic policy model is improved, the Transformer encoder is fused with the double-delay deep deterministic policy gradient algorithm model, and intelligent dynamic pricing of taxi charging is realized.

[0123] As shown in an embodiment of the present application, Figure 3 The improved double-delay deep deterministic policy model outputs the charging action of the charging station agent based on the state information at the current moment and the historical state information, and the charging action comprises:

[0124] In S301, historical state information of a plurality of historical time steps before the current moment is obtained, the historical state information is input into the Transformer encoder for feature extraction, and historical feature information is obtained.

[0125] In S302, the historical feature information and the state information at the current moment are spliced to obtain enhanced state information, and the enhanced state information is input into the current policy network.

[0126] In S303, the current policy network calculates and generates the corresponding charging action based on the enhanced state information.

[0127] In S304, the charging action is executed to interact with the environment of the double-layer multi-agent reinforcement learning model.

[0128] In S305, the state information at the next moment is determined based on the result of the interaction, and the reward value obtained by executing the charging action is determined based on the reward function of the charging station agent.

[0129] In S306, the state information at the current moment, the charging action, the state information at the next moment and the reward value constitute an experience sample stored in the experience pool.

[0130] Exemplarily, the Transformer encoder can be a lightweight Transformer network composed of multi-head self-attention layers. The Transformer encoder includes two layers of encoding layers, and each layer of encoding layers includes a four-head self-attention layer, a feedforward neural network, and a residual connection, etc.

[0131] In the prior art, the current state information is directly used as the input of the policy network, which ignores the time correlation of variables such as price, load, and queue. However, charging station pricing is essentially a time series decision problem, which needs to capture historical LMP sequences, historical site loads, vehicle arrival timing, historical pricing behavior, and charging queue length changes, etc. Therefore, in this example, a lightweight Transformer encoder is used to extract features from the historical state information, and the output historical feature information is spliced with the current state information to form enhanced state information, which is used as the state information input of the current policy network, so as to capture the long-term time dependence of historical charging prices and loads, and provide historical awareness for subsequent value estimation and policy decision.

[0132] In an embodiment of the present application, the improved double-delay deep deterministic policy model further includes an adversarial policy generator, and the method further includes training the improved double-delay deep deterministic policy model, and the training step includes:

[0133] randomly selecting an experience sample from the experience pool, assuming that the experience sample is identified as (state information s t , current charging action a t , current reward value r t , state information s t+1 of the next moment);

[0134] inputting the historical state information into the Transformer encoder for feature extraction to obtain historical feature information, splicing the historical feature information with the state information s t of the current moment to obtain enhanced state information;

[0135] the current policy network outputs a charging action a t based on the enhanced state information;

[0136] the adversarial policy generator adds adversarial disturbance to the charging action a t to generate the worst action, and determines an action after confrontation based on the charging action a t and the worst action;

[0137] two Q networks respectively calculate the estimated Q value of the action after confrontation, and the minimum value in the calculated estimated Q value is taken as the minimum estimated Q value;

[0138] The target policy network outputs a charging action a based on the state information s at the next time t+1 The target policy network outputs a charging action a based on the state information s at the next time t+1 ;

[0139] The two target Q networks respectively calculate the target Q value of the charging action a at the next time t+1 The minimum value among the calculated target Q values is taken as the minimum target Q value, and the current reward value r t and the minimum target Q value are used to determine the target value y.

[0140] The mean square error between the evaluation target value and the minimum estimated Q value is minimized by using the gradient descent algorithm to update the parameters of the current critic network.

[0141] In a competitive charging market, the optimal strategy of each charging site depends on the behavior of other agents. In order to achieve a strong and adaptive strategy, an adversarial perturbation mechanism is adopted in this embodiment. By adding adversarial perturbation to the charging action through the adversarial strategy generator, the worst action is obtained by gradient stepping in the direction of minimizing the Q value of the charging station agent, to simulate the worst response of other agents.

[0142] The adversarial action a' is expressed as:

[0143] ;

[0144] wherein, represents the charging action output by the current policy network of the i-th charging station agent based on the state information s, represents the perturbation strength, represents the parameters of the Transformer encoder, represents the action value output by the current critic network based on the adversarial action a' for evaluation, represents the derivative, and represents the gradient of the current critic network with respect to the charging action, which is used to construct the worst action direction and can generate the worst action. The current critic network inputs the adversarial action a' for evaluation and outputs the estimated Q value, which takes into account the influence of the adversarial perturbation on the action value, to enhance the robustness of the strategy.

[0145] The target value y is expressed as follows:

[0146] ;

[0147] wherein, represents the reward value, represents the discount factor, represents the output of the i-th target critic network, represents the state at time t+1,​ Indicates the target policy network in state The given action, Indicates noise. Indicates a Gaussian distribution. It represents the standard deviation.

[0148] For example, the error between the target value and the minimum estimated Q value is the loss function. Represented as:

[0149] ;

[0150] in, This represents the parameters of the current critic network, where B represents the experience pool. Indicates the current critic network (parameter is) For the state is Actions The minimum estimated Q value.

[0151] In one embodiment of this application, the training further includes:

[0152] After updating the parameters of the current critic network several times, update the parameters of the current policy network, including:

[0153] The current policy network outputs charging action a1 based on state information s1;

[0154] The adversarial strategy generator adds adversarial perturbation to charging action a1 to generate the worst action, and determines the action after adversarial action based on charging action a1 and the worst action.

[0155] Two Q-networks are used to calculate the estimated Q-values ​​of the actions after the adversarial process, and the minimum of the calculated estimated Q-values ​​is taken as the minimum estimated Q-value.

[0156] The policy gradient is determined based on the gradient of the minimum estimated Q value and the gradient of the current policy network. The parameters of the current policy network are then updated based on the policy gradient in the direction of gradient ascent.

[0157] For example, the policy gradient is represented as:

[0158] ;

[0159] in, This represents the loss function of the current critic network. The parameters of the current policy network for the i-th charging station agent are represented. Indicates in the parameter The current policy of the policy network. This indicates that the current critic network evaluates the value of the output actions based on the actions after the adversarial process. It can be understood that the action value is derived near the "worst action".

[0160] In this embodiment, after updating the parameters of the current critic network, the policy is pressured by the adversarial action in the "adversarial environment", so that the policy still tries to improve the action value in the worst case, and forces the current policy network to learn to still have the optimal policy under the adversarial response, so that the policy has robustness to the worst competitive behavior.

[0161] Exemplarily, the parameters of the target policy network and the parameters of the target critic network are updated in a soft update manner, and the formula is as follows:

[0162] ;

[0163] ;

[0164] wherein, represents the parameters of the i-th current critic network, represents the parameters of the i-th target critic network, represents the parameters of the i-th current policy network, represents the parameters of the i-th target policy network, represents a soft update coefficient.

[0165] Exemplarily, in order to prevent oscillation in value function estimation, a Wasserstein smoothing regularization term is introduced during training. This term constrains the distribution distance between samples, maintaining the continuity of the joint value function.

[0166] In an embodiment of the present application, the chain rule is used to derive the parameters of the Transformer encoder and the parameters of the current critic network, and the gradient descent algorithm is used to update the parameters of the Transformer encoder and the parameters of the current critic network synchronously, to ensure that the feature learning of the Transformer encoder and the value estimation are synchronized to converge, and the parameters of the Transformer encoder and the parameters of the current critic network are updated end-to-end.

[0167] Exemplarily, the chain formula is as follows:

[0168] ;

[0169] wherein, represents the gradient of the output of the Transformer encoder with respect to its parameters , represents the gradient of the loss function of the current critic network with respect to the historical features , parameters of the Transformer encoder.

[0170] Exemplarily, the overall optimization objective of the charging station agent is represented as:

[0171] ;

[0172] In the above formula, the first term represents the minimization of the value estimation error, and the second term represents the maximization of the expected return of the strategy, so that the overall optimization objective of the charging station agent becomes a minimax game. The taxi agent and the charging station agent are trained through an alternating training mechanism, in which, in each training cycle, one group of agents optimizes its strategy, while the strategy of the other group remains fixed. This mechanism decouples the learning process, alleviates the non-stationarity problem in simultaneous training, and avoids the convergence problem. Through alternating optimization of scheduling and pricing, the framework enables the two agent groups to adaptively co-adapt stably, gradually discover better joint strategies, and ultimately achieve better overall system performance.

[0173] In this embodiment, by fusing the Transformer encoder and the double-delay deep deterministic policy gradient algorithm model, the long-term dependence of historical prices and loads is captured, and deep deterministic policy optimization with the properties of historical perception, robustness and adversarialness is realized in the multi-agent electric vehicle charging pricing scenario, so that the charging station agent can learn a more stable and robust pricing strategy when facing uncertain load fluctuations, dynamic electricity prices and multi-agent interaction, and realize the robustness and dynamic adjustment capability of charging pricing.

[0174] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement any of the above methods for jointly determining a taxi scheduling strategy and a charging station pricing strategy.

[0175] Figure 4 Fig. 1 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. Figure 4 The electronic device shown in the figure comprises a processor 401, a communication interface 402, a memory 403 and a communication bus 404. The processor 1401, the communication interface 402 and the memory 403 complete communication with each other through the communication bus 404. Among them, Figure 4 The connection mode between the processor 401, the communication interface 402 and the memory 403 shown in the figure is only exemplary. In the implementation process, the processor 401, the communication interface 402 and the memory 403 can also be connected with each other in communication through other connection modes other than the communication bus 404.

[0176] The storage 403 can be configured to store computer programs, which can include instructions and data, to implement the steps of any of the above methods for jointly determining a taxi dispatching strategy and a charging station pricing strategy. In embodiments of the present application, the storage 403 can be various types of storage media, such as random access memory (RAM), read only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, and registers, etc. The storage 403 can include a hard disk and / or a memory.

[0177] The processor 401 can be a general-purpose processor, which can be a processor configured to perform specific steps and / or operations by reading and executing computer programs (e.g., computer programs) stored in a storage (e.g., the storage 403), and can use data stored in the storage (e.g., the storage 403) in the process of executing the steps and / or operations. The general-purpose processor can be, for example but not limited to, a central processing unit (CPU). In addition, the processor 401 can also be a special-purpose processor, which can be a processor specially designed to perform specific steps and / or operations. The special-purpose processor can be, for example but not limited to, ASIC and FPGA, etc. In addition, the processor 401 can also be a combination of multiple processors, such as a multi-core processor.

[0178] The communication interface 402 can include input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device, and interfaces for interconnecting the network device with other devices (e.g., network devices). The communication network can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 402 can be a module, a circuit, a transceiver, or any device capable of communication.

[0179] In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 401 or the instructions in the form of software. The method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution completion, or executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field, such as random access memory flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory 403, and the processor 401 reads the information in the memory 403, and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0180] Although the preferred embodiments of the present application have been disclosed for illustrative purposes, those of ordinary skill in the art will realize that various modifications, additions and substitutions are possible without departing from the scope and spirit of the present application disclosed by the appended claims.

Claims

1. A method for jointly determining a taxi dispatching policy and a charging station pricing policy, characterized in that, The method comprises: building a double-layer multi-agent reinforcement learning model comprising a first multi-agent reinforcement learning model and a second multi-agent reinforcement learning model, each agent in the first multi-agent reinforcement learning model being a taxi agent, each agent in the second multi-agent reinforcement learning model being a charging station agent, each taxi agent being provided with a corresponding policy network, and each charging station agent being provided with a corresponding improved double-delay deep deterministic policy model, the improved double-delay deep deterministic policy model being a fusion model of a Transformer encoder and a double-delay deep deterministic policy gradient algorithm model; obtaining state information of the taxi agents and the charging station agents at a current time when the taxi agents and the charging station agents interact with an environment, and inputting the state information at the current time into the double-layer multi-agent reinforcement learning model, wherein the state information comprises global vehicle order information, state information of the taxi agents, and state information of the charging station agents; the policy network calculates and generates a dispatch action of the taxi agent based on the state information at the current time, and determines a dispatch strategy of the taxi agent based on the dispatch action, wherein the dispatch strategy comprises a taxi agent that is not assigned an order and a taxi agent that is not fully loaded and is assigned to a target order; the improved double-delay deep deterministic policy model calculates and generates a charging action of the charging station agent based on the state information at the current time and historical state information, and determines a charging strategy of the charging station agent based on the charging action, wherein the charging strategy comprises a charging price of the charging station agent and a charging power allocated to each taxi. 2.The method of jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 1, wherein, Building a double-layer multi-agent reinforcement learning model comprising a first multi-agent reinforcement learning model and a second multi-agent reinforcement learning model comprises: establishing a Markov decision process, setting an agent set, a state space, a joint action space, a reward function, a state transition function, and a discount factor of the Markov decision process, wherein, the state space is a set of state information, the state information comprising global vehicle order information, state information of each taxi agent, and state information of a charging station agent, the global vehicle order information comprising order information to be processed and order information being processed of all taxi agents, the state information of each taxi agent comprising a current position, a remaining seat capacity, a cumulative reward value, a battery level, and an accepted order, and the state information of the charging station agent comprising a number of vehicles waiting to be charged, a number of vehicles being charged, a charging price of each charging station, and a charging demand of each taxi; the joint action space comprises a dispatch action of a taxi agent and a charging action of a charging station agent, the dispatch action of the taxi agent comprising the taxi agent not being assigned an order and a taxi agent that is not fully loaded and is assigned to a target order, and the charging action of the charging station agent comprising a charging price of the charging station agent and a charging power allocated to each taxi agent; the policy network calculates and generates a dispatch action of the taxi agent based on the state information at the current time, and determines a dispatch strategy of the taxi agent based on the dispatch action, wherein the dispatch strategy comprises a taxi agent that is not assigned an order and a taxi agent that is not fully loaded and is assigned to a target order; the improved double-delay deep deterministic policy model calculates and generates a charging action of the charging station agent based on the state information at the current time and historical state information, and determines a charging strategy of the charging station agent based on the charging action, wherein the charging strategy comprises a charging price of the charging station agent and a charging power allocated to each taxi. The reward function includes a reward function of a taxi agent and a reward function of a charging station agent, and the reward function of the charging station agent is determined based on a difference between an obtained charging cost and a power grid cost. 3.The method of jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 1, wherein, The policy network outputs a scheduling action of the taxi agent based on the state information at the current time, including: Inputting the state information at the current time into the policy network, the policy network calculates and generates a scheduling action of the taxi agent; Running the scheduling action in the environment to interact with the double-layer multi-agent reinforcement learning model environment; Based on the result of the interaction, the state information at the next time is obtained, and the reward value obtained by executing the scheduling action is determined based on the reward function of the taxi agent; Determine the advantage function based on the reward value, the reward mean value, the reward standard deviation and the carpooling timeout penalty at the current time, and determine the optimization objective function of the policy network based on the advantage function; Based on the advantage function, calculate the policy gradient, and update the parameters of the policy network according to the policy gradient, so that the optimization objective function is optimized in a predetermined direction.

4. The method for jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 3, wherein, Determine the advantage function based on the reward value, the reward mean value, the reward standard deviation and the carpooling timeout penalty, and determine the optimization objective function of the policy network based on the advantage function, including: Advantage function is represented as: ； ； ； ； wherein, represents a taxi agent i in state down-scheduling action the obtained reward value, represents the reward mean value at the current time, represents the reward standard deviation at the current time, represents the total number of taxi agents at the current time, represents the abbreviation of standard deviation, represents the carpooling timeout penalty of taxi agent i, represents the carpooling combination of taxi agent i, represents the average extra time caused by taxi agent i carpooling, represents the maximum tolerance value of the average extra time caused by carpooling, represents a hyperparameter; Optimization objective function is represented as: ； wherein, represents the desired, represents the probability that the policy network outputs a scheduling action, represents a regularization coefficient, represents a parameter Mean squared error before and after update.

5. The method for jointly determining taxi dispatching policy and charging station pricing policy according to claim 1, wherein, The double-delay deep deterministic policy gradient algorithm model includes a current policy network, a target policy network, a current critic network and a target critic network, the current critic network includes two Q networks, the target critic network includes two target Q networks, the improved double-delay deep deterministic policy model outputs a charging action of the charging station based on the state information at the current time and the historical state information, including: Obtain historical state information at a plurality of historical time steps before the current time; Input the historical state information into the Transformer encoder for feature extraction to obtain historical feature information; Concatenate the historical feature information and the state information at the current time to obtain enhanced state information, and input the enhanced state information into the current policy network; The current policy network calculates and generates a corresponding charging action based on the enhanced state information; Execute the charging action to interact with the environment of the double-layer multi-agent reinforcement learning model; Determine the state information at the next time based on the result of the interaction, and determine the reward value obtained by executing the charging action based on the reward function of the charging station agent; The current state information, charging action, next state information and reward value constitute an experience sample and are stored in an experience pool.

6. The method for jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 5, wherein, The improved double-delay deep deterministic policy model also includes an adversarial policy generator, and the method further includes training the improved double-delay deep deterministic policy model, and the training includes: randomly selecting an experience sample from the experience pool, assuming that the experience sample is identified as (state information s t , current charging action a t , current reward value r t , state information s t+1 of the next moment) The historical state information is input into the Transformer encoder for feature extraction to obtain historical feature information, and the historical feature information and the state information s of the current moment are input into the Transformer decoder to obtain enhanced state information s t The splicing is performed to obtain the enhanced state information; The current policy network outputs a charging action a based on the augmented state information t ; The counter strategy generator generates a worst action based on the charging action a t The counter strategy generator generates a worst action based on the charging action a t The counter strategy generator generates a worst action based on the charging action a The two Q networks calculate the estimated Q value of the adversarial action respectively, and the minimum value in the calculated estimated Q value is taken as the minimum estimated Q value; The target policy network is based on the state information s of the next time t+1 Output the charging action a of the next time t+1 ; The two target Q networks respectively calculate target Q values of charging actions a at a next time t+1 The minimum target Q value is determined from the calculated target Q values, and the target value is determined from the current reward value r t and the minimum target Q value. The parameters of the current critic network are updated after several times of updating the parameters of the current critic network, including:

7. The method for jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 6, wherein, The training further includes: After several times of updating the parameters of the current critic network, the parameters of the current policy network are updated, including: The charging action a1 output by the current policy network based on the state information s1; The adversarial policy generator generates a worst action by adding an adversarial disturbance to the charging action a1, and determines an adversarial action based on the charging action a1 and the worst action; The two Q networks respectively calculate the estimated Q values of the adversarial action, and take the minimum value in the calculated estimated Q values as the minimum estimated Q value; A policy gradient is determined based on the gradient of the minimum estimated Q value and the gradient of the current policy network, and the parameters of the current policy network are updated in the direction of gradient ascent based on the policy gradient.

8. The method for jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 6, wherein, The adversarial policy generator generates the charging action a t Joining the adversarial disturbance, output the action after the confrontation, including: Actions after the confrontation is represented as: ； wherein, represents a charging action output by the current policy network of the ith charging station agent based on the state information, represents a disturbance intensity, represents a parameter of a Transformer encoder, represents an action value output by the current critic network based on the action after the confrontation, represents derivation.

9. The method for jointly determining a taxi dispatching policy and a charging station pricing policy according to claim 6, wherein, The training includes: The parameters of the Transformer encoder and the parameters of the current critic network are derived using the chain rule, and the parameters of the Transformer encoder and the parameters of the current critic network are updated simultaneously using a gradient descent algorithm.

10. An electronic device comprising a memory and a processor, characterized in that The memory stores a computer program, and the processor executes the computer program to implement the steps of the method for jointly determining a taxi dispatching strategy and a charging station pricing strategy according to any one of claims 1 to 9.

Images