Method and apparatus for joint dispatching and charging of a fleet of autonomous taxis

Abstract

The present application relates to the technical field of automatic driving and intelligent traffic management, in particular to a method and device for joint scheduling and charging of an unmanned taxi fleet. The method comprises: obtaining current state information of the system, and establishing a decision target with the goal of maximizing platform revenue; converting the decision target into a Markov decision process, and establishing a composite reward function for multi-objective optimization; constructing a bipartite graph for joint scheduling and charging, and using a reinforcement learning model to obtain dynamic weights of the composite reward function to calculate the comprehensive weights of each edge in the bipartite graph, matching the bipartite graph to obtain the current optimal matching decision and executing it. The present application solves the problems of vehicle supply and demand imbalance, charging station congestion and low long-term system revenue caused by the separation of order scheduling and vehicle charging decision in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving and intelligent traffic management, in particular to a joint order scheduling and battery charging decision method and device for electric unmanned taxis based on multi-agent deep reinforcement learning. BACKGROUND

[0002] With the development of automatic driving technology, unmanned taxis are gradually replacing traditional human-driven taxis (HDTs). Unlike HDTs, unmanned taxis are centrally controlled by the platform, which improves scheduling efficiency but introduces a new challenge: the coupling problem of charging and scheduling. As electric vehicles, the state of charge (SoC) of unmanned taxis must be actively managed by the platform. When the SoC is below a threshold (usually 20%), the vehicle needs to go to a charging station (CS) for charging.

[0003] Existing technologies usually separate scheduling and charging decisions: first, assign orders based on immediate revenue (such as the KM algorithm), and then assign charging stations for low-battery vehicles. This decoupled decision-making approach has significant drawbacks: first, it is short-sighted, maximizing immediate revenue while ignoring the impact of charging behavior on the availability of vehicles for future orders, which undermines long-term revenue. Second, the imbalance between supply and demand is exacerbated. Order scheduling consumes electricity, affecting charging demand; charging behavior occupies vehicle time, affecting order supply. This easily leads to a large number of vehicles requiring charging at the same time, causing charging station congestion (charging side supply and demand imbalance), while the reduction in available vehicles due to vehicle charging exacerbates the order side supply and demand imbalance. Therefore, there is an urgent need for a method that can optimize scheduling and charging decisions together to maximize the long-term overall revenue of the system.

[0004] In related prior art, such as Chinese patent document CN117035369A, an intelligent scheduling method for unmanned vehicle resources is disclosed. First, the state parameters of the unmanned vehicle are obtained; the state correction parameters of the unmanned vehicle are obtained based on the state parameters of the unmanned vehicle, the battery energy loss coefficient of the unmanned vehicle is calculated based on the state correction parameters of the unmanned vehicle, the resource scheduling priority graph of the unmanned vehicle is obtained based on the state correction parameters of the unmanned vehicle, the charging preemption coefficient of the unmanned vehicle is calculated based on the resource scheduling priority graph of the unmanned vehicle, and the priority weight of the unmanned vehicle is calculated based on the battery energy loss coefficient of the unmanned vehicle and the resource scheduling priority graph of the unmanned vehicle. The best charging area is obtained based on the priority weight of the unmanned vehicle, and then the best charging area is used for resource scheduling of the unmanned vehicle. This method improves the rationality of resource scheduling.

[0005] And, Chinese patent document CN120764986A discloses an intelligent scheduling method and device for unmanned taxis, wherein the method comprises: based on received vehicle data, passenger data and environmental data, constructing a spatial topology structure for connecting target unmanned taxis, passengers and charging stations; capturing the spatial dependency relationship between target unmanned taxi nodes, passenger nodes and charging station nodes to generate spatial coding features and time coding features and fuse them in combination with corresponding weights to generate spatiotemporal features of the target unmanned taxi, so as to generate passenger matching results and / or path planning and / or charging decisions of the target unmanned taxi based on the spatiotemporal features.

[0006] Therefore, the present application provides a method for joint order scheduling and battery charging decision of electric unmanned taxis based on multi-agent deep reinforcement learning, to solve the problems of vehicle supply and demand imbalance, charging station congestion and low long-term system revenue caused by the separation of order scheduling and vehicle charging decision in the prior art. SUMMARY

[0007] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and provides a method for joint scheduling and charging of an unmanned taxi fleet, which realizes joint decision of scheduling and charging of the unmanned taxi fleet through multi-agent deep reinforcement learning and physically inspired coding mechanisms, thereby maximizing the long-term comprehensive revenue of the platform.

[0008] The present application also discloses a device for joint scheduling and charging of an unmanned taxi fleet.

[0009] The detailed technical solutions of the present application are as follows:

[0010] A method for joint scheduling and charging of an unmanned taxi fleet is applied to an unmanned taxi fleet scheduling and charging management system, and the method comprises:

[0011] Obtaining current state information of the system and establishing a decision target aiming at maximizing the revenue of the platform;

[0012] Converting the decision target into a Markov decision process and establishing a composite reward function for multi-objective optimization;

[0013] Constructing a bipartite graph for joint scheduling and charging, and using a reinforcement learning model to obtain dynamic weights of the composite reward function to calculate comprehensive weights of each edge in the bipartite graph, matching the bipartite graph to obtain a current optimal matching decision and executing it.

[0014] According to the present application, the established decision target is preferably:

[0015]

[0016] In Equation (1): is an order set or a charging station set , and refers to an order or a charging station ; is a binary decision variable, =1 indicates that a taxi goes to an order or a charging station ; represents the platform revenue at time t, and the revenue at all times is accumulated by ; and:

[0017]

[0018] In Equation (2): represents the income of a taxi completing an order; represents the expenditure of a taxi charging at a charging station; and:

[0019]

[0020] In Equation (3): represents the income of a taxi completing an order ; represents the expenditure of a taxi charging at a charging station ; represents the total distance of the estimated travel path of an order; represents the unit price of the order service; represents the percentage value of the remaining power of the power battery of a taxi at the current time; represents the standard total capacity of the power battery of a taxi; represents the charging price;

[0021] And / or, the constraint conditions of the decision objective include:

[0022] Vehicle power constraints for ensuring that a taxi has sufficient power to complete an order and safely arrive at a charging station:

[0023]

[0024] In Equation (4): represents the vehicle energy consumption rate; represents the distance from the current location of a taxi to the starting point of an order o; represents the minimum safe power;

[0025] The passenger can only be in the state of charging or service order, and cannot receive multiple orders at the same time when the service order is in the constraint:

[0026]

[0027] In formula (5): represents the taxi In the matching decision of all charging stations, there can be at most one value of 1, that is, charging at one charging station; represents the taxi In the matching decision of all orders, there can be at most one value of 1, that is, providing service for one order.

[0028] According to the present application, the compound reward function of the multi-objective optimization established is preferably:

[0029]

[0030] In formula (9): is an element in the order set or the charging station set , which refers to the order or the charging station ; represents the platform revenue at time t; represents the battery power consumed by the taxi to execute the current order; represents the long-term charging revenue; , , is the weight parameter of each component; and:

[0031]

[0032]

[0033] In formula (6), (7): represents the vehicle energy consumption rate; represents the charging price; represents the total distance of the estimated travel path of the order; represents the distance from the current position of the taxi to the starting point of the order o; represents the percentage value of the remaining power of the power battery of the taxi at the current time; represents the standard total capacity of the power battery of the taxi; represents the unit price of the power purchased by the platform from the power grid; represents the taxi at the charging station charging time; for the penalty weight, indicating the taxi the degree of influence of charging; and:

[0034]

[0035] in formula (8): indicating the taxi arriving at the charging station driving time; indicating the taxi time required for charging; indicating the taxi charging queue time;

[0036] the weight parameters of each component , , the sum is constrained to be 1, that is:

[0037] .

[0038] According to the application, the method further comprises estimating the expected driving time required by the taxi to serve the order:

[0039]

[0040] in formula (11): indicating the expected driving time required by the taxi to serve the order, which is used to update the set of unmanned taxis available at each time.

[0041] According to the application, the dynamic weight of the compound reward function is obtained by using a reinforcement learning model, which specifically comprises:

[0042] Based on the Coulomb force model, the current state information of the system is encoded into a first Coulomb force vector between the taxi and the order and a second Coulomb force vector between the taxi and the charging station;

[0043] Based on the first Coulomb force vector and the second Coulomb force vector, the observation state of the reinforcement learning model is constructed, the dynamic weight of the compound reward function is taken as the action of each taxi in the reinforcement learning model, and the value calculated according to the compound reward function after bipartite graph matching is taken as the reward of the reinforcement learning model;

[0044] The reinforcement learning model is trained in a loop, and finally outputs the dynamic weight parameter of each taxi in the compound reward function.

[0045] According to the application, the system current state information is encoded as a first Coulomb force vector between the taxi and the order and a second Coulomb force vector between the taxi and the charging station, specifically including:

[0046] Based on the system current state information, estimate the driver-passenger matching side supply-demand gap information and the charging side supply-demand gap information; wherein the calculation of the driver-passenger matching side supply-demand gap information is:

[0047]

[0048] In formula (12): represents the area In the current time, the driver-passenger matching side supply-demand gap information; represents the area The total number of all pending orders within the area; represents the area The total number of all available taxis within the area;

[0049] The calculation of the charging side supply-demand gap information is:

[0050]

[0051] In formula (13): represents the area In the current time, the charging side supply-demand gap information; represents the set of charging stations; represents the charging station The number of all charging pile devices contained within the charging station; represents the charging station The ordered sequence information of the vehicles currently queuing for charging in the charging station; represents the taxi The percentage value of the remaining power of the power battery at the current time; represents the standard total capacity of the taxi power battery; represents the charging station The rated output charging power value of the charging pile n in the charging station.

[0052] According to the application, the system current state information is encoded as a first Coulomb force vector between the taxi and the order and a second Coulomb force vector between the taxi and the charging station, specifically including:

[0053] Based on the current state information of the system, and the supply-demand gap information of the driver-passenger matching side and the charging side, a first Coulomb force vector between the taxi and the order is obtained by using an order encoder, and a second Coulomb force vector between the taxi and the charging station is obtained by using a charging station encoder;

[0054] The first Coulomb force vector between the taxi and the order obtained by using the order encoder is specifically:

[0055] The order encoder maps its input to a Gaussian distribution , and samples the latent representation of the charge quantity product between the taxi v and the order o from the distribution ;

[0056] The first Coulomb force vector between the taxi and the order is calculated based on the Coulomb's law formula;

[0057] The second Coulomb force vector between the taxi and the charging station obtained by using the charging station encoder is specifically:

[0058] The charging station encoder maps its input to a Gaussian distribution , and outputs the latent representation of the charge quantity product between the taxi v and the charging station ; ;

[0059] The second Coulomb force vector between the taxi and the charging station is calculated based on the Coulomb's law formula;

[0060] Wherein, represents the feature vector of the taxi at time t; represents the feature vector of the order ; represents the mean; represents the variance.

[0061] According to the application, the observation state of the constructed reinforcement learning model is:

[0062]

[0063] In formula (16): represents the observation state; represents the first Coulomb force vector set of the nearest orders near the geographic position of the taxi ; represents the second Coulomb force vector set of the nearest charging stations near the geographic position of the taxi ; ​​

[0064] The action of each taxi in the reinforcement learning model is represented as:

[0065]

[0066] In formula (17): represents the dynamic weight of the composite reward function, representing the action of the taxi.

[0067] According to the present application, the reinforcement learning model is preferably trained using a multi-agent soft actor-critic framework, specifically including:

[0068] An actor network, a critic network, and a joint optimization target of the actor network and the critic network are created, and the joint optimization target is:

[0069]

[0070] In formula (18): represents the optimization target; represents the policy; represents the system expectation under the observation state and the action ; represents the long-term discount coefficient; is the weight coefficient of the entropy regularization term; represents the entropy of the policy under the observation state ;

[0071] Enter the loop training: at each training step, after the environment is reset, each agent outputs an action according to the current local observation state through the actor network, the platform executes joint decision, the environment state is transferred, and the reward of each agent is calculated;

[0072] Enter the soft update of the reinforcement learning model:

[0073] The constructed critic network includes a soft Q-value network and a soft V-value network , wherein the input of the soft Q-value network is , and the output is the soft Q-value; the input of the soft V-value network is , and the output is the soft V-value; the critic network updates the parameters by minimizing the soft Bellman residual according to the current observation state , the action , and the target value of the next state, and the objective function is:

[0074]

[0075] In formula (19): Objective function representing critic network Expectation Discount factor Taxi Observation state at t+1

[0076] The input of the actor network constructed is , and the output is , and the exploration is carried out by adding Gaussian noise , and finally the random action subject to the strategy is output; the actor network updates the strategy parameters by maximizing the weighted sum of the expected return and the strategy entropy according to the function calculated by the critic network:

[0077]

[0078] In formula (20): Objective function representing actor network Objective function representing Divergence System strategy with network parameters Normalizing constant

[0079] In another aspect of the present application, a device for joint scheduling and charging of a fleet of unmanned taxis is provided, which is loaded with the method for joint scheduling and charging of a fleet of unmanned taxis as described above, and the device comprises:

[0080] A data acquisition module is configured to acquire current state information of the system and establish a decision objective with the objective of maximizing platform revenue;

[0081] An objective construction module is configured to convert the decision objective into a Markov decision process and establish a composite reward function for multi-objective optimization;

[0082] A decision generation module is configured to construct a bipartite graph for joint scheduling and charging, and to obtain dynamic weights of the composite reward function by using a reinforcement learning model, so as to calculate comprehensive weights of edges in the bipartite graph, match the bipartite graph, obtain a current optimal matching decision, and execute the current optimal matching decision.

[0083] Compared with the prior art, the present application has the following advantages:

[0084] ​The application provides a method for joint scheduling and charging of an unmanned taxi fleet, establishes a decision target for maximizing platform revenue, and converts it into a Markov decision process. The dynamic weight parameters of the compound reward function based on multi-objective optimization are calculated using deep reinforcement learning. Then, based on the dynamic weight parameters, the comprehensive weight of each edge in the bipartite graph of joint scheduling and charging is calculated, and the bipartite graph is matched to obtain the optimal matching decision at the current time. Finally, the platform converts the optimal matching decision into corresponding control instructions and executes the decision scheme. This method realizes collaborative decision-making of scheduling and charging, improves the service rate of the platform, and maximizes the long-term revenue of the system. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 is a flowchart of the method for joint scheduling and charging of an unmanned taxi fleet according to the application.

[0086] Figure 2 is a flowchart of the Coulomb force coding algorithm used in embodiment 1 of the application.

[0087] Figure 3 is a network structure diagram of the reinforcement learning model based on multi-agent soft actor-critic used in embodiment 1 of the application. DETAILED DESCRIPTION

[0088] The application will be further described below in conjunction with the drawings and embodiments.

[0089] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0090] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0091] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0092] Embodiment 1,

[0093] Reference Figure 1 The embodiment provides a method for joint scheduling and charging of an unmanned taxi fleet, which is applied to an unmanned taxi fleet scheduling and charging management system, and the method comprises the following steps:

[0094] S1, acquire system current state information, and establish a decision goal of maximizing platform revenue.

[0095] In this embodiment, the system state information can include unmanned taxi state feature information, order feature information, and charging station feature information.

[0096] Let the set composed of all taxi vehicles at time t be denoted as unmanned taxi set , which contains the taxis The feature vector at time t is denoted as . Wherein, represents the geographic area location information of the taxi at the current time; represents the average cruising speed of the taxi ; represents the standard total capacity of the power battery of the taxi; represents the percentage value of the remaining power of the power battery of the taxi at the current time, usually, there is a minimum threshold value that must be maintained to ensure sufficient energy to reach the charging station for cruising; represents the operating mode state of the taxi at the current time.

[0097] It should be understood that, = 1 indicates that the taxi is currently serving an order, and this state will not change until the passenger arrives at the destination; = -1 indicates that the taxi is currently scheduled to be charged, which can be on the way to the charging station (CS), in the charging queue, or being charged, in which case the state will not change until it is fully charged; = 0 indicates that the taxi is currently in an idle driving state, which is neither charging nor serving an order, and will go to an area where there are dense orders or excess charging resources in the future according to the supply and demand prediction results.

[0098] Let the set composed of all orders be denoted as order set , which contains the orders The feature vector of the order is denoted as . Wherein, represents the starting area location information of the passenger pickup; represents the end area location information of the passenger drop-off; represents the total distance of the estimated driving path of the order; a billing unit price representing an order service.

[0099] Let the set consisting of all charging stations be denoted as a charging station set , which contains charging stations . The feature vector of is denoted as . Wherein, represents the geographical location area information of the charging station ; represents the number of all charging pile devices contained in the charging station ; represents the rated output charging power value of the charging pile n in the charging station ; represents the ordered sequence information of vehicles currently queuing for charging in the charging station ;

[0100] Understandably, after obtaining the above system current state information, the data can be cleaned first, then divided according to the area, and then mapped in the regional grid after cleaning the data.

[0101] Based on the above state information, the embodiment establishes a decision goal of maximizing the platform revenue:

[0102]

[0103] In formula (1): is an element in the order set or the charging station set , which refers to an order or a charging station; is a binary decision variable, =1 indicates that the taxi goes to the order or the charging station ; represents the platform revenue at time t, and the revenues of all time are accumulated by ; and:

[0104]

[0105] In formula (2): represents the income of the taxi completing the order; represents the expenditure of the taxi charging; and:

[0106]

[0107] In formula (3): represents the income of the taxi completing the order ; Indicates taxi At the charging station Expenses related to charging; This indicates the electricity price for charging.

[0108] The following constraints must be met during this process:

[0109] 1) Vehicle battery power constraint:

[0110]

[0111] In equation (4): Indicates vehicle energy consumption rate; Indicates taxi The distance from the current location to the starting point of order o; Indicates the minimum safe power level;

[0112] This constraint is designed to ensure that taxis have sufficient battery power to complete the order and arrive safely at the charging station.

[0113] 2) Passengers can only be in a charging or service order state, and cannot accept multiple orders simultaneously while in a service order state, to ensure the logical rationality and safety of the operation:

[0114]

[0115] In equation (5): Indicates in taxi In all the decisions made by charging stations, there can only be at most one value of 1, that is, charging at one charging station; Indicates in taxi In all order matching decisions, there can only be at most one value of 1, which means that service is provided for one order.

[0116] S2. Transform the decision objective into a Markov decision process and establish a multi-objective optimization composite reward function.

[0117] Specifically, to overcome short-sighted decision-making caused by a single immediate gain objective, this embodiment designs a composite reward function that integrates three independent components—immediate gain, long-term cost, and long-term benefit—through a linear weighted sum. .

[0118] The first component is the immediate benefit item. Its calculation parameter is formula (2), which directly reflects the revenue obtained from executing the order or the cost of electricity paid for charging.

[0119] The second component is the long-term service cost item. :

[0120]

[0121] This component is activated only when the decision involves order assignment, and its core idea is to quantify the battery power consumed by executing the current order, which means that this part of power consumption implies the need for future charging to be replenished, so this item converts the future expected charging cost to the current decision to remind the system to pay attention to the negative impact of the current service on the future vehicle endurance.

[0122] The third component is the long-term charging benefit term

[0123]

[0124] In formula (7): denotes the taxi charging at the charging station time; denotes the penalty weight, indicating the impact of the taxi charging; and:

[0125]

[0126] In formula (8): denotes the driving time of the taxi to the charging station ; denotes the time required for the taxi to charge; denotes the time of the taxi charging queue.

[0127] This component is activated only when the decision involves charging scheduling, and this term evaluates the potential future value brought by the current charging behavior, including the expected revenue that can be generated by serving more future orders due to the recovery of battery power, while deducting the opportunity cost caused by the time-consuming charging process.

[0128] These three components are scaled by a dynamic weight parameter respectively:

[0129]

[0130] And the sum of the weight parameters of the three components is constrained to be 1, that is:

[0131]

[0132] By adjusting the size of these weights, the system can flexibly adjust the emphasis on the three goals in different operating environments.

[0133] Further, in the present embodiment, in order to update the set of unmanned taxis available for scheduling​ In addition to estimating the time required for a taxi to charge , the platform also needs to estimate the expected travel time required for a taxi to serve an order , to update the set of autonomous taxis available at each time :

[0134]

[0135] S3, construct a bipartite graph of joint scheduling and charging, and use a reinforcement learning model to obtain the dynamic weights of the composite reward function to calculate the comprehensive weights of each edge in the bipartite graph, match the bipartite graph, obtain the current optimal matching decision and execute.

[0136] In this embodiment, a bipartite graph of joint scheduling and charging is established , wherein, represents the charging piles of the charging station at the time the remaining available queuing positions; represents the set of edges of the bipartite graph. Specifically, the multiple charging piles of each charging station are split into multiple virtual nodes to ensure one-to-one matching, and a bipartite graph containing two disjoint vertex sets can be obtained: the first vertex set consists of all currently idle taxis that can be dispatched; the second vertex set consists of all currently available idle charging queue positions in all charging stations. By setting a distance threshold, only reasonable edges in space are retained, greatly reducing the size of the graph and improving the solving speed.

[0137] Based on the bipartite graph constructed above , this embodiment first uses a deep reinforcement learning model to obtain the dynamic weight parameters of the composite reward function, and then calculates the comprehensive weights of each edge in the bipartite graph according to the dynamic weight parameters.

[0138] Further, in this embodiment, each autonomous taxi is regarded as an agent, and a fixed-dimensional observation state is constructed for each idle taxi , the action of each agent is the weight parameter vector in the composite reward function, the reward is the value calculated according to the composite reward function after executing the bipartite graph matching, and through multiple rounds of reinforcement learning training, the dynamic weight parameter value of the composite reward function is finally obtained.

[0139] Specifically, the dynamic weights of the composite reward function are obtained using a reinforcement learning model, including:

[0140] S31. Based on the Coulomb force model, encode the current state information of the system into a first Coulomb force vector between a taxi and an order, and a second Coulomb force vector between the taxi and a charging station;

[0141] S32. Based on the first Coulomb force vector and the second Coulomb force vector, construct the observation state of the reinforcement learning model, use the dynamic weight of the composite reward function as the action of each taxi in the reinforcement learning model, and use the value calculated according to the composite reward function after performing bipartite graph matching as the reward of the reinforcement learning model;

[0142] S33. Continuously train the reinforcement learning model, and its final output is the dynamic weight parameter of each taxi in the composite reward function.

[0143] For Figure 2 , in S31 above, encoding the current state information of the system into a first Coulomb force vector between a taxi and an order, and a second Coulomb force vector between the taxi and a charging station specifically includes:

[0144] S311. Based on the current state information of the system, estimate the supply-demand gap information on the driver-passenger matching side and the supply-demand gap information on the charging side.

[0145] Specifically, taking the area as an example, the platform calculates the supply-demand gap information on the driver-passenger matching side in the current moment as: For:

[0146]

[0147] In formula (12): represents the total number of all pending orders in the area ; represents the total number of all available taxis in the area ;

[0148] The supply-demand gap information on the charging side in the current moment is: For:

[0149]

[0150] Furthermore, the calculation method of the supply-demand gap information is defined as the total number of all pending orders in this area minus the total number of all available taxis in this area , and the obtained value reflects this area The tension or abundance of the inner vehicle supply.

[0151] Understandably, Indicates that the area vehicle is in short supply, and the passenger waiting time may be prolonged, and the order cancellation risk is high; Indicates that the vehicle is in excess supply, and there may be empty vehicles, and the resource utilization rate is low. This index is the core driving force of scheduling decision.

[0152] Charging side supply-demand gap value The calculation comprehensively considers the total charging demand of all charging stations in the region The instantaneous charging supply capacity of all charging piles, indicating the tension of charging resources in the region .

[0153] Understandably, Indicates that the charging demand exceeds the current charging pile instant service capacity, and the charging station will be congested, and the vehicle waiting time will be longer; Indicates that there is a surplus of charging resources.

[0154] S312, based on the current state information of the system, and the supply-demand gap information on the driver-passenger matching side and the charging side, the first coulomb force vector between the taxi and the order is obtained by using the order encoder, and the second coulomb force vector between the taxi and the charging station is obtained by using the charging station encoder.

[0155] Specifically, a physical simulation method based on electrostatic Coulomb force principle is introduced, which maps different entities in the system to physical objects with different properties of electric charge, that is, the taxi is regarded as a positive charge, and the order and the charging station are regarded as a negative charge, aiming to simulate the basic behavior mode of the vehicle attracted by the order and the charging station.

[0156] Two variational autoencoders, namely order encoder and charging station encoder, are maintained. Both autoencoders are composed of 3 fully connected layers, using ReLU activation function in the middle. The autoencoder architecture is composed of multiple fully connected (FC) layers in the encoder and decoder components, which is trained in an unsupervised manner, and the training loss function is:

[0157]

[0158] In formula (14), the first term is the reconstruction loss, aiming to make the latent variable able to predict the supply-demand gap at the next moment; the second term represents the KL divergence loss under all orders and charging decisions, which is used to constrain the distribution of the latent variable to be close to the prior distribution.

[0159] For any combination of a taxi v and an order o, according to the taxi state, the order feature, the dispatch supply-demand gap and the charging supply-demand gap, we have As the input of the order encoder, it is mapped to a Gaussian distribution , and a latent representation of the charge amount product between the taxi v and the order o is sampled from the distribution .

[0160] The charging station encoder is similar, which is used to map As the input of the charging station encoder, it is mapped to a Gaussian distribution , and a latent representation of the charge amount product between the taxi v and the charging station . .

[0161] According to the Coulomb's law formula, the interaction force between the taxi and the order and the charging station is calculated respectively:

[0162]

[0163] In equation (15): represents the Coulomb force vector; is used to refer to or ; is used to refer to or ; is used to represent the Euclidean distance between the current position of the taxi v and the pickup point of the order o or the location of the charging station ; is the Coulomb constant; is used to represent the unit directional vector from the current position of the taxi v to the pickup point of the order o or to the location of the charging station .

[0164] The size of the Coulomb force vector is proportional to the value of the charge product carried between the vehicle and the order or the charging station, and inversely proportional to the square of the Euclidean distance between the current position of the vehicle and the pickup point of the order or the location of the charging station; its direction is defined by the unit directional vector from the current position of the vehicle to the pickup point of the order or the location of the charging station. After the system runs, new interaction data can be added to the training set to fine-tune the encoder online to adapt to changes in supply and demand patterns.

[0165] In the above S32, an observation state of a fixed dimension is constructed for each idle taxi , which includes the state parameters of the vehicle itself , the geographic location near a first set of coulomb force vectors of the recent orders and geographical positions near the a second set of coulomb force vectors of the recent charging stations

[0166]

[0167] The action of each agent is the weight parameter vector in its composite reward function:

[0168]

[0169] By adjusting the weights, the agent dynamically adjusts the decision strategy under different supply and demand environments, and realizes the trade-off strategy for the three strategic objectives of "pursuing immediate cash income", "controlling long-term operating cost" and "investing in future service capacity". Its reward .

[0170] Further Figure 3 In the above S33, the reinforcement learning model is trained using a multi-agent soft actor-critic framework, and the workflow sequence is as follows:

[0171] First, the framework is initialized, the actor network, critic network and corresponding target network are created, and the network parameters and experience replay pool are initialized. The joint optimization goal of the actor network and critic network is to maximize the expected cumulative soft return:

[0172]

[0173] In formula (18): represents the optimization goal; represents the strategy; represents the system expectation under the observation state and action . represents the long-term discount coefficient; is the weight coefficient of the entropy regularization term; represents the entropy of the strategy under the observation state .

[0174] Subsequently, enter the loop training phase: at each training step, after the environment is reset, each agent outputs an action according to the current local observation state through the actor network, the platform executes joint decision, the environment state is transferred, and the reward of each agent is calculated, i.e. the value calculated according to the composite reward function after the bipartite graph matching is performed.

[0175] Next, the soft update of the model is performed. The critic network includes a soft Q value network and a soft V value network ​where the soft Q-value network takes as input and outputs a soft Q-value; the soft V-value network takes as input and outputs a soft V-value. The critic network updates its parameters by minimizing the soft Bellman residual, which is defined as:

[0176]

[0177] In equation (19), we have: denotes the objective function of the critic network; denotes the expectation; denotes the discount factor; denotes the observation state of the taxi at time t+1.

[0178] The input of the actor network is the observation state , and the output is the weight parameter of the action, which is finally outputted as a random action following the policy by adding Gaussian noise for exploration. The actor network updates its policy parameters by maximizing the weighted sum of the expected return and the policy entropy, which is defined as:

[0179]

[0180] In equation (20), we have: denotes the objective function of the actor network; denotes the divergence; denotes the system policy with network parameters ; and denotes the normalization constant.

[0181] Finally, the target network parameters are slowly synchronized with the current network parameters in a soft manner to stabilize the training. The above process is repeated until the model policy converges or the pre-set training epoch is reached.

[0182] The model finally outputs the dynamic weight parameter of each taxi in the compound reward function.

[0183] Further, for the bipartite graph constructed above, a weight value is assigned to each connection edge, and the calculation of the weight value directly depends on the compound reward function . For the bipartite graph ​​The KM algorithm is used to solve for the maximum weighted matching problem. It iterates through initialization, row reduction, column reduction, cover check, and matrix adjustment until a complete matching is found, which is the optimal matching decision at the current time step. .

[0184] The platform will solve for the optimal matching decision. The proposed solution is translated into specific control commands and issued to the corresponding driverless taxi fleet dispatch and charging management system. Vehicles receiving an order will have their status updated to "Service in Progress" and will begin navigation to the passenger pick-up point. Vehicles receiving a charging task will have their status updated to "Charging in Progress" or "Heading to Charging Station" and will begin navigation to the designated charging station to join the queue.

[0185] After the command is issued, the platform synchronously updates the status identifiers of all vehicles, the processing status of all orders, and the queue information of all charging stations in the entire system, thereby completing all operations in the current decision-making cycle, advancing the system time, and preparing to enter the next decision-making cycle.

[0186] The effectiveness of the method of the present invention will be verified by using specific examples below.

[0187] Using Manhattan's road network data as an example, it can be divided into 63 irregular geographical areas based on the heat map distribution of order requests. 1500 driverless taxis are initialized in the system and randomly distributed across these areas. The key parameters for each vehicle are as follows:

[0188] taxi At present Geographical location of the time Initial random allocation;

[0189] taxi Average cruising speed : Follows a normal distribution km / h;

[0190] The standard total capacity of taxi power batteries (B) is uniformly 30.4 kWh (referencing a typical electric vehicle model).

[0191] taxi initial charge The data is uniformly and randomly distributed within the interval [30%, 80%] to simulate real-world operational scenarios.

[0192] taxi initial state Set all to 0 (idle state).

[0193] Charging network configuration: 50 charging stations will be configured within the area. Key parameters for each charging station are as follows:

[0194] Number of charging piles : Randomly distributed between [5, 15];

[0195] Rated output charging power value of charging pile equipment : Randomly assigned as 7kW (L1), 22kW (L2), or 50kW (L3);

[0196] Charging unit price : Set uniformly at 1.2 yuan / kWh;

[0197] Initial queue for queuing for charging : Initially empty.

[0198] The core physical parameters are as follows:

[0199] Vehicle energy consumption rate : Set to 0.2 kWh / km, representing the average level of urban driving;

[0200] Minimum safe power : Set to 20%, ensuring that the vehicle has enough power to reach the charging station.

[0201] The system platform collects the following state information:

[0202] Vehicle state: idle vehicle The state is , indicating that it is located in area 25, speed 58km / h, battery capacity 30.4kWh, remaining power 35%, and is in idle state.

[0203] Order information: new order The characteristics are , indicating from area 25 to area 30, distance 8.5 kilometers, unit price 3.5 yuan / km.

[0204] The immediate income of this order is yuan.

[0205] Charging station information: charging station near the vehicle has an idle pile with a power of kW.

[0206] Charging cost: yuan.

[0207] Calculate the supply-demand gap of area 25:

[0208] , in short supply;

[0209] kWh, the charging demand is greater than the supply.

[0210] For vehicles and orders , the order encoder inputs .

[0211] The order encoder outputs , sampling gets , and negative values represent attractiveness.

[0212] Calculate the coulomb force vector between vehicles and orders . , assuming the distance km, then , the size and direction of the force are used to construct the state.

[0213] Similarly, calculate the coulomb force vector between vehicles and nearby charging stations .

[0214] Finally, construct the local observation state for the vehicle :

[0215] ;

[0216] It contains its own state, the coulomb force information of the nearby 2 charging stations and 6 orders.

[0217] Based on the above, the actor network outputs dynamic weight parameters according to the state :

[0218] ;

[0219] This set of weights indicates that in the case of current tight supply and demand but the vehicle The power is low (35%), the system is pursuing immediate income (weight 0.5) while also paying considerable attention to the long-term power consumption cost brought by service (weight 0.4), and the immediate charging income (weight 0.1) is not prioritized.

[0220] Build a bipartite graph : its first vertex set contains all idle vehicles (such as ), and the second vertex set contains all orders (such as ) and all idle charging piles.

[0221] Calculate the edge weight: for edge , the comprehensive reward is:

[0222] ​

[0223] ;

[0224] For edge , the comprehensive reward is:

[0225]

[0226] ;

[0227] Among the long-term charging revenue items, p takes the average order unit price 3.0 yuan / km, estimated at 45 minutes, 0.1.

[0228] Hungarian algorithm matching: the platform runs the KM algorithm based on the weight matrix of all edges. The calculation result shows that assigning to (edge weight 17.33) can bring higher long-term revenue of the system than assigning (edge weight 15.74). Therefore, the optimal matching decision contains .

[0229] Subsequently, the decision is executed and the system is updated: the platform instructs the vehicle to go to the charging station for charging. The state of the vehicle is updated to -1 (charging). The queue of the charging station joins the vehicle . The order is served by other vehicles with more sufficient power in the system (such as ). And the experience tuple of this decision is stored in the experience replay pool. When the amount of data in the experience replay pool is sufficient, sample a small batch of data, and update the parameters of the critic network and the actor network as described above.

[0230] Through the above example, it can be seen how the method of the present application makes intelligent trade-offs among complex multi-objectives by quantitatively evaluating the long-term impact of different decisions (such as charging a low-power vehicle in advance to avoid running out of power in the future during peak hours), thereby maximizing the long-term revenue of the system.

[0231] It should be understood that the details of the present application that are not described are known in the art and will not be described again.

[0232] Embodiment 2,

[0233] The embodiment provides a device for joint scheduling and charging of a self-driving taxi fleet, which is loaded with a method for joint scheduling and charging of the self-driving taxi fleet, and the device comprises:

[0234] a data acquisition module, which is used for acquiring current state information of the system and establishing a decision target with a target of maximizing platform revenue;

[0235] a target construction module, which is used for converting the decision target into a Markov decision process and establishing a composite reward function of multi-objective optimization;

[0236] a decision generation module, which is used for constructing a bipartite graph of joint scheduling and charging, acquiring dynamic weights of the composite reward function by using a reinforcement learning model, calculating comprehensive weights of edges in the bipartite graph, matching the bipartite graph, obtaining a current optimal matching decision and executing the current optimal matching decision.

[0237] Obviously, the above embodiment of the present application is only an example for clearly illustrating the technical scheme of the present application, and is not a limitation on the specific embodiments of the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for joint scheduling and charging of a fleet of autonomous taxis, applied to a scheduling and charging management system for a fleet of autonomous taxis, characterized in that, The method comprises: acquiring system current state information, and establishing a decision goal of maximizing platform revenue; transforming the decision goal into a Markov decision process, and establishing a composite reward function of multi-objective optimization; The composite reward function of multi-objective optimization established is: In formula (9), is an order set or a charging station set ; , ; represents the platform revenue at time t; represents the battery power consumed by a taxi executing the current order; represents the long-term charging revenue; , , is the weight parameter of each component; A bipartite graph of joint scheduling and charging is constructed, and a dynamic weight of the composite reward function is obtained by using a reinforcement learning model to calculate a comprehensive weight of each edge in the bipartite graph, and the bipartite graph is matched to obtain a current optimal matching decision and execute the same; The dynamic weight of the composite reward function is obtained by using the reinforcement learning model, specifically comprising: Based on the Coulomb force model, the system current state information is coded into a first Coulomb force vector between taxis and orders and a second Coulomb force vector between taxis and charging stations; Based on the first Coulomb force vector and the second Coulomb force vector, an observation state of the reinforcement learning model is constructed, the dynamic weight of the composite reward function is taken as an action of each taxi in the reinforcement learning model, and a value calculated according to the composite reward function after executing the bipartite graph matching is taken as a reward of the reinforcement learning model; The reinforcement learning model is trained in a loop, and finally outputs a dynamic weight parameter of each taxi in the composite reward function; The system current state information is coded into a first Coulomb force vector between taxis and orders and a second Coulomb force vector between taxis and charging stations, specifically comprising: Estimate the supply-demand gap information on the driver-passenger matching side based on the current state information of the system and the charging side ; Based on the system current state information, and the supply-demand gap information on the driver-passenger matching side and the supply-demand gap information on the charging side, a first Coulomb force vector between taxis and orders is obtained by using an order encoder, and a second Coulomb force vector between taxis and charging stations is obtained by using a charging station encoder; The first Coulomb force vector between taxis and orders is obtained by using the order encoder, specifically comprising: The order encoder maps its input into a Gaussian distribution and samples from this distribution a latent representation of the amount of electric charge product between a taxi v and an order o ; Based on the Coulomb's law formula, a first Coulomb force vector between taxis and orders is calculated; The second Coulomb force vector between taxis and charging stations is obtained by using the charging station encoder, specifically comprising: The charging station encoder inputs its data. Mapped to Gaussian distribution Output taxi v and charging station Potential representation of the product of charges ; Based on the Coulomb's law formula, a second Coulomb force vector between taxis and charging stations is calculated; wherein, representing a taxi characteristic vector at time t; representing an order characteristic vector; representing a mean; representing a variance.

2. The method for joint dispatching and charging of a fleet of unmanned taxis according to claim 1, characterized in that, The decision goal established is: In formula (1): is an order set or a charging station set , and refers to an order or a charging station ; is a binary decision variable, =1 indicates that a taxi goes to an order or a charging station ; represents the platform revenue at time t, and the revenue at all times is accumulated by ; and: In formula (2): representing a taxi revenue from completed orders; representing a taxi expenditure for charging; And: in formula (3): represents a taxi completes an order ; and represents a taxi at a charging station ; and represents the total distance of the estimated driving path of an order represents the unit price of the charging of an order service represents a taxi at the current moment represents the standard total capacity of the power battery of a taxi represents the charging price And / or, the constraint condition of the decision goal comprises: A vehicle power constraint for ensuring that a taxi has sufficient power to complete an order and safely arrive at a charging station: In formula (4): represents the energy consumption rate of the vehicle; represents a taxi distance from the current location to the order o origin; represents the minimum safe power A constraint that a passenger can only be in a charging or service order state, and cannot receive multiple orders at the same time when serving an order: In formula (5): denotes a taxi In the matching decision of all charging stations, there can be at most one value of 1, i.e. charging at one charging station; denotes a taxi In the matching decision of all orders, there can be at most one value of 1, i.e. serving one order.

3. The method of jointly scheduling and charging for a fleet of unmanned taxis of claim 1, wherein, The taxi Battery power consumed to execute the current order And long-term charging benefits In detail as follows: in formulas (6), (7): represents the energy consumption rate of the vehicle; represents the charging price; represents the total distance of the estimated driving path of the order; represents the taxi distance from the current location to the starting point of the order o; represents the taxi at the current time point, the remaining percentage of the power battery; represents the standard total capacity of the power battery of the taxi; represents the unit price of the power purchased by the platform from the power grid; represents the taxi at the charging station charging time; is a penalty weight, representing the influence degree of the taxi charging; And: In formula (8): representing a taxi driving time to the charging station ; representing a taxi time required for charging; representing a taxi time spent in the queue for charging; weight parameters of the components , , the sum of which is constrained to 1, i.e.: 。 4. The method for joint dispatching and charging of a fleet of unmanned taxis according to claim 3, characterized in that, The method further comprises estimating an expected driving time required for a taxi to serve an order: In formula (11): denotes the expected driving time required by a taxi for serving an order, which is used to update the set of available autonomous taxis at each time instant, denotes the average cruising driving speed of a taxi .

5. The method for joint dispatching and charging of a fleet of unmanned taxis according to claim 1, wherein, The calculation of the supply-demand gap information on the driver-passenger matching side is: In formula (12): representing area In the current passenger matching side supply-demand gap information at the time; representing area total number of all pending orders within the area; representing area total number of all available taxis within the area; The calculation of the supply-demand gap information on the charging side is: In formula (13): representing the area at the current charging side supply-demand gap information at the current time; representing the charging station set; representing the charging station the number of all charging pile devices contained inside; representing the charging station the ordered sequence information of vehicles currently queuing for charging in the charging station; representing the taxi at the current time, the remaining percentage value of the power battery; representing the standard total capacity of the taxi power battery; representing the charging station the rated output charging power value of the charging pile n in the charging station.

6. The method for joint dispatching and charging of a fleet of unmanned taxis according to claim 1, wherein, The observation state of the reinforcement learning model constructed is: In formula (16): represents an observed state; represents a taxi geographical location vicinity a first set of Coulomb force vectors of the individual most recent orders; represents a taxi geographical location vicinity a second set of Coulomb force vectors of the individual most recent charging stations; The action of each taxi in the reinforcement learning model is represented as: In formula (17): The dynamic weight representing the composite reward function, used to represent the action of the taxi.

7. The method of jointly scheduling and charging for a fleet of unmanned taxis of claim 6, wherein, The reinforcement learning model adopts a multi-agent soft actor-critic framework for training, specifically comprising: An actor network, a critic network, and a joint optimization goal of the actor network and the critic network are created, and the joint optimization goal is: In formula (18): denotes an optimization objective; denotes a policy; denotes a system expectation under an observation state and an action ; denotes a long-term discount factor; is a weight coefficient of an entropy regularization term; denotes an entropy of a policy under an observation state ; Enter the cycle training: in each training step, after the environment is reset, each agent outputs actions according to the current local observation state through the actor network, the platform executes joint decision, the environment state is transferred, and the reward of each agent is calculated; Enter the soft update of the reinforcement learning model: The critic network is constructed to include a soft Q-value network and a soft V-value network , wherein the input of the soft Q-value network is and the output is a soft Q-value; the input of the soft V-value network is and the output is a soft V-value; the critic network updates the parameters by minimizing the soft Bellman residual according to the current observation state , action and the target value of the next state, and the objective function is: In formula (19): represents the objective function of the critic network; represents the expectation; represents the discount factor; represents a taxi the observation state at time t+1; The input of the actor network constructed is , the output is , exploration is carried out by adding Gaussian noise , and finally a random action subject to the policy is output; the actor network updates the policy parameters by maximizing the weighted sum of the expected return and the policy entropy according to the function calculated by the critic network: In formula (20): a target function representing an actor network; a target function representing an actor network; a divergence; a system policy with network parameters a system policy with network parameters a normalization constant.

8. An apparatus for joint dispatching and charging of a fleet of autonomous taxis, loaded with the method for joint dispatching and charging of a fleet of autonomous taxis according to any one of claims 1 to 7, characterized in that, The device comprises: A data acquisition module is configured to acquire current state information of the system and establish a decision target of maximizing platform revenue; A target construction module is configured to convert the decision target into a Markov decision process and establish a composite reward function for multi-objective optimization; A decision generation module is configured to construct a bipartite graph of joint scheduling and charging, acquire dynamic weights of the composite reward function by using a reinforcement learning model, calculate comprehensive weights of edges in the bipartite graph, match the bipartite graph, obtain a current optimal matching decision, and execute the current optimal matching decision.

Images