Unmanned online car-hailing order allocation method based on online switching

Abstract

The invention relates to an unmanned online car-hailing order allocation method based on online switching, and the method comprises the steps: collecting order data, and carrying out the preprocessing of the order data, and forming a data set; generating a cost matrix according to the geographic position of the city and the traffic network; classifying the online car-hailing orders and the vehicle states in the data set; for the classified online car-hailing orders and vehicles, respectively calculating order cost according to a reinforcement learning algorithm and a traditional greedy algorithm, and selecting an order distribution strategy; and allocating the car-hailing order to the idle car according to the order allocation strategy in the step S4. According to the unmanned online car-hailing order allocation method based on online switching provided by the invention, multi-car cooperation and global scheduling optimization are realized, and the pursuit of long-term cost minimization under the condition of global optimization during current unmanned online car-hailing order allocation is met. The method can effectively deal with uncertainty caused by environment change or data scarcity, and can adapt to vehicles and order scales of different orders of magnitude.

Description

Technical Field

[0001] The present invention relates to the field of unmanned online car-hailing scheduling and operation, and in particular to an unmanned online car-hailing order allocation method based on online switching. Background Art

[0002] With the continuous development of driverless online ride-hailing technology, the order allocation of driverless online ride-hailing has become a new research direction. Compared with the traditional online ride-hailing model, driverless online ride-hailing orders will not be affected by the driver's subjective judgment, and better algorithms are needed for multi-vehicle coordination and global scheduling optimization to ensure that vehicles can be efficiently allocated and utilized.

[0003] Traditional algorithms are based on preset rules or optimization models, lacking flexibility in dealing with dynamic changes, and aiming at the best match at the current moment, ignoring long-term cost minimization. Reinforcement learning algorithms aim to minimize costs, but the results obtained in the application model are local optimal solutions, not global optimal solutions. Summary of the invention

[0004] The technical problem of the present invention is: when the number of driverless online ride-hailing vehicles and the number of orders continue to increase, multi-vehicle coordination and global scheduling optimization can be achieved through online switching, while satisfying the global optimality of the current driverless online ride-hailing vehicle order allocation and pursuing long-term cost minimization.

[0005] The purpose of the present invention is to solve the above problems and propose an unmanned online car-hailing order allocation method based on online switching, which includes the following steps: S1: collect order data and preprocess them to form a data set; S2: Generate a cost matrix based on the city’s geographical location and transportation network; S3: Classify the online car-hailing orders and vehicle status in the dataset; S4: For the classified online car-hailing orders and vehicles, the order costs are calculated based on the reinforcement learning algorithm and the traditional greedy algorithm, and the order allocation strategy is selected; S5: Allocate the ride-hailing order to the idle vehicles according to the order allocation strategy of step S4.

[0006] Preferably, online ride-hailing orders are classified according to order status, order time, starting location and destination.

[0007] Preferably, the vehicle status is classified according to the current position and idle state of the unmanned vehicle, and the vehicle status is updated.

[0008] Furthermore, the cost matrix is ​​generated, including the following sub-steps: 1) Obtain the latitude and longitude coordinates of various locations in the city and information about the city’s transportation network, including road connections, traffic signals, speed limits, and traffic flow; 2) Take the straight-line distance between locations as the cost and calculate the travel time between locations based on the traffic flow and speed limit of the road; 3) The locations of various places in the city are regarded as nodes, the roads between the places are regarded as edges, and each edge is attached with a weight as the cost between the nodes; 4) Create a city based on the number of locations and the cost between locations n×n The cost matrix of .

[0009] Furthermore, order priority is set based on order status, with orders with longer waiting times receiving higher priority; Preferably, step S4, the greedy algorithm includes the following sub-steps: 1) Calculate the distance between two locations in a plane coordinate system: ; In the formula, i, j Indicates location, d( i, j ) indicates location i and location j The distance x , y Represents coordinates; 2) Calculate the time cost between two locations: ; In the formula, T(i, j) Indicate location i and location j The time cost between Indicates from location i To the location j Average speed; 3) The matching weight of vehicles and orders is calculated as follows: ; In the formula, Indicates the starting location of the vehicle. represents the destination of the order, W( ) represents the matching weight between the vehicle and the order, k represents the congestion coefficient; 4) Order priority allocation: sort the matching weights and allocate the order to the idle vehicle with the largest matching weight; 5) Maximize the total matching reward; 6) Update the vehicle status, update the idle status of the vehicle assigned to the order to busy, and change the vehicle's location to the destination location of the order.

[0010] Preferably, the reinforcement learning algorithm in step S4 includes the following sub-steps: It includes the following sub-steps: 1) Initialization Actor network, Critic Network, target network and experience pool modules; 2) According to the current order allocation plan, the order status and rewards are obtained and stored in the experience pool module; 3) Use the target network to calculate the target Q value; 4) Use Critic The network calculates the evaluation value of the current order allocation and constructs a loss function for gradient update; 5) Use Actor The network outputs an order allocation plan based on the current order status and calculates the gradient; 6) Update the parameters of the target network; 7) Repeat steps 2-7 until the training end condition is reached.

[0011] Preferably, the reinforcement learning algorithm in step S4 adopts Actor-Critic Structural IMP-TD3 Algorithms, including Actor Network, 2 Critic network, Actor Target network and 2 Critic Target network.

[0012] Further, 2 Critic The network is used to reduce the variance of Q-value estimation and improve the stability of the algorithm through delayed update and soft update strategies. Critic The calculation formula for network update is: ; ; In the formula, Indicates i The sampling weight of samples, i Indicates the counting unit. M express i The maximum value of represents the discount factor, represents the standard deviation of the noise, represents a normal distribution, represents the noise limit, θ express Critic Network parameters, Indicates the i Sample TD error, Indicates i Sample rewards, Indicatesi The status of the samples 。

[0013] Preferably, the minimum Q value is selected to calculate the target estimate, reducing the over-estimation bias, and, to avoid overfitting, moving towards the target Critic The network adds a noise signal as a regularization term.

[0014] Preferably, Actor Target Network Actor Target Network Actor The target network extracts behavioral strategies from the data set and makes decisions quickly when new orders arrive. Assuming the current strategy is , that is, in the state Next select action The probability distribution of , whose goal is to maximize the expected cumulative reward, update Actor The calculation formula of the network parameters is: ; In the formula, are the parameters of the Actor network, yes Critic The state-action value function computed by the network, represents the objective function, also known as the performance index, Indicates that the current policy is in a given state Next select action The probability distribution of Preferably, 2 Critic The target network is used to evaluate the vehicle status and the effect of order allocation. Critic The target network loss function usually uses the mean square error to measure the difference between the estimated Q value and the actual target Q value. Critic The loss function expression of the target network is: ; ; In the formula, and Respectively represent 2 Critic Target network state-action value function, and There are 2 Critic Target network parameters, It’s an instant reward. is the discount factor, s represents the current state, Indicates the next state, represents the action in state s, represents the next action in state s′, Indicates the firstCritic The parameters of the network, Indicates the first target Critic Parameters of the network.

[0015] Further, in step S4, the order allocation strategy is selected by comparing the cost value of the reinforcement learning algorithm with the cost value, decision value and offset value of the greedy algorithm. If the cost value of the reinforcement learning algorithm is greater, the order allocation strategy selects the reinforcement learning algorithm, otherwise the greedy algorithm is selected. The calculation formulas for the cost value, decision value, and offset value of the greedy algorithm are: Cost = ρ × RGREEDY - C ; In the formula, Cost represents the variable of the calculation result, C and ρ represent artificially set parameters, and their value ranges are ρ∈[0,1], C≥0, R GREEDY Represents the cost value of the greedy algorithm.

[0016] Compared with the prior art, the beneficial effects of the present invention include: 1) The present invention provides an unmanned online car-hailing order allocation method based on online switching. Through the online switching method, it solves the problem of large data volume caused by the continuous increase of unmanned online car-hailing vehicles and orders, realizes multi-vehicle coordination and global scheduling optimization, and also satisfies the pursuit of long-term cost minimization under the global optimal condition when allocating orders for unmanned online car-hailing vehicles.

[0017] 2) The present invention provides an unmanned vehicle-hailing order allocation method based on online switching. When the reinforcement learning strategy performs well, the system gives priority to using it; when the reinforcement learning strategy performs poorly, it automatically switches to a more stable greedy algorithm, which can effectively deal with the uncertainty caused by environmental changes or data scarcity, and can adapt to vehicles and order sizes of different orders of magnitude. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] Figure 1 This is a flow chart of a method for allocating driverless online car-hailing orders based on online switching according to an embodiment of the present invention; Figure 2 A network structure diagram of a reinforcement learning algorithm according to an embodiment of the present invention; Figure 3 This is a comparison chart of order response rates of the KM, DQN and MAAC algorithms and the online switching algorithm according to an embodiment of the present invention; Figure 4 This is a total cost comparison diagram of the KM, DQN and MAAC algorithms and the online switching algorithm in an embodiment of the present invention; DETAILED DESCRIPTION like Figure 1 As shown, the method for allocating driverless online car-hailing orders based on online switching includes the following steps: S1: collect order data and preprocess them to form a data set; Online ride-hailing orders are categorized based on order status, order time, starting location, and destination.

[0020] Order priority is set according to the order status. The longer the waiting time, the higher the priority of the order; S2: Generate a cost matrix based on the city’s geographical location and transportation network; Generate a cost matrix, including the following sub-steps: 1) Obtain the latitude and longitude coordinates of various locations in the city and information about the city’s transportation network, including road connections, traffic signals, speed limits, and traffic flow; 2) Take the straight-line distance between locations as the cost and calculate the travel time between locations based on the traffic flow and speed limit of the road; 3) The locations of various places in the city are regarded as nodes, the roads between the places are regarded as edges, and each edge is attached with a weight as the cost between the nodes; 4) Create an n×n cost matrix based on the number of locations in the city and the costs between locations.

[0021] S3: Classify the online car-hailing orders and vehicle status in the dataset; The vehicle status is classified according to the current position and idle state of the unmanned vehicle, and the vehicle status is updated.

[0022] S4: For the classified online car-hailing orders and vehicles, the order costs are calculated based on the reinforcement learning algorithm and the traditional greedy algorithm, and the order allocation strategy is selected; The greedy algorithm consists of the following sub-steps: 1) Calculate the distance between two locations in a plane coordinate system: ; In the formula, i, j Indicates location, d( i, j ) indicates location i and location j The distance x , y Represents coordinates; 2) Calculate the time cost between two locations: ; In the formula, T(i, j) Indicate location i and location j The time cost between Indicates from location iTo the location j Average speed; 3) The matching weight of vehicles and orders is calculated as follows: ; In the formula, Indicates the starting point of the vehicle. represents the destination of the order, W( ) represents the matching weight between the vehicle and the order, k represents the congestion coefficient; 4) Order priority allocation: sort the matching weights and allocate the order to the idle vehicle with the largest matching weight; 5) Maximize the total matching reward; 6) Update the vehicle status, update the idle status of the vehicle assigned to the order to busy, and change the vehicle's location to the destination location of the order.

[0023] like Figure 2 As shown, the reinforcement learning algorithm adopts Actor-Critic IMP-TD3 algorithm for structures, including Actor Network, 2 Critic network, Actor Target network and 2 Critic Target network.

[0024] 2 Critic The network is used to reduce the variance of Q-value estimation and improve the stability of the algorithm through delayed update and soft update strategies. Critic The calculation formula for network update is: ; ; In the formula, Indicates i The sampling weight of samples, i Indicates the unit of count. M express i The maximum value of represents the discount factor, represents the standard deviation of the noise, represents a normal distribution, represents the noise limit, θ represents the Critic network parameters, Indicates the i The TD error of samples is Indicates i Sample rewards, Indicates i The status of the samples 。

[0025] Select the minimum Q value to calculate the target estimate, reduce the over-estimation bias, and to avoid overfitting, move towards the target Critic The network adds a noise signal as a regularization term.

[0026] Actor The target network extracts behavioral strategies from the data set and makes decisions quickly when new orders arrive. Assuming the current strategy is , that is, in the state Next select action The probability distribution of , whose goal is to maximize the expected cumulative reward, update Actor The calculation formula of the network parameters is: ; In the formula, yes Actor The parameters of the network, yes Critic The state-action value function computed by the network, represents the objective function, also known as the performance index, Indicates that the current policy is in a given state Next select action The probability distribution of 2 Critic The target network is used to evaluate the vehicle status and the effect of order allocation. Critic The target network loss function usually uses the mean square error to measure the difference between the estimated Q value and the actual target Q value. Critic The loss function expression of the target network is: ; ; In the formula, and Respectively represent 2 Critic Target network state-action value function, and There are 2 Critic Target network parameters, It’s an instant reward. is the discount factor, s represents the current state, Indicates the next state, represents the action in state s, represents the next action in state s′, Indicates the first Critic The parameters of the network, Indicates the first target Critic Parameters of the network.

[0027] The reinforcement learning algorithm includes the following sub-steps: It includes the following sub-steps: 1) Initialization Actor network, Critic Network, target network and experience pool modules; 2) According to the current order allocation plan, the order status and rewards are obtained and stored in the experience pool module; 3) Use the target network to calculate the target Q value; 4) Use Critic The network calculates the evaluation value of the current order allocation and constructs a loss function for gradient update; 5) Use Actor The network outputs an order allocation plan based on the current order status and calculates the gradient; 6) Update the parameters of the target network; 7) Repeat steps 2-7 until the training end condition is reached.

[0028] Select an order allocation strategy, and compare the cost value of the reinforcement learning algorithm with the cost value, decision value, and offset value of the greedy algorithm. If the cost value of the reinforcement learning algorithm is greater, the order allocation strategy selects the reinforcement learning algorithm, otherwise the greedy algorithm is selected; The calculation formulas for the cost value, decision value, and offset value of the greedy algorithm are: Cost = ρ × RGREEDY - C In the formula, Cost represents the variable of the calculation result, C and ρ represent artificially set parameters, and their value ranges are ρ∈[0,1], C≥0, RGREEDY Represents the cost value of the greedy algorithm.

[0029] S5: Allocate the ride-hailing order to the idle vehicles according to the order allocation strategy of step S4.

[0030] The present invention uses 800,000 New York taxi order data collected by the New York City Taxi and Limousine Commission in December 2023 and the New York taxi area map provided by it as map data for experiments. The New York taxi geographical area is divided into 264 areas, and orders with missing or erroneous features in the data set are filtered out. And the China Automobile Fuel Consumption Query System 4 of the Ministry of Public Security of China is collected. M1 and M2 The fuel consumption data of urban vehicles of different types is used to calculate the cost matrix of vehicles taking orders across regions. In order to analyze and verify the performance of the method proposed in this paper, we adopt KM , DQN , MAAC as a baseline method.

[0031] In order to analyze and verify the performance of the method proposed in this paper, we use KM, DQN, MAAC This is compared with the allocation method of the present invention.

[0032] like Figure 3 The order response rates of the four methods shown in the figure are compared. The method of the present invention increases the order response rate by 5% or more when the number of unmanned online taxis is 1000, 1250, 1500, 1750, and 2000 respectively. KM For large-scale data sets, as the data size increases, the actual computing performance of the algorithm may become a bottleneck. DQN When the algorithm handles large-scale order allocation problems, its performance degrades and it is difficult to converge.

[0033] like Figure 4 The total order cost of the four methods shown in the figure is compared. The proposed method performs better than the other three algorithms. After analysis and summary, KM The algorithm is a short-sighted algorithm that only considers the benefits of the current round of vehicle allocation. DQN and MAAC Algorithms tend to maximize immediate rewards and fail to adequately capture and optimize long-term rewards.

[0034] The experimental results show that the method proposed in this invention is KM, DQN and MAAC The comparison of the methods shows that under different numbers of driverless online ride-hailing vehicles, the method has a higher response rate, lower total cost, and better average revenue of the worst part. This also shows that our proposed method can better allocate orders to driverless online ride-hailing vehicles, and improve user service satisfaction while ensuring the interests of the online ride-hailing platform.

Claims

1. A driverless online car-hailing order allocation method based on online switching, characterized in that: The steps include: S1: collect order data and preprocess them to form a data set; S2: Generate a cost matrix based on the city’s geographical location and transportation network; S3: Classify the online car-hailing orders and vehicle status in the dataset; S4: For the classified online car-hailing orders and vehicles, the order costs are calculated based on the reinforcement learning algorithm and the traditional greedy algorithm, and the order allocation strategy is selected; S5: Allocate the ride-hailing order to the idle vehicles according to the order allocation strategy of step S4.

2. The method for allocating driverless online car-hailing orders based on online switching according to claim 1 is characterized in that: In step S2, the generation of the cost matrix includes the following sub-steps: 1) Obtain the latitude and longitude coordinates of various locations in the city and information about the city’s transportation network; 2) Take the straight-line distance between locations as the cost and calculate the travel time between locations based on the traffic flow and speed limit of the road; 3) The locations of various places in the city are regarded as nodes, the roads between the places are regarded as edges, and each edge is attached with a weight as the cost between the nodes; 4) Create a cost matrix based on the number of locations in the city and the costs between the locations.

3. The method for allocating driverless online car-hailing orders based on online switching according to claim 1 is characterized in that: The step S3 also includes order priority, and the priority is set according to the order status. The longer the waiting time, the higher the priority of the order.

4. The method for allocating driverless online car-hailing orders based on online switching according to claim 1 is characterized in that: The step S4, the greedy algorithm includes the following sub-steps: 1) Calculate the distance between two locations in a plane coordinate system: ； In the formula, i,j Indicates location, d( i,j ) indicates location i and location j The distance x , y Represents coordinates; 2) Calculate the time cost between two locations: ； In the formula, T(i,j) Indicate location i and location j The time cost between Indicates from location i To the location j Average speed; 3) The matching weight of vehicles and orders is calculated as follows: ； In the formula, Indicates the starting location of the vehicle. represents the destination of the order, W( ) represents the matching weight between the vehicle and the order, k represents the congestion coefficient; 4) Order priority allocation: sort the matching weights and allocate the order to the idle vehicle with the largest matching weight; 5) Maximize the total matching reward; 6) Update the vehicle status, update the idle status of the vehicle assigned to the order to busy, and change the vehicle's location to the destination location of the order.

5. The method for allocating driverless online car-hailing orders based on online switching according to claim 1 is characterized in that: In step S4, the reinforcement learning algorithm is based on Actor-Critic Structural IMP-TD3 Algorithms, including Actors Network, 2 Critic network, Actors Target network and 2 Critic Target network.

6. The method for allocating driverless online car-hailing orders based on online switching according to claim 5 is characterized in that: 2 Critic The network is used to reduce the variance of Q-value estimation and improve the stability of the algorithm through delayed update and soft update strategies. Critic The calculation formula for network update is: ； ； In the formula, Indicates i The sampling weight of samples, i Indicates the counting unit. M express i The maximum value of represents the discount factor, represents the standard deviation of the noise, represents a normal distribution, represents the noise limit, θ express Critic Network parameters, Indicates the i The TD error of samples is Indicates i Sample rewards, Indicates i The status of a sample.

7. The method for allocating driverless online car-hailing orders based on online switching according to claim 5 is characterized in that: Said Actors The target network extracts the behavior strategy from the dataset, whose goal is to maximize the expected cumulative reward and update Actors The calculation formula of the network parameters is: ； In the formula, yes Actors The parameters of the network, yes Critic The state-action value function computed by the network, represents the objective function, also known as the performance index, Indicates that the current policy is in a given state Next select action The probability distribution of .

8. The method for allocating driverless online car-hailing orders based on online switching according to claim 5 is characterized in that: The two Critic The target network is used to evaluate the vehicle status and the effect of order allocation. Critic The loss function expression of the target network is: ； ； In the formula, and Respectively represent 2 Critic Target network state-action value function, and There are 2 Critic Target network parameters, It’s an instant reward. is the discount factor, s represents the current state, Indicates the next state, represents the action in state s, represents the next action in state s′, Indicates the first Critic The parameters of the network, Indicates the first target Critic Parameters of the network.

9. The method for allocating driverless online car-hailing orders based on online switching according to claim 1 is characterized in that: In step S4, the order allocation strategy is selected by comparing the cost value of the reinforcement learning algorithm with the cost value, decision value and offset value of the greedy algorithm. If the cost value of the reinforcement learning algorithm is greater, the order allocation strategy selects the reinforcement learning algorithm, otherwise the greedy algorithm is selected. The calculation formulas for the cost value, decision value, and offset value of the greedy algorithm are: Cost =ρ× R GREEDY - C ; In the formula, Cost represents the variable of the calculation result, C and ρ represent artificially set parameters, and their value ranges are ρ∈[0,1], C≥0, R GREEDY Represents the cost value of the greedy algorithm.

Images