Deep reinforcement learning for optimizing carpooling policies

Abstract

A method for operating a shareable ride vehicle, comprising: determining a target position of the shareable ride vehicle; determining a shareable ride strategy algorithm based on the determined target position of the shareable ride vehicle to determine the behavior of the shareable ride vehicle, the behavior including whether to accept a shareable ride for multiple persons, or maintain a single shareable ride and the route of a shareable ride for multiple persons (if any); and determining the behavior of the shareable ride vehicle based on the current position of the shareable ride vehicle and the determined shareable ride strategy algorithm, so that the shareable ride vehicle operates according to the determined behavior of the shareable ride vehicle.

Description

[0001] Cross-references

[0002] This application claims the benefit of priority to U.S. Non-Provisional Application No. 15 / 970,425, filed on May 3, 2018, entitled “Deep Reinforcement Learning for Optimizing Carpooling Policies,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to methods and apparatus for operating a ride-shareable vehicle. Background Art

[0004] The vehicle dispatch platform can automatically assign transportation requests to the corresponding vehicles to provide transportation services. Transportation services can include transporting a single passenger / passenger group or multiple passengers / passenger groups in a carpool. The driver of each vehicle will be paid for the transportation service provided. For drivers, it is important to maximize the compensation for the time they spend on the streets. Summary of the invention

[0005] Various embodiments of the present application may include systems, methods, and non-transitory computer-readable media configured for operating a shareable ride vehicle. According to one aspect, an exemplary method for operating a shareable ride vehicle may include: determining a shareable ride strategy algorithm based on a determined target location of the shareable ride vehicle to determine a behavior of the shareable ride vehicle, the behavior including whether to accept a multi-person shareable ride, or maintain a route of a single shareable ride and a multi-person shareable ride (if any), and determining a behavior of the shareable ride vehicle based on a current location of the shareable ride vehicle and the determined shareable ride strategy algorithm, and causing the shareable ride vehicle to operate according to the determined behavior of the shareable ride vehicle.

[0006] According to another aspect, the present application provides a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for operating a ride-sharing vehicle. The method may include steps that are the same or similar to the above exemplary method.

[0007] According to another aspect, the present application provides a system for providing a ride-sharing service including one or more ride-sharing vehicles, including a server including one or more processors and a memory storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform a method of operating the one or more ride-sharing vehicles. The method may include steps that are the same or similar to the above exemplary method.

[0008] In some embodiments, the determined ride-sharing policy algorithm may be configured based on a deep reinforcement learning method of a deep Q network (DQN). The exemplary method may further include determining a current date or a current time, and the ride-sharing policy algorithm may also be determined based on the current date or the current time.

[0009] Determining a ride-sharing strategy algorithm may include: when the target location is a first location, determining a first ride-sharing strategy algorithm as the ride-sharing strategy algorithm, and, when the target location is a second location different from the first location, determining a second ride-sharing strategy algorithm different from the first ride-sharing strategy as the ride-sharing strategy algorithm. The number of people at the first location may be greater than the number of people at the second location, and the first ride-sharing strategy algorithm may be configured to accept more multi-person ride-sharing than the second ride-sharing strategy algorithm. The first ride-sharing strategy algorithm may not be configured based on a deep reinforcement learning method of a deep Q network (DQN), and the second ride-sharing strategy algorithm may be configured based on a deep reinforcement learning method of the DQN.

[0010] The exemplary method may further include determining a density of ride requests for a target location of a shareable ride vehicle, and the shared ride strategy algorithm may be determined based on the determined density of ride requests. The exemplary method may further include determining a current date or a current time, and based on the current date or the current time, determining a density of ride requests for a target location of a shareable ride vehicle. Determining the shared ride strategy algorithm may include: determining a first shared ride strategy algorithm as a shared ride strategy algorithm when the ride request density is a first density; and determining a second shared ride strategy algorithm different from the first shared ride strategy algorithm as a shared ride strategy algorithm when the ride request density is a second density that is less dense than the first location. The first shared ride strategy algorithm may be configured to accept more multi-person shared rides than the second shared ride strategy algorithm. The first shared ride strategy algorithm may not be configured based on a deep reinforcement learning method of a deep Q network (DQN), and the second shared ride strategy algorithm may be configured based on a deep reinforcement learning method of a DQN.

[0011] The target location of the ride-sharing vehicle may include a target service area for the ride-sharing service. The target location of the ride-sharing vehicle may include a current location of the ride-sharing vehicle.

[0012] The features and characteristics of the systems, methods and non-transitory computer-readable media of the present application, as well as the functions of the related elements of the methods of operation, the combination of parts, the economy of manufacturing, etc., become apparent after considering the description of the drawings which are a part of this application, all of which constitute a part of this specification, wherein like reference numerals represent corresponding parts in the various drawings. However, it should be expressly understood that the drawings are for illustration and description purposes only and are not intended as a definition of the limitations of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Certain features of various embodiments of the present technology are particularly set forth in the appended claims. The features and advantages of the present technology may be better understood by reference to the following detailed description, which, taken in conjunction with the accompanying drawings, describes in detail an illustrative embodiment in which the principles of the invention are employed:

[0014] Figure 1 An exemplary environment for providing a vehicle navigation simulation environment in accordance with various embodiments is shown.

[0015] Figure 2 An exemplary environment for providing vehicle navigation in accordance with various embodiments is shown.

[0016] Figure 3A An exemplary reinforcement learning framework in accordance with various embodiments is shown.

[0017] Figure 3B-3E An exemplary algorithm for providing a vehicle navigation simulation environment according to various embodiments is shown.

[0018] Figure 3F Exemplary state transitions for providing a vehicle navigation simulation environment according to various embodiments are shown.

[0019] Figure 3G Exemplary routing for carpooling is shown in accordance with various embodiments.

[0020] Figure 4A A flowchart showing an exemplary method for providing a vehicle navigation simulation environment according to various embodiments

[0021] Figure 4B A flow chart of an exemplary method for providing vehicle navigation in accordance with various embodiments is shown.

[0022] Figure 5A An exemplary geographic area according to an experimental simulation for analyzing an established ride-sharing algorithm is shown.

[0023] Figure 5B The experimental results show that the Q values ​​of the DQN strategy and the table Q strategy in the area with fewer people deviate from the baseline strategy in (a) and (b).

[0024] Figure 5C The experimental results show that the Q values ​​of the DQN strategy and the table Q strategy in the area with a large number of people deviate from the baseline strategy in (a) and (b).

[0025] Figure 5D A table showing the average cumulative compensation on weekdays and weekends in two areas with low and high populations is shown.

[0026] Figure 6 A flow chart is shown of an exemplary method for operating a ride-shareable vehicle in accordance with various embodiments.

[0027] Figure 7 A block diagram is shown of an exemplary computer system in which any of the embodiments described herein may be implemented. DETAILED DESCRIPTION

[0028] A vehicle platform that can provide transportation services such as ride-sharing services. The vehicle platform, which may also be referred to as a vehicle calling or vehicle dispatching platform, can be accessed through a device such as a mobile phone with a platform application installed. Via the application, a user (transportation requester) can send a transportation request (e.g., pickup location, destination) to the vehicle platform. The vehicle platform can relay the request to the vehicle driver. Sometimes, two or more passengers / passenger groups may request a ride-sharing service. The vehicle driver can choose from the accepted requests, pick up and drop off passengers based on the accepted requests, and get paid accordingly.

[0029] Existing platforms only provide basic information about current transportation requests, which drivers cannot use to determine the best strategy to maximize their income (e.g., who to pick up, whether to accept carpooling). Or, if the platform automatically matches vehicles with service requesters, the matching is based only on simple conditions, such as the closest distance. In addition, using current technology, drivers cannot determine the best route when sharing a ride. Therefore, in order to help drivers maximize their income and / or help passengers minimize their ride time, it is important for vehicle platforms to provide automatic decision-making functions that can improve vehicle services.

[0030] Various embodiments of the present application include systems, methods, and non-transitory computer-readable media configured to provide a vehicle navigation simulation environment, and systems, methods, and non-transitory computer-readable media configured to provide vehicle navigation. The provided vehicle navigation simulation environment may include a simulator for training strategies that help maximize vehicle driver compensation and / or minimize passenger travel time. The provided vehicle navigation can be based on trained strategies to guide actual vehicle drivers in actual situations.

[0031] The disclosed system and method provide an algorithm for constructing a vehicle navigation environment (also referred to as a simulator) for training an algorithm or model based on historical data (e.g., various historical trips and rewards with respect to time and location). Based on the training, the algorithm or model can provide a trained strategy. The trained strategy can maximize the reward for the vehicle driver, minimize the time cost for the passenger, maximize the efficiency of the vehicle platform, maximize the efficiency of the vehicle service, and / or optimize other parameters based on the training. The trained strategy can be deployed on a server and / or computing device of the platform used by the driver. Different strategies can be applied based on various applicable parameters (e.g., geographic location, population density, ride request density, time and date, etc.).

[0032] System architecture:

[0033] Figure 1 An exemplary environment 100 for providing a vehicle navigation simulation environment according to various embodiments is shown. Figure 1 As shown, the exemplary environment 100 may include at least one computing system 102a, which includes one or more processors 104a and memory 106a. The processor 104a may include a CPU (central processing unit), a GPU (graphics processing unit) and / or an alternative processor or integrated circuit. The memory 106a may be non-temporary and readable. The memory 106a may store instructions that, when executed by one or more processors 104a, cause one or more processors 104a to perform various operations described herein. The system 102a may be implemented on various devices such as servers, computers, etc. The system 102a may be installed with appropriate software and / or hardware (e.g., wires, wireless connections, etc.) to access other devices of the environment 100. In some embodiments, the vehicle navigation environment / simulator disclosed herein may be stored in the memory 106a as an algorithm.

[0034] Environment 100 may include one or more data stores (e.g., data store 108a) and one or more computing devices (e.g., computing device 109a) accessible to system 102a. In some embodiments, system 102a may be configured to obtain data (e.g., historical trip data) from data store 108a (e.g., a database or data set of historical transportation trips) and / or computing device 109a (e.g., a computer, server, mobile phone used by a driver or passenger to obtain transportation trip information such as time, location, and cost). System 102a may use the obtained data to train an algorithm or model for vehicle navigation. The location may include GPS (Global Positioning System) coordinates of the vehicle.

[0035] Figure 2 An exemplary environment 200 for providing vehicle navigation in accordance with various embodiments is shown. Figure 2 An exemplary environment 200 for providing a vehicle navigation simulation environment according to various embodiments is shown. Figure 2 As shown, the exemplary environment 200 may include at least one computing system 102b, which includes one or more processors 104b and memory 106b. The memory 106b may be non-temporary and readable. The memory 106b may store instructions that, when executed by one or more processors 104b, cause one or more processors 104b to perform various operations described herein. The system 102b may be implemented on or as various devices such as mobile phones, servers, computers, wearable devices (smart watches), etc. The system 102b may be installed with suitable software and / or hardware (e.g., wired, wireless connections, etc.) to access other devices of the environment 200.

[0036] Systems 102a and 102b may correspond to the same system or different systems. Processors 104a and 104b may correspond to the same processor or different processors. Memories 106a and 106b may correspond to the same memory or different memories. Data stores 108a and 108b may correspond to the same data store or different data stores. Computing devices 109a and 109b may correspond to the same computing device or different computing devices.

[0037] Environment 200 may include one or more data stores (e.g., data store 108b) and one or more computing devices (e.g., computing device 109b) accessible to system 102b. In some embodiments, system 102b may be configured to obtain data (e.g., maps, location, current time, weather, traffic, driver information, user information, vehicle information, transaction information, etc.) from data store 108b and / or computing device 109b. The location may include GPS coordinates of the vehicle.

[0038] Although shown as a single component in the figure, it should be understood that the system 102b, data storage 108b, and computing device 109b can be implemented as a single device or two or more devices coupled together, or two or more of them can be integrated together. The system 102b can be implemented as a single system or multiple systems coupled to each other. Typically, the system 102b, computing device 109b, data storage 108b, and computing devices 110 and 111 may be able to communicate with each other via one or more wired or wireless networks (e.g., the Internet), and data can be communicated through the wired or wireless network.

[0039] In some embodiments, system 102b may implement an online information or service platform. The service may be associated with a vehicle (e.g., a car, bicycle, ship, airplane, etc.), and the platform may be referred to as a vehicle (ride-hailing service or shared order dispatch) platform. The platform may accept transportation requests, identify vehicles that meet the request, arrange for pickups, and process transactions. For example, a user may use a computing device 111 (e.g., a mobile phone with a software application associated with the platform installed) to request transportation from the platform, and system 102b may receive the request and forward it to various vehicle drivers (e.g., by publishing the request to a mobile phone carried by the driver). One of the vehicle drivers may use a computing device 110 (e.g., another mobile phone with an application associated with the platform installed) to receive the published transportation request and obtain boarding location information. Similarly, carpooling requests from multiple passengers / passenger groups may be processed. Fee (e.g., transportation fee) transactions may be conducted between system 102b and computing devices 110 and 111. Compensation for transportation services may be provided to drivers. Some platform data may be stored in memory 106 b or may be retrieved from data store 108 b and / or computing devices 109 b , 110 , and 111 .

[0040] Environment 100 may further include one or more computing devices (e.g., computing devices 110 and 111) coupled to system 102b. Computing devices 110 and 111 may include devices such as mobile phones, tablets, computers, wearable devices (smart watches), etc. Computing devices 110 and 111 may send data to system 102b or receive data from system 102b.

[0041] refer to Figure 1 and Figure 2 In various embodiments, the environment 100 can train a model to obtain a policy, and the environment 200 can implement the trained policy. For example, the system 102a can obtain data (e.g., training data) from the data storage 108 and / or the computing device 109. The training data may include historical trips of passengers / passenger groups. Each historical trip may include information such as the pickup location, pickup time, drop-off location, drop-off time, and cost. The obtained data may be stored in the memory 106a. The system 102a can train a model using the obtained data, or train an algorithm using the obtained data to learn a model for vehicle navigation. In the latter example, an algorithm that learns a model without providing a state transition probability model and / or a value function model may be referred to as a model-free reinforcement learning (RL) algorithm. Through simulation, the RL algorithm can be trained to provide a policy that can be implemented in an actual device to help the driver make the best decision.

[0042] Policy configuration:

[0043] Figure 3A An exemplary reinforcement learning framework according to various embodiments is shown. As shown in the figure, for an exemplary RL algorithm, a software agent 301 takes actions in an "environment" 302 (or referred to as a "simulator") to maximize the agent's "reward". The agent and the environment interact in discrete time steps. During training, at time step t, the agent observes the system state (e.g., state S t ), generating an action (e.g., action a t ) and receive a reward for the result (e.g., a reward of r t+1 ) and the next state (e.g., state S t+1 ). Accordingly, at time step t, the environment provides one or more states to the subject (e.g., state S t ), obtain the action taken by the subject (for example, action a t ), the forward state (for example, state S t+1 ), and determine the reward (e.g., reward r t+1 ). Regarding the vehicle service environment, the training may simulate the vehicle driver waiting at the current location, picking up a passenger group or two passenger groups sharing a vehicle (compared with the subject's behavior), regarding time (compared with various states), movement of vehicle and passenger locations (compared with various states), benefits (compared with rewards), etc. Each passenger group may include one or more passengers.

[0044] Back in the simulation, in order to generate the best policy governing the decision at each step, the corresponding state-action value function of the driver can be estimated. The value function can show the advantage of making a decision at a specific location and time of day relative to the long-term goal (e.g., maximizing revenue). At each step, in the state provided by the environment, the agent performs an action (e.g., wait for or transport one passenger group, two passenger groups, three passenger groups, etc.), and accordingly, the agent receives a reward from the environment and updates the state. That is, the agent selects an action from a set of available actions, and the agent moves to a new state and determines the reward associated with the transition for the action. Transitions can be executed recursively, and the agent's goal is to obtain as much reward as possible.

[0045] For simulation, RL algorithms are based on Markov decision processes (MDPs). The MPD may depend on the observable state space S, the action space a, the state transition probabilities, the reward function r, the starting state, and / or the reward discount rate, some of which are described in detail below. The state transition probabilities and / or the reward function r may be known or unknown (referred to as model-free methods).

[0046] State, S: The state of the simulated environment may include location and / or time information. For example, the location information may include the geographic coordinates of the simulated vehicle and the time (e.g., time of day in seconds): S = (I, t), where I is a GPS coordinate pair (latitude, longitude) and t is the time. S may contain other features that characterize the spatiotemporal space (I, t).

[0047] Action, a: This action is an assignment to the driver, which may include: waiting at the current location, picking up a certain passenger / passenger group, picking up multiple passengers / passenger groups and transporting them via a rideshare, etc. An assignment regarding transportation may be defined by pickup location(s), pickup time(s), drop-off location(s), and / or drop-off time(s).

[0048] Remuneration, r: Remuneration can take many forms. For example, in simulation, remuneration may be represented by a nominal number determined based on distance. For example, in a single passenger trip, remuneration may be determined based on the distance between the start and end points of the trip. As another example, in a two-passenger carpool trip, remuneration may be determined based on the sum of: a first distance between the first passenger's origin and destination, and a second distance between the second passenger's origin and destination. In real life, remuneration may be related to the total cost of the transport, such as the compensation a driver receives for each transport. The platform may determine such remuneration based on distance traveled or other parameters.

[0049] Segment: A segment can include any time period, such as a full day from 0:00am to 23:59pm. Therefore, the terminal state is the state where the t component corresponds to 23:59pm. Alternatively, other segment definitions within a period of time can be used.

[0050] Policy, π: A function that maps states to a distribution over the action space (e.g., a random policy) or a specific action (e.g., a deterministic policy).

[0051] In various embodiments, the trained policy from RL beats existing decision data and other inferior policies in terms of cumulative reward. The simulation environment can be trained using travel history data of a historical group of passengers, such as a historical taxi travel data set within a given city. Historical data can be used to guide sample passenger trip requests for simulation. For example, given a month of travel data, one possible way to generate a full day of travel for a simulation run is to sample a quarter of the travel every hour for each day of the week given in a month. For another example, it can be assumed that after the driver delivers the passenger to the destination, a new trip request assigned is accepted from the vicinity of the destination. According to the action search and / or route described below, the action of the simulated vehicle can be selected by a given strategy, which may include the cost of the trip, waiting actions, etc. The simulation can be run for multiple segments (e.g., multiple days), and these sets can be calculated and averaged to obtain cumulative rewards.

[0052] Reference below Figure 3B-3G Detailed algorithms for providing the environment are provided. The environment can support various modes. In the reservation mode, the simulated vehicle knows in advance the transportation requests from passengers and makes carpooling decisions (e.g., whether to make multiple passengers share the ride) when the vehicle is vacant, i.e., without passengers. In RL terms, the driver's (agent's) state may contain (location, time) pairs, the agent's actions, and the rewards collected after performing each action.

[0053] In some embodiments, an exemplary method for providing a vehicle navigation simulation environment may include recursively executing steps (1)-(4) over a period of time. Steps (1)-(4) may include: step (1) providing one or more states (e.g., state S) of a simulation environment to a simulation subject, wherein the simulation subject includes a simulated vehicle, and the state includes a first current time (e.g., t) and a first current position (e.g., I) of the simulated vehicle; step (2) when the simulated vehicle has no passengers, obtaining an action through the simulated vehicle, wherein the action is selected from: waiting at the first current position of the simulated vehicle, transporting M passenger groups, each of the M passenger groups includes one or more passengers, and each two of the M passenger groups have at least one of the following: different boarding locations or different alighting locations; step (3) determining a reward (e.g., reward r) for the simulated vehicle that performs the action; step (4) updating one or more states based on the action to obtain one or more updated states for providing to the simulated vehicle, wherein the updated state includes a second current time and a second current position of the simulated vehicle.

[0054] In some embodiments, "passenger groups" are used to distinguish passengers who board from different locations and / or get off at different locations. If passengers share the same boarding and alighting locations, they may belong to the same passenger group. Each passenger group may include only one passenger or multiple passengers. In addition, the simulated vehicle can accommodate N passengers, and at any time during transportation, the total number of passengers on board must not exceed N. When referring to passengers in this article, drivers are not counted.

[0055] In some embodiments, obtaining an action of the simulated vehicle when the simulated vehicle has no passengers includes: actions obtained by the simulated vehicle only when the simulated vehicle has no passengers; the simulated vehicle recursively performing each operation.

[0056] In some embodiments, if the action in step (2) is to transport M passenger groups, then in step (4), the second current time is the current time corresponding to the departure of all M passenger groups, and the second current position is the current position of the vehicle at the second current time.

[0057] In some embodiments, in booking mode, the actions and transportation allocation sequence of M passenger groups (including waiting at the current location when M=0) are assigned to simulated vehicles. The subject can learn a strategy that covers only primary actions (for example, determining the number M of transporting M passenger groups, including waiting at the current location when M=0) or primary and secondary actions (for example, which second passenger group to pick up after picking up the first passenger group, which route to take when the passenger groups carpool, etc.). In the first case, the learning strategy makes the first-level decisions, while the second-level decisions can be determined by Algorithms 2 and 3. In the second case, the policy is responsible for determining M and the route and plan of the carpool trip. The various actions are described in detail below with reference to the corresponding algorithms. For RL training, at the beginning of the segment, D0 is the initial state of the vehicle S0=(I0, t0), and the actual starting point of the vehicle's transportation trip is O1, and S O1 =(I O1 , t O1 ) is the vehicle in the intermediate state when picking up the first passenger. This notation and similar terms are used in the following algorithms.

[0058] Figure 3B An exemplary algorithm 1 for providing a vehicle navigation simulation environment according to various embodiments is shown. Figure 3B The operations shown in and presented below are exemplary.

[0059] Algorithm 1 may correspond to a waiting action (W). That is, M = 0, and the simulated vehicle is assigned to wait at its current location without picking up any passenger group. When a waiting action is assigned to a vehicle in state S0 = (I0, t0), at time t0, the vehicle is moved forward by t dWhen t0+t1, the vehicle stays at the current position I0. Therefore, the driver's next state will be (I0, t0+t2 d ), as described in line 4 of Algorithm 1. That is, if the action in step (2) is to wait at the current position of the simulated vehicle, then the second current time is the sum of the first current time plus the time period t d The corresponding current time, and the second current position is the same as the first current position.

[0060] Figure 3C An exemplary algorithm 2 for providing a vehicle navigation simulation environment according to various embodiments is shown. Figure 3C The operations shown in and presented below are intended to be illustrative.

[0061] Algorithm 2 may correspond to a Take-1 action (transport 1 passenger group). That is, M=1. Given an initial state S0, a transport trip is assigned to a simulated vehicle for which the vehicle can reach the starting point O1 of the transport trip in a shorter time than the historical boarding time of the passenger group. For example, referring to line 4 of Algorithm 2, the transport request search area may be reduced by finding all historical transport trips with pickup times ranging from t0 to (t0+T), regardless of the starting point of the historical trip, where T defines a search time window (e.g., 600 seconds). Referring to line 5 of Algorithm 2, the transport trip search area may be further reduced by finding all historical vehicle trips that the simulated vehicle can reach before the historical boarding time from the initial state S0 of the simulated vehicle. Here, t(D0, O1) may represent the time to advance from state D0 to state O1. Since historical transport data may indicate when and where transport demand occurs, filtering the transport request search by historical boarding time in line 4 may obtain candidate users that match the time window, regardless of how far these candidate users are. In addition, filtering the transport request search by the vicinity of the vehicle location in line 5 can further narrow down the group of potential passengers that are most suitable for maximizing the reward. Referring to lines 6-7 of Algorithm 2, if there is no such trip origin, similar to Algorithm 1, the simulated vehicle continues to wait at its current location I0, but the time is advanced to (t0+t d ), the vehicle state becomes S1 = (I0, t0 + t d ). The reward of the waiting action is 0. Referring to lines 9-10 of Algorithm 2, if such a historical vehicle trip exists, the historical vehicle trip with the minimum boarding time (the shortest time to reach the boarding location) is assigned to the simulated vehicle. Finally, the simulated vehicle picks up the passenger group from the starting point of the assigned trip, delivers the passenger group to the destination, and updates its state to S1=(I D1 , t D1 ). Here, I D1 represents the alighting position of the passenger group, tD1 is the time of day when the simulated vehicle arrives at destination D1.

[0062] Therefore, in some embodiments, the method for providing a vehicle navigation simulation environment may further include: based on historical data of the itinerary experienced by the historical passenger groups: searching for one or more first historical passenger groups, wherein: (condition A) the time points of picking up the first historical passenger groups from the first boarding location are within a first time threshold from the first current time, and (condition B) the time points of the simulated vehicle arriving at the first boarding location from the first current location are not later than the historical time points of the first passenger group boarding; in response to not finding the first historical passenger group that meets (condition A) and (condition B), assigning the simulated vehicle to wait at the first current location, and correspondingly determining that the reward for the current action is zero.

[0063] In some embodiments, if M=1, and in response to finding one or more first historical passenger groups that satisfy (condition A) and (condition B), the method may further include assigning a simulated vehicle to transport a passenger group P associated with the first pickup location that takes the least time to reach the first pickup location from the first current location, and determining the reward for the current action accordingly based on the travel distance of the assigned passenger group P, wherein the passenger group P is one of the first historical passenger groups found.

[0064] Figure 3D An exemplary algorithm 3 for providing a vehicle navigation simulation environment according to various embodiments is shown. Figure 3D The operations shown in and presented below are exemplary.

[0065] Algorithm 3 may correspond to the Take-2 action (transporting 2 passenger groups in a carpool). That is, M = 2. Referring to lines 3-7 of Algorithm 3, given the initial state S0, similar to the Take-1 action, the first transportation task is assigned to the simulated vehicle. Once the first transportation task is assigned, the simulated vehicle arrives at the starting position O1 to pick up the first passenger group, and its intermediate state is updated to S O1 =(I O1 , t O1 ).

[0066] From the intermediate state S O1 , in lines 9-24 of Algorithm 3, how to assign the second transport task to the simulated vehicle is described, where the second transport task is assigned to the driver and the state of the simulated vehicle is updated to S by following a similar process as that of assigning the first transport task. O2 =(I O2 , t O2 ). Refer to line 12 of Algorithm 3. The difference from Algorithm 2 is the search range of the boarding time of the transportation trip. For the second transportation task, by selectingO1 to (t O1 +(T c *t(O1, D1))) to narrow the itinerary search area, regardless of the starting position of the historical transportation trips. Here, t(O1, D1) can represent the time to transport the first passenger group from its departure point to the destination alone. When searching for the second transportation request, the simulated vehicle may have to be in the intermediate state S O1 The maximum stay (T c *t(O1, D1)) seconds. Here, T c It is in the range of (0, 1) and is an important parameter for controlling the trip search area for the second transport task allocation.

[0067] The search area for the second transport task may not be fixed. For example, assume that the search time window size is fixed to T = 600s, similar to the first transport task. The boarding time search range for the second transport task becomes (t O1 , t O1 +T). From the historical data set, if the historical vehicle can complete the assigned trip of the first passenger group from O1 to D1 within t(O1, D1) = 500s < T, it is more efficient to assign the Take-1 action to the simulated vehicle instead of assigning the Take-2 action. Therefore, a dynamic boarding time search range is required to select the second transportation task. Referring to line 13 of Algorithm 3, after reducing the boarding time search area of ​​the second transportation task, it can be selected from its intermediate state S O1 Start at time t in history O2 All historical transport trips that the vehicle can reach are previously simulated to further reduce the search area.

[0068] Therefore, in some embodiments, the method for providing a vehicle navigation simulation environment may further include: if M=2 and in response to finding one or more first historical passenger groups that satisfy the above-mentioned (condition A) and (condition B), assigning a simulated vehicle to pick up a passenger group P associated with a first boarding location, wherein it takes the least time to reach the first boarding location from the first current location, and the passenger group P is one of the first historical passenger groups found; determining the time T for transporting the passenger group P from the first boarding location to the destination of the passenger group P; searching for one or more second historical passenger groups, wherein: (condition C) the time points of picking up the second historical passenger groups from the second boarding location are within a second time threshold from the time point of picking up the passenger group P, the second time threshold is a part of the determined time T, and (condition D) the time points of the simulated vehicle arriving at the second boarding location from the time when the passenger group P boards the vehicle are not later than the historical time points when the second historical passenger groups were picked up; in response to not finding a second historical passenger group that satisfies (condition C) and (condition D), assigning a simulated vehicle to wait at the first boarding location of the passenger group P.

[0069] After determining that M=2 is the number of passenger groups to be transported, the simulated vehicle has picked up one first passenger group and determined the second passenger group to be picked up. (The first and second passenger groups have different destinations D1 and D2). The second passenger group to be selected and which of the first and second passenger groups to get off first can be determined according to lines 17-24 of Algorithm 3. Referring to line 18 of Algorithm 3, the simulated vehicle can select the minimum value (T Extl +T Extll ) corresponds to the second passenger group. Extl and T Extll You can refer to Figure 3E The algorithm described in 4, Figure 3E An exemplary algorithm 4 for providing a vehicle navigation simulation environment according to various embodiments is shown.

[0070] In one example, the problem being solved here can be deterministic, and this decision can be summarized as part of the auxiliary decision. Figure 3F , Figure 3F Exemplary state transitions for providing a vehicle navigation simulation environment according to various embodiments are shown. Figure 3F The operations shown in and described below are exemplary only. Figure 3F A segment of a day is shown in which multiple state transitions (corresponding to the above recursion) can be performed. An exemplary state transition involving carpooling two passenger groups is provided. As described above, the simulated vehicle can start in state D0 at T0 and O1 Move to state O1 to pick up the first passenger group, and then O2At T1, the simulated vehicle may move to state O2 to pick up the second passenger group. After both passenger groups get off, at T1, the simulated vehicle may move to the next state transition.

[0071] After picking up the second group of passengers, the simulated vehicle may choose to drop off either the first or second group of passengers. Figure 3G Exemplary routing for carpooling is shown in accordance with various embodiments. Figure 3G The operations shown in and presented below are exemplary. Figure 3G Two approaches to solving the routing problem are shown. That is, after carpooling the two passenger groups, the simulated vehicle can follow either of the following approaches:

[0072] D0→O1→O2→D1→D2 Figure 3G It is shown as path I in

[0073] or

[0074] D0→O1→O2→D2→D1 Figure 3G Shown as Path II.

[0075] In path I, D2 is the final state of the simulated vehicle of the current state transition and the initial state of the next state transition. In path II, D1 is the final state of the simulated vehicle of the current state transition and the initial state of the next state transition.

[0076] Referring back to lines 17-24 of Algorithm 3 and Algorithm 4, the second transport task with the least total extra passenger travel time can be assigned to the simulated vehicle. In some embodiments, when selecting between paths, when the vehicle travels from X to Y, the extra passenger travel time Ext selected for path P can be defined. P (X, Y). Additional travel time Ext P (.,.) is an estimate of the extra time each passenger group spends during carpooling, or zero if carpooling is not taking place. Figure 3G In the example, the actual travel time of passenger group 1 picked up from O1 without carpooling is t(O1, D1) and the actual travel time of passenger group 2 picked up from O2 is t(O2, D2). However, for carpooling, the travel time of passenger group 1 picked up from O1 is t(O1, O2) + t Est (O2, D1), the travel time of passenger group 2 picked up from O2 is t Est (O2, D1)+t Est (D1, D2). Estimated travel time t Est(.,.) can be the output of a prediction algorithm, examples of which are discussed in the following reference, which is incorporated herein by reference in its entirety: I. Jindal, Tony, Qin, X. Chen, M. Nokleby, and J. Ye., A Unified Neural Network Approach for Estimating Travel Time and Distance for a Taxi Trip, ArXiv print, October 2017.

[0077] Reference again Figure 3E , Algorithm 4 shows how to obtain the additional passenger travel time for the two paths. The additional passenger travel time is always zero after the Take-1 action is assigned, but here the Take-2 action is assigned. Therefore, when following path l, the additional travel time for passenger group 1 is:

[0078] Ext l (O1, D1) = t(O1, O2) + t Est (O2, D1)-t(O1, D1)

[0079] The additional travel time for passenger group 2 when following route I is:

[0080] Ext l (O2, D2) = t Est (O2, D1)+t Est (D1, D2)-t(O2, D2)

[0081] The additional travel time for passenger group 1 when following route II is:

[0082] Ext ll (O1, D1)=t(O1, O2)+t(O2, D2)+t Est (O2, D1)-t(O1, D1)

[0083] The additional travel time for passenger group 2 when following route II is:

[0084] Ext ll (O2, D2)=t(O2, D2)-t(O2, D2)=0

[0085] From the individual extra travel times of the passenger groups on the two paths, the total extra travel time for the passengers on each path can be derived. That is, for path I, Total Extl =T Extl =Ext l (O1, D1)+Ext l(O2, D2). For path II, Total Extll =T Extll =Ext ll (O1, D1)+Ext ll (O2, D2). Therefore, referring to lines 20-23 of Algorithm 3, in order to minimize the additional time cost of passengers, if Total Extl <Total Extl The simulated vehicle can choose path I, otherwise it follows path II.

[0086] After the conversion is completed (in Figure 3F T1 in the example, the environment can calculate the reward for this transition. Referring to line 24 of Algorithm 3, the reward can be based on the effective travel distance satisfied by the carpool trip and the sum of the original individual travel distances d(O1, D1) + d(O2, D2). The subject then prepares a new action to perform the above action set. Similarly, as long as a Take-3 action, a Take-4 action, or any Take-M action that is consistent with the vehicle capacity can be derived.

[0087] Therefore, in some embodiments, the method for providing a vehicle navigation simulation environment may further include: in response to finding one or more second historical passenger groups that meet (condition C) and (condition D), assigning a simulated vehicle to transport passenger group Q, wherein: passenger group Q is one of the second historical passenger groups found; the sum of the costs of transporting the passenger group P and the passenger group Q is the least of the total additional travel time of the passengers of (route option 1) and the total additional travel time of the passengers of (route option 2); (route option 1) includes: picking up passenger group Q, dropping off passenger group P, and dropping off passenger group Q; (route option 2) includes: picking up passenger group Q, dropping off passenger group Q, and dropping off passenger group P; the total additional passenger travel time of (route option 1) is the sum of the additional time spent by the simulated vehicle transporting passenger group P and passenger group Q according to (route option 1) compared with transporting them in groups without carpooling; the total additional passenger travel time of (route option 2) is the sum of the additional time spent by the simulated vehicle transporting passenger group P and passenger group Q according to (route option 2) compared with transporting them in groups without carpooling.

[0088] In some embodiments, the method for providing a vehicle navigation simulation environment may further include: if the total additional passenger travel time of (route option 1) is less than the total additional passenger travel time of (route option 2), assigning a simulated vehicle to follow (route option 1); if the total additional passenger travel time of (route option 1) is greater than the total additional passenger travel time of (route option 2), assigning a simulated vehicle to follow (route option 2).

[0089] Thus, the disclosed environment can be used to train models and / or algorithms for vehicle navigation. The prior art has not yet developed a system and method that can provide a robust mechanism for training strategies for vehicle services. The environment is the key to providing an optimization strategy that can guide vehicle drivers effortlessly while maximizing the driver's benefits and minimizing the time cost of passengers. That is, based on the historical data of the travel of historical passenger groups, the above steps (1)-(4) are recursively executed to train a strategy that maximizes the cumulative reward within the time period; and when the actual vehicle has no passengers, the trained strategy determines the action for the actual vehicle in the actual environment, and the action for the actual vehicle in the actual environment is selected from: (action 1) waiting at the current position of the actual vehicle, (action 2) determining a value M to transport M actual passenger groups, each passenger group including one or more passengers. For the actual vehicle in the actual environment, (action 2) may also include: determining M actual passenger groups from the available actual passenger groups requesting vehicle services; if M is greater than 1, determining the following order: picking up each of the M actual passenger groups and dropping off each of the M passenger groups; transporting the determined M actual passenger groups according to the determined order. Therefore, the provided simulation environment paves the way for generating automated vehicle guidance that can make passenger pickup, waiting decisions, and carpool routing decisions for actual vehicle drivers, which is not possible with existing technologies.

[0090] Figure 4A 1 is a flow chart showing an exemplary method 400 for providing a vehicle navigation simulation environment according to various embodiments of the present application. The exemplary method 400 may be performed in a system including, for example, Figure 1 The exemplary method 400 may be implemented in various environments of the environment 100 of the present invention. The exemplary method 400 may be implemented by one or more components of the system 102a (e.g., the processor 104a, the memory 106a). The exemplary method 400 may be implemented by multiple systems similar to the system 102a. The operations of the method 400 presented below are intended to be illustrative. Depending on the implementation, the exemplary method 400 may include additional, fewer, or alternative steps that are performed in various orders or in parallel.

[0091] Exemplary method 400 may include recursively performing steps (1)-(4) over a period of time (e.g., one day). At box 401, step (1) may include providing one or more states of a simulated environment to a simulated subject. The simulated subject includes a simulated vehicle, and the state includes a first current time and a first current position of the simulated vehicle. At box 402, step (2) may include obtaining an action by the simulated vehicle when the simulated vehicle has no passengers. The action is selected from: waiting at the first current position of the simulated vehicle, and transporting M passenger groups. Each of the M passenger groups includes one or more passengers. Every two of the M passenger groups have at least one of the following: different boarding locations or different alighting locations. At box 403, step (3) may include determining a reward for the simulated vehicle for the action. At box 404, step (4) may include updating one or more states based on the action to obtain one or more updated states for providing to the simulated vehicle. The updated state includes a second current time and a second current position of the simulated vehicle.

[0092] In some embodiments, the exemplary method 400 may be performed to obtain a simulator / simulation environment for training an algorithm or model as described above. For example, the training may obtain historical trip data to obtain a strategy that maximizes the cumulative reward during the time period. The historical data may include details of historical passenger trips, such as historical time points and pick-up locations.

[0093] Thus, the trained policies can be implemented on a variety of computing devices to help service vehicle drivers maximize their pay while working on the street. For example, a service vehicle driver can install a software application on a mobile phone and use the application to access the vehicle platform to receive business. The trained policies can be implemented in the application to recommend pay optimization actions to the driver. For example, when there are no passengers in the vehicle, the executed trained policies may provide the following suggestions: (1) wait at the current location, (2) pick up one passenger group, (3) pick up two passenger groups, (3) pick up 3 passenger groups, etc., each passenger group including one or more passengers. The passenger groups to be picked up have already requested transportation from the vehicle platform, and their requested pickup locations are known to the application. Refer to the following Figure 4B Describe the details that determine the recommendation.

[0094] Figure 4B A flowchart of an exemplary method 450 for providing vehicle navigation according to various embodiments of the present application is shown. Figure 2The exemplary method 450 is implemented in various environments of the environment 200 of the system 102b. The exemplary method 450 can be implemented by one or more components of the system 102b (e.g., the processor 104b, the memory 106b) or the computing device 110. For example, the method 450 can be executed by a server to provide instructions to the computing device 110 (e.g., a mobile phone used by a vehicle driver). The method 450 can be implemented by multiple systems similar to the system 102b. For another example, the method 450 can be executed by the computing device 110. The operations of the method 450 presented below are intended to be illustrative. Depending on the implementation, the exemplary method 450 may include additional, fewer, or alternative steps performed in various orders or in parallel.

[0095] At box 451, the current actual number of passengers on the actual vehicle can be determined. In one example, this step can be triggered when the vehicle driver activates the corresponding function from the application. In another example, this step can be continuously performed by the application. Since the vehicle driver relies on the application to interact with the vehicle platform, the application tracks whether the current transportation task has been completed. If all tasks have been completed, the application can determine that there are no passengers on the vehicle. At box 452, in response to determining that there are no actual passengers on the actual vehicle, instructions for transporting M actual passenger groups are provided based at least on a training strategy that maximizes the cumulative reward of the actual vehicle. Reference above Figure 1 , Figure 2 , Figures 3A-3G and Figure 4A The training of the strategy is described. Each of the M passenger groups may include one or more passengers. Every two of the M passenger groups may have at least one of the following: different boarding locations or different drop-off locations. The actual vehicle is located at a first current location. For M=0, the instruction may include waiting at the first current location. For M=1, the instruction may include transporting passenger group R. For M=2, the instruction may include transporting passenger groups R and S in a carpool. The boarding location of passenger group R can be reached from the first current location in the least time. The sum of the cost of transporting the passenger group R and the passenger group S is the least of the total additional travel time of the passengers of (route option 1) and the total additional travel time of the passengers of (route option 2). (Route option 1) may include picking up passenger group S, dropping off passenger group R, and dropping off passenger group S. (Route option 2) may include picking up passenger group S, dropping off passenger group S, and dropping off passenger group R. The total additional travel time of passengers of (route option 1) may be the sum of the additional time when the actual vehicle of (route option 1) transports passenger group R and passenger group S compared to group-by-group transportation without carpooling. The total additional travel time for passengers of (route option 2) may be the sum of the additional time when the actual vehicles of (route option 2) are used to transport passenger group R and passenger group S compared to group-by-group transportation without carpooling.

[0096] In some embodiments, if the total additional travel time for passengers of (route option 1) is less than the total additional travel time for passengers of (route option 2), the instructions may include (route option 1). If the total additional travel time for passengers of (route option 1) is greater than the total additional travel time for passengers of (route option 2), the instructions may include adopting (route option 2).

[0097] In some embodiments, when the vehicle has no passengers, the trained policy can determine M for providing instructions. After determining that M=1, the trained policy can automatically determine the passenger group R from the current user requesting the vehicle service. After determining that M=2, the trained policy can automatically determine the passenger groups R and S from the current user requesting the vehicle service, and determine the optimal route selection as described above. Similarly, the trained policy may determine the passenger group and route selection for M=3, M=4, etc. For each determination, the trained policy can maximize the compensation for the vehicle driver, minimize the time cost for the passenger, maximize the efficiency of the vehicle platform, maximize the efficiency of the vehicle service, and / or optimize other parameters according to the trained policy. Alternatively, the trained policy can determine M, and the passenger group determination and / or route determination can be performed by an algorithm (e.g., an algorithm similar to Algorithms 2 to 4, and installed in a computing device or installed in a server of the computing device).

[0098] In some embodiments, the training strategy for maximizing the cumulative reward can adopt a deep reinforcement learning method (deep Q network (DQN)), in which function approximation techniques are used on tabular Q learning. The simplest way to obtain a strategy is tabular Q learning, in which the algorithm records the value function in tabular form. However, when the state and / or action space is large, it is expensive to maintain such a large table. Therefore, in some embodiments, the table is approximately learned using function approximation techniques. For example, in DQN, a deep neural network is used to approximate the Q function or the value function. Deep reinforcement learning (DeepRL) has become popular due to its success in game technology with hundreds of functions in the state space. In contrast, in carpooling, the state space is much larger because the state consists of latitude and longitude coordinates and one continuous variable (time of day). Therefore, in some embodiments, DQN is suitable for generating the best strategy to maximize the cumulative reward of carpooling.

[0099] In some embodiments, when establishing a policy, it is assumed that the vehicle (e.g., a taxi) relies entirely on RL to decide to carpool by learning the value function corresponding to the vehicle state from the experience generated by the carpooling simulator. Specifically, since the subject (e.g., a vehicle) has no knowledge of state transitions and reward distributions, a model-free RL method is used to learn the optimal policy. In one embodiment, the policy includes a mapping function that models the subject's action selection given a state, where the value of the state is determined by the state-behavior value function Vπ(s)=E[R|s,π]. Here, R represents the sum of discounted rewards. The value function estimates the performance of the subject in a given state and the optimal policy is associated with the maximum possible value of Vπ(s). Given the optimal policy and the behavior in a given state s, the value of the behavior under the optimal policy is defined by Q(s,a)=E[R|s,a,π].

[0100] In some embodiments, the Q-value function Q(s,a) is estimated by updating a lookup table used to determine the Q-value function as Q(s,a) using time difference Q learning (Table Q). t , a): =Q(s t ,a)+α[r+γmaxa Q(s t +1,a)-Q(s t , a)]. Here, 0≤γ<1 is the discount rate, simulating the behavior when the agent chooses long-term rewards (γ→1) instead of immediate rewards (γ=0), and 0<α≤1 is the learning rate that controls the size of the step. In training, an epsilon-greedy strategy is adopted, in which the agent in state s chooses the action a with probability 1-∈ with the highest value Q(s,a) (mining), and the agent in state s chooses a random action a to ensure mining.

[0101] In some cases, tabular Q-learning is useful for smaller MPD problems. However, in cases where the state-action space is huge or the state space is continuous, a function approximation model for the Q(s,a)=fθ(s,a) model will be useful. The best example of a function approximator is a neural network (universal function approximator). The basic neural network architecture is useful for large MPD problems, where the neural network takes a state space (longitude, latitude, time of day) as input and outputs multiple Q values ​​corresponding to the actions (W, TK1, TK2). In order to approximate the Q function, a three-layer deep neural network that learns the state-action-value function may be useful. In some embodiments, the state transitions (experiences) are stored in a replay memory, and each iteration samples a mini-batch from the replay memory. In the DQN framework, mini-batch updates via backpropagation are essentially solving a problem with a loss function (Q(S t , a|θ)-r(s t , a)-γmax aQ(S t+1 , a|θ')) 2 , where θ′ is the Q network parameter of the previous iteration.

[0102] In some embodiments, the maximum operator is used to select and evaluate actions that destabilize the Q-network training. To improve the stability of training, in some embodiments, Double-DQN can be used, where the target Q-network is kept and periodically synchronized with the original Q-network. Therefore, the modified loss function is defined as: In some embodiments, the discount factor γ is preferably set to 0.95 to maximize the daily revenue of the vehicle.

[0103] Thus, a vehicle driver can rely on strategies and / or algorithms to determine that the vehicle service is performed in an efficient manner while maximizing revenue and / or minimizing the time cost of passengers. The vehicle service may involve a single passenger / passenger group ride and / or a multiple passenger / passenger group ride. The optimization results obtained by the disclosed system and method are not obtainable by existing systems and methods. Currently, even if a location map of the current vehicle service request is provided, the driver cannot determine the best move that will bring more reward than other options. Existing systems and methods cannot balance between waiting and picking up passengers, cannot determine which passenger to pick up, and cannot determine the best route for a carpool trip. Therefore, the disclosed system and method at least alleviates or overcomes such challenges in providing vehicle services and providing navigation.

[0104] Experimental simulation:

[0105] In the following, reference Figures 5A-5CExperiments for analyzing configured carpooling policies are discussed. In the experiments, various carpooling policies including DQN policies and Tabular Q policies are examined in different geographical environments to analyze the optimal carpooling policies for different geographical environments. An example of the experiment is discussed in the following reference, which is hereby incorporated by reference in its entirety: I. Jindal, Tony, Qin, X. Chen, M. Nokleby, and J. Ye, Deep Reinforcement Learning for Optimizing Carpooling Policies, October 2017. In the experiments, a single-agent carpooling policy search is used under the assumption that the decisions made by one agent (e.g., a taxi) are independent of other agents. In a single-agent or multi-agent RL learning framework, the agent is a ride-sharing platform that makes decisions for the taxis. In this experiment, it is assumed that the ride-sharing platform makes decisions for only one taxi, and then the taxi itself acts as the agent. To learn the Tabular Q policy, the selected geographical area is discretized into square cells of latitude 0.0002 × latitude 0.0002 (approximately 200 m × 200 m) to form a two-dimensional grid, and the time of the day is also discretized into 600 s as the sampling period, while for learning the DQN policy, any variables are discretized.

[0106] In this experiment, the performance of different carpooling strategies is evaluated on weekdays and weekends by comparing the average cumulative rewards of a fixed strategy (baseline) and a fixed strategy that always adopts the form Q strategy (baseline). In the experiment, the empirical sample is taken from the reference above. Figure 3B-3G The described carpooling simulator is generated in real time.

[0107] In the experiment, we respectively Figure 5A The performance of different carpooling strategies is studied for two different taxi call density areas in Manhattan residential area and Manhattan downtown area shown in (a) and (b). Specifically, for Manhattan residential area, Figure 5A As shown in (a), a square area with longitude [-73.9694, -73.9274] and latitude [40.805, 40.8438] in the northern part of Manhattan is selected. For downtown Manhattan, Figure 5A As shown in (b), a square area of ​​Manhattan city with longitude [-74.0094, -73.9774] and latitude [40.715, 40.7438] is selected.

[0108] Figure 5B Figure 2 shows the Q-value deviations of the DQN policy and the Tabular Q policy for the residential Manhattan area, relative to the fixed policy as the baseline for (a) and (b). Specifically, Figure 5B In, for Figure 5BThe DQN policy in (a) plots the average action value (Q value) of the gradient descent. For table Q, the Q value is a calculation of a working day. Figure 5B Multiple fragments in (b). Figure 5C The Q-value deviations of the DQN policy and the Tabular Q policy for Manhattan city areas relative to the fixed policy as the baseline in (a) and (b) are shown. Figure 5B Similarly, running values ​​for the DQN policy and the Tabular Q policy are plotted on weekdays in (a) and (b), respectively. In both policies and both regions, the mean Q is found to converge smoothly after a few thousand episodes when training of the RL network is stopped.

[0109] Figure 5D A table showing the average cumulative rewards for the residential and downtown areas on weekdays and weekends is shown. As shown in the figure, on weekdays, both the DQN strategy and the fixed strategy perform equally well. This result is achieved because downtown Manhattan is an area where taxi driver calls are dense and conducive to carpooling. On the other hand, on weekends, the taxi call density is reduced, and the DQN strategy learns the optimal strategy better than the baseline strategy.

[0110] Tabular Q strategies consistently perform the worst because the state-action space is large and obtaining Q-values ​​for such a state-action space is impractical. In all experimental strategies, very sparse Q-value tables are obtained. At test time, there are states where all actions have the same Q-value of zero.

[0111] In downtown Manhattan, where taxi rides are very frequent, the DQN policy consistently prefers carpooling and generates similar rewards to the fixed policy. On the other hand, in residential Manhattan, where taxi rides are less frequent, the DQN policy drives taxis into high-value areas by taking actions TK1 or W. To better understand the revenue earned, we randomly selected location I in residential Manhattan and ran the entire episode to generate action and reward sequences for the fixed policy and the DQN policy. In the morning, the DQN policy and the fixed policy follow the same sequence of actions, but then the DQN policy starts to reduce the immediate reward and earns more long-term cumulative rewards by driving taxis into high action-value areas.

[0112] Best strategy match:

[0113] Figure 6 A flowchart 600 of an exemplary method for a ride-sharing vehicle is shown in accordance with various embodiments. The flowchart illustrates blocks (and possible decision points) organized in a manner that facilitates understanding. However, it should be appreciated that blocks may be reorganized for parallel execution, reordered, modified (changed, deleted, or augmented) as circumstances warrant. Figure 6In the example of , the blocks of flowchart 600 are executed by an applicable device (e.g., a server) located outside the ride-sharing vehicle, by an applicable device located inside the ride-sharing vehicle, for example, a mobile device carried by the driver or a computing device embedded in or connected to the ride-sharing vehicle, or a combination thereof.

[0114] exist Figure 6 In the example of , flowchart 600 begins at block 601 where a target location of a ride-sharing vehicle is determined. In some embodiments, the target location of the ride-sharing vehicle may be a target service area for a ride-sharing service. For example, the target service area may be an applicable geographic area, such as the New York metropolitan area, the New York City area, residential Manhattan, etc. In some embodiments, the target location of the ride-sharing vehicle may be a current location of the ride-sharing vehicle. For example, the current location of the ride-sharing vehicle may be indicated by GPS information.

[0115] exist Figure 6 In the example of , flow chart 600 continues to block 602 to determine the current date or current time. In some embodiments, the current date can be represented by a day of the week (e.g., Sunday, Monday, etc.), a weekday or weekend, a day and month (e.g., July 12), etc. In some embodiments, the current time can be represented by a time range in a day (e.g., morning, afternoon, evening, etc.), a time period in a day (e.g., 0-6AM, 6-12AM, 0-6PM, 6-12PM, etc.), etc.

[0116] exist Figure 6 In the example of , flowchart 600 continues to box 603, determining the ride request density at the target location of the determined ride-sharing vehicle. In some embodiments, the actual ride request density obtained from the statistical ride-sharing data can be determined as the ride request density. In some embodiments, the estimated ride request density is determined as the ride request density. In a specific implementation, the estimated ride request density can be determined based on demographic information (e.g., population density) and / or the current date or current time. For example, the ride request density in a higher population density area during the day can be estimated to be higher than the ride request density in a lower population density area at night. In some embodiments, when the target location of the ride-sharing vehicle is its current location, the actual ride request density and / or the estimated ride request density can be calculated as an average value of a small area (e.g., a 200mx200m square area) that includes the current location.

[0117] exist Figure 6In the example of , flowchart 600 continues to box 604, determining a shared ride strategy algorithm to determine the behavior of the shared ride vehicle. In some embodiments, the candidate shared ride strategy algorithm to be selected may include one or more of a DQN strategy algorithm, a table Q strategy algorithm, and a fixed strategy algorithm. In some embodiments, the shared ride strategy algorithm is configured to determine the behavior of the shared ride vehicle, and the behavior may include whether to accept multi-person shared rides, or maintain single-person shared rides and multi-person shared rides (if any) The route, thereby increasing (e.g., maximizing) the revenue of the shared ride vehicle while reducing (e.g., minimizing) passenger travel time. In some embodiments, the use of computing resources or power consumption to execute the shared ride strategy algorithm may also be considered, particularly when a computing device in a shared ride strategy vehicle executes the shared ride strategy algorithm. In some cases, compared to the DQN strategy algorithm, the fixed strategy algorithm may require less computing resources, and thus less power consumption, because multi-trip sharing is always accepted. In some embodiments, a ride-sharing strategy algorithm is determined based on one or more of a determined target location for a ride-sharing vehicle (box 601), a determined current date or current time (box 602), and a determined ride request density (box 603).

[0118] In one specific implementation, when the target location is a first location, the first ride-sharing strategy algorithm is determined as the ride-sharing strategy algorithm; when the target location is a second location different from the first location, a second ride-sharing strategy algorithm different from the first ride-sharing strategy algorithm is determined as the ride-sharing strategy algorithm. For example, when the first location has a larger population than the second location, the first ride-sharing strategy algorithm is configured to accept more multi-person ride-sharing than the second ride-sharing strategy algorithm. In this case, for example, the first ride-sharing strategy algorithm is a fixed strategy algorithm, and the second ride-sharing strategy algorithm is a DQN strategy algorithm.

[0119] In one specific implementation, when the ride request density is a first density, the first ride-sharing strategy algorithm is determined as the ride-sharing strategy algorithm; when the ride request density is a second density less dense than the first location, a second ride-sharing strategy algorithm different from the first ride-sharing strategy algorithm is determined as the ride-sharing strategy algorithm. The first ride-sharing strategy algorithm is configured to accept more multi-person ride-sharing than the second ride-sharing strategy algorithm. In this case, for example, the first ride-sharing strategy algorithm is a fixed strategy algorithm, and the second ride-sharing strategy algorithm is a DQN strategy algorithm.

[0120] exist Figure 6In the example of , flowchart 600 continues to block 605 to determine the behavior of the ride-sharing vehicle based on the current position of the ride-sharing vehicle and the determined ride-sharing strategy algorithm. In some embodiments, the behavior of the ride-sharing vehicle can include waiting, transporting one passenger group, two passenger groups (e.g., accepting a second passenger group), three passenger groups (e.g., accepting a third passenger group), etc.

[0121] exist Figure 6 In the example of , flowchart 600 continues to block 606, where the ride-sharing vehicle is caused to operate according to the determined behavior of the ride-sharing vehicle. In some embodiments, the instructions for operating the ride-sharing vehicle are transmitted from a server external to the ride-sharing vehicle to a mobile device carried by a driver of the ride-sharing vehicle, so that the human driver drives according to the instructions. In some embodiments, the instructions for causing the ride-sharing vehicle are transmitted from a server external to the ride-sharing vehicle to a computing device embedded in or connected to the ride-sharing vehicle, so that the human agent performs autonomous driving according to the instructions. In some embodiments, based on the execution of the determined ride-sharing strategy algorithm, instructions for operating the ride-sharing vehicle are generated within the ride-sharing vehicle, and the generated instructions are provided (e.g., displayed) to the driver or human agent.

[0122] exist Figure 6 In the example of , flowchart 600 continues to block 607, where the ride-sharing vehicle is caused to send ride-sharing data for feedback. In some embodiments, the ride-sharing data includes multiple heartbeat information, such as geographic location, vehicle status (e.g., waiting, Take-1, Take-2, etc.) and time. In some embodiments, the ride-sharing data may include information variables including pickup latitude, pickup longitude, pickup time, drop-off latitude, drop-off longitude, drop-off time, travel time, and travel distance. In some embodiments, the ride-sharing data is sent to a server for feedback, where the ride-sharing strategy algorithm is updated based on enhanced machine learning according to the ride-sharing data.

[0123] Hardware architecture:

[0124] The technology described herein is implemented by one or more special computing devices. The special computing device can be hardwired to perform the technology, or it can include circuits or digital electronic devices, such as one or more permanently programmed as programmable application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) to perform the technology, or it can include one or more hardware processors that are programmed to perform the technology according to program instructions in firmware, memory, other memories, or combinations. Such special computing devices can also combine customized hardwired logic, ASICs or FPGAs with customized programming to implement the technology. The special computing device can be a desktop computer system, a server computer system, a portable computer system, a handheld device, a network device, or any other device or device combination that combines hardwiring and / or program logic to implement the technology. The computing device is usually controlled and coordinated by operating system software. Conventional operating systems control and schedule process operations, perform memory management, provide file systems, networking, I / O services, and provide user interface functions such as graphical user interfaces ("GUIs").

[0125] Figure 7 7 is a block diagram illustrating a computer system 700 on which any applicable embodiments described herein may be implemented. In some embodiments, system 700 may correspond to system 102a or 102b described above. In some embodiments, system 700 may correspond to computing devices 109a, 109b, 110, and / or 111. Computer system 700 includes a bus 702 or other communication mechanism for communicating information, one or more hardware processors 704 coupled to bus 702 for processing information. Hardware processor 704 may be, for example, one or more general-purpose microprocessors. Processor 704 may correspond to processor 104a or 104b described above.

[0126] The computer system 700 also includes a main memory 706, such as a random access memory (RAM), cache, and / or other dynamic device, coupled to the bus 702 for information and instructions to be executed by the processor 704. The main memory 706 may also be used for temporary variables or other intermediate information during the execution of instructions to be executed by the processor 704. When these instructions are stored in a storage medium accessible to the processor 704, they render the computer system 700 into a special-purpose machine dedicated to performing the operations specified in the instructions. The computer system 700 also includes a read-only memory (ROM) 708 or other static storage device coupled to the bus 702 for storing static information and instructions for the processor 704. A storage 710, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 702 to store information and instructions. The main memory 706, ROM 708, and / or storage 710 may correspond to the memory 106a or 106b described above.

[0127] The computer system 700 may implement the techniques described herein using custom hardwired logic, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), firmware, and / or program logic that, in conjunction with the computer system, enables or programs the computer system 700 to become a special purpose machine. According to one embodiment, the techniques described herein are implemented by the computer system 700 in response to the processor 704 executing one or more instructions contained in the main memory 706. The instructions may be read into the main memory 706 from another storage medium such as the memory 710. Executing the sequences of instructions contained in the main memory 706 causes the processor 704 to perform the processes described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0128] Main memory 706, ROM 708 and / or memory 710 may include non-transitory storage media. As used herein, the term "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that cause the machine to operate in a specific manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as memory 710. Volatile media include dynamic memory, such as main memory 706. Common forms of non-transitory media include, for example, floppy drives, floppy disks, hard disks, solid-state drives, tapes or any other storage media, CD-ROMs, any other optical data storage media, any physical media with a hole pattern, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), non-volatile memory (FLASH-EPROM), basic non-volatile memory (NVRAM), any other storage chip or cassette tape and network versions thereof.

[0129] The computer system 700 also includes a communication interface 718 coupled to the bus 702. The communication interface 718 provides a two-way data communication coupled to one or more network links connected to one or more local networks. For example, the communication interface 718 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. For another example, the communication interface 718 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or a WAN component that communicates with a WAN). A wireless link can also be implemented. In any such implementation, the communication interface 718 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0130] Computer system 700 can send messages and receive data, including program code, through the network, network links and communication interface 718. In the example of the Internet, a server can send a request code for an application program through the Internet, an ISP, a local area network and communication interface 718.

[0131] The received code may be executed by processor 704 as it is received, and / or stored in memory 710 or other non-volatile storage for later execution.

[0132] Each of the processes, methods and algorithms described in the foregoing sections may be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, or fully or partially automated by them. The processes and algorithms may be partially or fully implemented in dedicated circuits.

[0133] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present application. In addition, in some embodiments, certain methods or processes may be omitted. The methods and processes described herein are also not limited to any particular order, and the boxes or states associated therewith may be executed in other appropriate orders. For example, the boxes or states described may be executed in an order different from that specifically disclosed, or multiple boxes or states may be combined in a single box or state. The example boxes or states may be executed serially, in parallel, or in other ways. Blocks or states may be added to or removed from the disclosed example embodiments. The example systems and components described herein may be configured differently from those described. For example, elements may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0134] Throughout the specification, multiple instances may implement components, operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as individual operations, one or more individual operations may be performed simultaneously, and the operations do not need to be performed in the order shown. The structures and functions represented as individual components in the example configuration may be implemented as combined structures or components. Similarly, the structures and functions presented as individual components may be implemented as individual components. Such and other variations, modifications, additions and improvements fall within the scope of the subject matter of this article.

[0135] Although an overview of the subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present application. Such embodiments of the subject matter may be referred to herein, individually or collectively, by the term "invention" for convenience purposes only, and if there are in fact multiple disclosures or concepts, it is not intended that the scope of the present application is automatically limited to any single disclosure or concept disclosed.

[0136] The detailed description is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A method for operating a ride-sharing vehicle, comprising: Determine target locations for ride-sharing vehicles; Determining a ride-sharing policy algorithm based on the determined target location of the ride-sharing vehicle, the ride-sharing policy algorithm comprising a DQN policy algorithm, a table Q policy algorithm, or a fixed policy algorithm, the ride-sharing policy algorithm being configured to determine a behavior of the ride-sharing vehicle, the behavior comprising whether to accept a multi-person ride-sharing, or to maintain a single-person ride-sharing and a route for the multi-person ride-sharing; Determining a behavior of the ride-sharing vehicle based on the current position of the ride-sharing vehicle and the determined ride-sharing strategy algorithm, the behavior of the ride-sharing vehicle comprising at least one of: waiting, transporting one passenger group, transporting two passenger groups, and transporting three passenger groups; as well as The ride-shareable vehicle is caused to operate according to the determined behavior of the ride-shareable vehicle, which is capable of maximizing revenue for the ride-shareable vehicle while minimizing passenger travel time.

2. The method according to claim 1, characterized in that: It also includes determining a current date or a current time, and determining the ride-sharing strategy algorithm based on the current date or the current time.

3. The method according to claim 1, characterized in that The algorithm for determining the shared ride strategy includes: When the target location is the first location, determining a first ride-sharing strategy algorithm as the ride-sharing strategy algorithm; and When the target location is a second location different from the first location, a second ride-sharing strategy algorithm different from the first ride-sharing strategy algorithm is determined as the ride-sharing strategy algorithm.

4. The method according to claim 3, characterized in that: The first location has a greater number of people than the second location, and the first ride-sharing strategy algorithm is configured to accept more multi-person ride-sharing than the second ride-sharing strategy algorithm.

5. The method according to claim 4, characterized in that The first ride-sharing strategy algorithm is not based on a deep reinforcement learning method configuration of a deep Q network (DQN), and the second ride-sharing strategy algorithm is based on a deep reinforcement learning method configuration of a DQN.

6. The method according to claim 1, characterized in that It also includes determining a ride request density at the target location of the shareable ride vehicle, wherein the shareable ride strategy algorithm is determined based on the determined ride request density.

7. The method according to claim 6, characterized in that Further comprising determining a current date or a current time, and determining the density of ride requests for the target location of the ride-sharing vehicle based on the current date or the current time.

8. The method according to claim 6, characterized in that The algorithm for determining the shared ride strategy includes: When the ride request density is a first density, determining a first ride-sharing strategy algorithm as the ride-sharing strategy algorithm; and When the ride request density is a second density lower than the first location density, a second ride sharing strategy algorithm different from the first ride sharing strategy algorithm is determined to be the ride sharing strategy algorithm.

9. The method according to claim 8, characterized in that The first ride-sharing strategy algorithm is configured to accept more multi-person ride-sharing than the second ride-sharing strategy algorithm.

10. The method according to claim 9, characterized in that The first ride-sharing strategy algorithm is not configured based on a deep reinforcement learning method of a deep Q network (DQN), and the second ride-sharing strategy algorithm is configured based on a deep reinforcement learning method of a DQN.

11. The method according to claim 1, characterized in that: The target location of the ride-sharing vehicle includes a target service area for the ride-sharing service.

12. The method according to claim 1, characterized in that The target location of the ride-sharing vehicle includes the current location of the ride-sharing vehicle.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for operating a ride-sharing vehicle, the method comprising: Determine target locations for ride-sharing vehicles; Determining a ride-sharing policy algorithm based on the determined target location of the ride-sharing vehicle, the ride-sharing policy algorithm comprising a DQN policy algorithm, a table Q policy algorithm, or a fixed policy algorithm, the ride-sharing policy algorithm being configured to determine a behavior of the ride-sharing vehicle, the behavior comprising whether to accept a multi-person ride-sharing, or to maintain a single-person ride-sharing and a route for the multi-person ride-sharing; Determining a behavior of the ride-sharing vehicle based on the current position of the ride-sharing vehicle and the determined ride-sharing strategy algorithm, the behavior of the ride-sharing vehicle comprising at least one of: waiting, transporting one passenger group, transporting two passenger groups, and transporting three passenger groups; as well as The ride-shareable vehicle is caused to operate according to the determined behavior of the ride-shareable vehicle, which can maximize the revenue of the ride-shareable vehicle while minimizing the passenger travel time.

14. The non-transitory computer-readable storage medium of claim 13, wherein: The method also includes determining a current date or a current time, and determining the ride-sharing strategy algorithm based on the current date or the current time.

15. The non-transitory computer-readable storage medium of claim 13, wherein: The method also includes determining a density of ride requests for a target location of the ride-shareable vehicle, wherein the ride-share strategy algorithm is determined based on the determined density of ride requests.

16. A system for providing a shared ride service, comprising: A server comprising one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method of operating one or more ride-sharing vehicles, wherein the method comprises: determining a target position of a target vehicle in the one or more ride-sharing vehicles; Determining a ride-sharing strategy algorithm based on the determined target location of the target vehicle, the ride-sharing strategy algorithm comprising a DQN strategy algorithm, a table Q strategy algorithm or a fixed strategy algorithm, the ride-sharing strategy algorithm being configured to determine a behavior of the target vehicle, the behavior comprising whether to accept a multi-person ride-sharing, or to maintain a single-person ride-sharing and a route of a multi-person ride-sharing; Determining a behavior of the target vehicle based on the current position of the target vehicle and the determined ride-sharing strategy algorithm, the behavior comprising at least one of: waiting, transporting one passenger group, transporting two passenger groups, and transporting three passenger groups; and The target vehicle is caused to operate according to the determined behavior of the target vehicle, wherein the determined behavior of the ride-shareable vehicle is capable of maximizing revenue for the ride-shareable vehicle while minimizing passenger travel time.

17. The system according to claim 16, characterized in that At least one of the one or more ride-sharing vehicles is an autonomous vehicle.

Images