A method for joint selection of travel mode and departure time based on deep reinforcement learning

Abstract

The present invention relates to the field of smart transportation, and specifically to a travel mode and departure time joint selection model based on deep reinforcement learning. The technical solution is: a travel mode and departure time joint selection model based on deep reinforcement learning, comprising the following steps: step 1) constructing a Markov decision process; step 2) training a deep reinforcement learning model. The present invention takes advantage of deep reinforcement learning (DRL) and proposes a new JTMDTC model to maximize the travel utility of individuals in a dynamic multimodal transportation network. In order to achieve computational efficiency and large-scale application, the clustering method is embedded in the modeling framework, and by designing a reinforcement learning environment and algorithm, it is ensured that the model can provide individuals with better travel options and higher utility in a complex traffic environment.

Description

Technical Field

[0001] The present invention relates to the field of smart transportation, and specifically to a method for jointly selecting a travel mode and a departure time based on deep reinforcement learning. Background Art

[0002] With population growth and urbanization, urban travel demand has increased rapidly, leading to widespread traffic congestion around the world. Various traffic management strategies have emerged to address this problem, but their successful implementation depends on how travelers’ travel choices are understood and modeled. Travel choices involve multiple dimensions, such as departure time, travel mode, and route, which can be described using discrete or continuous choice models. Unlike traditional one-dimensional models, multidimensional models consider the correlations between different choices, especially the choices of travel mode and departure time. These choices reflect individual preferences and affect the spatiotemporal demand of transportation networks. In order to evaluate the effectiveness of transportation policy initiatives, a modeling framework that considers both travel mode and departure time choices is needed. Most existing studies on joint travel mode and departure time choice (JTMDTC) modeling use discrete choice decision methods, which may be limited by the formulation of random utility functions in complex decision-making processes, which may lead to suboptimal results. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a method for jointly selecting travel mode and departure time based on deep reinforcement learning.

[0004] The technical solution of the present invention is:

[0005] A method for jointly selecting travel mode and departure time based on deep reinforcement learning, comprising the following steps:

[0006] Step 1) construct a Markov decision process;

[0007] Step 2) Train the deep reinforcement learning model.

[0008] The step 1) includes: step 1.1) setting the action space; step 1.2) setting the state space; step 1.3) setting the reward function.

[0009] The step 1) comprises:

[0010] Step 1.1) Set up the action space

[0011] Action Space

[0012] Among them, m is the traffic mode and t is the travel time;

[0013] Step 1.2) Set up the state space

[0014] Environmental status

[0015] Agent Status

[0016] Step 1.3) Set up the reward function

[0017] Reward Function

[0018]

[0019]

[0020] in, is the total travel cost under transportation mode m, is the total travel time under traffic mode m, δ(t i ) is the total trip delay under traffic mode m, is the other travel-related costs under transportation mode m.

[0021] The step 2) comprises:

[0022] 2.1) Select efficient agent representatives through DBSCAN clustering;

[0023] 2.2) Train agent representatives based on deep Q-network.

[0024] 5. A travel mode and departure time joint selection model based on deep reinforcement learning according to claim 4, characterized in that: step 2.1) comprises:

[0025] Step 2.1.1) Select an unvisited data point as the starting point. If the number of points in the neighborhood of the starting point meets the threshold, mark the starting point as a core point and create a new cluster.

[0026] Step 2.1.2) Starting from the core point, expand the new cluster by density reachability; check each data point in the neighborhood of the core point, if the number of points in the neighborhood of the data point meets the threshold, mark the data point as a core point and add the data point to the current cluster; if the number of points in the neighborhood of the data point does not meet the threshold, mark the data point as a non-core point and add the data point to the current cluster;

[0027] Step 2.1.3) If the data point is not a core point or a non-core point, mark the data point as a noise point;

[0028] Step 2.1.4) Repeat the above steps until all data points have been visited;

[0029] Step 2.1.5) Select a core point in each cluster as the agent representative.

[0030] The step 2.2) comprises:

[0031] Step 2.2.1) Initialize the replay memory D, online network parameters w and target network parameters v; In each iteration of training, perform steps 2.2.2) to 2.2.5);

[0032] Step 2.2.2) observes the initial state S0 from the environment, and in each time step, performs steps 2.2.3) to 2.2.4);

[0033] Step 2.2.3) Use the ε-greedy strategy to obtain the current action and interact with the environment to obtain the reward r t ;

[0034] Step 2.2.4) Record the environment information and obtain the next state s t+1 , storage experience group (s t ,a t ,r t ,s t+1 ) in replay memory D;

[0035] Step 2.2.5) Randomly sample from D, let

[0036]

[0037] Determine whether c time steps have passed. If so, make v←w t ;

[0038] Step 2.2.6) Iterate continuously until the training is completed.

[0039] In the ε-greedy strategy,

[0040] The beneficial effects of the present invention are:

[0041] This paper takes advantage of deep reinforcement learning (DRL) and proposes a new JTMDTC model to maximize the travel utility of individuals in a dynamic multimodal transportation network. In order to achieve computational efficiency and large-scale application, the clustering method is embedded into the modeling framework, and the reinforcement learning environment and algorithm are designed to ensure that the model can provide individuals with better travel choices and higher utility in a complex traffic environment.

[0042] First, by utilizing deep reinforcement learning methods, the model can make decisions based on multiple factors (such as travel mode, departure time, etc.) to maximize the travel utility of individuals over multiple days. It can provide different individuals with near-optimal joint travel modes and departure time options while considering real-time traffic conditions and individual preferences.

[0043] Secondly, the model makes travel choice decisions in a dynamic multimodal transportation network and can cope with complex traffic environments. By establishing a Markov decision process and using a deep Q network for decision learning, the model can flexibly adjust travel strategies according to real-time traffic conditions and individual preferences to cope with challenges such as traffic congestion and multiple transportation modes.

[0044] Finally, in order to be applicable to multiple individuals who need to make travel decisions, the clustering method is integrated with DRL to obtain representative individuals for model training. This computationally efficient training method can reduce the consumption of computing resources and time, and reduce the complexity of training while maintaining model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic flow chart of the method of the present invention.

[0046] Figure 2 This is an example diagram of a multi-mode traffic simulation network constructed based on SUMO in the present invention.

[0047] Figure 3 It is a schematic diagram of the convergence mode of the loss function of the present invention and the reward changes of four typical individuals.

[0048] Figure 4-1 This is a schematic diagram of the changes in action selection during the first typical individual training process of the present invention.

[0049] Figure 4-2 This is a schematic diagram of the changes in action selection during the second typical individual training process of the present invention.

[0050] Figure 4-3 This is a schematic diagram of the changes in action selection during the third typical individual training process of the present invention.

[0051] Figure 4-4 This is a schematic diagram of the changes in action selection during the fourth typical individual training process of the present invention.

[0052] Figure 5 It is a schematic diagram comparing the effects of the present invention and the basic DQN algorithm. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] like Figure 1 As shown, a method for jointly selecting travel mode and departure time based on deep reinforcement learning includes the following steps:

[0055] Step 1) Construct a Markov decision process suitable for the joint selection of travel mode and departure time training environment

[0056] Step 1.1) Set up the action space

[0057] The action space of the Markov decision process includes travel mode selection and departure time selection. The former considers three modes of transportation, including private cars, public transportation, and cycling. Public transportation is divided into buses and subways, and transfers can be made between the two. In order to use public transportation as quickly as possible, it is assumed that travelers walk to the bus stop or subway station closest to the starting point.

[0058] For the choice of departure time, it is assumed that each traveler has an initial or expected departure time and is allowed to depart within a certain time window [t min ,t max ] within this departure time, where t min and t max are the earliest and latest possible departure times, respectively. In summary, the action space can be described by the following vectors:

[0059] Action Space

[0060] Among them, m is the travel mode choice, t is the departure time choice, and N is the number of travel modes (N=4, respectively private car, public transportation bus, public transportation subway, and bicycle).

[0061] Step 1.2) Set up the state space

[0062] Consider the construction of two states: environment state and agent state.

[0063] The environmental state includes factors that affect the cost of traveling by different modes of transportation. For public transportation, two factors are considered, namely accessibility and fare. Here, the present invention defines and measures accessibility p as the total walking distance required to complete the first and last legs of the journey:

[0064] Reachability p = d origin +d destination

[0065] Among them, d origin and d destination is the walking distance from the nearest bus stop or subway station to the origin and destination respectively. Public transport fare is the monetary cost that people have to pay to use public transport services.

[0066] For a private car, the fuel price is considered as an influencing factor and placed in the state. Therefore, the state vector specific to the environment information is as follows:

[0067] Environmental status

[0068] The agent state mainly includes the travel information that the agent has experienced. First, it includes the travel distance L and the memory travel time T associated with each mode of transportation. The former is the travel distance L expressed by the travel mode. m , the latter is the remembered travel time expressed by travel mode Two other variables that are part of the trip information are the initial departure time t0 and the departure time difference or offset Δt relative to t0:

[0069]

[0070] In order to focus on the value of time (VOT), the present invention pays more attention to the travel time, thereby relaxing the travel distance in the agent state, which is written as follows:

[0071] Agent Status

[0072] Step 1.3) Set up the reward function

[0073] When the agent selects and executes an action, it will cause the environment to change from the current state to a new state, and this change is accompanied by a feedback or reward that aims to improve the agent's decision logic. The present invention defines the reward as the utility of the trip, which is mainly composed of various costs. Specifically, the reward obtained in the i-th step is calculated as follows:

[0074]

[0075] in, is the total travel cost under transportation mode m, and E1 and E2 are empirical constants used to map and scale this cost to reward.

[0076] The total travel cost is further broken down into three parts, namely, the total travel time Total trip delay δ(t i ) and other travel-related costs

[0077]

[0078] Among them, α is the time value.

[0079] The total travel time for private cars and bicycles is essentially the travel time spent on the road, while public transportation needs to consider three different types of travel time, including the travel time in the car. inv , waiting time t wt and transfer time t tr . Therefore, the total travel time for a particular mode is expressed as:

[0080]

[0081] in, is the total travel time under traffic mode m.

[0082] The problem with only considering the total travel time is that the actual arrival time is ignored. Although the total travel time is low, the arrival time may be far from the expected time. Therefore, a scheduling delay is introduced. When the actual arrival time deviates from the expected time, the scheduling delay cost increases linearly:

[0083]

[0084] Among them, δ(t i ) is the total travel delay under traffic mode m, β and γ are the unit costs of early arrival and late arrival, respectively, t d is the expected arrival time.

[0085] Finally, other travel-related costs mainly refer to the total fuel cost of private cars (oil price o) which is linearly related to the travel distance, or the public transportation costs related to the trip:

[0086]

[0087] in: is the other travel-related costs under transportation mode m; is an indicator function, which returns 1 when a travel mode is selected, otherwise it returns 0; the bus fare f bus No change, subway fare f subway Subway fares increase with distance traveled.

[0088] Step 2) Train the deep reinforcement learning model

[0089] 2.1) Selecting efficient agent representatives through DBSCAN clustering

[0090] Representative individuals are identified through clustering and used for model training, so that the JTMDTC problem of different individuals can be solved in a computationally efficient manner.

[0091] In order to obtain representative individuals, the present invention adopts a widely used clustering algorithm called Density-Based Spatial Clustering of Applications with Noise (DBSCAN). It is a non-parametric method that does not require assumptions or specification of the distribution of the data. The core of DBSCAN is to first identify high-density samples and then gradually connect these similar samples into larger clusters. Two travel characteristics are considered as inputs for clustering, namely travel distance and accessibility to public transportation, which together provide an overall picture of the decision-making environment faced by individuals. The algorithm divides data points into different clusters by evaluating the density of data points, and is able to identify noise points (points that do not belong to any cluster). Thereafter, an individual will be selected from each cluster as a representative agent for training.

[0092] Step 2.1.1) First, perform DBSCAN clustering on the data set. Select an unvisited data point as the starting point. If the number of points in the neighborhood of the starting point is greater than or equal to the preset threshold (MinPts), mark the starting point as a core point and create a new cluster;

[0093] Step 2.1.2) Starting from the core point, expand the new cluster through density reachability. Density reachability means that from a core point, it is possible to connect (density reachable) to other points along the density.

[0094] Check each data point in the neighborhood of the core point. If the number of points in the neighborhood of the data point meets the threshold, mark the data point as a core point, add the data point to the current cluster, and then continue to explore its neighborhood until no more points that meet the conditions are found; if the number of points in the neighborhood of the data point does not meet the threshold, mark the data point as a non-core point and add the data point to the current cluster;

[0095] Step 2.1.3) If the data point is not a core point or a non-core point, mark the data point as a noise point;

[0096] Step 2.1.4) Repeat the above steps until all data points have been visited, and DBSCAN clustering is completed;

[0097] Step 2.1.5) Select a core point in each cluster as the agent representative and train it in the next step.

[0098] 2.2) Training agent representatives based on deep Q-network

[0099] The present invention proposes a customized deep Q network (DQN algorithm) as a solution for training the above-mentioned agent representatives obtained by clustering. In order to solve the JTMDTC problem, the present invention adopts model-free value-based RL as a solution algorithm. It learns the state-action value associated with the MDP, and on this basis, derives the optimal strategy by always choosing the action that leads to the maximum state-action value:

[0100] Q π (s t ,a t )=E[U t ∣S t =s t ,A t =a t ]

[0101]

[0102] In the actual iteration process, the action selection adopts the ε-greedy strategy:

[0103]

[0104] The DQN algorithm is at the forefront of model-free value-based reinforcement learning, which uses neural networks to approximate the optimal state-action value function, thereby extending the application of traditional Q-Learning to high-dimensional and continuous space problems:

[0105] Q(s t ,a t ; w)≈Q * (s t ,a t )

[0106] In order to obtain a successfully trained agent, the state-action value function must be iteratively updated through temporal difference (TD) until convergence or stability:

[0107]

[0108] Then we need to optimize the weights of the neural network that approximates the state-action value to minimize the TD error, which is in the form of a loss function:

[0109]

[0110] In order to obtain better training performance, the present invention improves the DQN algorithm with two key technologies, one is experience replay and the other is target network. The former refers to the construction of a replay memory that stores experience in real time. Each memory pool consists of the experience of selected representatives from each cluster. Compared with directly using current experience for training, experience relay not only allows historical experience to be reused, but also eliminates the undesirable correlation between consecutive experience samples. Specifically, each time the agent selects an action and interacts with the environment, it records the environment and agent information, and stores the experience group (s t ,a t ,r t ,s t+1 ) in the replay memory D. The latter technique refers to preparing and using two neural network parameters w and v, one as the target network parameter v and the other as the online network parameter w. Both networks have the same initial weights, but the target network parameters replicate the online network parameters every fixed number of steps c. c is an empirical constant.

[0111] Specifically, the online network parameters are updated at each update:

[0112]

[0113] When the interval step number c is reached, update the target network parameter v←w t .

[0114] After clustering, the agent representatives can participate in the final travel mode and departure time decision through training. reduced After that, it will select the travel mode and departure time action with the highest reward based on the training results. This action will be deployed on all agents belonging to this category. In this way, the model complexity is reduced and the computational efficiency is improved while ensuring a higher reward.

[0115] Example

[0116] Step 1) Construct a Markov decision process suitable for the joint selection of travel mode and departure time training environment

[0117] On the one hand, SUMO is used as a traffic simulation engine. The selected road network is a part of the urban area of ​​Suzhou, China, covering an area of ​​about 20 square kilometers. In order to build and simulate such a network, OpenStreetMap (OSM) is used to obtain the geometry and configuration of the network (2,423 nodes and 4,970 edges), and this information is input into SUMO to establish the simulation environment. As a multimodal network, public transportation must be configured, including buses and subways. This is achieved by first extracting public transportation operation information from a map service application and then performing map matching. The extracted information includes line ids, station ids, and their geographical locations. These locations such as Figure 2 shown.

[0118] On the other hand, using the existing knowledge of the utility function parameters, the typical morning rush hour (7:00 a.m. to 9:00 a.m.) JTMDTC was considered in the simulation and trained on five consecutive weekdays. The simulation duration was 2 hours, and the simulation range was divided into four 30-minute time intervals (the unit interval t unit ). For each weekday, there is a typical demand pattern during the morning rush hour, and for each time interval, travel demand is treated as a random variable following a Gaussian distribution. Therefore, during simulation and training, each time slice of the day is associated with a slightly different travel demand. This randomness is consistent with the daily traffic fluctuations (periodic congestion) observed in reality around a certain pattern.

[0119] Table 1 Simulation parameter settings

[0120]

[0121]

[0122] Step 2) Train the deep reinforcement learning model

[0123] Step 2.1) Select efficient agent representatives through DBSCAN clustering.

[0124] 60 time-related travel individuals in the network are selected, and their respective travel characteristics are input into the clustering method. By setting the minimum number of points to 10 and the minimum distance threshold to 0.05, four clusters are finally obtained, each of which contains a set of density-connected data points and a set of noise points. A representative is selected from each of the four clusters as the agent representative to participate in the subsequent reinforcement learning training, and its experience is put into the public memory pool for training DQN. The location of the trip OD (starting point and end point of the trip) of the four agents in the network is shown in the figure. Figure 2 shown.

[0125] Figure 2In the figure, the itinerary OD marks the starting point and end point of each category. In the figure, people are divided into four categories by color, and each category has two location points, which serve as the starting point and the end point respectively.

[0126] Step 2.2) Train the agent representative based on the deep Q network

[0127] All numerical experiments are performed on a standard computer with an Intel Core(TM) i5-9400F 2.90GHz CPU and 8GB RAM. Figure 3 The figure shows the convergence pattern of the loss function after 800 episodes of training. The overall downward trend is obvious, and in the early stages, there are obvious fluctuations in the loss function value due to behavioral exploration and the fact that the agent knows nothing about the environment at the beginning. However, this fluctuation mainly lasts for the first 300 steps. From about 500 steps, the change in the loss function value is minimal and almost falls on a straight line. This observation clearly shows the convergence of the algorithm. Regardless of the absolute value of the reward, all representative agents show an increasing trend on the curve, which means that they continue to improve their travel choices by interacting and learning with the environment. In about 700 steps, the reward of each representative agent is basically stable and there is not much change thereafter.

[0128] exist Figure 4-1 , Figure 4-2 , Figure 4-3 , Figure 4-4 The actions followed by four representative agents during training are graphically represented in Figure 1. The agent class refers to the agent representative in each cluster described above that participates in the subsequent reinforcement learning training. The action exploration phase is the beginning of the training, and the actions of the representative agents change frequently. However, once we enter the action development phase, we no longer see such fluctuations, and each representative agent seems to have found its own optimal solution to the JTMDTC problem. The occasional action changes during the development process may be the result of random action selection triggered by the ε-greedy strategy.

[0129] exist Figure 5 In the figure, the basic DQN algorithm and the customized DQN algorithm of the present invention each generate 50 sets of solutions and compare them with the reference values.

[0130] For the customized DQN algorithm of the present invention, more than 80% of the solutions give solutions exceeding 95% of the reference value, which means that these solutions are close to optimality.

[0131] For the base DQN algorithm, only about 30% of the solutions exceed 95% of the same reference value, and quite a few solutions are even below 60% of the reference value. Therefore, the comparison results demonstrate the effectiveness of the proposed method in solving the JTMDTC problem with many individuals, and the important role played by the representative agent in completing this task. Since the test individuals are not part of the training, the results show that the proposed method has good portability.

[0132] The decision-making method and model disclosed in the present invention can be applied to the fields of intelligent transportation systems, travel planning and traffic management, and improve the overall efficiency and sustainability of the transportation network. The present invention has important practical application value in improving traffic flow, reducing congestion, and providing more efficient traffic management strategies based on this.

[0133] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above specific embodiments. The above specific embodiments and the description in the specification are only for further illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the claims and their equivalents.

Claims

1. A method for jointly selecting travel mode and departure time based on deep reinforcement learning, comprising the following steps: Step 1) construct a Markov decision process; Step 2) Training the deep reinforcement learning model; The step 1) comprises: Step 1.1) Set up the action space; Among them, m is the travel mode and t is the departure time; Step 1.2) Set up the state space; Among them, accessibility represents the total walking distance required to complete the first and last legs of the journey; Step 1.3) Set up the reward function; When the agent selects and executes an action, it causes the environment to change from its current state to a new state, accompanied by a reward; in, is the total travel cost under transportation mode m, E1 and E2 are empirical constants used to map and scale this cost to reward, is the total travel time under traffic mode m, t inv ,t wt and t tr They represent the travel time, waiting time and transfer time in the vehicle, δ(t i ) is the total travel delay under traffic mode m, β and λ are the unit costs of early arrival and late arrival, respectively, t d is the expected arrival time, is the other travel-related costs under transportation mode m, L car is the travel distance of private cars, is an indicator function, which returns 1 when a travel mode is selected, otherwise it returns 0. The bus fare f bus No change, subway fare f subway It increases with the distance traveled; α is the time value; The step 2) comprises: 2.1) Select efficient agent representatives through DBSCAN clustering; 2.2) Training agent representatives based on deep Q-network; The step 2.1) comprises: Step 2.1.1) The input of clustering is travel distance and public transportation accessibility. Select an unvisited data point as the starting point. If the number of points in the neighborhood of the starting point meets the threshold, mark the starting point as a core point and create a new cluster. Step 2.1.2) Starting from the core point, expand the new cluster through density reachability, which means that a core point can be connected to other points along the density; check each data point in the neighborhood of the core point. If the number of points in the neighborhood of the data point meets the threshold, mark the data point as a core point and add the data point to the current cluster; if the number of points in the neighborhood of the data point does not meet the threshold, mark the data point as a non-core point and add the data point to the current cluster; Step 2.1.3) If the data point is not a core point or a non-core point, mark the data point as a noise point; Step 2.1.4) Repeat the above steps until all data points have been visited; Step 2.1.5) Select a core point in each cluster as the agent representative; The step 2.2) comprises: Step 2.2.1) Initialize the replay memory D, online network parameters w and target network parameters v; In each iteration of training, perform steps 2.2.2) to 2.2.5); Step 2.2.2) observes the initial state S0 from the environment, and in each time step, performs steps 2.2.3) to 2.2.4); Step 2.2.3) Use the ε-greedy strategy to obtain the current travel mode and departure time action, and interact with the environment to obtain the reward r i ; Step 2.2.4) Record the environment information and obtain the next state s i+1 , storage experience group (s i ,a i ,r i ,s i+1 ) in replay memory D; Step 2.2.5) Randomly sample from D, let Determine whether c time steps have passed. If so, make v←w i ; Step 2.2.6) Iterate continuously until the training is completed; The clustered agent representatives pass the training, i.e., participate in the final travel mode and departure time decision; In the ε-greedy strategy,

Images