A bus scheduling method based on reinforcement learning

Abstract

Embodiments of the present application provide a bus scheduling method based on reinforcement learning, comprising the following steps: S110, initializing road network data, and performing regional grid division on the road network; S120, dividing trajectory data according to time periods, and calculating traffic flow of different regions in different time periods; S130, combining real-time ride request data, calculating and updating the traffic flow; S140, when a bus arrives at a route endpoint, obtaining a current road network state from multiple dimensions; S150, based on a reinforcement learning neural network, combining the updated traffic flow obtained in step S130 and the road network state obtained in step S140, calculating feedback data of the bus driving to different routes; and S160, repeating steps S130-S150, obtaining reinforcement learning model parameters, and performing bus scheduling based on the trained reinforcement learning model. Through this method, the current road network state and traffic flow can be fully considered, urban bus resources can be effectively utilized, bus operation efficiency can be improved, and urban traffic congestion can be alleviated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Vehicles bus scheduling, and particularly relates to a bus scheduling method based on reinforcement learning. BACKGROUND

[0002] Bus scheduling refers to the operation of a dispatcher directing the driving direction of a specified bus according to the current road network environment and passenger flow. Bus scheduling mainly needs to consider two problems, one is whether the specified driving route at the moment can ensure that the bus efficiently receives requests and relieve traffic pressure, and the second is that the current scheduling scheme will affect the number of buses stopping at each station and the number of requests in each region in the next period, thereby affecting the subsequent scheduling scheme. However, the existing "experience-based" scheduling scheme often leads to the occurrence of "traffic congestion" and "empty bus" phenomenon due to inaccurate estimation of the current road network condition.

[0003] How to efficiently construct the scheduling model of public transportation system has attracted much attention in recent years. Most public transportation companies still use experienced schedulers to analyze the characteristics of bus lines and make bus schedules by referring to historical data of bus lines (Wang Y, Tian Y. A Schedule Optimization for Weihai Bus System[C]. 2015 International Conference on Service Science (ICSS), 2015, 45-48.). Some scholars have studied the setting of vehicle dispatching interval in vehicle scheduling problem. For example, a mixed dispatching model for different vehicle types on the same route is proposed (Ye M. Novel Field-Bus Real-Time Network Scheduling Method[C]. 2010 Second International Conference on Computer Modeling and Simulation, 2010, 3:253-256.).Or under the premise of reasonable assumptions, to build a scheduling model with the goal of optimizing the cost of bus operation and passenger waiting cost (see Zhu Q, Li Y. Improved Harmony search Algorithm for Bus Scheduling Optimization [C]. 2019 Chinese Control And Decision Conference (CCDC), 2019, pp. 815-818.), considering passenger comfort (see Quan Z, Xin W, Di N. Research on Intelligent Bus Scheduling Based on QPSO Algorithm [C]. 2015 8th International Conference on Intelligent Computation Technology and Automation (ICICTA), 2015, 730-733.), mainly to optimize the operation cost of the bus company as the goal, based on the factors such as passenger waiting cost and vehicle empty seat penalty to model and solve (see Hassold S, Ceder A A. Multiobjective Approach to Creating Bus Timetables with Multiple Vehicle Types [J]. Transportation Research Record: Journal of the Transportation Research Board, 2012, 2276 (1): 56-62.).

[0004] However, the above research is mainly aimed at maximizing operating revenue, without considering the current road network situation and the utilization rate of bus resources. SUMMARY

[0005] The embodiments of the present application propose a bus scheduling method based on reinforcement learning to solve the technical problems mentioned in the background art section.

[0006] In a first aspect, the embodiments of the present application provide a bus scheduling method based on reinforcement learning, comprising the following steps:

[0007] S110, initialize the road network data, and divide the road network into regional grids;

[0008] S120, divide the trajectory data according to the time period, and calculate the traffic flow of each time period in different regions;

[0009] S130, calculate and update traffic flow in combination with real-time ride request data;

[0010] S140, when the bus is driving to the route endpoint, obtain the current road network state from multiple dimensions, wherein the dimensions include average sharing rate sa, average passenger transport rate dr, average walking distance wd and average waiting time wt;

[0011] S150, based on the reinforcement learning neural network, combining the updated traffic flow obtained in step S130 and the road network state obtained in step S140, calculate the feedback data of the bus driving to different routes; and

[0012] S160, repeat steps S130-S150 to obtain the reinforcement learning model parameters, and perform bus scheduling based on the trained reinforcement learning model.

[0013] Through this method, by obtaining multi-dimensional road network state, data closest to the actual road conditions can be obtained. The current road network state and traffic flow can be fully considered, urban public transport resources can be effectively utilized, public transport operation efficiency can be improved, and urban traffic congestion can be alleviated.

[0014] In some embodiments, in step S150, the feedback data R of the bus driving to different routes is obtained by the following formula:

[0015]

[0016] wherein, represents the value after x is normalized and mapped to the interval [0, 1], sa represents the average sharing rate of a bus until the current period, dr represents the average passenger transport rate of a bus through the service domain until the current period, wd represents the average walking distance of a bus received request until the current period, and wt represents the average waiting time of a bus received request until the current period.

[0017] Through the above formula, the feedback brought by the departure behavior can be obtained to update the road network and bus state in real time.

[0018] In some embodiments, the average sharing rate sa is obtained by the following formula:

[0019]

[0020] wherein, D represents a set of ride requests successfully received by the bus, l(req.o, req.d) represents the shortest path distance of a ride request req from the pickup point req.o to the destination req.d, F represents a set of equipped vehicles, and tl(c) represents the total driving distance of vehicle c.

[0021] The average sharing rate sa is obtained to reflect the average number of passengers sharing a vehicle in the entire trip.

[0022] In some embodiments, the average passenger transport rate dr is obtained by the following formula:

[0023]

[0024] where |D| represents the number of successful bus received ride requests, and |R| represents the total number of ride requests in the traffic flow.

[0025] The average passenger transport rate dr is obtained to reflect the proportion of requests that can be met by the elastic bus.

[0026] In some embodiments, the average walking distance wd is obtained by the following formula:

[0027]

[0028] where l(x1, x2) represents the straight-line distance from x1 to x2, represents the actual boarding location of the passenger issuing the ride request req(t, o, d, w), represents the actual alighting location of the passenger issuing the ride request req(t, o, d, w), and |D| represents the number of successful bus received ride requests.

[0029] The average walking distance wd is obtained to reflect the average distance that a passenger needs to walk from the expected boarding / alighting location to the actual boarding / alighting location.

[0030] In some embodiments, the average waiting time wt is obtained by the following formula:

[0031]

[0032] where represents the boarding time of the passenger issuing the request req(t, o, d, w), and |D| represents the number of successful bus received ride requests.

[0033] The average waiting time wt is obtained to reflect the average waiting time required from when the passenger issues the ride request to when the passenger boards the bus.

[0034] In some embodiments, in step S160, the training process of the reinforcement learning model includes the following steps:

[0035] S161, input the current road network state s of a bus into the evaluation network to obtain the evaluation of the current state value: Q(s, a; θ i );

[0036] Wherein, the current road network state s={sa, dr, wt, wd, ts}, sa represents the average sharing rate of a bus until the current period, dr represents the average passenger transport rate of a bus passing through the service area until the current period, wd represents the average walking distance of a bus receiving requests until the current period, wt represents the average waiting time of a bus receiving requests until the current period, and ts represents the current period number;

[0037] S162, input the next road network state s' into the target network to obtain the estimated value of the value of the next state: max a′ Q(s', a'; θ i-1 );

[0038] S163, obtain the loss function according to the Bellman formula, and the loss function formula is:

[0039] L i (θ i )=E s,a,r,s′～ρ(.) [(y i -Q(s,a;θ i )) 2 ],

[0040] y i =r+γmax a′ Q(s',a';θ i-1 )

[0041] Wherein, y i represents the time difference target, y i -Q represents the time difference error, a represents the departure behavior, a' represents the next departure behavior, p represents the behavior distribution of the current road network state s of the associated bus, the departure behavior a, the feedback r of the departure behavior a and the next road network state s', θ i represents the parameters of the evaluation network, θ i-1 represents the parameters of the target network, γ represents the discount factor, and E s,a,r,s′～ρ( . ) represents the expected function;

[0042] S164, parameter tuning of the evaluation network is performed using back propagation according to the loss function;

[0043] S165, the target network is updated using the stable evaluation network parameters; and

[0044] S166, repeat steps S161-S165 until the reinforcement learning model converges.

[0045] Through multiple learning of historical data, the bus scheduling model parameters are obtained, and according to the model, each tuple (v i , pj ) the feedback generated by the bus scheduling model based on the reinforcement learning policy.

[0046] In some embodiments, step S120 comprises the following steps:

[0047] S121, constructing a traffic flow table Γ(z) according to the trajectory data, the type of the tuple in the table being <t, g o ,g d ,num> where z represents a grid area, t represents a time period, g o represents a grid area to which a pickup point of a ride request belongs, g d represents a grid area to which a drop-off point of a ride request belongs, and num represents a corresponding traffic flow;

[0048] S122, obtaining the number T of time units included in the entire operation period of the bus, the number T of time units being obtained by the following formula:

[0049]

[0050] where U represents a smallest time unit, the specific duration of which can be set by the user, OT represents the total duration of the operation of the bus, represents the upward rounding of the numerical value x; and

[0051] S123, obtaining the traffic flow of each grid, the traffic flow of each grid being obtained by the following formula:

[0052]

[0053] slot(t, i) = [t + (i - 1) * U, t + i * U, i = 1, 2, …, T

[0054] where fg(z, t) represents the traffic flow of each grid; x represents a tuple in the traffic flow table Γ(z), slot(t, i) represents the i-th micro time period with U as the time unit starting from time t; and

[0055] S124, obtaining the total passenger flow f t of a potential route r(u, v, t), where r(u, v, t) represents a route with u and v as the starting station and the terminal station, respectively, starting from time t, the total passenger flow f t of the potential route r(u, v, t) being obtained by the following formula:

[0056]

[0057] where G is the set of all grid areas through which the potential route r(u, v, t) passes.

[0058] The total passenger flow is obtained by the method, so as to store the data of the bus interacting with the environment as a training sample.

[0059] In a second aspect, the present application provides a computer readable storage medium, having stored thereon a computer program which, when executed by a processor, implements any of the above methods.

[0060] The bus scheduling method based on reinforcement learning provided by the embodiments of the present application can effectively utilize urban bus resources, improve bus operation efficiency, and relieve urban traffic congestion. BRIEF DESCRIPTION OF DRAWINGS

[0061] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments thereof, read in conjunction with the accompanying drawings:

[0062] Figure 1 is a flowchart of an embodiment of the bus scheduling method based on reinforcement learning according to the present application;

[0063] Figure 2 is a schematic diagram of a road network in Xiamen Island according to an embodiment of the present application;

[0064] Figure 3 is a schematic diagram of a bus scheduling model according to an embodiment of the present application;

[0065] Figure 4 is a flowchart of bus scheduling according to an embodiment of the present application;

[0066] Figure 5a is a schematic diagram of the sharing rate change of different scheduling schemes;

[0067] Figure 5b is a schematic diagram of the transport rate change of different scheduling schemes;

[0068] Figure 5c is a schematic diagram of the average walking distance change of different scheduling schemes;

[0069] Figure 5d is a schematic diagram of the average waiting time change of different scheduling schemes;

[0070] Figure 6 is a schematic diagram of the comprehensive index Q value change of different scheduling schemes;

[0071] Figure 7 is a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0072] The application will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application and are not a limitation of the application. In addition, it should be noted that only parts related to the application are shown in the drawings for ease of description.

[0073] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and embodiments.

[0074] Figure 1 A flowchart of one embodiment of the bus scheduling method based on reinforcement learning of the present application is shown, the method 100 comprising the following steps:

[0075] S110, initializing road network data, and performing regional grid division on the road network;

[0076] S120, dividing trajectory data according to time periods, and calculating traffic flow of different regions in different time periods;

[0077] S130, combining real-time ride request data, calculating and updating traffic flow.

[0078] In the present embodiment, the required trajectory data is taxi operation data, and the basic format is as follows:

[0079] R(olongitude,olatitude,dlongitude,dlatitude,otime,dtime)

[0080] Wherein, olongitude represents the longitude of the pickup point, olatitude represents the latitude of the pickup point, dlongitude represents the longitude of the drop-off point, dlatitude represents the latitude of the drop-off point, otime represents the time of the pickup point, and dtime represents the time of the drop-off point.

[0081] The road network of a city is composed of a series of road segments E={e1,e2,…,e m} and a series of intersections N={n1,n2,…,n n}, wherein the basic format of the road segment E is as follows:

[0082] e(id,length,s,e)

[0083] Wherein, id is the number of the road segment, length is the length of the road segment, s is the starting intersection of the road segment, and e is the ending intersection of the road segment.

[0084] The basic format of the intersection N is as follows:

[0085] n(id,lon,lat)

[0086] Where id is the intersection number, lon is the longitude of the intersection, and lat is the latitude of the intersection.

[0087] Figure 2 A schematic diagram of the road network within Xiamen Island according to an embodiment of this application is shown, as follows: Figure 2 As shown, in order to facilitate the calculation of regional passenger flow, this application divides the road network into regional grids, dividing it into 100*100 grid areas.

[0088] In this embodiment, taking a route r(u,v,t) that travels from station u to station v during time period t as an example, the calculation of the traffic flow in this trip includes the following steps:

[0089] (1) For the service domain A of this route i This retrieves the set of grid regions it covers. A certain grid g is served by service domain A. i The conditions for coverage are as follows:

[0090]

[0091] Where area(X) represents the region range of polygon X, and θ is a predefined threshold with a range of [0,1].

[0092] (2) Use table Γ(z) to store OD (traffic flow) data, in order to <t,g o ,g d Each tuple is represented in the form of ,num>, where z represents the grid region, t represents the time period, and g o ,g d These represent the grid areas of the requested pick-up and drop-off points, respectively, and num represents the number of corresponding OD (traffic flow) pairs. U is used to represent a minimum time unit, the specific duration of which can be customized, such as 5 minutes or 10 minutes. Therefore, the number of time units included in the entire bus operation period is: OT represents the total operating time of the bus, in minutes. This indicates rounding up the numerical value x. For a certain micro-segment, the following definition is used: slot(t,i)=[t+(i-1)*U,t+i*U],i=1,2,…,T, where slot(t,i) is the i-th segment starting from time t, with U as the time unit.

[0093] (3) Computing service domain A i The number of trips across all covered grids starting from time period t. The flow rate for each grid is defined as fg(z,t):

[0094]

[0095] slot(t,i) = [t + (i - 1) * U, t + i * U], i = 1, 2, …, K

[0096] where x is a tuple in table T(z). From time t, the total passenger flow in route r(u, v, t) from station u to station v is calculated as: G is the set of all grids that route r(u, v, t) passes through.

[0097] With reference to Figure 1 , the bus scheduling method based on reinforcement learning further comprises:

[0098] S140, when the bus arrives at the route terminal, the current road network state is obtained from multiple dimensions, wherein the dimensions include average sharing rate sa, average passenger transport rate dr, average walking distance wd and average waiting time wt;

[0099] S150, based on the reinforcement learning neural network, the feedback data of the bus driving to different routes is calculated in combination with the updated traffic flow obtained in step S130 and the road network state obtained in step S140.

[0100] In step S140, the update of the road network state includes the following contents:

[0101] (1) State space, using S = {s1, s2, … s n} represents the state space of the bus. A tuple s = {sa, dr, wt, wd, ts} is used to represent the current state of a bus. Wherein sa, dr, wt, wd, ts are defined as follows:

[0102] 1) sa represents the average sharing rate of the bus up to the current period, which reflects the average number of passengers sharing the vehicle in the entire journey, which is defined as follows:

[0103]

[0104] where D represents the set of all successfully received requests, l(req.o, req.d) represents the shortest path distance of a request req from the pickup point req.o to the destination req.d, F represents the set of equipped vehicles, and tl(c) represents the total driving distance of vehicle c.

[0105] 2) dr represents the average passenger transport rate of the bus through the service domain up to the current period, which reflects the proportion of requests that can be met by the flexible bus, which is defined as follows:

[0106]

[0107] where |D| is the number of requests successfully received by the bus, and |R| represents the total number of requests.

[0108] 3) wt represents the average waiting time of the requests received by the bus up to the current period, which refers to the average waiting time required from the passenger sending a ride request to the passenger boarding the bus, and is defined as follows:

[0109]

[0110] where, represents the boarding time of the passenger sending the request req(t, o, d, w).

[0111] 4) wd represents the average walking distance of the requests received by the bus up to the current period, which refers to the average distance walked by the passenger from the desired boarding / alighting location to the actual boarding / alighting location, and is defined as follows:

[0112]

[0113] where l(x1, x2) represents the straight-line distance from x1 to x2, and is the actual boarding / alighting location of the passenger sending the request req(t, o, d, w).

[0114] 5) ts represents the current period number.

[0115] (2) Action space, using A = {a1, a2, … a n} to represent the action space of the bus. Where a k is the current bus selectable departure direction, including the operation of staying at the original station.

[0116] (3) Reward feedback, from the resource sharing rate (sa), the passenger transport rate (dr), the passenger average walking distance (wd) and the passenger average waiting time (wt) four dimensions to evaluate the feedback R brought by each behavior, by default each dimension has the same importance, that is, each is one quarter:

[0117]

[0118] where, represents the value after normalizing x to the interval [0, 1].

[0119] Every time a bus travels to the end of its current route, the road network and bus state are updated.

[0120] Figure 3 Fig. 1 shows a bus scheduling model schematic diagram according to an embodiment of the present application, in conjunction with reference Figure 1 andFigure 3 The bus scheduling method based on reinforcement learning further comprises:

[0121] S160, repeating steps S130-S150 to obtain the reinforcement learning model parameters, and performing bus scheduling based on the trained reinforcement learning model.

[0122] In this embodiment, the data of the bus interacting with the environment in steps S130-S150 is stored as a training sample, and the data format can be represented as <s, r, a, s'>. The training process of the reinforcement learning model in step S160 comprises the following sub-steps:

[0123] S161, inputting the current road network state s of a bus into the evaluation network to obtain the evaluation of the current state value: Q(s, a; θ i );

[0124] S162, inputting the next road network state s' into the target network to obtain the evaluation of the value of the next state: max a′ Q(s', a'; θ i-1 );

[0125] It should be noted that the target network and the evaluation network have the same architecture, but their parameters are different.

[0126] S163, obtaining a loss function according to the Bellman formula, and the loss function formula is:

[0127] L i (θ i )=E s,a,r,s′～ρ(.) [(y i -Q(s,a;θ i )) 2 ],

[0128] y i =r+γmax a′ Q(s',a';θ i-1 )

[0129] Wherein, y i represents the time difference target, y i -Q represents the time difference error, a represents the departure behavior, a' represents the next departure behavior, p represents the behavior distribution associated with the current road network state s of the bus, the departure behavior a, the feedback r of the departure behavior a and the next road network state s', θ i represents the parameters of the evaluation network, θ i-1 represents the parameters of the target network, γ represents the discount factor, and E s,a,r,s′～ρ(.) represents the expected function.

[0130] S164, parameter tuning is performed on the evaluation network using back propagation according to the loss function;

[0131] S165, the target network is updated using stable evaluation network parameters; wherein the stable evaluation network parameters are the parameters after tuning.

[0132] S166, steps S161-S165 are repeated until the reinforcement learning model (DQN) converges.

[0133] Through multiple learning of historical data, bus scheduling model parameters are obtained, and based on the model, feedback brought by each tuple (v i ,p j ) in the current state can be predicted, and a daily scheduling strategy is generated based on the feedback predicted by the bus scheduling model according to the policy of reinforcement learning.

[0134] Figure 4 A flowchart of bus scheduling according to one embodiment of the application is shown, in conjunction with reference Figures 1-4 In this embodiment, the specific bus scheduling process includes the following steps:

[0135] S1, according to the area and station heat, using DBSCAN clustering algorithm to mine K hot stations from historical request data;

[0136] S2, based on the hot stations generated in S1, bus routes are constructed two by two to form a route sequence P={p1, p2,…p n};

[0137] S3, when the operation starts, the system automatically deploys N buses, and randomly selects N departure stations and N departure directions;

[0138] S4, using U to represent a minimum time unit, the specific duration can be set by the user. When a bus arrives at the end of the route, the reinforcement learning model obtains the current micro-period road network environment state from four dimensions of resource sharing rate (sa), resource sharing rate (sa), passenger average walking distance (wd) and passenger average waiting time (wt);

[0139] S5, the bus selects the next departure route, and each departure behavior (v i ,p j ) will bring a feedback Reward to the system, and the bus state and road network environment state are updated, and the feedback Reward is defined as:

[0140]

[0141] S6, repeating steps S4-S5, obtaining the bus scheduling model parameters through multiple learning of historical data, the bus scheduling model being capable of predicting feedback Reward brought by each tuple (v i ,p j ) under the current state; and

[0142] S7, generating a one-day scheduling strategy based on the strategy of reinforcement learning and the feedback Reward predicted by the bus scheduling model.

[0143] Through the method, the current road network state and traffic flow can be fully considered, urban bus resources can be effectively utilized, bus operation efficiency can be improved, and urban traffic congestion can be relieved.

[0144] Among them, the related variables in the embodiment are defined as follows:

[0145] (1) bus, using V={v1,v2,…v m} to represent the vehicle sequence allocated in the scheduling process, wherein v i represents a specific bus.

[0146] (2) station, selecting the K most popular bus stations according to the regional heat and station heat, and based on these stations, building bus routes two by two. Using B={b1,b2,…b K} to represent the K most popular bus stations generated by clustering, wherein b i represents a specific station.

[0147] (3) route, using P={p1,p2,…p n} to represent all bus routes contained in the map, wherein p i represents a specific route, and is uniquely determined by the tuple (b o ,b d ), b o represents the starting station of the route, and b d represents the terminal station of the route.

[0148] (4) departure behavior, using the tuple (v i ,p j ) to represent the bus v i will select the route p j as the departure direction.

[0149] For the bus scheduling method based on reinforcement learning proposed in the present application, the following experimental verification is carried out.

[0150] The experiment is programmed by JAVA language. The data set used in the experiment is a taxi trajectory data set of Xiamen in July 2014, which contains about 5000 taxis, 220 million GPS positioning point records and 8 million real-time route data. The trajectory data from 6:00 to 10:00 on July 2, 2014 is selected for simulation, which contains 59311 trajectories.

[0151] When multiple buses are cruising in the same service domain, the requests in the service domain are received by the earlier buses according to the principle of "first come, first served"; for the requests in the overlapping area of adjacent service domains, the receiving principle of "first come, first served" is also followed. In the experiment part, 5 buses are deployed by default, 12 bus stops are elected based on historical data in the map, the operation period of the bus is set to 7:00 in the morning to 10:00 at night, a total of 15 hours, and the time unit U is set to 10 minutes by default, so the total operation time of the day is divided into 90 micro time periods.

[0152] The experiment is based on the bus scheduling method of reinforcement learning (Deep Q Network-Based Bus Scheduling Algorithm, DQN), and the sharing rate, transportation rate, average passenger walking distance and average waiting time are used as indicators. In the following, DQN refers to the method of the application. In addition, the comparison scheme is a scheduling scheme (fixed) that specifies a fixed route for the bus and three scheduling schemes based on the greedy algorithm, but respectively taking the sharing rate (SAFS), transportation rate (DRFS) and passenger comfort (PCFS) as the target.

[0153] Figures 5a-5d The abscissa represents the number of buses simulated, Figure 5a The ordinate respectively represents the sharing rate, transportation rate, average passenger walking distance and average waiting time. From Figures 5a-5d It can be seen that the simulation effect of the application is generally better than the other four scheduling schemes. The fixed scheme has a relatively low overall sharing rate, but it shows the most obvious upward trend. Because the increase of bus resources provides more opportunities for requests to be accepted, the passenger transportation rate indicators of the five schemes all show an upward trend with the increase of the number of deployed buses, among which the three scheduling schemes based on the greedy algorithm, DRFS, SAFS and PCFS, have similar overall performance. With the increase of the number of buses, the comfort of passengers in all schemes will be improved, among which the two scheduling schemes that give priority to the sharing rate and passenger transportation rate have similar performance, and the average waiting time of passengers in the DQN scheme is the shortest. The overall performance of the PCFS scheme that gives priority to passenger comfort is not as good as DQN.

[0154] Figure 6The comprehensive index Q value changes obtained by simulation according to different scheduling schemes are shown, and the definition of Q value is as follows:

[0155]

[0156] From Figure 6 It can be seen that the scheduling scheme of the DQN of the present application can make the comprehensive operation benefit of the bus respectively higher than that of the traditional scheduling scheme and the scheduling scheme based on the greedy algorithm by 86% and 29%.

[0157] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements any one of the above methods when executed by a processor.

[0158] As Figure 7 shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage portion 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0159] The following components are connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output portion 707 including a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 708 including a hard disk, and the like; and a communication portion 709 including a network interface card such as a LAN card, a modem, and the like. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is installed into the storage portion 708 as necessary.

[0160] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-described functions defined in the methods of the present application are executed. Note that the computer readable medium described in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF or the like, or any suitable combination of the above.

[0161] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0162] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0163] The modules described in the embodiments of the present application can be implemented through software or hardware. The modules described can be set in a processor, for example, can be described as: a processor includes an acquisition module, an analysis module and an output module. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0164] The above description is merely the preferred embodiments of the present application and the technical principles used. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the above technical features or equivalent features in any combination without departing from the above inventive concept. For example, the above features are replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A bus scheduling method based on reinforcement learning, characterized in that, Includes the following steps: S110. Initialize road network data and perform regional grid division of the road network; S120. Divide trajectory data into time periods and calculate traffic flow in different areas at different times; S130. Calculate and update the traffic flow based on real-time ride request data; S140. When the bus reaches the end of the route, the current road network status is obtained from multiple dimensions, including the average sharing rate. Average passenger transport rate Average walking distance and average waiting time ; S150. Based on a reinforcement learning neural network, and combining the updated traffic flow obtained in step S130 and the road network status obtained in step S140, calculate feedback data on buses heading to different routes; and S160. Repeat steps S130-S150 to obtain the parameters of the reinforcement learning model and perform bus scheduling based on the trained reinforcement learning model. In step S160, the training process of the reinforcement learning model includes the following steps: S161. The current road network status of a bus. Input the value into the evaluation network to obtain an estimate of the current state's value: ; Among them, the current road network status , This indicates the average sharing rate of a bus up to the current time period. This indicates the average passenger transport rate of a bus within its service area up to the current time period. This shows the average walking distance received by a bus up to the current time period. This represents the average waiting time for a bus as of the current time period. Indicates the current time period number; S162, Change the next road network status An estimate of the value of inputting the data into the target network to obtain the next state: ; S163. Obtain the loss function according to the Bellman formula. The formula for the loss function is: ； in, Indicates a time-difference objective. Indicates time difference error. Indicates the act of starting a vehicle. Indicates the next departure action. Indicates the current road network status of the associated bus. Departure behavior Departure behavior Feedback and the next road network status behavioral distribution, Indicates the parameters of the evaluation network. The parameters representing the target network, Indicates the discount factor. Represent the expected function; S164. Parameter tuning of the evaluation network is performed using backpropagation based on the loss function; S165. Update the target network using stable evaluation network parameters; and S166. Repeat steps S161-S165 until the reinforcement learning model converges.

2. The bus scheduling method based on reinforcement learning according to claim 1, characterized in that, In step S150, the feedback data R of the bus heading to different routes is obtained using the following formula: ； in, , , and These represent normalizing and mapping sa, dr, wd, and wt to intervals, respectively. The value after that, This indicates the average sharing rate of a bus up to the current time period. This indicates the average passenger transport rate of a bus within its service area up to the current time period. This represents the average walking distance received by a bus up to the current time period. This represents the average waiting time for a bus as of the current time period.

3. The bus scheduling method based on reinforcement learning according to claim 2, characterized in that, The average sharing rate It can be obtained through the following formula: ； in, This represents the set of ride requests successfully received by the bus company. This indicates a ride request. From the boarding point Arrive at the destination The shortest path distance, Indicates the set of vehicles provided. Indicates vehicle Total driving distance.

4. The bus scheduling method based on reinforcement learning according to claim 2, characterized in that, The average passenger transport rate It can be obtained through the following formula: ; in, This indicates the number of successful bus ride requests received by the bus company. This represents the total number of ride requests in the traffic flow.

5. The bus scheduling method based on reinforcement learning according to claim 2, characterized in that, The average walking distance It can be obtained through the following formula: ; in, Indicates from arrive The straight-line distance Indicates a request for a ride. The actual boarding position of the passengers. Indicates a request for a ride. The actual drop-off location of the passengers. This indicates the number of ride requests successfully received by the bus company.

6. The bus scheduling method based on reinforcement learning according to claim 2, characterized in that, The average waiting time It can be obtained through the following formula: ; in, Indicates that a request has been made. The boarding time of the passengers, This indicates the number of ride requests successfully received by the bus company.

7. The bus scheduling method based on reinforcement learning according to claim 1, characterized in that, Step S120 includes the following steps: S121. Construct a traffic flow table based on the trajectory data. The tuples in the table are of type 1 ,in Indicates a grid area. Indicates time period, This indicates the grid area to which the boarding point of the passenger request belongs. This indicates the grid area to which the drop-off point of the ride request belongs. This indicates the corresponding traffic flow; S122. Obtain the number T of time units in the entire bus operation period. The number T of time units is obtained by the following formula: ； in, This represents the smallest unit of time, the specific duration of which can be customized. This indicates the total operating time of the bus. Represents logarithmic values Round up; and S123. Obtain the traffic flow for each grid, which is calculated using the following formula: ； in, This represents the traffic flow in each grid cell; Traffic flow table tuples in Indicates from time Beginning, with For the first time unit A micro-period; and S124. Obtain the total passenger flow of potential routes r(u, v, t). Where r(u, v, t) represents the total passenger flow of the potential route r(u, v, t) starting from time t, with u and v as the starting and ending stations respectively. It can be obtained through the following formula: ； in, The potential route is All grid sets traversed.

8. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the method as described in any one of claims 1-7.

Images