Multi-line efficient transfer bus planning method based on deep reinforcement learning

Abstract

The invention discloses a multi-line efficient transfer bus planning method based on deep reinforcement learning, and the method comprises the following steps: S1, obtaining the bus travel demand data of a traffic zone of a research region, and obtaining the bus passenger flow of a road segment of the research region; s2, according to the bus passenger flow volume of the road section of the research area in the step S1, a frequent item set mining method is adopted to carry out bus route planning, and a bus route sequence of the research area is obtained; s3, according to the bus route sequence of the research area in the step S2, bus route planning is converted into a Markov decision process, and a bus departure time table scheduling model is constructed; s4, according to the bus departure time table scheduling model constructed in the step S3, searching an optimal bus departure time by adopting a dual deep Q network method; according to the method, the bus travel efficiency is improved, the problem of low bus resource allocation is solved, and flexible and efficient allocation of bus resources is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of transportation, and particularly relates to a multi-line efficient transfer bus planning method based on deep reinforcement learning. Background Art

[0003] Currently, there are many studies on bus line planning and bus departure scheduling problems, mainly focusing on establishing optimization models based on travel costs, operating costs, vehicle and pedestrian constraints, etc., and using different methods to solve the models. For the planning model, most of the existing studies are for single-line buses, aiming to pursue the shortest path, and rarely consider the multi-line transfer problem; for the model solving method, most of the existing studies use heuristic-based algorithms for solving, such as genetic algorithms, simulated annealing algorithms, etc., which are difficult to apply in complex scenarios such as multi-line collaborative transfer; for specific scenarios, the existing studies mainly focus on single problems of road bus planning or customized buses, rapid buses, etc. These measures can only optimize the local area of urban traffic and cannot completely solve the problems existing in the existing road buses. Therefore, the existing bus line planning and bus departure scheduling methods have problems of low bus resource allocation and low bus travel efficiency, and it is necessary to innovate the bus planning method. Summary of the Invention

[0004] In view of the above deficiencies in the prior art, the present invention provides a multi-line efficient transfer bus planning method based on deep reinforcement learning. For the complex traffic scenario of multi-line collaborative transfer of multiple bus lines in a region, by analyzing bus vehicle data, the bus stop travel demand data in the research area is obtained, and the bus vehicle line planning is carried out according to the passenger flow of the sections between bus stops. The bus vehicle line planning in the real scenario is regarded as a combinatorial optimization problem, and the bus vehicle is used as an intelligent agent, and the double deep Q-network method is used to search for the optimal solution to improve the bus travel efficiency and solve the problem of low bus resource allocation efficiency.

[0005] In order to achieve the above invention purpose, the technical solution adopted by the present invention is as follows:

[0006] A multi-line efficient transfer bus planning method based on deep reinforcement learning, comprising the following steps:

[0007] S1. Obtain the bus travel demand data of traffic cells in the research area to obtain the passenger flow of bus vehicles on the sections in the research area;

[0008] S2. According to the passenger flow of bus vehicles on the sections in the research area in step S1, use the frequent item set mining method to carry out bus vehicle line planning to obtain the bus vehicle line sequence in the research area;

[0009] S3. Convert the bus vehicle line planning into a Markov decision process according to the bus vehicle line sequence in the research area in step S2, and construct a bus vehicle departure schedule dispatching model;

[0010] S4. According to the bus vehicle departure schedule dispatching model constructed in step S3, use the double deep Q-network method to search for the optimal bus vehicle departure time.

[0011] Further, step S1 specifically includes:

[0012] S11. Obtain the passenger flow volume data of traffic zones in the research area;

[0013] S12. Map the passenger flow volume data of traffic zones in the research area obtained in step S11 to the bus stops near the traffic zones, and calculate the passenger flow volume between bus stops, that is:

[0014]

[0015] where q i,j,n,m represents the passenger flow volume from traffic zone n and traffic zone m between bus stop i and bus stop j, T n,m represents the passenger flow volume between traffic zone n and traffic zone m, exp(·) represents the exponential function, a and b respectively represent empirical coefficients, L i,j represents the line length between bus stop i and bus stop j, E n represents the set of bus stops near traffic zone n, E m represents the set of bus stops near traffic zone m, L ii,jj represents the line length between bus stop ii and bus stop jj, q i,j represents the passenger flow volume between bus stop i and bus stop j;

[0016] S13. According to the passenger flow volume between bus stops calculated in step S12, use the following formula to obtain the bus vehicle passenger flow volume of the road section in the research area, that is:

[0017]

[0018] where q k represents the bus vehicle passenger flow volume of road section k in the research area, R k represents the pair of bus stops passing through road section k.

[0019] Further, step S2 specifically includes:

[0020] S21. Initialize the bus vehicle passenger flow volume of the road section in the research area in step S1 to obtain the bus vehicle line set and set the support threshold, that is:

[0021]

[0022] Where D represents the bus route set, br1 represents the first bus route, br i represents the i-th bus route, br j represents the jth bus route, br n represents the nth bus line, o represents the starting station of the ith bus line, d represents the destination station of the ith bus line, ss represents the set of bus stations that the ith bus line passes through, and nb represents the passenger flow of the ith bus line;

[0023] S22, select the bus route with the largest passenger flow from the bus route set in step S21. i As the high passenger flow section, select the bus route from the bus route set again i The bus route with the largest passenger flow connecting the starting station and the destination station j , and determine the selected bus route br j Is it greater than the set support threshold? If so, the bus route br i With bus routesbr j The bus routes are connected and stored in the bus route collection as new bus routes, and the original bus routes are deleted. i With bus routesbr j Otherwise, the bus routes will be directly i Save to the bus route collection and delete the original bus routebr i ;

[0024] S23, repeat step S22 until each bus route in the bus route set is less than the set support threshold, then the bus route planning of all high passenger flow sections is completed, and the bus route sequence of the study area is obtained, that is:

[0025] D′=(br′1,br′2,…,br′ n )

[0026] Where D′ represents the bus route sequence in the study area, br′1 represents the first bus route after update, br′2 represents the second bus route after update, and br′ n Indicates the updated nth bus route.

[0027] Furthermore, step S3 specifically includes:

[0028] According to the bus vehicle route sequence in the research area in step S2, taking the bus vehicle as an agent in reinforcement learning, by respectively constructing the state space, action space and comprehensive reward and punishment function of the bus vehicle, a bus vehicle departure schedule dispatching model is obtained.

[0029] Furthermore, taking the bus vehicle as an agent in reinforcement learning, the state space of the bus vehicle is constructed. The specific process is as follows:

[0030] First, according to the running route, running state and passenger capacity of the bus vehicle, the self-state of the bus vehicle is obtained, that is:

[0031]

[0032] Among them, S1 represents the self-state of the bus vehicle at the t-th moment, represents the running route of the bus vehicle at the t-th moment, represents the running state of the bus vehicle at the t-th moment, represents the passenger capacity of the bus vehicle at the t-th moment;

[0033] Secondly, according to the number of waiting passengers of the bus vehicle at the station, the passenger information of the bus vehicle at the station is obtained, that is:

[0034]

[0035] Among them, S2 represents the passenger information of the bus vehicle at the station at the t-th moment, represents the number of waiting passengers of the bus vehicle at station i at the t-th moment;

[0036] Finally, according to the self-state of the bus vehicle and the passenger information of the bus vehicle at the station, the state space of the bus vehicle is constructed, that is:

[0037] S t =(S1, S2)

[0038] Among them, S t represents the state space of the bus vehicle at the t-th moment.

[0039] Furthermore, taking the bus vehicle as an agent in reinforcement learning, the action space of the bus vehicle is constructed. The specific process is as follows:

[0040] According to the state space of the bus vehicle, by defining an action to determine whether the bus vehicle needs to depart, the action space of the bus vehicle is constructed, that is:

[0041] A t =(a1, a2)

[0042] Among them, A tDenote the action space of the bus vehicle at the $t$-th moment. $a_1$ represents the action of dispatching a bus, that is, selecting a bus from the currently non-operating buses for dispatching, and $a_2$ represents the action of not dispatching a bus, that is, not performing any action.

[0043] Furthermore, taking the bus vehicle as an agent in reinforcement learning, a comprehensive reward and punishment function for the bus vehicle is constructed. The specific process is as follows:

[0044] First, based on the operating income and operating cost of the bus vehicle, calculate the enterprise benefit of the bus vehicle, that is:

[0045]

[0046] where $R$ e represents the enterprise benefit of the bus vehicle, $n$ represents the number of bus vehicle routes, $k$ represents the number of operating buses configured for each bus vehicle route, $d$ i,j represents the bus operating distance between bus stop $i$ and bus stop $j$, $p$ ticket represents the current unit passenger fare of the bus, $p$ fuel represents the current unit fuel consumption cost of the bus, $p$ fixed represents the fixed cost of each bus operation, including vehicle wear and tear, vehicle purchase, and personnel salaries, and $D'$ represents the bus vehicle route sequence in the research area;

[0047] Secondly, based on the time cost of passengers on the bus, the time cost of passengers waiting for the bus, and the time cost of passengers walking, calculate the passenger benefit of the bus vehicle, that is:

[0048]

[0049] where $R$ p represents the passenger benefit of the bus vehicle, $R$ on represents the time cost of passengers on the bus, $R$ off represents the time cost of passengers waiting for the bus and the time cost of passengers walking, $M$ represents the set of passengers, $T$ off represents the time when passengers get off the bus, $T$ on represents the time when passengers get on the bus, $\alpha$ represents the time value of bus passengers, $\beta$ represents the time cost of passengers walking and waiting, $Z$ represents the number of bus stops, represents the average walking distance of passengers at bus stop $j$, $v$ walk represents the average walking speed of passengers, $q$ j represents the bus passenger demand at bus stop $j$, $D'$ j represents the decision variable. If $D'$ j = 1, it means the bus vehicle passes through bus stop $j$. If $D'$ j= 0 indicates that the bus does not pass through bus stop j, and max{·} represents taking the maximum value, T j represents the departure time of the bus from bus stop j, T l represents the latest departure time of the bus from bus stop j;

[0050] Finally, based on the calculated enterprise benefit of the bus and the calculated passenger benefit of the bus, the comprehensive reward and punishment function of the bus is obtained, that is:

[0051] R = R p + R e

[0052] where R represents the comprehensive reward and punishment function of the bus.

[0053] Furthermore, step S4 specifically includes:

[0054] S41. Input the state space, action space, comprehensive reward and punishment function, and greedy exploration rate of the bus in the bus departure time table scheduling model in step S3 into the training network Q, and set the parameter update frequency of the target network Q' and the initial bus route departure time table;

[0055] S42. Initialize the network parameters of the training network Q and the target network Q', randomly select the state space of the bus at the current moment, take the departure time under the state space of the bus at the current moment as the initial departure time, and add it to the initial bus route departure time table;

[0056] S43. Input the state space of the bus at the current moment into the training network Q to obtain the Q(s,a,θ) values of all actions, where Q(·) represents the Q value obtained after taking action a under the state space of the bus at the current moment, s represents the state space of the bus at the current moment, that is s = S t , S t is the state space of the bus at the t-th moment, a represents the action space of the bus at the current moment, that is a = A t , A t is the action space of the bus at the t-th moment, θ represents the network parameters of the training network Q, and the greedy algorithm is used to select the action corresponding to the bus from the current Q(s,a,θ) values;

[0057] S44. Execute the action corresponding to the bus in step S43, and obtain the immediate reward of the bus under this action based on the comprehensive reward and punishment function, to obtain the updated state space of the bus;

[0058] S45. Determine whether the updated state space of the bus is the final state. If so, execute step S46; otherwise, execute step S48;

[0059] S46. Increment the number of training times of the training network Q by one, and determine whether the state space of the current bus vehicle is the final state. If so, end the training of the training network Q and execute step S49; otherwise, execute step S47.

[0060] S47. Determine whether the number of training times of the training network Q has reached the maximum value. If so, end the training of the training network Q and execute step S49; otherwise, execute step S42.

[0061] S48. Store the updated state space, action space, and comprehensive reward and punishment function of the bus vehicle in the memory bank, calculate the target Q value using the target network Q', and update the network parameters of the training network Q through the gradient backpropagation of the network using the mean square error loss function according to the calculated target value to obtain the trained training network Q.

[0062] S49. Use the trained training network Q to schedule the bus vehicle according to the state space, action space, and comprehensive reward and punishment function of the bus vehicle in the bus departure schedule model to obtain the optimal bus departure time.

[0063] Furthermore, the calculation formula for the target Q value in step S48 is:

[0064]

[0065] where y i represents the final target Q value, r j represents the immediate reward obtained by taking action a at the current moment j, where the immediate reward is calculated by the comprehensive reward and punishment function for the current action, γ represents the reward discount factor, s represents the state space of the bus vehicle at the current moment, where s = S t and S t represents the state space of the bus vehicle at the t-th moment, a represents the action space of the bus vehicle at the current moment, where a = A t and A t represents the action space of the bus vehicle at the t-th moment, θ represents the network parameters of the training network Q, θ′ represents the network parameters of the target network Q', Q′(·) represents inputting the action corresponding to the maximum Q value estimated by the training network Q into the target network Q' for Q value calculation, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current moment, argmax a′ Q(s,a,θ) represents the action corresponding to the estimated maximum Q value obtained by the training network Q, and s′ is the updated state space of the bus vehicle.

[0066] Furthermore, the calculation formula for the mean square error loss function in step S48 is:

[0067] L(θ) = E[y i - Q(s, a, θ)] 2

[0068] where L (θ) represents the mean squared error loss function, E[·] represents the mean calculation, y i represents the final target Q value, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current moment, s represents the state space of the bus vehicle at the current moment, i.e., s = S t , S t is the state space of the bus vehicle at the t-th moment, a represents the action space of the bus vehicle at the current moment, i.e., a = A t , A t is the action space of the bus vehicle at the t-th moment, and θ represents the network parameters of the training network Q.

[0069] The present invention has the following beneficial effects:

[0070] A multi-line efficient transfer bus planning method based on deep reinforcement learning proposed by the present invention, aiming at the complex traffic scenario of coordinated transfer of multiple bus lines in a region, obtains the travel demand data of bus stops in the research area by analyzing bus vehicle data, and plans the bus vehicle lines according to the passenger flow volume of the sections between bus stops, formulates the bus vehicle line planning in the real scenario as a combinatorial optimization problem, and takes the bus vehicle as an intelligent agent, and uses the double deep Q network method to search for the optimal solution, which improves the travel efficiency of bus vehicles, solves the problem of low bus resource allocation, and realizes the flexible and efficient allocation of bus resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a schematic flow chart of a multi-line efficient transfer bus planning method based on deep reinforcement learning proposed by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0072] The following describes the specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the inventive concept of the present invention are within the scope of protection.

[0073] As Figure 1 shown, a multi-line efficient transfer bus planning method based on deep reinforcement learning includes the following steps S1 - S4:

[0074] S1. Obtain the bus travel demand data of traffic zones in the research area, and get the passenger flow of bus vehicles on the roads in the research area.

[0075] In this embodiment, import the processed passenger flow data of each traffic zone, and map it to the bus stops around the traffic zones. By calculating the passenger flow between bus stops, the travel volume distribution of bus vehicle lines is realized. At the same time, superimpose the travel volumes between the pairs of bus stops corresponding to each bus line passing through a certain road section to obtain the passenger flow of bus vehicles on each road section in the research area.

[0076] Specifically, step S1 specifically includes S11 - S13:

[0077] S11. Obtain the passenger flow data of traffic zones in the research area.

[0078] S12. Map the passenger flow data of traffic zones in the research area obtained in step S11 to the bus stops near the traffic zones, and calculate the passenger flow between bus stops, that is:

[0079]

[0080] Among them, q i,j,n,m represents the passenger flow from bus stop i to bus stop j from traffic zone n and traffic zone m, T n,m represents the passenger flow between traffic zone n and traffic zone m, exp(·) represents the exponential function, a and b respectively represent empirical coefficients, L i,j represents the line length between bus stop i and bus stop j, E n represents the set of bus stops near traffic zone n, E m represents the set of bus stops near traffic zone m, L ii,jj represents the line length between bus stop ii and bus stop jj, q i,j represents the passenger flow between bus stop i and bus stop j.

[0081] S13. According to the passenger flow between bus stops calculated in step S12, use the following formula to obtain the passenger flow of bus vehicles on the road sections in the research area, that is:

[0082]

[0083] Among them, q k represents the passenger flow of bus vehicles on road section k in the research area, R k represents the pair of bus stops passing through road section k.

[0084] S2. According to the passenger flow of bus vehicles on the road sections in the research area in step S1, use the frequent item set mining method to plan the bus vehicle lines, and obtain the bus vehicle line sequence in the research area.

[0085] In this embodiment, according to the passenger flow of bus vehicles on the road sections in the research area, the frequent item set mining method (Apriori) is used to generate bus vehicle line sequences, so as to realize the selection, extension and generation of bus vehicle lines.

[0086] Specifically, step S2 specifically includes S21 - S23:

[0087] S21. Initialize the passenger flow of bus vehicles on the road sections in step S1 to obtain a bus vehicle line set and set a support threshold, that is:

[0088]

[0089] Among them, D represents the bus vehicle line set, br1 represents the first bus vehicle line, br i represents the i-th bus vehicle line, br j represents the j-th bus vehicle line, br n represents the n-th bus vehicle line, o represents the starting station of the i-th bus vehicle line, d represents the destination station of the i-th bus vehicle line, ss represents the set of bus stops passed by the i-th bus vehicle line, and nb represents the passenger flow of the i-th bus vehicle line.

[0090] In this embodiment, the support threshold is represented by Th.

[0091] S22. Select the bus vehicle line br i with the largest passenger flow from the bus vehicle line set in step S21 as the high-passenger flow section, and select again from the bus vehicle line set the bus vehicle line br i with the largest passenger flow whose starting station and destination station are connected to those of the bus vehicle line br j , and determine whether the selected bus vehicle line br j is greater than the set support threshold. If so, connect the bus vehicle line br i and the bus vehicle line br j and store them as a new bus vehicle line in the bus vehicle line set, and at the same time delete the original bus vehicle line br i and the bus vehicle line br j . Otherwise, directly store the bus vehicle line br i in the bus vehicle line set and delete the original bus vehicle line br i .

[0092] In this embodiment, the process of selecting bus vehicle lines is as follows: compare each line in the bus vehicle line set (D), and select the bus vehicle line br iAs a high passenger flow section; the bus line extension process is: select the bus line from the bus line set (D) and then i The bus route with the largest passenger flow connecting the starting station and the destination station j , and determine the selected bus route br j Is it greater than the set support threshold? If so, the bus route br i With bus routesbr j The bus routes are connected and stored in the bus route collection as new bus routes, and the original bus routes are deleted. i With bus routesbr j , otherwise, it means the bus route br i If the conditions for extension are not met, the bus routes will be directly extended. i Stored in the bus route collection.

[0093] S23, repeat step S22 until each bus route in the bus route set is less than the set support threshold, then the bus route planning of all high passenger flow sections is completed, and the bus route sequence of the study area is obtained, that is:

[0094] D′=(br′1,br′2,…,br′ n )

[0095] Where D′ represents the bus route sequence in the study area, br′1 represents the first bus route after update, br′2 represents the second bus route after update, and br′ n Indicates the updated nth bus route.

[0096] S3. According to the bus route sequence in the research area in step S2, the bus route planning is converted into a Markov decision process, and a bus departure timetable scheduling model is constructed.

[0097] In this embodiment, according to the bus route sequence in the study area, the bus route planning is converted into a Markov decision process, that is, the bus is used as an intelligent agent in reinforcement learning, and a bus departure schedule scheduling model is constructed, that is, the state space, action space and comprehensive reward and punishment function of the bus are set respectively.

[0098] Specifically, step S3 includes:

[0099] According to the bus route sequence in the research area in step S2, the bus is used as an intelligent agent in reinforcement learning, and the bus departure timetable scheduling model is obtained by constructing the state space, action space and comprehensive reward and punishment function of the bus respectively.

[0100] In this embodiment, the agent mainly makes decision judgments based on the current vehicle's own state information S1 and the passenger information S2 at each bus stop at the current moment. Therefore, the state of the bus vehicle at time t is defined as Meanwhile, the passenger information at the bus stop at time t is set as Then the state space of the bus vehicle at time t can be expressed as S t =(S1, S2).

[0101] Specifically, taking the bus vehicle as the agent in reinforcement learning and constructing the state space of the bus vehicle, the specific process is as follows: First, according to the running route, running state, and passenger capacity of the bus vehicle, the state of the bus vehicle itself is obtained, that is:

[0102]

[0103] where S1 represents the state of the bus vehicle itself at the t-th moment, represents the running route of the bus vehicle at the t-th moment, represents the running state of the bus vehicle at the t-th moment, represents the passenger capacity of the bus vehicle at the t-th moment.

[0104] In this embodiment, represents the running state of the bus vehicle at the t-th moment. When mov = 0, it means running, and when mov = 1, it means in the process of running.

[0105] Secondly, according to the number of waiting passengers of the bus vehicle at the stop, the passenger information of the bus vehicle at the stop is obtained, that is:

[0106]

[0107] where S2 represents the passenger information of the bus vehicle at the stop at the t-th moment, represents the number of waiting passengers of the bus vehicle at stop i at the t-th moment.

[0108] Finally, according to the state of the bus vehicle itself and the passenger information of the bus vehicle at the stop, the state space of the bus vehicle is constructed, that is:

[0109] S t =(S1, S2)

[0110] where S t represents the state space of the bus vehicle at the t-th moment.

[0111] In this embodiment, the bus vehicle makes a decision according to the state information S at the current moment t tTo decide whether a bus needs to depart, thus, define action a1 as the need to depart, that is, select a bus from the currently non - operating buses to depart; define action a2 as that a certain bus line does not need to depart, so no action needs to be taken; therefore, the action space of the bus at time t is A t =(a1,a2).

[0112] Specifically, taking the bus as an agent in reinforcement learning, construct the action space of the bus. The specific process is as follows: According to the state space of the bus, determine whether the bus needs to depart by defining actions, and construct the action space of the bus, that is:

[0113] A t =(a1,a2)

[0114] Among them, A t represents the action space of the bus at the t - th moment. a1 represents the action of needing to depart, that is, select a bus from the currently non - operating buses to depart, and a2 represents the action of not needing to depart, that is, do not take any action.

[0115] In this embodiment, the reward - punishment function is set according to the principle of maximizing the total social benefit. The total social benefit of bus operation is divided into enterprise benefit and passenger benefit. Therefore, the bus route planning needs to consider the needs of both enterprises and passengers. The enterprise benefit is the operating profit of the bus enterprise. The calculation method is the operating income minus the operating cost. The operating income is mainly the passenger fare income, and the operating cost includes the fuel consumption cost related to the bus operation route and the fixed costs such as vehicle purchase, vehicle loss, and personnel salary. The passenger benefit is mainly the time - cost benefit of passengers during the entire bus journey, including the time - cost of passengers on the bus, the time - cost of passengers waiting for the bus, and the time - cost of passengers walking.

[0116] Specifically, taking the bus as an agent in reinforcement learning, construct the comprehensive reward - punishment function of the bus. The specific process is as follows: First, calculate the enterprise benefit of the bus according to the operating income and operating cost of the bus, that is:

[0117]

[0118] Among them, R e represents the enterprise benefit of the bus, n represents the number of bus lines, k represents the number of operating buses configured for each bus line, d i,j represents the bus operation distance between bus stop i and bus stop j, p ticket represents the current unit passenger fare of the bus, p fuel represents the current unit fuel consumption cost of the bus, p fixedIt represents the fixed cost of each bus operation, including vehicle wear and tear, vehicle purchase, and personnel salaries. D′ represents the bus line sequence in the research area.

[0119] Secondly, according to the time cost of passengers on the bus, the time cost of passengers waiting for the bus, and the walking time cost of passengers, calculate the passenger benefit of the bus, that is:

[0120]

[0121] Among them, R p represents the passenger benefit of the bus, and R on represents the time cost of passengers on the bus, and R off represents the time cost of passengers waiting for the bus and the walking time cost of passengers. M represents the set of passengers, T off represents the time when passengers get off the bus, and T on represents the time when passengers get on the bus. α represents the time value of bus passengers, β represents the walking and waiting time cost of passengers, Z represents the number of bus stops, represents the average walking distance of passengers at bus stop j, v walk represents the average walking speed of passengers, q j represents the bus passenger demand at bus stop j, D′ j represents the decision variable. If D′ j =1, it means the bus passes through bus stop j. If D′ j =0, it means the bus does not pass through bus stop j. max{·} represents taking the maximum value. T j represents the departure time of the bus from bus stop j, and T l represents the latest departure time of the bus from bus stop j.

[0122] In this embodiment, as long as the bus departs before the latest departure time, passengers will not incur the time cost of waiting for the bus.

[0123] Finally, according to the calculated enterprise benefit of the bus and the calculated passenger benefit of the bus, obtain the comprehensive reward and punishment function of the bus, that is:

[0124] R = R p + R e

[0125] Among them, R represents the comprehensive reward and punishment function of the bus.

[0126] S4. According to the bus departure schedule dispatching model constructed in step S3, use the double deep Q-network method to search for the optimal bus departure time.

[0127] In this embodiment, the double deep Q-network method is used to search for the optimal departure time of the bus vehicle.

[0128] Specifically, step S4 specifically includes S41 - S49:

[0129] S41. Input the state space, action space, comprehensive reward and punishment function, and greedy exploration rate of the bus vehicle in the bus vehicle departure schedule model in step S3 into the training network Q, and set the parameter update frequency of the target network Q' and the initial bus vehicle line departure schedule.

[0130] In this embodiment, the greedy exploration rate is ε, and the parameter update frequency of the target network Q' is C, that is, the network parameters of the target network Q' are updated every C steps.

[0131] S42. Initialize the network parameters of the training network Q and the target network Q', randomly select the state space of the bus vehicle at the current moment, use the departure time in the state space of the bus vehicle at the current moment as the initial departure time, and add it to the initial bus vehicle line departure schedule.

[0132] S43. Input the state space of the bus vehicle at the current moment into the training network Q to obtain the Q(s, a, θ) values of all actions, where Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current moment, s represents the state space of the bus vehicle at the current moment, that is, s = S t ,S t is the state space of the bus vehicle at the t-th moment, a represents the action space of the bus vehicle at the current moment, that is, a = A t ,A t is the action space of the bus vehicle at the t-th moment, θ represents the network parameters of the training network Q, and the greedy method is used to select the action corresponding to the bus vehicle from the current Q(s, a, θ) values.

[0133] S44. Execute the action corresponding to the bus vehicle in step S43, obtain the immediate reward of the bus vehicle under this action based on the comprehensive reward and punishment function, and obtain the updated state space of the bus vehicle.

[0134] In this embodiment, the immediate reward of the bus vehicle is the reward value calculated according to the comprehensive reward and punishment function.

[0135] S45. Determine whether the updated state space of the bus vehicle is the final state. If so, execute step S46; otherwise, execute step S48.

[0136] S46. Increment the training times of the training network Q by one, and determine whether the current state space of the bus vehicle is the final state. If so, end the training of the training network Q and execute step S49; otherwise, execute step S47.

[0137] S47. Determine whether the number of training times of the training network Q reaches the maximum value. If so, end the training of the training network Q and execute step S49; otherwise, execute step S42.

[0138] S48. Store the updated state space, action space, and comprehensive reward and punishment function of the bus vehicle in the memory bank. Calculate the target Q value using the target network Q', and update the network parameters of the training network Q through the gradient backpropagation of the network using the mean square error loss function based on the calculated target value to obtain the trained training network Q.

[0139] Specifically, the calculation formula for the target Q value in step S48 is:

[0140]

[0141] where y i represents the final target Q value, r j represents the immediate reward obtained by taking action a at the current time j. The immediate reward is calculated by the current action through the comprehensive reward and punishment function. γ represents the reward discount factor, s represents the state space of the bus vehicle at the current time, where s = S t and S t represents the state space of the bus vehicle at the t-th time, a represents the action space of the bus vehicle at the current time, where a = A t and A t represents the action space of the bus vehicle at the t-th time, θ represents the network parameters of the training network Q, θ' represents the network parameters of the target network Q', Q′(·) represents inputting the action corresponding to the maximum Q value estimated by the training network Q into the target network Q' for Q value calculation, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current time, argmax a′ Q(s,a,θ) represents the action corresponding to the estimated maximum Q value obtained by the training network Q, and s′ is the updated state space of the bus vehicle.

[0142] In this embodiment, γ represents the reward discount factor, and its value range is [0,1];

[0143] Specifically, the calculation formula for the mean square error loss function in step S48 is:

[0144] L (θ) = E[y i - Q(s,a,θ)] 2

[0145] where L (θ) represents the mean square error loss function, E[·] represents the mean calculation, and y iIt represents the final target Q value, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current moment, s represents the state space of the bus vehicle at the current moment, that is, s = S t , S t is the state space of the bus vehicle at the t-th moment, a represents the action space of the bus vehicle at the current moment, that is, a = A t , A t is the action space of the bus vehicle at the t-th moment, and θ represents the network parameters of the training network Q

[0146] S49. Using the trained training network Q, according to the state space, action space and comprehensive reward and punishment function of the bus vehicle departure schedule model, the optimal departure time of the bus vehicle is obtained

[0147] A multi-line efficient transfer bus planning method based on deep reinforcement learning proposed by the present invention aims at the complex traffic scenario of coordinated transfer of multiple bus lines in a region. By analyzing bus vehicle data, the travel demand data of bus stops in the research area is obtained, and the bus vehicle line planning is carried out according to the passenger flow on the road sections between bus stops. The bus vehicle line planning in the real scenario is regarded as a combinatorial optimization problem, and the bus vehicle is regarded as an intelligent agent. The double deep Q network method is used to search for the optimal solution, which improves the travel efficiency of the bus vehicle and solves the problem of low bus resource allocation, thus realizing the flexible and efficient allocation of bus resources

[0148] In the present invention, specific embodiments are used to elaborate the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation of the present invention

[0149] Those of ordinary skill in the art will realize that the embodiments described here are for helping readers understand the principle of the present invention. It should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not deviate from the essence of the present invention according to the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention

Claims

1. A multi-line efficient transfer bus planning method based on deep reinforcement learning, characterized in that It includes the following steps: S1. Obtain the bus travel demand data of traffic zones in the research area, and get the passenger flow of buses on the roads in the research area; S2. According to the passenger flow of buses on the roads in the research area in step S1, use the frequent item set mining method to plan the bus routes, and obtain the bus route sequence in the research area; S3. According to the bus route sequence in the research area in step S2, convert the bus route planning into a Markov decision process, and construct a bus departure schedule dispatching model; S4. According to the bus departure schedule dispatching model constructed in step S3, use the double deep Q-network method to search for the optimal bus departure time.

2. The multi-route efficient transfer bus planning method based on deep reinforcement learning according to claim 1, characterized in that, Step S1 specifically includes: S11. Obtain the passenger flow data of traffic zones in the research area; S12. Map the passenger flow data of traffic zones in the research area obtained in step S11 to the bus stops near the traffic zones, and calculate the passenger flow between bus stops, that is: where q i,j,n,m represents the passenger flow from traffic zone n to traffic zone m between bus stop i and bus stop j, T n,m represents the passenger flow between traffic zone n and traffic zone m, exp(·) represents the exponential function, and a and b respectively represent empirical coefficients, L i,j represents the line length between bus stop i and bus stop j, E n represents the set of bus stops near traffic zone n, E m represents the set of bus stops near traffic zone m, L ii,jj represents the line length between bus stop ii and bus stop jj, q i,j represents the passenger flow between bus stop i and bus stop j; S13. According to the passenger flow between bus stops calculated in step S12, use the following formula to get the passenger flow of buses on the roads in the research area, that is: where q k represents the passenger flow of buses on section k in the research area, and R k represents the pairs of bus stops passing through section k.

3. A multi-route efficient transfer bus planning method based on deep reinforcement learning according to claim 1, characterized in that Step S2 specifically includes: S21. Initialize the passenger flow of buses on the roads in the research area in step S1, obtain the bus route set and set the support threshold, that is: Among them, D represents the set of bus vehicle routes, br1 represents the first bus vehicle route, and br i represents the i-th bus vehicle route, and br j represents the j-th bus vehicle route, and br n represents the n-th bus vehicle route, o represents the starting station of the i-th bus vehicle route, d represents the destination station of the i-th bus vehicle route, ss represents the set of bus stops passed by the i-th bus vehicle route, and nb represents the passenger flow of the i-th bus vehicle route; S22, select the bus route with the largest passenger flow from the bus route set in step S21. i As the high passenger flow section, select the bus route from the bus route set again i The bus route with the largest passenger flow connecting the starting station and the destination station j , and determine the selected bus route br j Is it greater than the set support threshold? If so, the bus route br i With bus routesbr j The bus routes are connected and stored in the bus route collection as new bus routes, and the original bus routes are deleted. i With bus routesbr j Otherwise, the bus routes will be directly i Save to the bus route collection and delete the original bus routebr i ; S23. Repeat step S22 until each bus route in the bus route set is less than the set support threshold, then complete the bus route planning for all high-passenger-flow roads, and obtain the bus route sequence in the research area, that is: D′ = (br′1, br′2, …, br′ n ) Among them, D′ represents the bus vehicle line sequence in the research area, br′1 represents the first updated bus vehicle line, br′2 represents the second updated bus vehicle line, and br′ n represents the nth updated bus vehicle line.

4. A multi-route efficient transfer bus planning method based on deep reinforcement learning according to claim 1, characterized in that Step S3 specifically includes: According to the bus route sequence in the research area in step S2, take the bus as an agent in reinforcement learning, and by constructing the state space, action space and comprehensive reward and punishment function of the bus respectively, obtain the bus departure schedule dispatching model.

5. A multi-line efficient transfer bus planning method based on deep reinforcement learning according to claim 4, characterized in that Take the bus as an agent in reinforcement learning, and construct the state space of the bus. The specific process is: First, according to the running route, running status and passenger load of the bus, obtain the self-state of the bus, that is: Among them, S1 represents the self - state of the bus vehicle at the t - th moment, represents the operating route of the bus vehicle at the t - th moment, represents the operating state of the bus vehicle at the t - th moment, represents the number of passengers on the bus vehicle at the t - th moment; Second, according to the number of waiting passengers at the bus stop of the bus, obtain the passenger information of the bus at the bus stop, that is: Among them, \(S_2\) represents the passenger information of the bus at the station at the \(t\)th moment, indicating the number of passengers waiting for the bus at station \(i\) at the \(t\)th moment; Finally, according to the self-state of the bus and the passenger information of the bus at the bus stop, construct the state space of the bus, that is: S t =(S1, S2) Among them, S t represents the state space of the bus at the t-th moment.

6. The multi-line efficient transfer bus planning method based on deep reinforcement learning according to claim 4, wherein, Take the bus as an agent in reinforcement learning, and construct the action space of the bus. The specific process is: According to the state space of the bus, define an action to determine whether the bus needs to depart, and construct the action space of the bus, that is: A t =(a1,a2) Among them, A t represents the action space of the bus vehicle at the t-th moment. a1 represents the action of dispatching a bus, that is, selecting a bus from the currently non-operating buses to dispatch. a2 represents the action of not dispatching a bus, that is, not performing any action.

7. A multi-route efficient transfer bus planning method based on deep reinforcement learning according to claim 4, characterized in that Take the bus as an agent in reinforcement learning, and construct the comprehensive reward and punishment function of the bus. The specific process is: First, according to the operating income and operating cost of the bus, calculate the enterprise benefit of the bus, that is: Among them, R e represents the enterprise benefit of bus vehicles, n represents the number of bus vehicle lines, k represents the number of operating bus vehicles configured for each bus vehicle line, and d i,j represents the bus operation distance between bus stop i and bus stop j, and p ticket represents the current bus fare per passenger, and p fuel represents the current unit fuel consumption cost of the bus, and p fixed represents the fixed cost per bus operation, including vehicle loss, vehicle purchase, and personnel salary. D′ represents the bus vehicle line sequence in the research area; Second, according to the time cost of passengers on the bus, the time cost of passengers waiting for the bus and the walking time cost of passengers, calculate the passenger benefit of the bus, that is: Among them, R p represents the passenger benefit of the bus vehicle, R on represents the time cost of passengers on the bus vehicle, R off represents the time cost of passengers waiting for the bus vehicle and the time cost of passengers' walking. M represents the set of passengers, T off represents the time when passengers get off the bus, T on represents the time when passengers get on the bus. α represents the time value of bus passengers, β represents the time cost of passengers' walking and waiting, Z represents the number of bus stops, represents the average walking distance of passengers at bus stop j, v walk represents the average walking speed of passengers, q j represents the bus passenger demand at bus stop j, D j ′ represents a decision variable. If D j ′ = 1, it means the bus vehicle passes through bus stop j. If D j ′ = 0, it means the bus vehicle does not pass through bus stop j. max{·} represents taking the maximum value, T j represents the departure time of the bus vehicle from bus stop j, T l represents the latest departure time of the bus vehicle from bus stop j; Finally, according to the calculated enterprise benefit of the bus and the calculated passenger benefit of the bus, obtain the comprehensive reward and punishment function of the bus, that is: R = R p + R e Among them, R represents the comprehensive reward and punishment function of the bus.

8. The multi-line efficient transfer bus planning method based on deep reinforcement learning according to claim 1, characterized in that Step S4 specifically includes: S41. Input the state space, action space, comprehensive reward and punishment function, and greedy exploration rate of the bus in the bus departure schedule dispatching model in step S3 into the training network Q, and set the parameter update frequency of the target network Q' and the initial bus route departure schedule; S42. Initialize the network parameters of the training network Q and the target network Q', randomly select the state space of the bus at the current moment, use the departure time under the state space of the bus at the current moment as the initial departure time, and add it to the initial bus route departure schedule; S43. Input the state space of the bus vehicle at the current moment into the training network Q to obtain the Q(s,a,θ) values of all actions. Here, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current moment, s represents the state space of the bus vehicle at the current moment, i.e., s = S t , S t is the state space of the bus vehicle at the t-th moment, a represents the action space of the bus vehicle at the current moment, i.e., a = A t , A t is the action space of the bus vehicle at the t-th moment, θ represents the network parameters of the training network Q, and the greedy method is used to select the action corresponding to the bus vehicle from the current Q(s,a,θ) values; S44. Execute the action corresponding to the bus in step S43, obtain the immediate reward of the bus under this action based on the comprehensive reward and punishment function, and obtain the updated state space of the bus; S45. Determine whether the updated state space of the bus is the final state. If so, execute step S46; otherwise, execute step S48; S46. Increment the training times of the training network Q by one, and determine whether the current state space of the bus is the final state. If so, end the training of the training network Q and execute step S49; otherwise, execute step S47; S47. Determine whether the training times of the training network Q reach the maximum value. If so, end the training of the training network Q and execute step S49; otherwise, execute step S42; S48. Store the updated state space, action space, and comprehensive reward and punishment function of the bus in the memory bank, calculate the target Q value using the target network Q', and update the network parameters of the training network Q through the gradient backpropagation of the network using the mean square error loss function according to the calculated target value to obtain the trained training network Q; S49. Use the trained training network Q to obtain the optimal bus departure time according to the state space, action space, and comprehensive reward and punishment function of the bus in the bus departure schedule dispatching model.

9. A multi-route efficient transfer bus planning method based on deep reinforcement learning according to claim 8, wherein The calculation formula for the target Q value in step S48 is: Among them, y i represents the final target Q value, r j represents the immediate reward obtained by taking action a at the current time j. Even if the reward is calculated by the current action through the comprehensive reward and punishment function, γ represents the reward discount factor, s represents the state space of the bus vehicle at the current time, where s = S t and S t represents the state space of the bus vehicle at the t-th time, a represents the action space of the bus vehicle at the current time, where a = A t and A t represents the action space of the bus vehicle at the t-th time, θ represents the network parameters of the training network Q, θ′ represents the network parameters of the target network Q', Q′(·) represents inputting the action corresponding to the maximum Q value estimated by the training network Q into the target network Q' for Q value calculation, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current time, argmax a′ Q(s,a,θ) represents the action corresponding to the estimated maximum Q value obtained by the training network Q, and s′ is the updated state space of the bus vehicle.

10. A multi-line efficient transfer bus planning method based on deep reinforcement learning according to claim 8, characterized in that The calculation formula for the mean square error loss function in step S48 is: L (θ) = E[y i - Q(s,a,θ)] 2 Among them, L (θ) represents the mean squared error loss function, E[·] represents the mean calculation, and y i represents the final target Q value, Q(·) represents the Q value obtained after taking action a in the state space of the bus vehicle at the current moment, s represents the state space of the bus vehicle at the current moment, that is, s = S t , S t is the state space of the bus vehicle at the t-th moment, a represents the action space of the bus vehicle at the current moment, that is, a = A t , A t is the action space of the bus vehicle at the t-th moment, and θ represents the network parameters of the training network Q.

Images