Idle taxi dispatching method based on many-to-many deep reinforcement learning algorithm

Abstract

The application discloses a kind of idle taxi scheduling methods based on multiple-to-multiple deep reinforcement learning algorithm, comprising: taxi scheduling area is divided into grid, and the taxi relocation task is defined as partially observable Markov decision process, and optimization target is defined to construct taxi relocation model;Wherein, optimization target is the balance of taxi supply and demand relationship, and relocation model includes critic part and actor part, and actor part includes multiple policy networks, and each grid corresponds to a policy network;Critic part includes value network and target network, and two networks cooperate to predict the global state value of a certain specific time as accurately as possible;Using multiple-to-multiple deep reinforcement learning algorithm to train the taxi relocation model;Idle taxi scheduling is realized using the trained taxi relocation model, and the scheduling result is obtained.The relocation strategy obtained by the application can make supply and demand more balanced, improve response rate and reduce taxi response time.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of traffic engineering and artificial intelligence, and in particular to an idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm for the purpose of balancing supply and demand. Background Technology

[0002] Currently, ride-hailing platforms are very popular. Passengers simply send their pick-up and drop-off locations to the platform, which automatically matches them with a suitable taxi. The platform monitors taxi locations in real time and responds to requests as quickly as possible. Due to the complexity of urban transportation systems and the large geographical areas of cities, supply and demand imbalances are a frequent problem for these platforms. Some areas have many available vehicles but low demand for taxis; while other areas have high demand but relatively few vehicles. In areas with oversupply, drivers have to compete for several rides, while in areas with undersupply, passengers have to wait a long time to get a taxi. For example, in... Figure 2 In the context of the problem, region Q1 has many available taxis but no passengers. Regions Q2, Q3, and Q4 have some passengers but no taxis, indicating a supply-demand imbalance. To achieve balance, we should consider the different demand levels in Q2, Q3, and Q4 to effectively reallocate vehicles in Q1.

[0003] To balance supply and demand, many studies treat the taxi relocation problem as a network flow problem and solve it using operations research algorithms. However, due to the time-consuming nature of these algorithms, some recent research has treated the relocation task as a Markov decision process and used multi-agent reinforcement learning to train the policy model. Assuming the entire city is divided into a hexagonal grid, the trained policy model can guide idle taxis to move from one cell to another. However, in multi-agent reinforcement learning, overly complex actions can make the model difficult to converge. Therefore, recent research has had to relocate taxis in a one-to-one manner, meaning all idle taxis either remain in the current cell or cluster in another cell. Existing HTRS (Homogeneous Taxi Relocation System) guides taxis to an adjacent cell, while META (Heterogeneous Taxi Relocation System) guides taxis to a nearby cell that may not be adjacent. This approach is not practical in real-world applications, and the scheduling results are not ideal. Summary of the Invention

[0004] This invention provides an idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm to solve the technical problem of imbalance between supply and demand of urban taxis.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] On one hand, this invention provides an idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm, the idle taxi scheduling method based on the many-to-many deep reinforcement learning algorithm comprising:

[0007] The taxi dispatch area is divided into grids, and the taxi relocation task is defined as a partially observable Markov decision process. An optimization objective is defined to construct a taxi relocation model. The optimization objective is to balance the supply and demand of taxis. The taxi relocation model includes a commentator part and an actor part. The actor part includes multiple policy networks, with one policy network corresponding to each grid. The taxi determines its action based on the policy network of its grid. The commentator part includes a value network and a target network. The value network and the target network cooperate to predict the global state value at a specific time as accurately as possible.

[0008] The taxi relocation model was trained using a many-to-many deep reinforcement learning algorithm.

[0009] The trained taxi relocation model is used to schedule idle taxis, and the scheduling results are obtained.

[0010] Furthermore, the taxi dispatch area is divided into grids, the taxi relocation task is defined as a partially observable Markov decision process, and an optimization objective is defined, including:

[0011] The taxi dispatch area is divided into grids, with the dispatch area equally divided into multiple hexagonal grids;

[0012] Define intelligent agents, treating each taxi as an intelligent agent and the taxi operation scenario as the environment;

[0013] Define actions; each action set includes seven actions: stay in place and move to the other six adjacent grids; each grid corresponds to an action set, and taxis entering the same grid share the same action set; the global action space is the combination of all action sets, and the global action at a certain moment is a vector composed of the actions performed by all taxis;

[0014] The state is defined by representing the state of a single grid using a three-dimensional vector. The elements of the three-dimensional vector represent the demand quantity, the number of available taxis, and the number of taxis currently in service in the current grid, respectively. The global state at any given time consists of the states of all grids. Taxi located in a grid does not have a global view, but only a partial view called a local observation, which includes the state of the taxi's current grid and its six neighboring grids. Taxi in the same grid share the same local observation. The global state will be used to train the commentator part of the taxi relocation model, while the local observations will be used to train the grid-related policy network in the actor part of the taxi relocation model.

[0015] Define a policy function and use a neural network to approximate the policy function. Taxis in the same grid use the same policy function. At a certain moment, the taxis in the grid perform actions based on their corresponding policy functions by observing the local observations at the current moment.

[0016] Define rewards: after performing an action, the taxi receives a reward from the environment. Each grid provides two types of rewards, namely, the first type of reward. Second type of reward in, The expression is:

[0017]

[0018] in, Indicates the grid c where the taxi is located. j The demand quantity in c j The number of available taxis in the area; The result of the calculation is a decimal between 0 and 1. Then c j Supply and demand will reach equilibrium; The closer to 0, the better for c. j The greater the imbalance between supply and demand;

[0019] The expression is:

[0020]

[0021] when When, then c j When supply and demand reach a state of equilibrium; When c is negative, then j Supply exceeds demand; when When c is a positive number, then j Supply cannot meet demand;

[0022] use Define the global reward obtained by all taxis after performing operations at a given moment. The global reward is used to train the commentator component; The expression is:

[0023]

[0024] in, It is grid c j The first type of reward obtained at time t;

[0025] use Define the local reward obtained by the taxi after performing an action at a certain moment. Right now:

[0026]

[0027] in, Indicates time t by c j The second type of reward provided; Indicates time t by c j The second type of reward is provided by the nbr-th adjacent grid; This indicates that the taxi should remain in place. This indicates that the taxi performs the action of moving towards the nth adjacent grid; ω j and ω j nbr These are preset parameters used to measure the signal from c. j and c j The nth grid c j nbr The reward;

[0028] Define the optimization problem. To evaluate future long-term returns, use γ as a discount factor. The global discounted returns are as follows:

[0029] U t =R t +γ·R t+1 +γ 2 ·R t+2 +γ 3 ·R t+3 +…

[0030] Among them, R t Let represent the random variable of the global reward at time t, and the global action-value function be as follows:

[0031] Q(s t ,a t )=E[U t |S t =s t At =a t ]

[0032] Among them, s t a represents the global state at time t. t Let represent the global action at time t; E represents the expected value of the global discounted reward.

[0033] And the global state-value function is as follows:

[0034] V(s t ) = E A [Q(s t A)]

[0035] Here, A refers to the random variable of the global action at time t; E A This represents the expectation of the global action-value function under condition A;

[0036] The optimization problem is:

[0037]

[0038] Where S is a random variable, representing the global state at time t, and θ 1 ,θ 2 ,…,θ k Let E represent the parameters of the 1st policy network, the 2nd policy network, ..., the kth policy network, respectively; S This represents the expectation of solving the global state-value function under the condition of S;

[0039] The optimization problem is to minimize the expectation in the equation by adjusting the parameters of a network of k policy agents.

[0040] Furthermore, the step of using a many-to-many deep reinforcement learning algorithm to train the taxi relocation model includes:

[0041] An agent performs actions, with each available taxi performing an action in the grid; where the action is based on a probability distribution of random sampling of a given subset of observations;

[0042] To obtain rewards, we obtain a global reward from the environment after all idle taxis have performed an action. Local rewards And the new global state s t+1 ;

[0043] Update the commentator section by predicting the state value at time t using the value network. Right now Predict the state value at time t+1 using the target network. Right now Where v(.) represents the prediction function of the value network and the target network, w now This represents the current network parameters of the value network. This represents the current network parameters of the target network;

[0044] The method for updating the value network is as follows:

[0045]

[0046] Where α is the learning rate. Denotes the gradient of v. It is the global TD error; w new This represents the updated network parameters of the value network.

[0047] The calculation method is as follows:

[0048]

[0049] The target network is updated using the new parameters from the critic network, namely:

[0050]

[0051] Where τ is a weight; This represents the updated network parameters of the target network.

[0052] Update the policy network by using local TD error To update the policy network for each grid;

[0053] After the training process is completed, the agent uses a policy network to make decisions.

[0054] Furthermore, the process of updating the policy network includes:

[0055] The design includes a reward and a local reward combiner, which divides all idle taxis into seven groups based on their different actions: in, Indicates remaining at c j The taxi, and to This means moving to six adjacent cells. Taxis in the same group share the same local TD error;

[0056] Summarize the TD errors and update the policy network parameters θ as follows: j :

[0057]

[0058] in, It is by The probabilities of the seven predicted actions. These are the numbers of taxis performing seven different actions; They represent Local TD error corresponding to grouping These are partial observations, where β represents the learning rate. Represents parameter θ j The gradient, where π represents the policy network function. This represents the probability that the policy network corresponding to the parameter will output action 0 under partial observation conditions. This represents the updated network parameters for the strategy network corresponding to cell j. This indicates the current network parameters of the strategy network corresponding to cell j.

[0059] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0060] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction that is loaded and executed by a processor to implement the above-described method.

[0061] The beneficial effects of the technical solution provided by this invention include at least the following:

[0062] The technical solution of this invention defines the taxi relocation task as a partially observable Markov decision process and defines the optimization objective; designs the model's training and decision-making process to help the model converge quickly; designs a reward and a local reward combiner to make the policy network update more efficient; and thus, based on multi-agent reinforcement learning, reschedules idle taxis in a many-to-many manner. The relocation strategy obtained through the technical solution of this invention can achieve a more balanced supply and demand, improve the response rate, and reduce the taxi response time. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the execution flow of the idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm provided in an embodiment of the present invention;

[0065] Figure 2 This is a diagram illustrating the problem of relocating idle taxis;

[0066] Figure 3 This is a taxi relocation model and environmental diagram provided in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the information flow during the training of the taxi relocation model provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0069] First Embodiment

[0070] To ensure that idle taxis move independently rather than clustering together, this embodiment provides an idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm, which can relocate taxis in a many-to-many manner. This method can be implemented by electronic devices, and its execution flow is as follows: Figure 1 As shown, it includes the following steps:

[0071] S1. Divide the taxi dispatch area into grids, define the taxi relocation task as a partially observable Markov decision process, and define the optimization objective to construct a taxi relocation model.

[0072] Among them, such as Figure 3 As shown, in our taxi relocation problem, the agents are idle taxis. Each agent has seven actions, including staying in the current cell and moving to six adjacent cells. The environment is the entire city's taxi traffic system, and our goal is to balance the supply and demand in the environment. The taxi relocation model is a variant of the multi-agent cooperative A2C (Advantage Action Review) method, consisting of an actor part and a reviewer part. The actor part has k policy networks, one for each cell. The taxi determines its action based on the policy network of its current cell. The reviewer part has a value network and a target network, which cooperate to predict the global state value at a specific time as accurately as possible, thereby evaluating the state of the environment after the agent performs an action.

[0073] S2, The taxi relocation model is trained using a many-to-many deep reinforcement learning algorithm;

[0074] S3 utilizes the trained taxi relocation model to schedule idle taxis and obtain the scheduling results.

[0075] Specifically, in this embodiment, the taxi relocation task is defined as a partially observable Markov decision process, and an optimization objective is defined, including the following steps:

[0076] Define an intelligent agent; each taxi is an intelligent agent. Let g represent the set of intelligent agents, where g i (i = 1, 2, ..., m) represents a single taxi (or agent), where m is the number of agents;

[0077] Define an action: at some moment t, taxi g i Drive into cell c j Taxi g i Can perform an action Action Set It contains seven different actions, namely {0, 1, 2, 3, 4, 5, 6}. In the text, 0 indicates that the taxi is stationary (i.e., c). j ), 1 (or 2, 3, 4, 5, 6) indicates that the taxi has moved to an adjacent cell. (or We use To represent c j There are six adjacent cells. In our problem setting, taxis entering the same cell share the same action set. There are a total of k action sets, and the global action space is the combination of all action sets, i.e.: in, Indicates the first A set of actions, Let be the number of cells. At some time t, the global action is... It is an m-dimensional vector; where... (i = 1, 2, ..., m) represents taxi g i The action performed at time t, where m is the number of agents.

[0078] Define a state in the grid to describe a single cell c. j In terms of state, we use three-dimensional vectors in, c j The demand quantity in c j The number of available taxis in the area c j The number of taxis currently in service. At a certain time t, the global state s consists of the states of all cells, that is: Among them, s t This represents the global state at time t. Let represent the state of the j-th cell at time t, k be the number of cells, and ° represent the connection vector. Therefore, when considering actions, the state transition function is:

[0079] p(s t+1 |s t ;a t )=P[S t+1 =s t+1 |S t =s t A t =a t ]

[0080] Among them, s t+1 p(s) represents the global state at time t+1. t+1 |s t ;a t ) indicates the global state s t Do action a t The global state then becomes s t+1 The probability, P[.] represents the state transition probability function, S t A represents the random variable A representing the global state at time t. t Let a be a random variable representing the global action at time t. t This represents the global action at time t;

[0081] In our question settings, located in cell c j taxi g i It has no global perspective. It only has one observation, called a local observation. j Partial perspectives, including c j The state of its six adjacent cells, i.e.:

[0082]

[0083] in, This indicates the cell at time t. j taxi g i Local observations c j The state of the nth adjacent cell, nbr = 1, 2, 3, 4, 5, 6;

[0084] Because of the same cell c j The taxis in the data share the same local observations and are unambiguous, so we use Indicates in c j Local observations of all taxis in the system. In the next section, the global state s will be used to train the critic part of the multi-agent reinforcement learning model, while the local observations o i This will be used in the actor training section, in cell cj Related policy networks;

[0085] Define a policy function; our goal is to design an effective relocation strategy for idle taxis (i.e., the agent). At some time t, in cell c... j taxis in the middle i Based on its strategy function π j Through observation Execute action Right now:

[0086]

[0087] In this equation, we use π instead of π j To simplify. In our problem setting, we consider the same cell c j The taxis in the system use the same policy function π. j Therefore, there are a total of k policy functions. We use those with parameter θ. j Neural networks to approximate complex π j To avoid ambiguity, we use θ. j To represent the policy network;

[0088] Define a reward for taxi g after the action is performed. i Earn rewards from the environment. Each cell c j Two types of rewards can be provided, namely and in, The calculation formula is:

[0089]

[0090] It is a decimal between 0 and 1. If c j Supply and demand will reach equilibrium. If If the value is close to 0, then supply and demand will be unbalanced.

[0091] Defined by the following formula, namely:

[0092]

[0093] when At that time, supply and demand reach a state of equilibrium. When the value is negative, supply exceeds demand. When the value is positive, supply cannot meet demand. We use the first type of reward. Define the global reward obtained by all taxis after performing operations at a certain time t. Right now:

[0094]

[0095] in, It is cell c j The first type of reward is given at time t. The global reward can be used to train the commentator part. The second type of reward is used. We define the action that the taxi performs at some time t. The subsequent partial reward, namely:

[0096]

[0097] taxi g i In cell c j China Times, Rewards Provided by this cell, and adjacent to c j cell Provide rewards In the above formula, we allocate rewards to taxis in two different ways: if the taxi's action is to remain stationary... Then use the first formula to calculate the reward. If the taxi's action is to move to an adjacent cell... The second formula is then used to calculate the reward. The taxi that performs the movement action should collect the reward. and rewards Because taxis can affect the starting cell (c j ) and endpoint cell The supply and demand situation. In the above formula, we use ω. j and To measure from cell c j and The reward;

[0098] We define the optimization problem, and to evaluate the long-term returns, we use γ as a discount factor. The global discounted returns are as follows:

[0099] U t =R t +γ·R t+1 +γ 2 ·R t+2 +γ 3 ·R t+3 +…

[0100] Among them, R t Let represent the random variable of the global reward at time t, and the global action-value function be as follows:

[0101] Q(s t ,a t )=E[U t |S t =s tA t =a t ]

[0102] Among them, s t a represents the global state at time t. t Represents the global action at time t; E represents the expected value of the global discounted reward.

[0103] And the global state-value function is as follows:

[0104] V(s t ) = E A [Q(s t A)]

[0105] Here, A refers to the random variable representing the global action at time t. E A This represents the expectation of the global action-value function under condition A;

[0106] The optimization problem is:

[0107]

[0108] Where S is a random variable, representing the global state at time t, and θ 1 ,θ 2 ,…,θ k Let E represent the parameters of the 1st policy network, the 2nd policy network, ..., the kth policy network, respectively; S This represents the expectation of solving the global state-value function under the condition of S;

[0109] In other words, the problem is to minimize the expectation in the equation by adjusting the parameters of the k policy networks.

[0110] Furthermore, to train the model, this embodiment has carefully designed the model training and decision-making process to help the model converge quickly; the specific training and decision-making process is as follows:

[0111] Execute the operation, each available taxi g i In cell c j China's implementation of actions This action was based on a given set of observations. The probability distribution of random sampling. Figure 4 This describes the information flow of this step, namely:

[0112]

[0113] in, It is cell c j The network parameters for the taxi-sharing strategy.

[0114] To obtain rewards, we obtain a global reward from the environment after all idle taxis have performed an action. Local rewards And the new global state s t+1 ;

[0115] In the updated commentator section, we predict the state value at time t using the value network. Right now: Predict the state value at time t+1 using the target network. Right now: Where v(.) represents the prediction function of the value network and the target network, w now This represents the current network parameters of the value network. This represents the current network parameters of the target network;

[0116] The method for updating the value network is as follows:

[0117]

[0118] Where α is the learning rate. Denotes the gradient of v. This is the global TD error. new This represents the updated network parameters of the value network;

[0119] calculate The method is as follows:

[0120]

[0121] The target network is updated using the new parameters from the critic network. Figure 4 This explains the information flow in this step; that is:

[0122]

[0123] Here, τ is a weight. This represents the updated network parameters of the target network.

[0124] In the update strategy section, we use local TD error. Update each cell c j The policy network π j ,Right now:

[0125]

[0126] Where i∈{1,…,m}, j∈{1,…,k}, and β represents the learning rate. Represents parameter θ j The gradient, where π represents the policy network function. This represents the probability that the policy network corresponding to the parameter will output action 0 under partial observation conditions. This represents the updated network parameters for the strategy network corresponding to cell j. This indicates the current network parameters of the strategy network corresponding to cell j;

[0127] and The calculation method is as follows:

[0128]

[0129] Here, there are a total of k policy networks {θ} 1 ,θ 2 ,…,θ k} Figure 4 This describes the information flow of this step;

[0130] After completing the training process, the agent (i.e., the taxi) uses a policy network to make decisions. At decision time t, assume there is an available taxi g. i In cell c j It should execute according to the probability distribution. Actions obtained from sampling Here θ j These are the learning parameters of the network, and These are partial observations;

[0131] Furthermore, during the training process, if m is present at time t... j If a taxi is available, then each policy network will perform m... j The next update. To make this step more efficient, we carefully designed a reward and a local reward combiner to make policy network updates more effective. When updating the policy network, we use the state values ​​predicted from the critic part, the local rewards in the environment, and the local states. Thus, we propose a local reward combination method aimed at θ j Update once. The local reward combiner will m j The taxis were divided into seven groups based on their different actions, namely in, Indicates remaining at c j The taxi, and to This means moving to six adjacent cells. The taxis in the same group share the same local TD (time difference) error, i.e.:

[0132]

[0133] Superscript j. j. …,j. c j The seven taxi groups in the middle, They represent Local TD error corresponding to grouping.

[0134] Therefore, we summarize the TD error and update θ as follows: j :

[0135]

[0136] in, It is by The probabilities of the seven predicted actions. These represent the number of taxis performing seven different actions.

[0137] During training, we use the above formula to update the policy network.

[0138] In summary, this embodiment provides an idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm. It defines the taxi relocation task as a partially observable Markov decision process and defines the optimization objective; designs the model's training and decision-making processes to help the model converge quickly; and designs a reward and a local reward combiner to make policy network updates more efficient. Thus, based on multi-agent reinforcement learning, it re-schedules idle taxis in a many-to-many manner. The relocation strategy obtained through the technical solution of this invention can achieve a more balanced supply and demand, improve the response rate, and reduce taxi response time.

[0139] Second Embodiment

[0140] This embodiment provides an electronic device, which includes a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment.

[0141] The electronic device can vary considerably depending on its configuration or performance, and may include one or more processors (central processing units, CPUs) and one or more memories, wherein the memories store at least one instruction, which is loaded by the processor and described above.

[0142] Third Embodiment

[0143] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0144] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0145] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0148] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for scheduling idle taxis based on a many-to-many deep reinforcement learning algorithm, characterized in that, The idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm includes: The taxi dispatch area is divided into grids, and the taxi relocation task is defined as a partially observable Markov decision process. An optimization objective is defined to construct a taxi relocation model. The optimization objective is to balance the supply and demand of taxis. The taxi relocation model includes a commentator part and an actor part. The actor part includes multiple policy networks, with one policy network corresponding to each grid. The taxi determines its action based on the policy network of its grid. The commentator part includes a value network and a target network. The value network and the target network cooperate to predict the global state value at a specific time as accurately as possible. The taxi relocation model was trained using a many-to-many deep reinforcement learning algorithm. The trained taxi relocation model is used to schedule idle taxis, and the scheduling results are obtained. The process of dividing the taxi dispatch area into grids, defining the taxi relocation task as a partially observable Markov decision process, and defining the optimization objective includes: The taxi dispatch area is divided into grids, with the dispatch area equally divided into multiple hexagonal grids; Define intelligent agents, treating each taxi as an intelligent agent and the taxi operation scenario as the environment; Define actions; each action set includes seven actions: stay in place and move to the other six adjacent grids; each grid corresponds to an action set, and taxis entering the same grid share the same action set; the global action space is the combination of all action sets, and the global action at a certain moment is a vector composed of the actions performed by all taxis; The state is defined by representing the state of a single grid using a three-dimensional vector. The elements of the three-dimensional vector represent the demand quantity, the number of available taxis, and the number of taxis currently in service in the current grid, respectively. The global state at any given time consists of the states of all grids. Taxi located in a grid does not have a global view, but only a partial view called a local observation, which includes the state of the taxi's current grid and its six neighboring grids. Taxi in the same grid share the same local observation. The global state will be used to train the commentator part of the taxi relocation model, while the local observations will be used to train the grid-related policy network in the actor part of the taxi relocation model. Define a policy function and use a neural network to approximate the policy function. Taxis in the same grid use the same policy function. At a certain moment, the taxis in the grid perform actions based on their corresponding policy functions by observing the local observations at the current moment. Define rewards: after performing an action, the taxi receives a reward from the environment. Each grid provides two types of rewards, namely, the first type of reward. Second type of reward ;in, The expression is: ； in, Indicates the grid where the taxi is located. The demand quantity in express The number of available taxis in the area; The result of the calculation is a decimal between 0 and 1. ,but Supply and demand will reach equilibrium; The closer it is to 0, the more it indicates The greater the imbalance between supply and demand; The expression is: ； when At that time, When supply and demand reach a state of equilibrium; When it is negative, then Supply exceeds demand; when When it is a positive number, then Supply cannot meet demand; use Define the global reward obtained by all taxis after performing operations at a given moment. The global reward is used to train the commentator component; The expression is: ； in, It is a grid At any moment The first type of reward obtained at that time; use Define the local reward obtained by the taxi after performing an action at a certain moment. ,Right now: ； in, Indicates time t by The second type of reward provided; Indicates time t by The The second type of reward provided by each adjacent grid; This indicates that the taxi should remain in place. Indicates that the taxi is heading towards the [direction]. Actions of adjacent grids; and These are preset parameters used to measure data from... and The grid The reward; Define the optimization problem, and in order to evaluate future long-term returns, use As a discount factor, the global discount return is as follows: ； in, express The global reward at time step is a random variable, and the global action-value function is as follows: ; in, Indicates time The global state at that time, Indicates time Let E represent the global action at time E, and let E represent the expected value of the global discounted reward; and the global state-value function is as follows: ； Here, It means The random variable of global action at any given moment; Indicates in Under the given conditions, solve for the expectation of the global action-value function; The optimization problem is: ； in, It is a random variable, representing the time... The global state, Let them represent the first policy network, the second policy network, ..., the third policy network, respectively. Parameters of a policy network; This represents the expectation of solving the global state-value function under the condition of S; The optimization problem is solved by adjusting... The parameters of the policy network are used to minimize the expectation in the equation.

2. The idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm as described in claim 1, characterized in that, The process of training the taxi relocation model using a many-to-many deep reinforcement learning algorithm includes: An agent performs actions, with each available taxi performing an action in the grid; where the action is based on a probability distribution of random sampling of a given subset of observations; To obtain rewards, we obtain a global reward from the environment after all idle taxis have performed an action. Local rewards And the new global state ; Update the commentator section to predict moments using value networks. State value ,Right now And use the target network to predict the time. State value ,Right now ;in, (.) represents the prediction function for the value network and the target network. This represents the current network parameters of the value network. This represents the current network parameters of the target network; The method for updating the value network is as follows: ； in, It's the learning rate. express gradient, It is the global TD error; This represents the updated network parameters of the value network. The calculation method is as follows: ； The target network is updated using the new parameters from the critic network, namely: ； in, It is a weight; This represents the updated network parameters of the target network. Update the policy network by using local TD error To update the policy network for each grid; After the training process is completed, the agent uses a policy network to make decisions.

3. The idle taxi scheduling method based on a many-to-many deep reinforcement learning algorithm as described in claim 2, characterized in that, The process of updating the policy network includes: The design includes a reward and a local reward combiner, which divides all idle taxis into seven groups based on their different actions: ,in, Indicates staying at The taxi, and to This means moving to six adjacent cells. Taxis in the same group share the same local TD error; Summarize the TD errors and update the policy network parameters as follows: : ； in, It is by The probabilities of the seven predicted actions. These are the numbers of taxis performing seven different actions; They represent Local TD error corresponding to grouping These are partial observations. Indicates the learning rate. Indicates parameters gradient, Represents the policy network function, This represents the probability that the policy network corresponding to the parameter will output action 0 under partial observation conditions. Represents a cell The corresponding policy network updated network parameters, Represents a cell The current network parameters of the corresponding policy network.

Images