A centralized shared operation car scheduling method based on deep reinforcement learning

Abstract

The application discloses a centralized sharing operation car scheduling method based on deep reinforcement learning, and relates to the technical field of artificial intelligence. The method comprises the following steps: modeling a Markov decision process for a city vehicle scheduling problem, defining a state space, an adjacent perception action space, state transition and a reward; constructing a strategy network based on a space-time attention mechanism to capture space-time dependency; and training the strategy network for vehicle scheduling decision. The application proposes a centralized training framework with an adjacent perception coordination mechanism, adjusts global actions based on global supply and demand dynamics and local interaction, uses a space-time attention mechanism to capture space-time dependency, and thus efficiently processes large-scale vehicle scheduling problems. In addition, by calculating an adjacent vehicle scheduling proportion, the application relieves internal competition among drivers, effectively reduces redundant scheduling behaviors, and has good robustness for problems of drivers and passengers changing with space and time in real situations.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly to a centralized shared operation car dispatching method based on deep reinforcement learning. BACKGROUND

[0002] With the acceleration of urbanization and the increasing demand for travel, efficient and flexible urban transportation becomes particularly important. On-demand mobility (MoD) services, such as ride-hailing platforms, provide flexible ride services to users through digital platforms, making urban transportation more convenient and efficient. These systems rely on efficient real-time dispatching to allocate idle vehicle resources to different user demand points during operation. Shared mobility platform business, such as Figure 1 as shown.

[0003] In reality, there may be imbalances in supply and demand due to various circumstances, such as rush hours, holidays, etc. Therefore, vehicle dispatching is needed to balance the supply and demand imbalance between regions, such as Figure 2 as shown. The optimization goal becomes how to make the vehicle fleet cooperate in dispatching to minimize the total dispatching cost. Vehicle dispatching involves multi-vehicle path planning and task allocation, aiming to meet customer demand in the best way and maintain supply and demand balance. This problem is of great significance in many fields such as logistics distribution, ride-hailing, emergency rescue, etc. In particular, in modern large-scale shared mobility systems, how to reasonably dispatch vehicles to meet dynamically changing user demand, maintain supply and demand balance, and improve the overall efficiency of the system while ensuring service quality is one of the current research hotspots and difficulties.

[0004] The dispatching problem can be traced back to the classic Travelling Salesman Problem (TSP) and Vehicle Routing Problem (VRP). For these problems, how to quickly and efficiently approximate solutions has high practical value in supply chain configuration, circuit layout, network flow planning, etc., and has received extensive attention. Domestic and foreign scholars have conducted in-depth research on algorithm optimization, model establishment, and application scenario expansion.

[0005] In large-scale environments, vehicle relocation becomes increasingly complex, especially when demand patterns have high dynamics, uncertainty, and spatial heterogeneity. Existing vehicle fleet relocation methods mainly focus on maximizing order fulfillment rate while reducing dispatching cost. These methods can be broadly divided into three categories: optimization-based methods, imitation learning methods, and reinforcement learning methods.

[0006] The optimization-based approach models the repositioning problem as a constrained optimization problem by computing the utility of dispatching a driver at each spatiotemporal location and determining the dispatching decisions by solving the optimization problem to maximize the overall utility under supply constraints. This approach suffers from a significant decrease in computational efficiency for large-scale fleets and passengers.

[0007] The imitation learning approach, on the other hand, uses expert demonstrations (typically the driving trajectories of high-income drivers) to guide the learning of novice drivers' strategies. These methods include adversarial imitation learning and inverse reinforcement learning, which extract driver strategies from historical trajectories to enable novices to imitate the driving behavior of experienced drivers in a data-driven manner. This approach is limited by the state and strategy distribution in the dataset, with a significant decrease in performance in out-of-distribution scenarios, and it also fails to address the competition between drivers effectively.

[0008] In recent years, reinforcement learning methods have been widely applied in the field of intelligent transportation. This approach typically models the dispatching or order matching task as a Markov Decision Process (MDP) and guides the agent to learn the optimal strategy by interacting with the environment to obtain reward signals, as shown in the algorithm flow Figure 3 According to the degree of coordination of the system, existing mainstream reinforcement learning methods can be broadly classified into centralized and decentralized methods.

[0009] Centralized methods typically treat the entire dispatching system as a single agent, making unified decisions by aggregating global information. One approach uses neural networks to estimate the value of each region to guide vehicle dispatching, while another approach generates globally optimal dispatching instructions for all vehicles by a centralized control unit that integrates supply and demand information from all regions, thereby achieving efficient allocation of overall resources.

[0010] In comparison, decentralized methods model each driver or grid cell as an independent agent, making decisions and learning based on local information and reward signals. Some methods model the coordination process of multiple agents as a sequential decision-making problem, treating each driver as a single agent and taking turns to execute the decision-making process. Other methods use a "centralized training, distributed execution" architecture, sharing a global evaluation network during the training phase to improve learning stability and making independent decisions by each agent during the execution phase to enhance the flexibility and scalability of the system.

[0011] Despite the progress made, most of the reinforcement learning-based relocation methods still focus on optimizing the strategy of a single driver. In contrast, the design of collaborative strategies for large-scale fleets, especially in scenarios involving internal competition and fluctuations in the number of agents participating, remains understudied. For example, changes in the number of system agents due to passenger pickups or driver on- and off-boarding introduce significant non-stationarity. This not only weakens the effectiveness of individual strategy training but also highlights the urgent need to develop more robust methods that can explicitly address fluctuations in real-world systems. SUMMARY

[0012] To address the above problems, the purpose of the present application is to provide a centralized shared operation car scheduling method based on deep reinforcement learning (DRL), which models the fleet relocation as a collaborative decision-making process, does not regard the driver as an isolated learning individual, but proposes a centralized training framework with an adjacent perception coordination mechanism, adjusts the global action based on the global supply and demand dynamics and local interaction, uses a spatio-temporal attention mechanism to capture spatio-temporal dependencies, and thus efficiently handles large-scale vehicle scheduling problems. Moreover, by calculating the adjacent vehicle scheduling proportion, the internal competition of the driver is alleviated, and the behavior of redundant scheduling is effectively reduced, which has good robustness to the problems of drivers and passengers changing with time and space in real situations.

[0013] To solve the above technical problems, the present application provides the following technical solutions:

[0014] On the one hand, a centralized shared operation car scheduling method based on deep reinforcement learning is provided, which comprises the following steps:

[0015] Modeling the urban vehicle scheduling problem as a Markov decision process, defining the state space, adjacent perception action space, state transition and reward;

[0016] State space: the urban area is divided into multiple grids, time is discretized into multiple time steps, and the global state is represented as the number of available drivers and the number of orders in each grid, with the global state as the input of the policy network;

[0017] Adjacent perception action space: define adjacent grids, and for each pair of adjacent grids, the action is represented as the vehicle scheduling proportion between the grids;

[0018] State transition: after receiving the relocation instruction, the drivers in each grid determine the proportion of relocation to the adjacent grid according to the action output of the grid;

[0019] Reward: introduce the distance between the driver and the order distribution, and use the number of order matches as the reward to promote the alignment of the driver and the order distribution;

[0020] A strategy network based on spatiotemporal attention mechanism is constructed, the temporal attention layer is used for modeling the dependency between continuous time steps, and the spatial attention layer is used for modeling the dependency between grids at each time step;

[0021] The strategy network is trained, and the trained strategy network is used for vehicle scheduling decision.

[0022] Optionally, in the state space, the entire urban area is divided into n*n uniform grids, each grid corresponds to a specific geographic location; time is discretized into multiple time steps, at each time step t, the global state Composed of two channels, denoted as , wherein represent the number of available drivers and the number of orders in each grid, respectively.

[0023] Optionally, in the adjacency-aware action space, if two grids are in the same neighborhood, i.e. the center cell and eight adjacent cells, they are defined as adjacent grids; for each pair of adjacent grids (i, j)→(i', j'), the action represents the vehicle scheduling ratio between grid (i, j) and grid (i', j'), positive action represents forward scheduling, and negative action represents reverse scheduling;

[0024] The global action a t is defined as:

[0025]

[0026] For each pair of adjacent grids, the actual vehicle scheduling ratio is calculated as:

[0027]

[0028] wherein represent all adjacent grids of grid (i, j), and m represents one of all adjacent grids, is an indicator function, which ensures that only positive or non-negative actions are considered by the current grid.

[0029] Optionally, in the state transition, after relocation is completed, it enters the order matching stage, according to the supply and demand relationship, the drivers or orders that fail to be successfully matched will be continued to the next time step; the drivers who successfully match orders will deliver passengers to the destination and will be temporarily unavailable during transportation; if a certain order fails to be served within continuous t = 3 time steps, it will be removed from the system.

[0030] Optionally, the global reward at time step t is defined as follows:

[0031]

[0032] where σ represents the Sigmod activation function, M t represents the number of matched orders at time step t; P t represents the number of orders in each grid at time step t; is the Wasserstein distance between the driver distribution and the order distribution at time step t + 1; is the driver distribution after the vehicles are dispatched at time step t + 1 but before the orders are matched; β is a hyperparameter that controls the trade-off between maximizing order matching and maintaining supply-demand balance.

[0033] Optionally, the policy network is a Transformer-based policy network, comprising a convolutional embedding module, a spatio-temporal attention module, and an output module.

[0034] The input end adopts a state stacking representation method, that is, in each decision step, the input x t contains the stacked state of the latest k consecutive time steps:

[0035]

[0036] The convolutional embedding module applies a continuous 1x1 convolutional layer in the channel dimension to embed the input of each time step from 2 channels to a d-dimensional feature space, and the output of the convolutional embedding module is represented as:

[0037]

[0038] where L represents the number of convolutional layers;

[0039] In the spatio-temporal attention module, firstly, the embedding tensor is permuted and reshaped to obtain ; subsequently, the time attention layer models the dependency relationship between the k consecutive time step stacked frames, and the spatial attention layer models the dependency relationship between the grids within each time step; in order to encode the two-dimensional coordinate information, a rotated position encoding is added in the spatial attention layer to integrate the horizontal and vertical coordinates of each grid into the query vector and the key vector;

[0040] In the output module, a global pooling operation is used on the output of the spatio-temporal attention module to extract global features; then a multi-layer perceptron with a Leaky ReLU activation function is used to generate a parameterized Gaussian distribution; finally, the final action is sampled in the following way:

[0041]

[0042] where represents an element-wise product, μ, is the mean and variance of the Gaussian distribution, and the random variable It follows a multivariate normal distribution with a mean of 0 and a covariance matrix of identity matrix I; to ensure that the action value is within the interval of -1 to 1, the sampling results are activated by the Tanh function.

[0043] Optionally, the SAC algorithm is used to train the policy network to obtain the optimal policy. A strategy defined as maximizing the sum of expected reward and policy entropy is expressed as:

[0044]

[0045] Where ρ π This represents the state distribution induced by policy π. Represents the state distribution ρ induced by policy π. π The expected value of r(s) t a t ) indicates that time step t is in state s t Perform action a t The reward This represents the current policy in state s. t The entropy is α, where α is a temperature parameter used to balance the ratio between reward and entropy.

[0046] Optionally, the training steps of the policy network are as follows:

[0047] Step 1: Initialize the policy network Critics Network Corresponding target critic network At the same time, set the temperature parameter α, the soft update coefficient τ, and the number of time steps stacked k; initialize the experience playback buffer D to store historical interaction data;

[0048] Step 2: In each iteration of interaction with the environment, first obtain the states of the k most recent consecutive time steps, forming a stacked state. The stacked state is input into the policy network, and the current policy action 'a' is obtained by sampling. t Execute action a t Interact with the environment, trigger matchmaking, and receive rewards from the environment. t and the next state s t+1 ; the quadruple Store in the experience replay buffer D;

[0049] Step 3: During the network parameter update phase, a batch of samples is randomly sampled from the experience replay buffer D; the policy network and the critic network are updated according to the loss function, and the target critic network is updated using a soft update mechanism.

[0050] Step four: repeat the above step two and step three until the policy network converges or reaches a preset number of iterations.

[0051] In another aspect, an electronic device is provided, the electronic device comprising:

[0052] a processor;

[0053] a memory having computer readable instructions stored thereon, the computer readable instructions, when loaded and executed by the processor, implement the steps of the above deep reinforcement learning based centralized shared operation car dispatching method.

[0054] In another aspect, a computer readable storage medium is provided, the computer readable storage medium having program code stored therein, the program code being executable by a processor to perform the steps of the above deep reinforcement learning based centralized shared operation car dispatching method.

[0055] The technical solutions provided by the application have at least the following beneficial effects:

[0056] (1) Adjacent perception continuous action representation;

[0057] The application converts the traditional discrete grid level scheduling action space into a continuous action space; the action representation is a driver allocation ratio across regions, and the region adjacency information is explicitly introduced; the action is modeled by a directed flow, thereby realizing a smoother and more flexible scheduling strategy, avoiding invalid scheduling, thereby reducing competition between drivers and improving resource utilization efficiency.

[0058] (2) Spatio-temporal attention policy network;

[0059] The application models the region spatio-temporal features by combining state stacking and spatio-temporal attention mechanisms, realizes more fine and forward-looking global strategy generation, can be seamlessly integrated with reinforcement learning algorithms such as SAC, and realizes end-to-end training.

[0060] (3) Centralized vehicle dispatching framework;

[0061] The application no longer takes a single driver as an independent agent, but globally and uniformly dispatches, reduces the action dimension explosion problem in the multi-agent method, makes the model maintain stable training and efficient inference under a large-scale urban grid; the centralized decision can explore a suitable cooperation mechanism while ensuring scalability, thereby achieving a globally optimal scheduling result; no longer relies on collaborative training between multiple independent agents, significantly reduces communication complexity and system coupling.

[0062] In summary, the application achieves significant improvement in scheduling efficiency, resource utilization, stability and scalability through the synergistic effect of the above three innovations. Experimental results in real and simulated environments show that the application can effectively improve order response rate and driver utilization rate, reduce redundant scheduling behavior, and maintain excellent performance stability in large-scale travel systems. Therefore, the application has wide application prospects and engineering value in shared travel scheduling, logistics distribution, medical rescue and other scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 is a shared travel platform business schematic diagram;

[0065] Figure 2 is a vehicle scheduling schematic diagram;

[0066] Figure 3 is a reinforcement learning algorithm training process schematic diagram;

[0067] Figure 4 is a framework schematic diagram of the centralized shared operation vehicle scheduling method based on deep reinforcement learning provided by the embodiment of the present application;

[0068] Figure 5 is a motion representation schematic diagram based on adjacency relationship provided by the embodiment of the present application;

[0069] Figure 6 is a structure schematic diagram of the policy network provided by the embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme of the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0071] The present application is mainly aimed at how to reasonably dispatch vehicles in a city to meet the balance between supply and demand, and the dispatching specifically refers to a passenger-seeking strategy of idle vehicles, that is, a driving direction decision of idle vehicles. The decision problem is usually modeled as a Markov decision process (MDP). A general Markov decision process can be represented by a five-tuple . Wherein S represents a state set, A represents an action set, P(s'|s,a) is a state transition function, representing the probability of transitioning from state s to state s' by performing action a. r(s,a) is a reward function, representing the reward of performing action a in state s. γ is a discount factor. The strategy of an agent is usually represented by π(a|s), which means the probability function of performing action a in state s. When the strategy is a deterministic strategy, the probability value only contains 0 and 1. When the strategy is a random strategy, the output of the function is a probability distribution of an action. The solution of the decision problem is to find the optimal strategy function in the MDP process to meet the maximum expected reward.

[0072] Based on the above problem, an embodiment of the present application provides a centralized shared operation vehicle scheduling method based on deep reinforcement learning. The method can be implemented by an electronic device, which can be a terminal or a server. As shown in Figure 4 , the processing flow of the method can include the following steps:

[0073] A Markov decision process is modeled for the city vehicle scheduling problem, and a state space, an adjacent perception action space, a state transition and a reward are defined.

[0074] State space: the entire city area is divided into n x n uniform grids, and each grid corresponds to a specific geographic location; time is discretized into multiple time steps, and at each time step t, the global state is composed of two channels and is represented as , wherein represent the number of available drivers and the number of orders (i.e. the number of passenger vehicle requests) in each grid, respectively. In the present application, the global state is used as the input of the policy network, rather than an independent state input for each driver, which can enhance scalability and more effectively cope with large-scale vehicle scenarios.

[0075] Adjacent perception action space: the present application proposes an action space A, which enables scalable and directional allocation of vehicle resources between adjacent grid units, as shown in Figure 5 . If two grids are in the same neighborhood, i.e. the center unit and the eight adjacent units, they are defined as adjacent grids. Based on the definition of adjacency, there are a total of 4n 2- 6n + 2 pairs of adjacent relationship. For each pair of adjacent grid (i, j)→(i', j'), the action represents the vehicle dispatching ratio between grid (i, j) and grid (i', j') (i.e. the arrow in the figure). Each pair of adjacent grids is assigned a predefined forward direction, and a positive action value represents forward dispatching, while a negative action value represents reverse dispatching.

[0076] The global action a t is defined as:

[0077]

[0078] For each pair of adjacent grids, the actual vehicle dispatching ratio is calculated as:

[0079]

[0080] where represents all adjacent grids of grid (i, j), and m represents one of the adjacent grids, is an indicator function that ensures only forward or non-negative actions are considered by the current grid. This action space based on adjacent relationship integrates neighborhood information into actions, achieving directional and fine-grained vehicle resource allocation.

[0081] Unlike traditional agent-centered methods (each driver chooses a destination or matches a specific order), the grid-level design can better capture the dynamics of the entire fleet, reduce the dimensionality problem caused by action combinations, and achieve efficient global coordination. In particular, the relocation process is based on vehicle ratio, which makes this method naturally robust to fluctuations in the number of drivers and highly adaptable in real-world environments.

[0082] State transition: After receiving the relocation instruction, the drivers in each grid determine the proportion of the adjacent grid to which they will relocate according to the action output of the grid. After relocation is completed, the order matching phase begins, and based on the supply and demand relationship, drivers or orders (passengers) who fail to successfully match will be extended to the next time step. Drivers who successfully match orders will deliver passengers to their destinations and will be temporarily unavailable during transportation. Drivers will appear at the destination of the passenger after a certain period of time. If an order cannot be served within three consecutive time steps (t = 3), it will be removed from the system.

[0083] Reward: The global reward at time step t is defined as follows:

[0084]

[0085] where σ represents the Sigmod activation function, M t represents the number of matched orders at time step t; Pt denotes the number of orders in each grid at time step t; is the Wasserstein distance between the driver distribution and the order distribution at time step t + 1; is the driver distribution after the vehicle is dispatched at time step t + 1 but before the order matching; β is a hyperparameter that controls the trade-off between maximizing order matching and maintaining supply and demand balance.

[0086] The Wasserstein distance can capture the size and spatial mismatch of the supply and demand distribution. When the number of order matches reaches a saturation state or the orders are sparse, it promotes the alignment between the driver and passenger distribution, helps to reduce empty driving and resource waste, and thus improves long-term efficiency. Unlike previous studies that mainly use gross merchandise volume (GMV) as the reward signal, the present invention selects the number of order matches as the reward, which can avoid the variance and bias introduced by order price fluctuations.

[0087] Further, a strategy network based on a spatio-temporal attention mechanism is constructed to capture spatio-temporal dependencies.

[0088] The present invention adopts a strategy network based on Transformer to capture spatio-temporal dependencies in the environment and output a global vehicle dispatching strategy. As shown in Figure 6 The strategy network architecture mainly includes three components: a convolutional embedding module, a spatio-temporal attention module, and an output module. The review network has a similar architecture to the strategy network, except that the spatio-temporal attention module is removed.

[0089] Since a single frame (time step) lacks sufficient temporal information (e.g., for inferring the movement or speed of objects), the present invention uses a state stacking representation at the input end, i.e., in each decision step, the input x t The stacked state contains the latest k consecutive time steps:

[0090]

[0091] The convolutional embedding module applies a continuous 1x1 convolutional layer in the channel dimension to embed the input of each time step from 2 channels to a d-dimensional feature space. The output of the convolutional embedding module is represented as:

[0092]

[0093] where L represents the number of convolutional layers. In actual implementation, a two-layer convolutional network is used for embedding, and a ReLU activation function is applied after each layer.

[0094] In the spatiotemporal attention module, to enable the model to focus on important spatiotemporal regions, a temporal attention layer and a spatial attention layer are added to the policy network, selectively aggregating temporal and spatial information using a spatiotemporal attention mechanism. First, the embedded tensor... Perform permute and reshape operations to obtain... Subsequently, a temporal attention layer models the dependencies between stacked frames across k consecutive time steps, while a spatial attention layer models the dependencies between grids within each time step. To encode 2D coordinate information, rotational position encoding is added to the spatial attention layer, integrating the horizontal and vertical coordinates of each grid into the query vector and key vector. This invention employs a standard Transformer architecture, with each attention block containing a 4-head self-attention and feedforward neural network (FFN), and residual connections and normalization are added to ensure training stability.

[0095] In the output module, global pooling is applied to the output of the spatiotemporal attention module to extract global features; then, a parameterized Gaussian distribution is generated using a multilayer perceptron (MLP) with a Leaky ReLU activation function. The final action was obtained by sampling in the following way:

[0096]

[0097] in Represents element-wise product, μ, Let the mean and variance of the Gaussian distribution be the random variable. The data follows a multivariate normal distribution with a mean of 0 and a covariance matrix equal to the matrix I between the unit driver distribution and the order distribution. To ensure that the action values ​​are within the range of -1 to 1, the sampling results are activated using the Tanh function. During the evaluation phase, ν = 0 is set, and deterministic actions are output directly.

[0098] Furthermore, this invention employs the Soft Actor-Critic (SAC) algorithm to train the policy network and uses the trained policy network for vehicle scheduling decisions. This algorithm achieves a balance between exploration and exploitation by optimizing the maximum entropy objective. Within this framework, the optimal policy... A strategy defined as maximizing the sum of expected reward and policy entropy is expressed as:

[0099]

[0100] Where ρ π This represents the state distribution induced by policy π. Represents the state distribution ρ induced by policy π. π The expected value of r(s) t at represents the time step t in state s t performs action a t , the reward of represents the entropy of the current policy in state s t , and a is a temperature parameter used to balance the proportion between the reward and the entropy.

[0101] The critic network used to evaluate the state-action value is updated by minimizing the Soft Bellman Residual, and the loss function is defined as follows:

[0102]

[0103] The calculation formula of the target value y t is as follows:

[0104]

[0105] In the above formula, D represents an experience replay pool used to store historical interaction samples, respectively represent the current critic network and the target critic network, and the network parameters are represented by , is the log probability of the current policy generating action a t+1 in the next state s t+1 . To improve the stability and robustness of the algorithm, the double critic structure in the original method is not used, but random noise is injected into the action input of the target critic to realize Target Policy Smoothing, so as to reduce the overestimation phenomenon.

[0106] The actor network, i.e., the policy network, is updated by minimizing the following loss:

[0107]

[0108] Where the parameters of the policy network are represented by θ.

[0109] The training steps of the policy network are as follows:

[0110] Step one: initialize the policy network , the critic network , the corresponding target critic network , and set the temperature parameter a, the soft update coefficient τ, and the number of time step stacks k; initialize the experience replay buffer D for storing historical interaction data;

[0111] Step two: in each iteration of interaction with the environment, first obtain the states of the last k consecutive time steps to form a stacked state The stacked state is input into the policy network, and a current policy action a is sampled t The action a is executed t Interact with the environment, trigger matching, and obtain reward r from the environment t And the next state s t+1 The four-tuple is stored in the experience replay buffer D.

[0112] Step three: in the network parameter updating phase, a batch of samples is randomly sampled from the experience replay buffer D; the policy network and the critic network are updated according to the loss function respectively, and the target critic network is updated using a soft update mechanism:

[0113]

[0114] Step four: repeat the above step two and step three until the policy network converges or reaches a preset iteration number.

[0115] In the embodiment of the application, a centralized vehicle relocation framework is proposed, and the joint scheduling decision of multiple drivers is modeled as continuous actions on a space-time grid. Unlike traditional multi-agent reinforcement learning or discrete action centralized methods, the application introduces a new adjacent perception action representation, where each action vector specifies the direction proportion flowing to the adjacent grid. This design effectively reduces the redundancy of the relocation of the action space, reduces the demand for complex inter-agent coordination, and enhances the robustness of the system to actual supply and demand fluctuations.

[0116] At the same time, in order to solve the non-Markovian property and spatial dependence of the scheduling decision, the application adds frame stacking and spatio-temporal attention mechanism to the SAC-based policy network, so that the model can capture the changing demand trend and local interaction. This design enables the policy to capture temporal trends and spatial dependencies with minimal computational overhead, enabling more reasonable and forward-looking decisions. The centralized method of the application has good scalability and can adapt to the scheduling needs of large-scale vehicle fleets, and can achieve efficient and stable resource allocation and decision optimization in complex and variable real environments.

[0117] In summary, compared with existing distributed or independent decision-based vehicle scheduling methods, the method of the application has the following beneficial effects:

[0118] (1) Improve the global scheduling efficiency and system scalability;

[0119] The application directly models the driver allocation proportion in each region by introducing an adjacent perception continuous action space, effectively avoiding the dimension explosion problem of traditional discrete action methods in large-scale grids, and achieving efficient scalability in large-scale scenarios.

[0120] (2) Fully utilize the space-time dependent information to enhance the intelligence of the strategy;

[0121] By introducing the space-time Transformer structure, the application can simultaneously capture the spatial correlation between regions and the time dynamic characteristics, thereby more accurately modeling the change trend of travel demand and the imbalance between supply and demand between regions, and realizing more accurate relocation decisions.

[0122] (3) Utilize the state stacking mechanism to enhance the short-term dynamic modeling capability;

[0123] The application adopts a frame stacking mechanism to encode the state of consecutive time steps, so that the model can identify short-term demand fluctuations and time series rules, thereby maintaining the stability and responsiveness of the strategy in a dynamically changing travel environment.

[0124] (4) Effectively reduce redundant scheduling and regional competition phenomena;

[0125] The adjacent perception action representation of the application can explicitly constrain the resource migration proportion between adjacent regions, significantly reducing repeated dispatch and over-concentration phenomena, and improving the overall utilization efficiency of driver resources.

[0126] (5) Achieve significant performance improvement in multiple key indicators;

[0127] Experimental results show that the application is superior to existing mainstream methods in order response rate, redundant scheduling quantity, and driver utilization rate, etc., verifying the comprehensive advantages of the application in large-scale scheduling scenarios.

[0128] (6) Has good generalization ability and expansion potential;

[0129] Compared with traditional scheduling methods which are limited by the number of agents and problem size, the application method can be extended to large-scale or even arbitrary number of vehicle scheduling scenarios, and is suitable for more types of scheduling and resource allocation tasks, and has wide application prospects.

[0130] In an exemplary embodiment, the application also provides an electronic device, which comprises:

[0131] a processor;

[0132] a memory having computer readable instructions stored thereon, the computer readable instructions being loaded and executed by the processor to implement the steps of the above-mentioned centralized shared operation car scheduling method based on deep reinforcement learning.

[0133] In the exemplary embodiments, the present application also provides a computer readable storage medium, in which at least one instruction is stored, the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned centralized shared operation car dispatching method based on deep reinforcement learning. For example, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0134] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or terminal device. Without more limitations, the elements defined by the statement "comprising" do not exclude the presence of additional identical elements in the process, method, article, or terminal device comprising the elements.

[0135] In the specification, "one embodiment", "an embodiment", "exemplary embodiment", "some embodiments", and the like indicate that the described embodiment can include a particular feature, structure, or characteristic, but not necessarily every embodiment. In addition, when a particular feature, structure, or characteristic is described in connection with an embodiment, it should be understood that such feature, structure, or characteristic can be implemented in connection with other embodiments, whether or not explicitly described.

[0136] It should be understood that in various embodiments of the present application, the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0137] In the embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0139] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0140] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0141] The present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application. In order to make the public have a thorough understanding of the present application, specific details are described in the following preferred embodiments of the present application, and the present application can also be completely understood without the description of these details to those skilled in the art. In addition, in order to avoid unnecessary confusion to the essence of the present application, well-known methods, processes, flows, elements and circuits, etc. are not described in detail.

[0142] The above is only the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A centralized shared vehicle scheduling method based on deep reinforcement learning, characterized in that, Includes the following steps: A Markov decision process is modeled for the urban vehicle scheduling problem, defining the state space, the adjacency-aware action space, the state transition, and the reward. State space: The city area is divided into n×n The grid is uniform, with each grid corresponding to a specific geographic location; time is discretized into multiple time steps, and at each time step... t global state It consists of two channels, represented as ,in These represent the number of available drivers and the number of orders in each grid, respectively, with the global state as the input to the policy network; Adjacency-aware action space: Define adjacency grids. For each pair of adjacency grids, the action is represented as the vehicle scheduling ratio between the grids. In the adjacency-sensing action space, if two grids are in the same neighborhood, i.e., the central cell and its eight adjacent cells, they are defined as adjacent grids; for each pair of adjacent grids ( i , j )→( i' , j' ),action Represents a grid ( i , j ) and grid ( i' , j' The vehicle dispatch ratio between () indicates forward dispatch, and a positive value indicates reverse dispatch. At time step t All-time global actions a t Defined as: ； For each pair of adjacent grids, the actual dispatched vehicle ratio is calculated as follows: ； in Represents grid ( i , j All adjacent meshes of ) As an indicator function, it ensures that only positive or non-negative actions are considered by the current grid. m Indicates one of all adjacent grid cells; State transition: After receiving the relocation command, the driver in each grid determines the proportion of the relocation flow to the adjacent grid based on the grid's action output; Incentives: Introduce the distance between drivers and order distributions, and use the number of order matches as an incentive to promote alignment between drivers and order distributions; A policy network based on a spatiotemporal attention mechanism is constructed. The temporal attention layer is used to model the dependencies between consecutive time steps, and the spatial attention layer is used to model the dependencies between grids at each time step. The policy network is a Transformer-based policy network, including: a convolutional embedding module, a spatiotemporal attention module, and an output module; The input uses a state stacking representation, meaning that in each decision step, the input... x t Includes the latest k Stacked states of consecutive time steps: ； The convolutional embedding module applies consecutive 1×1 convolutional layers along the channel dimension, embedding the input at each time step from 2 channels into... d In the dimensional feature space, the output of the convolutional embedding module is represented as: ； in L Indicates the number of convolutional layers; In the spatiotemporal attention module, the embedded tensor is first processed. Perform replacement and reshaping operations to obtain Subsequently, the temporal attention layer was modeled. k The spatial attention layer models the dependencies between stacked frames in consecutive time steps, and adds rotational position encoding to the spatial attention layer to integrate the horizontal and vertical coordinates of each grid into the query vector and key vector in order to encode the two-dimensional coordinate information. In the output module, global pooling is applied to the output of the spatiotemporal attention module to extract global features; then, a parameterized Gaussian distribution is generated through a multilayer perceptron with a Leaky ReLU activation function; the final action is sampled in the following manner: ； in Represents element-wise product. μ , Let the mean and variance of the Gaussian distribution be the random variable. It follows a multivariate normal distribution with a mean of 0 and a covariance matrix of identity matrix I; to ensure that the action value is within the interval of -1 to 1, the sampling results are activated by the Tanh function; The policy network is trained and then used for vehicle scheduling decisions.

2. The centralized shared vehicle scheduling method based on deep reinforcement learning according to claim 1, characterized in that, During the state transition, after the relocation is completed, the order matching stage begins. Based on the supply and demand relationship, drivers or orders that fail to be matched will be carried over to the next time step. Drivers who successfully match orders will deliver passengers to their destination and will be temporarily unavailable during transportation. If an order is in a continuous t If a service fails to be provided within 3 time steps, it will be removed from the system.

3. The centralized shared vehicle scheduling method based on deep reinforcement learning according to claim 1, characterized in that, Global rewards at time step t The definition is as follows: ； in σ Represents the Sigmoid activation function. M t Represents the time step t The number of matched orders at that time; P t Indicates time step t The number of orders within each grid; It is a time step t +1 is the Wasserstein distance between the driver distribution and the order distribution; It is a time step t +1 vehicle dispatched but driver distribution before order matching; β It is a hyperparameter that controls the trade-off between maximizing order matching and maintaining supply and demand balance.

4. The centralized shared vehicle scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The policy network is trained using the SAC algorithm to obtain the optimal policy. π A strategy defined as maximizing the sum of expected reward and policy entropy is expressed as: ； in ρ π Indicates by strategy π The induced state distribution, Indicating in strategy π Induced state distribution ρ π The following expectations; r ( s t ，a t ) indicates time step t In state s t Execute action a t The reward This represents the current strategy in state. s t The entropy below, α This is a temperature parameter used to balance the ratio between reward and entropy.

5. The centralized shared vehicle scheduling method based on deep reinforcement learning according to claim 4, characterized in that, The training steps for the policy network are as follows: Step 1: Initialize the policy network Critics Network Corresponding target critic network At the same time, set the temperature parameters. α Soft update coefficient τ and the number of time steps stacked k Initialize the experience playback buffer. D , used to store historical interaction data; Step 2: In each iteration of interaction with the environment, first obtain the most recent consecutive... k The states at each time step form a stacked state. The stacked state is input into the policy network, and the current policy action is obtained by sampling. a t Execute actions a t Interact with the environment, trigger matches, and receive rewards from the environment. r t and the next step status s t+1 ; the quadruple Store in the experience replay buffer D ; Step 3: During the network parameter update phase, retrieve data from the experience replay buffer. D A batch of samples is randomly sampled; the policy network and the critic network are updated according to the loss function, and the target critic network is updated using a soft update mechanism; Step 4: Repeat steps 2 and 3 above until the policy network converges or the preset number of iterations is reached.

6. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when loaded and executed by the processor, implement the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 5.

Images