Multi-vehicle type automatic driving online car dispatching and scheduling cooperation method

Abstract

The present application relates to the field of transportation planning and management technology, and more particularly to a multi-vehicle automatic driving network car dispatching and scheduling coordination method, comprising defining network car system model variables containing multiple service type vehicles and passengers; constructing an event-driven Markov decision process model based on a closed-loop queuing network; using a neural network-based post-decision state value function approximation model for value approximation, and optimizing the dispatching and vehicle relocation strategy through approximate dynamic programming. The present application effectively separates decision influence and random disturbance, improves the adaptability and learning efficiency of the model under non-stationary demand, and significantly improves the platform service completion rate and revenue.

Description

Technical Field

[0001] This invention relates to the field of transportation planning and management technology, and in particular to a method for dispatching and coordinating multi-vehicle autonomous ride-hailing services. Background Technology

[0002] With the booming development of mobile internet and the sharing economy, ride-hailing platforms have become a core component of urban transportation systems, providing the public with flexible and convenient on-demand travel services. In actual operation, these platforms need to process large-scale passenger travel requests in real time and dynamically allocate vehicle resources.

[0003] Existing ride-hailing dispatching and scheduling methods are largely based on the ideal assumption of homogeneity between vehicles and passengers, typically considering only a single service level scenario and ignoring complex factors such as multiple service levels, vehicle heterogeneity, and passenger preferences in real-world operations. In real-world scenarios where multiple service levels coexist, such as express and premium rides, passengers have varying sensitivities to price, service quality, and waiting time. Vehicles also possess different service capabilities due to positioning differences such as economy and premium models. This dramatically increases the combined dimensions of dispatching and relocation decisions, making the problem far more difficult to solve than traditional single-service models can handle.

[0004] More importantly, traditional methods mostly rely on steady-state assumptions or static allocation strategies based on fixed rules. When faced with scenarios such as rush hour, large events, or severe weather that cause non-stationary and drastic fluctuations in demand, these methods often fail to make timely and effective dynamic adjustments, directly leading to a series of chain problems such as severe imbalance in vehicle distribution between regions, failure to respond to a large number of passenger demands in a timely manner, and a significant decline in the overall revenue of the platform. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-vehicle autonomous ride-hailing dispatching and scheduling coordination method to solve the technical problems in existing ride-hailing systems, which suffer from low dispatching and vehicle relocation efficiency and insufficient platform revenue optimization when demand changes non-stationarily, due to the neglect of vehicle and passenger heterogeneity and the reliance on steady-state assumptions.

[0006] This invention provides a method for dispatching and coordinating multi-vehicle autonomous ride-hailing services, comprising:

[0007] S1. Define the state space, action space, instant reward, external information and state transition function of the ride-hailing system model. The state space includes at least the number of idle vehicles, the number of vehicles on the road and the number of relocated vehicles of different types in each region, as well as the origin and destination locations and service type preferences of passengers. The action space includes at least the passenger dispatch decision for selecting the service vehicle type for dual-compatible passengers and the vehicle relocation decision for assigning the dispatch target area to vehicles in the idle state.

[0008] S2. Based on the state space and action space, construct an event-driven Markov decision process model with passenger arrival event and vehicle state change event as triggering conditions;

[0009] S3. For the event-driven Markov decision process model, a post-decision state value function approximation model based on neural networks is used for value approximation, and an approximate dynamic programming method is used to solve the joint strategy of dispatching and vehicle relocation.

[0010] It should be noted that: the state space describes the complete information of the system at any given time, including vehicle status information and passenger order information. The vehicle status information includes the number of idle vehicles, the number of vehicles on the road, and the number of vehicles repositioning in each area. The passenger order information includes the passenger's origin and destination locations and service type preferences. The action space defines the set of decision operations that the platform can execute, specifically including passenger dispatch decisions and vehicle repositioning decisions. The passenger dispatch decision is used to select a specific service vehicle type for dual-compatible passengers, and the vehicle repositioning decision is used to select a target area for dispatching idle vehicles. By jointly optimizing the above two types of decisions, the problems of passenger heterogeneity and demand non-stationarity are addressed respectively. The immediate reward represents the immediate benefit or cost incurred after executing the dispatch or repositioning decision. The external information represents external random factors independent of the platform's decisions, specifically passenger demand information and trip completion information arriving between the current decision and the next decision time. The state transition function takes the current state, decision actions, and external information as input and outputs the system state at the next time step, thereby defining the evolution rules of the system state.

[0011] In some embodiments, in the event-driven Markov decision process model, the state transition events include passenger-triggered dispatch events and vehicle state change events; for passenger-triggered dispatch events, when the arriving passenger is a dual-compatible passenger and there are multiple available vehicle types in the area where the passenger is located, the passenger dispatch decision is triggered to determine the type of vehicle to serve the passenger, and then the number of idle vehicles and the number of vehicles on the way in the corresponding area are updated.

[0012] In some embodiments, for vehicle state change events, when a vehicle completes relocation, the vehicle transitions from the relocation state to the idle state; when a vehicle completes passenger service, the vehicle is transitioned to the idle state or directly relocated to the target area according to preset rules or real-time decisions, and the number of idle vehicles, the number of services on the way, and the number of relocations in each area are updated synchronously.

[0013] It should be noted that the event-driven state transition mechanism binds the system's state evolution and platform decision triggers to specific events. These state transition events include passenger-triggered dispatch events and vehicle status change events, and corresponding state updates and decision operations are performed based on different event types. For passenger-triggered dispatch events, when a passenger arrives, a matching operation is performed based on the passenger type and vehicle availability in the current area. For passengers who only support a single service type, no decision is needed, and the corresponding type of vehicle is directly assigned. For dual-service compatible passengers, when only one type of vehicle is available, that type of vehicle is directly assigned. Only when multiple types of vehicles are available is a dispatch decision performed to determine the type of vehicle to serve them. After the decision is completed, the number of idle vehicles and vehicles en route in the corresponding area is updated. For vehicle status transition events, when the vehicle status changes, the corresponding state update is performed. When a vehicle completes relocation, the relocated vehicle is changed to an idle vehicle. When a vehicle completes passenger service, according to preset rules or real-time decisions, the vehicle is either changed to an idle state or directly relocated to the target area, and the specific number of vehicles in each area that are idle, in service, or in relocation state is updated synchronously.

[0014] In some embodiments, a post-decision state value function approximation model based on neural networks is used for value approximation, including: setting a feature function vector and using a neural network to iteratively update the parameter vector, so that the post-decision state value function is represented by the mapping relationship between the feature function and the parameter vector, so as to approximate the future expected return of the system in the post-decision state.

[0015] In some embodiments, at each decision time, an action value function is constructed based on the approximation of the post-decision state value function, and by comparing the action value function values ​​corresponding to different possible actions, the action that maximizes the action value function value is selected as the current decision.

[0016] In some embodiments, the iterative training process of gradually updating the parameter vector introduces an experience replay mechanism and a target network update strategy, including:

[0017] State transition samples are generated in the simulation environment and stored in the experience replay buffer. The state transition samples include the current state, action, post-decision state, external information and transition benefits.

[0018] When the number of samples in the buffer reaches a preset condition, small-batch sampling is performed;

[0019] For each sampled sample, the target value is calculated based on the target network parameters, and the loss function is calculated by combining the value function estimate under the online network parameters;

[0020] An adaptive optimization algorithm is used to update the online network parameters, and the target network parameters are softly updated at a preset frequency, while the exploration rate is gradually decayed.

[0021] In some embodiments, the event-driven Markov decision process model is further subjected to sensitivity analysis by varying the fleet size and the proportion of vehicles of different service types to assess the stability of the system under different conditions.

[0022] In some embodiments, the vehicle status information in the state space includes at least the number of express cars and premium cars in each area that are in an idle state, a service state, and a relocation state, respectively; the service type preference in the passenger order information includes passengers who only accept express car services, passengers who only accept premium car services, and passengers who are compatible with both.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. This invention incorporates the joint optimization problem of dispatching and vehicle relocation into a unified approximate dynamic programming solution system through an event-driven reinforcement learning framework based on post-decision states. This effectively separates the deterministic influence of decision-making from external random disturbances, and improves the computational feasibility and learning efficiency of the model when dealing with high-dimensional state space problems without relying on complex structural assumptions.

[0025] 2. This invention alleviates fluctuations and instabilities during training by introducing an experience replay mechanism and a target network update strategy in the value function approximation process, and enhances the model's adaptability in non-stationary demand environments. As a result, it improves the service completion rate and platform revenue, core indicators, compared to traditional steady-state approximation methods. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the multi-vehicle autonomous ride-hailing dispatching and scheduling coordination method of the present invention. Detailed Implementation

[0028] The following will be based on embodiments of the present invention. Figure 1 The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] Application Overview

[0030] In actual operation, the platform needs to process large-scale passenger travel requests in real time and dynamically allocate vehicle resources. Its core operational decisions mainly include two aspects: First, passenger dispatching, where when a new passenger request is generated, the platform needs to quickly assign it to the most suitable available vehicle to shorten passenger waiting time and improve matching success rate; second, vehicle relocation, where when a vehicle is idle or has just completed a service, the platform needs to proactively relocate it to other areas to balance supply and demand in different areas and improve overall service coverage and system response efficiency. The real-time nature and accuracy of these two types of decisions directly determine the platform's service quality, operating costs, and ultimate revenue.

[0031] However, existing solutions have fundamental shortcomings in simultaneously addressing passenger and vehicle heterogeneity and demand non-stationarity, making it difficult to achieve efficient joint optimization of dispatching and vehicle relocation. Based on this, the present invention is proposed, detailed below.

[0032] Example

[0033] In one specific embodiment, a ride-hailing platform operates within a city road network comprised of multiple geographical regions. This platform manages two types of vehicles: economy express cars and premium private cars. Correspondingly, the platform's passenger base is divided into three types: price-sensitive passengers who only accept express car services, premium passengers who only accept private car services, and dual-service compatible passengers who accept both services. This heterogeneous configuration transforms dispatching decisions from a simple proximity-based allocation problem into a complex sequential decision-making problem requiring coordinated consideration of current and future supply and demand, and balancing resource utilization across different service levels.

[0034] To enable the method of this invention to be understood and executed by a computer system, it is necessary to formally model the ride-hailing system and define various variables that constitute the basis of the entire decision-making problem. According to the method of this invention, this modeling process includes defining the state space, action space, immediate rewards, external information, and state transition functions, which together describe the complete operational logic of a dynamic system.

[0035] The state space is used to characterize the instantaneous snapshot of the system at any decision moment, including two main parts: vehicle status information and passenger order information. Specifically, vehicle status information can be further subdivided into three dimensions of statistical data: the number of different types of vehicles in each region that are idle and waiting for orders, the number of vehicles on the road performing passenger pick-up and drop-off tasks, and the number of repositioned vehicles that are being dispatched from one region to another but have not yet arrived at their destination. In one embodiment, we divide the city into N non-overlapping regular hexagonal regions and maintain a nine-tuple of information for each region: the number of express vehicles idle, the number of express vehicles on the road, the number of express vehicles repositioned, the number of VIP vehicles idle, the number of VIP vehicles on the road, the number of VIP vehicles repositioned, and the number of express-only passengers, VIP-only passengers, and dual-compatible passengers newly arriving in that region at the current moment. This state definition enables the system to accurately perceive the supply and demand distribution hotspots and vehicle service status throughout the entire road network.

[0036] After defining the state, the next step is to define the operations that the platform can perform, i.e., the action space. In this invention, the action space consists of two core decision-making processes: passenger dispatching decisions and vehicle relocation decisions.

[0037] The passenger dispatch decision is triggered by passengers with a dual-service preference. This decision is discrete, assigning a vehicle from the available express or premium car options to this specific group. The vehicle relocation decision, on the other hand, is triggered by vehicles that are idle and not assigned orders. This decision assigns a target geographical area to the vehicle and orders it to drive empty to that area, aiming to balance supply and demand and meet potential passenger needs in the future. These two actions are the only control mechanisms the platform can actively apply; all subsequent optimization efforts revolve around choosing the optimal option at the right time. Executing either action brings an immediate benefit to the platform, which can be positive or negative. In this embodiment, for the dispatch action, the immediate reward is set as the estimated trip cost of the order minus the operating costs. For the relocation action, since empty driving itself does not generate revenue, its immediate reward is negative, representing the fuel or electricity costs of the empty drive and the vehicle's time depreciation costs.

[0038] Within the framework of Markov decision processes, the value of a state-action pair depends not only on the immediate reward it generates but also on the future state the action leads the system to. However, ride-hailing systems do not operate in a completely deterministic environment; the addition of passenger demand and the completion of vehicle trips are external random events independent of the platform's decisions. Therefore, this invention captures this uncertainty by defining external information. External information refers to all spontaneous external changes that occur between the completion of an action at the current decision time t and the arrival of the next decision time t+1. It mainly includes two random processes: first, the number of all types of newly arriving passengers in each region; and second, the vehicle information in each region that has completed a trip or relocation process, resulting in a change in its state.

[0039] By introducing these external random information, the state transition function becomes a mapping from determinism to randomness. That is, it takes the current state, the actions performed, and the external information that occurs in between as inputs, and after a series of event-driven update logics, it outputs the precise system state at the next decision moment. This mechanism perfectly simulates the dynamic evolution process of the real-world ride-hailing system.

[0040] Based on the above definitions, this invention further models the dynamic evolution process as a closed-loop queuing network with event-driven mechanisms at its core. The key is that state updates and decision triggers are not performed at fixed time intervals, but are closely tied to passenger arrival events and trip completion or vehicle state change events. The state transition mechanism of this invention is specifically manifested as two parallel and interleaved processing flows.

[0041] The first process is handling the dispatch event triggered by the passenger.

[0042] When a new passenger makes a travel request in a certain area, the system first identifies the passenger type. If the passenger is a single type of user, only requesting express or premium rides, the decision logic becomes very simple and direct: it immediately checks whether there are any available vehicles of the corresponding type in the area. If so, an available vehicle is assigned to serve the passenger, and the vehicle's status is updated from idle to en route, while recording the destination and estimated service time; if not, the order is declared a service failure, and the passenger is lost. This process does not require triggering optimization decisions, thus avoiding unnecessary computational overhead. However, when a dual-type passenger arrives, the situation becomes more complex, which is the core of the dispatch decision optimization in this invention. The system checks the number of available express and premium rides in the area. If both are zero, the passenger is also lost; if one type of vehicle is available but the other is not, the system has no choice but to directly assign an available vehicle of the available type, which also does not constitute a decision. The real decision point is when there is at least one available express ride and at least one available premium ride in area i, the platform must make an active choice. At this point, the dispatch decision variable defined in this invention is activated, and the algorithm needs to decide whether to assign an express car or a premium car to serve the passenger. Once a decision is made, the number of available vehicles corresponding to the selected vehicle type is decreased by one, while the number of vehicles en route is increased by one. This design is a low-computational-cost but efficient decision-triggering mechanism that ensures computing resources are allocated only to critical moments that have a substantial impact on the long-term benefits of the system.

[0043] The second parallel process is the handling of vehicle state transition events.

[0044] These events are triggered independently of passenger demand; they originate from the termination of the vehicle's own state. In one embodiment, there are two main types of these events. The first is when a vehicle performing a relocation task arrives at its target area, triggering a state update. The system changes the vehicle's state from a relocation vehicle to an idle vehicle and increases the number of idle vehicles of the corresponding type in the target area accordingly.

[0045] The second scenario is when a vehicle serving passengers completes its journey. The platform needs to decide the vehicle's next status after completing its service: whether to assign it a new relocation task and dispatch it directly to a high-demand potential area, or to leave it in its current area as idle and wait for the next order.

[0046] This decision also reflects the relocation decision under the joint optimization framework of this invention. After executing this decision, the status of the vehicle and the vehicle count in the area will be updated accordingly. Through the closed-loop operation of the above two core event processes, the spatial distribution of the three types of vehicles—idle, in service, and relocation—changes with each event trigger, thus forming a complex and realistic closed-loop queuing network that truly reflects the business logic.

[0047] After successfully constructing an event-driven Markov decision process model, the core challenge lies in finding the optimal decision strategy within a vast state and action space. Traditional dynamic programming methods are infeasible due to the curse of dimensionality. To address this bottleneck, this invention proposes an approximate dynamic programming method based on post-decision states. This method is represented by an approximate model of the post-decision state value function, as follows:

[0048]

[0049] in, Representation strategy, Let t represent the set of strategies, t represent the decision time, and T represent the length of the decision time domain. Indicates the discount factor. Indicates immediate benefits. This represents the transfer revenue generated from external information. Indicates the state before a decision is made. Representing the post-decision state, in the strategy Below, at the moment The actions taken Indicates external information. This represents the initial state of the system. The key to this approximate model lies in introducing the concept of the post-decision state, which refers to the instantaneous deterministic state of the system after executing action a but before the arrival of external information W.

[0050] In a specific embodiment, a fully connected neural network is used as the approximation function. The input of the network is the encoded vector of the post-decision state, and the output is the estimated value of the future cumulative reward of the state. The weight parameters of the neural network are the objects that we need to update and optimize through iterative training.

[0051] Furthermore, the constraints of the approximate model of the post-decision state value function include platform dual-compatible passenger dispatch constraints, vehicle quantity constraints, and 0-1 constraints on variables, which are expressed as follows:

[0052]

[0053]

[0054]

[0055] in, Indicates the region The middle type is The number of available vehicles Indicates from the region Departure, Heading to the Area The type is The number of vehicles in service. Indicates from the region Relocate to region The type is The number of repositioned vehicles. The type is The total number of vehicles. Should dual-compatible passengers be assigned to express trains? Indicate whether dual-compatible passengers will be assigned to private cars. : Indicates whether to use type The vehicles will be in the area Passengers were matched to destination areas The service.

[0056] At each decision-making moment *t*, the system enumerates all legal actions in the current state. For each candidate action, it first simulates the execution of that action to obtain a deterministic post-decision state. Then, it evaluates this post-decision state using an approximation model of the current post-decision state value function, calculating its state value. Next, it constructs an action value function, which consists of the sum of two parts: the immediate reward for executing the action, and a discount value to the post-decision state value. The action value function is expressed as:

[0057]

[0058] in, To construct an approximate expression of the post-decision state value function; θ is the parameter vector. The algorithm will ultimately select the action that maximizes the action value function as the optimal decision for the current moment and execute it. This process perfectly integrates short-term gains and long-term planning.

[0059] To train this neural network that serves as the action value function, this invention employs a core training algorithm that incorporates the concept of deep Q-networks. In the initial stage, two sets of network parameters need to be initialized: one set is online network parameters, used to evaluate actions and generate decisions in real time; the other set is target network parameters, which serve as a stable anchor for value evaluation, and whose parameters are slowly changed through soft updates.

[0060] Simultaneously, an experience replay buffer of a certain capacity is initialized, and the training process will repeat multiple complete simulation rounds. Within each round, the system starts from the observed initial state and makes a series of decisions until the end of the round. At each decision step, the algorithm combines an ε-greedy strategy for action selection, that is, it randomly selects an action for exploration with a small probability ε, while selecting the action that the current value function considers optimal with a probability of 1-ε. This mechanism balances the relationship between exploring the unknown and utilizing the known.

[0061] Once an action is executed, the environment simulator returns the state before the next decision, the state after the next decision, and the reward generated during the process, based on the state transition function, and generates a complete experience sample. This sample is stored in the experience replay buffer. When the number of samples stored in the buffer exceeds the preset minimum batch size, the training program will randomly select a small batch of samples from the buffer.

[0062] For each sample, we use the target network to compute its target value, which equals the immediate reward in the sample plus a discount to the value of the next post-decision state as evaluated by the target network. Then, we calculate the mean squared error between this target value and the current post-decision state value evaluated by the online network as a loss function. Based on this loss function, we perform backpropagation using the Adam optimizer to update the parameters of the online network.

[0063] At the same time, after each preset update frequency C, the parameters of the online network are copied to the target network in a soft update manner, that is, the target network parameters θ. - ← τθ + (1-τ)θ - Here, τ is a soft update rate much smaller than 1. As training progresses, the exploration rate ε gradually decays, transitioning from a high exploration rate to a high-efficiency utilization rate. Through repeated iterations, the neural network parameters gradually converge, eventually approximating the optimal post-decision state value function, thus obtaining a high-performance joint optimization strategy for order dispatching and relocation that can be used in actual production environments.

[0064] To further verify the robustness of this method and provide decision support for practical deployment, this invention also includes a sensitivity analysis of the obtained model. The sensitivity analysis mainly focuses on two key operational parameters: total fleet size and the proportion of vehicles of different service types.

[0065] In the first analysis, we fixed the ratio of express cars to premium cars at a certain value, and then gradually increased or decreased the total fleet size used for simulation. We observed the change curves of the optimization method of this invention on the two key performance indicators of service completion rate and total platform revenue, so as to evaluate the performance stability of the method under different capacity supply.

[0066] In the second analysis, we kept the total fleet size constant while varying the proportion of express vehicles in the fleet. The results showed that the system benefits were maximized when the proportion of express vehicles matched the proportion of express vehicle preference in actual passenger demand. This finding provides a data-driven scientific basis for the platform to formulate vehicle procurement or franchise plans.

[0067] Extensive numerical experiments demonstrate that, during simulated peak hours, evening peak hours, and off-peak hours, the method of this invention improves service completion rate by more than 5% compared to traditional methods based on steady-state approximation or multi-period static optimization, and also achieves a significant increase in overall platform revenue of nearly 8% during simulated peak hours, evening peak hours, and off-peak hours.

[0068] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0069] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for dispatching and coordinating multi-vehicle autonomous ride-hailing services, characterized in that, include: S1. Define the state space, action space, instant reward, external information and state transition function of the ride-hailing system model. The state space includes at least the number of idle vehicles, the number of vehicles on the road and the number of relocated vehicles of different types in each region, as well as the origin and destination locations and service type preferences of passengers. The action space includes at least the passenger dispatch decision for selecting the service vehicle type for dual-compatible passengers and the vehicle relocation decision for assigning the dispatch target area to vehicles in the idle state. S2. Based on the state space and action space, construct an event-driven Markov decision process model with passenger arrival event and vehicle state change event as triggering conditions; S3. For the event-driven Markov decision process model, a post-decision state value function approximation model based on neural networks is used for value approximation, and an approximate dynamic programming method is used to solve the joint strategy of dispatching and vehicle relocation.

2. The method according to claim 1, characterized in that, In the event-driven Markov decision process model, state transition events include passenger-triggered dispatch events and vehicle state change events. For passenger-triggered dispatch events, when the arriving passenger is a dual-compatible passenger and there are multiple types of vehicles available in the area, the passenger dispatch decision is triggered to determine the type of vehicle to serve, and then the number of idle vehicles and the number of vehicles on the way in the corresponding area are updated.

3. The method according to claim 2, characterized in that, For vehicle status change events, when a vehicle completes relocation, it transitions from the relocation state to the idle state; when a vehicle completes passenger service, it transitions to the idle state or directly relocates to the target area according to preset rules or real-time decisions, and simultaneously updates the number of idle vehicles, the number of services on the way, and the number of relocations in each area.

4. The method according to claim 1, characterized in that, The value approximation is performed using a post-decision state value function approximation model based on neural networks, including: setting a feature function vector and gradually updating the parameter vector through iterative training of the neural network, so that the post-decision state value function is represented by the mapping relationship between the feature function and the parameter vector, so as to approximate the future expected return of the system in the post-decision state.

5. The method according to claim 4, characterized in that, At each decision point, based on the approximation of the post-decision state value function, an action value function is constructed, and by comparing the action value function values ​​corresponding to different possible actions, the action that maximizes the action value function value is selected as the current decision.

6. The method according to claim 4, characterized in that, The iterative training process of gradually updating the parameter vector introduces an experience replay mechanism and a target network update strategy, including: State transition samples are generated in the simulation environment and stored in the experience replay buffer. The state transition samples include the current state, action, post-decision state, external information and transition benefits. When the number of samples in the buffer reaches a preset condition, small-batch sampling is performed; For each sampled sample, the target value is calculated based on the target network parameters, and the loss function is calculated by combining the value function estimate under the online network parameters; An adaptive optimization algorithm is used to update the online network parameters, and the target network parameters are softly updated at a preset frequency, while the exploration rate is gradually decayed.

7. The method according to claim 1, characterized in that, It also includes conducting sensitivity analysis on the event-driven Markov decision process model by changing the fleet size and the proportion of vehicles of different service types to evaluate the stability of the system under different conditions.

8. The method according to claim 1, characterized in that, The vehicle status information in the state space includes at least the number of express cars and premium cars in each area that are in an idle state, a service state, and a relocation state, respectively; the service type preferences in the passenger order information include passengers who only accept express car services, passengers who only accept premium car services, and passengers who are compatible with both.

Images