Multi-vehicle path planning system based on deep reinforcement learning
The path planning system, which combines multi-source information fusion and deep reinforcement learning, solves the problems of slow response speed and insufficient multi-vehicle coordination in patrol vehicle dispatching, and achieves efficient and collaborative path planning, improving dispatching efficiency and dynamic adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2026-04-08
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies for patrol vehicle dispatching suffer from slow response times and insufficient multi-vehicle coordination, making it difficult to achieve rapid response and real-time dynamic adaptation, resulting in path planning failures and resource waste.
A multi-source information fusion system is adopted, combining GPS and GIS data, and using an algorithm that combines deep reinforcement learning and integer programming to realize patrol vehicle route planning. This includes data cleaning, integer programming mathematical modeling, and reinforcement learning algorithm to dynamically generate the optimal route.
It enables precise, efficient, and collaborative route planning for patrol vehicles, improves dispatch efficiency and dynamic adaptability, and ensures rapid response and resource optimization.
Smart Images
Figure CN122050184A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent transportation technology, and in particular to a multi-vehicle path planning system based on deep reinforcement learning. Background Technology
[0002] In scenarios such as urban traffic emergency response, the efficient dispatch of patrol vehicles directly impacts incident response efficiency and public safety assurance capabilities. Currently, the mainstream patrol vehicle dispatching method relies primarily on centralized manual dispatch, depending on the experience and judgment of dispatchers. This method has the following core drawbacks: First, high response latency. When a sudden traffic incident occurs, the process of manually assessing road conditions and planning routes is time-consuming, hindering rapid response. Second, poor dynamic adaptability. It struggles to respond in real-time to dynamic changes such as road network congestion and road construction, easily leading to route planning failures. Third, insufficient multi-vehicle coordination and a lack of a global optimization perspective, easily resulting in multi-vehicle route conflicts and wasted road network resources.
[0003] To address the aforementioned issues, existing technologies have proposed path planning schemes based on traditional optimization methods such as integer programming and genetic algorithms. However, these methods suffer from slow convergence speed and low solution space search efficiency when dealing with large-scale road networks (e.g., more than 20 nodes), making it difficult to meet real-time scheduling requirements. In recent years, deep reinforcement learning technology has been gradually applied to the field of path planning due to its strong adaptability to dynamic environments. However, single deep reinforcement learning methods suffer from drawbacks such as strong exploration blindness, unstable training convergence, and susceptibility to getting trapped in local optima, making them unsuitable for direct application in patrol car scheduling scenarios with extremely high reliability requirements. Therefore, there is an urgent need for a collaborative algorithm architecture that integrates multi-source information processing, traditional integer programming, and deep reinforcement learning to balance scheduling efficiency, dynamic adaptability, and reliability, achieving accurate, efficient, and collaborative path planning for patrol cars. Summary of the Invention
[0004] To address the aforementioned problems, the present invention aims to solve the issues of slow response speed and insufficient multi-vehicle coordination in the prior art.
[0005] The present invention is achieved by at least one of the following technical solutions.
[0006] A multi-vehicle path planning system based on deep reinforcement learning, the system comprising:
[0007] The multi-source information acquisition and fusion system collects traffic flow information (accident locations, road conditions, etc.) through patrol car onboard cameras, and combines it with vehicle GPS location information and GIS geographic information to form basic perception data. The system further integrates these real-time collected data with historical traffic flow data and road network topology data to construct a multi-source information system.
[0008] The real-time decision-making system in the command and dispatch center integrates information and transmits it through a communication network to the command and dispatch center to build a patrol vehicle dispatch and route planning model. The dispatch and route planning model includes data cleaning, an integer programming mathematical model, a reinforcement learning algorithm, and a dynamic route generation module.
[0009] The task execution terminal is the patrol vehicle. Through this model, the system can intelligently allocate tasks and schedule routes for the patrol vehicle, guiding it to efficiently complete tasks such as traffic incident handling and road management.
[0010] Furthermore, the congestion factor calculation algorithm in the multi-source information fusion algorithm module is as follows:
[0011]
[0012] in For the first Congestion factors of this road section For the first Real-time traffic flow on this road segment For the first The capacity threshold for each road segment; the algorithm for calculating the flow balance error is as follows:
[0013]
[0014] in The road network topology matrix, This is the traffic flow vector for the road segment. This is the OD demand vector.
[0015] Furthermore, the data cleaning process is used to convert accident data coordinates into a form that meets the needs of analysis, retain complete data from the original accident data, remove data with duplicate information, missing content values, or data values that do not match reality, and select data items relevant to this paper from the retained data to facilitate data analysis.
[0016] The Integer Programming Mathematical Model (MILP) is defined as follows:
[0017]
[0018] in, The road network topology matrix defines the relationships between nodes and edges; For vehicle path information, "0" indicates that the edge is not included in the vehicle path, and "1" indicates that the edge is included in the vehicle path, meaning that the vehicle will drive through this road segment; This indicates OD (origin-destination) information, which is the origin and destination information of a vehicle; This refers to the maximum capacity or traffic flow throughput of the road segment. Congestion factor.
[0019] Furthermore, the reinforcement learning algorithm includes the following steps:
[0020] S1: Solve the initial feasible solution of the integer programming mathematical model using the lightweight MILP feasible solution generation module. Set a timeout during the solution process and stop when the first feasible solution is found;
[0021] S2: Map the feasible solution of MILP to the initial path probability of PPO, and inversely deduce the initial parameters of the Actor network, so that PPO is trained from the constraint compliance policy to avoid random exploration;
[0022] S3: Core training is performed through the PPO reinforcement learning scheduling module;
[0023] S4: The dynamic correction module determines the current traffic solution of PPO every 50 rounds. If a constraint is violated, a correction is triggered, and the lightweight MILP module is called to solve for the corrected feasible solution. This information is then fed back to the PPO module to update the initial strategy.
[0024] S5: Repeat steps S3~S4 until the PPO network converges. After training, only the PPO reinforcement learning scheduling module receives the real-time state vector, outputs the optimal path probability and issues scheduling instructions to achieve millisecond-level real-time inference.
[0025] Further, step S2 includes the following steps:
[0026] S21: Convert the segment flow vector into distributable flow for each available path. For any path... ( (K is the total number of optional paths), its allocable traffic Defined as the minimum traffic volume across all segments included in the path, i.e.:
[0027]
[0028] in For the first Traffic flow for each road segment (from MILP feasible solution). For path The set of road segment indexes included;
[0029] S22: Normalize the allocatable traffic for each path to obtain the initial policy probability distribution of the Actor network. The formula is:
[0030]
[0031] in To select a path The action, The sum of all available traffic for all paths, used for normalization;
[0032] S23: Backtracking the logits output of the last layer of the Actor network This makes the output exactly after Softmax. The formula is:
[0033]
[0034] in This is a numerically stable term to avoid the ln(0) error.
[0035] Furthermore, the training process of the PPO algorithm in S3 includes:
[0036] S31: Initialize the parameters of the Actor policy network and the Critic value network, and set the hyperparameter: discount factor. GAE coefficient Editing coefficients Learning rate = 3e-4;
[0037] S32: Constructing the normalized state vector , as input to the Actor and Critic networks;
[0038] S33: Patrol vehicles interact with the road network environment. Each time a patrol vehicle is assigned, it performs a route selection action once and records the status. ,action Reward function Next state and the logarithmic probability of actions Stored in the experience replay pool;
[0039] S34: When the sample size of the experience replay pool reaches a preset threshold, the advantages of each step are calculated based on GAE. With target value The Actor policy network is optimized using clipping loss, and the Critic value network is optimized using mean squared error loss.
[0040] S35: Repeat steps 2-3 until the average reward fluctuation is less than 1% for 100 consecutive rounds, then determine that the model has converged.
[0041] Furthermore, the reward function algorithm in the deep reinforcement learning unit is as follows:
[0042]
[0043] Where k1, k2, k3, and k4 are preset weighting coefficients. For the first Congestion factors of walking For the first The flow balance error of the step, For the first The time it took for the patrol car to arrive at the incident point, As an indicator variable, when all patrol vehicles complete their missions and the capacity and flow conservation constraints are satisfied... ,otherwise .
[0044] Furthermore, the formulas for the advantage value and target value of the GAE network are as follows:
[0045]
[0046] in For timing difference error, For single-step rewards, For the Critic network to state The value estimate, As a discount factor, For GAE coefficients, For the end mark ( This indicates that vehicle allocation has been completed.
[0047] Furthermore, the clipping loss formula for the Actor network is:
[0048]
[0049] in, This represents the probability ratio between the old and new strategies. The editing coefficient limits the fluctuation range of the probability ratio. These are the Actor network parameters before the update.
[0050] Furthermore, the constraint determination formula of the dynamic correction module is as follows:
[0051]
[0052] in: For the current solution of PPO, the first... Let A be the traffic flow of each road segment, A be the road network topology matrix, and b be the OD demand vector.
[0053] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0054] Figure 1 This is a block diagram of the overall architecture of the present invention;
[0055] Figure 2 This is a schematic diagram of the traffic network structure of the present invention;
[0056] Figure 3 This is a schematic diagram of the deep reinforcement learning process. Detailed Implementation
[0057] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0058] like Figure 1 As shown in this embodiment, a multi-vehicle path planning system based on deep reinforcement learning is provided. The system includes:
[0059] The multi-source information acquisition and fusion system collects traffic flow information (accident locations, road conditions, etc.) through patrol car-mounted cameras, and combines this with vehicle GPS location information and GIS geographic information to form basic perception data. The system further integrates this real-time collected data with historical traffic flow data and road network topology data to construct a multi-source information system. The real-time decision-making system in the command and dispatch center transmits the fused information through a communication network to the command and dispatch center to construct patrol car dispatch and route planning models. The dispatch and route planning models include data cleaning, integer programming mathematical models, reinforcement learning algorithms, and a dynamic route generation module.
[0060] The task execution terminal is the patrol vehicle. Through this model, the system can intelligently allocate tasks and schedule routes for the patrol vehicle, guiding it to efficiently complete tasks such as traffic incident handling and road management.
[0061] In one possible implementation, the congestion factor calculation algorithm in the multi-source information fusion algorithm module is as follows:
[0062]
[0063] in For the first Congestion factors of this road section For the first Real-time traffic flow on this road segment For the first The capacity threshold for each road segment; the algorithm for calculating the flow balance error is as follows:
[0064]
[0065] in The road network topology matrix, This is the traffic flow vector for the road segment. This is the OD demand vector.
[0066] In one possible implementation, the data cleaning is used to convert accident data coordinates into a form that meets the needs of analysis, retain complete data in the original accident data, remove data with duplicate information, missing content values, and data values that do not match the actual data, and select data items that are relevant to this paper from the retained data to facilitate data analysis.
[0067] In one possible implementation, the integer programming mathematical model (MILP) is defined as follows:
[0068]
[0069] in, The road network topology matrix defines the relationships between nodes and edges; For vehicle path information, "0" indicates that the edge is not included in the vehicle path, and "1" indicates that the edge is included in the vehicle path, meaning that the vehicle will drive through this road segment; This indicates OD (origin-destination) information, which is the origin and destination information of a vehicle; This refers to the maximum capacity or traffic flow throughput of the road segment. Congestion factor.
[0070] like Figure 2 As shown, in a traffic network, a fork in the road is defined as a node, which is an intersection in the actual traffic network; the connection between nodes is defined as an edge or arc, which is a road or its sub-segment in the actual traffic network (the definition of an edge is different from the definition of a road in the actual traffic network. An edge is a connection between nodes and does not contain other nodes in between, while an actual road usually contains many intersections); each edge is characterized by a feature attribute, namely, road impedance or weight, which can generally be expressed by length or travel time.
[0071] like Figure 3As shown, in one possible implementation, the reinforcement learning algorithm includes the following steps:
[0072] S1: Solve the initial feasible solution of the integer programming mathematical model using the lightweight MILP feasible solution generation module. Set a timeout during the solution process and stop when the first feasible solution is found;
[0073] S2: Map the feasible solution of MILP to the initial path probability of PPO, and inversely deduce the initial parameters of the Actor network, so that PPO is trained from the constraint compliance policy to avoid random exploration;
[0074] S3: Core training is performed through the PPO reinforcement learning scheduling module;
[0075] S4: The dynamic correction module determines the current traffic solution of PPO every 50 rounds. If a constraint is violated, a correction is triggered, and the lightweight MILP module is called to solve for the corrected feasible solution. This information is then fed back to the PPO module to update the initial strategy.
[0076] S5: Repeat steps S3~S4 until the PPO network converges. After training, only the PPO reinforcement learning scheduling module receives the real-time state vector, outputs the optimal path probability and issues scheduling instructions to achieve millisecond-level real-time inference.
[0077] S2 includes the following steps:
[0078] S21: Convert the segment flow vector into distributable flow for each available path. For any path... ( (K is the total number of optional paths), its allocable traffic Defined as the minimum traffic volume across all segments included in the path, i.e.:
[0079]
[0080] in For the first Traffic flow for each road segment (from MILP feasible solution). For path The set of road segment indexes included;
[0081] S22: Normalize the allocatable traffic for each path to obtain the initial policy probability distribution of the Actor network. The formula is:
[0082]
[0083] in To select a path The action, The sum of all available traffic for all paths, used for normalization;
[0084] S23: Backtracking the logits output of the last layer of the Actor network This makes the output exactly after Softmax. The formula is:
[0085]
[0086] in This is a numerically stable term to avoid the ln(0) error.
[0087] The training process of the PPO algorithm in S3 includes:
[0088] S31: Initialize the parameters of the Actor policy network and the Critic value network, and set the hyperparameter: discount factor. GAE coefficient Editing coefficients Learning rate = 3e-4;
[0089] S32: Constructing the normalized state vector , as input to the Actor and Critic networks;
[0090] S33: Patrol vehicles interact with the road network environment. Each time a patrol vehicle is assigned, it performs a route selection action once and records the status. ,action Reward function Next state and the logarithmic probability of actions Stored in the experience replay pool;
[0091] S34: When the sample size of the experience replay pool reaches a preset threshold, the advantages of each step are calculated based on GAE. With target value The Actor policy network is optimized using clipping loss, and the Critic value network is optimized using mean squared error loss.
[0092] S35: Repeat steps 2-3 until the average reward fluctuation is less than 1% for 100 consecutive rounds, then determine that the model has converged.
[0093] In one possible implementation, the Actor policy network is set to take a 15-dimensional state vector as input, pass through two fully connected layers (64-dimensional → 32-dimensional) and a Softmax output layer, and output the selection probability of K paths. The Critic value network is set to take a 15-dimensional state vector as input, pass through two fully connected layers (64-dimensional → 32-dimensional), and output the cumulative reward expectation of the current state.
[0094] The reward function algorithm in the deep reinforcement learning unit is as follows:
[0095]
[0096] Where k1, k2, k3, and k4 are preset weighting coefficients. For the first Congestion factors of walking For the first The flow balance error of the step, For the first The time it took for the patrol car to arrive at the incident point, As an indicator variable, when all patrol vehicles complete their missions and the capacity and flow conservation constraints are satisfied... ,otherwise .
[0097] The formulas for the advantage value and target value of the GAE network are as follows:
[0098]
[0099] in For timing difference error, For single-step rewards, For the Critic network to state The value estimate, As a discount factor, For GAE coefficients, For the end mark ( This indicates that vehicle allocation has been completed.
[0100] The clipping loss formula for the Actor network is:
[0101]
[0102] in, This represents the probability ratio between the old and new strategies. The editing coefficient limits the fluctuation range of the probability ratio. For the Actor network parameters before the update
[0103] In one possible implementation, the constraint determination formula of the dynamic correction module is:
[0104]
[0105] in: For the current solution of PPO, the first... Let A be the traffic flow of each road segment, A be the road network topology matrix, and b be the OD demand vector.
[0106] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.
Claims
1. A multi-vehicle path planning system based on deep reinforcement learning, characterized in that, The system includes: The multi-source information acquisition and fusion system collects traffic flow information (accident location, road conditions, etc.) through the patrol car's onboard camera, and combines it with the vehicle's GPS location information and GIS geographic information to form basic perception data. The system further integrates these real-time collected data with historical traffic flow data and road network topology data to construct a multi-source information system. The real-time decision-making system in the command and dispatch center integrates information and transmits it through a communication network to the command and dispatch center to construct a patrol vehicle dispatch and route planning model. The dispatch and route planning model includes data cleaning, an integer programming mathematical model, a reinforcement learning algorithm, and a dynamic route generation module. The task execution terminal is the patrol vehicle. Through this model, the system can intelligently allocate tasks and schedule routes for the patrol vehicle, guiding it to efficiently complete tasks such as traffic incident handling and road management.
2. The multi-vehicle path planning system based on deep reinforcement learning according to claim 1, characterized in that, The congestion factor calculation algorithm in the multi-source information fusion algorithm module is as follows: in For the first Congestion factors of this road section For the first Real-time traffic flow on this road segment For the first The capacity threshold for each road segment; the algorithm for calculating the flow balance error is as follows: in The road network topology matrix, This is the traffic flow vector for the road segment. This is the OD demand vector.
3. The multi-vehicle path planning system based on deep reinforcement learning according to claim 1, characterized in that, The data cleaning process is used to convert accident data coordinates into a form that meets the needs of analysis, retain complete data in the original accident data, remove data with duplicate information, missing content values, and data values that do not match the actual data, and select data items that are relevant to this paper from the retained data to facilitate data analysis. The Integer Programming Mathematical Model (MILP) is defined as follows: in, The road network topology matrix defines the relationships between nodes and edges; For vehicle path information, "0" indicates that the edge is not included in the vehicle path, and "1" indicates that the edge is included in the vehicle path, meaning that the vehicle will drive through this road segment; This indicates OD (origin-destination) information, which is the origin and destination information of a vehicle; This refers to the maximum capacity or traffic flow throughput of the road segment. Congestion factor.
4. The multi-vehicle path planning system based on deep reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithm includes the following steps: S1: Solve the initial feasible solution of the integer programming mathematical model using the lightweight MILP feasible solution generation module. Set a timeout during the solution process and stop when the first feasible solution is found; S2: Map the feasible solution of MILP to the initial path probability of PPO, and inversely deduce the initial parameters of the Actor network, so that PPO is trained from the constraint compliance policy to avoid random exploration; S3: Core training is performed through the PPO reinforcement learning scheduling module; S4: The dynamic correction module determines the current traffic solution of PPO every 50 rounds. If a constraint is violated, a correction is triggered, and the lightweight MILP module is called to solve for the corrected feasible solution. This information is then fed back to the PPO module to update the initial strategy. S5: Repeat steps S3~S4 until the PPO network converges. After training, only the PPO reinforcement learning scheduling module receives the real-time state vector, outputs the optimal path probability and issues scheduling instructions to achieve millisecond-level real-time inference.
5. A multi-vehicle path planning system based on deep reinforcement learning according to claim 4, characterized in that, S2 includes the following steps: S21: Convert the segment flow vector into distributable flow for each available path. For any path... ( (K is the total number of optional paths), its allocable traffic Defined as the minimum traffic volume across all segments included in the path, i.e.: in For the first Traffic flow for each road segment (from MILP feasible solution). For path The set of road segment indexes included; S22: Normalize the allocatable traffic for each path to obtain the initial policy probability distribution of the Actor network. The formula is: in To select a path The action, The sum of all available traffic for all paths, used for normalization; S23: Backtracking the logits output of the last layer of the Actor network This makes the output exactly after Softmax. The formula is: in This is a numerically stable term to avoid the ln(0) error.
6. A multi-vehicle path planning system based on deep reinforcement learning according to claim 4, characterized in that, The training process of the PPO algorithm in S3 includes: Step 1: Initialize the parameters of the Actor policy network and the Critic value network, and set the hyperparameter: discount factor. GAE coefficient Editing coefficients Learning rate = 3e-4; Step 2: Construct the normalized state vector , as input to the Actor and Critic networks; Step 3: The patrol car interacts with the road network environment. Each time a patrol car is assigned, it performs a route selection action once and records the status. ,action Reward function Next state and the logarithmic probability of actions Stored in the experience replay pool; Step 3: When the sample size of the experience replay pool reaches a preset threshold, calculate the advantages of each step based on GAE. With target value The Actor policy network is optimized using clipping loss, and the Critic value network is optimized using mean squared error loss. Step 4: Repeat steps 2-3 until the average reward fluctuation is less than 1% for 100 consecutive rounds, at which point the model is considered converged.
7. A multi-vehicle path planning system based on deep reinforcement learning according to claim 6, characterized in that, The reward function algorithm in the deep reinforcement learning unit is as follows: Where k1, k2, k3, and k4 are preset weighting coefficients. For the first Congestion factors of walking For the first The flow balance error of the step, For the first The time it took for the patrol car to arrive at the incident point, As an indicator variable, when all patrol vehicles complete their missions and the capacity and flow conservation constraints are satisfied... ,otherwise .
8. A multi-vehicle path planning system based on deep reinforcement learning according to claim 6, characterized in that, The formulas for the advantage value and target value of the GAE network are as follows: in For timing difference error, For single-step rewards, For the Critic network to state The value estimate, As a discount factor, For GAE coefficients, For the end mark ( This indicates that vehicle allocation has been completed.
9. A multi-vehicle path planning system based on deep reinforcement learning according to claim 6, characterized in that, The clipping loss formula for the Actor network is: in, This represents the probability ratio between the old and new strategies. The editing coefficient limits the fluctuation range of the probability ratio. These are the Actor network parameters before the update.
10. A multi-vehicle path planning system based on deep reinforcement learning according to claim 4, characterized in that, The constraint determination formula of the dynamic correction module is: in: For the current solution of PPO, the first... Let A be the traffic flow of each road segment, A be the road network topology matrix, and b be the OD demand vector.