An intelligent generation method and system of an urban road network based on deep reinforcement learning

Abstract

The application discloses an intelligent generation method and system of an urban road network based on deep reinforcement learning, and belongs to the technical field of urban traffic planning, intelligent traffic systems and deep reinforcement learning. The application models the urban road network generation process as a sequential decision problem on a candidate road edge set, which can convert the complex urban road planning process into a graph decision problem that can be solved by deep reinforcement learning, thereby reducing the dependence on artificial experience and fixed road network templates. The application constructs a candidate road boundary graph based on land use units, plot boundaries and spatial adjacency relationships, so that the generated road network can be coordinated with the urban spatial structure, and is suitable for new district road planning, district road completion and low-grade road network generation scenes. The application introduces a budget hard constraint and an action feasibility mask, directly embeds the construction budget into the road generation process, and avoids the infeasible scheme problem caused by simply controlling the cost through rewards and punishments.

Description

Technical Field

[0001] This invention relates to the fields of urban traffic planning, intelligent transportation systems, and deep reinforcement learning, and in particular to an intelligent generation method and system for urban road networks based on deep reinforcement learning. Background Technology

[0002] Urban road networks are critical infrastructure supporting urban spatial development, residents' daily travel, accessibility of public services, and emergency traffic organization. A rational road network structure not only affects traffic efficiency but also directly relates to connectivity between urban areas, the capacity to meet travel demands, and the resilience of the transportation system under disturbance conditions. With the increasing demand for new urban development, district renewal, and the replacement of low-grade roads, how to automatically generate a rationally structured, highly capable, and resilient road network within a limited construction budget has become a significant technical challenge in urban transportation planning and intelligent transportation systems.

[0003] Current urban road network planning methods primarily rely on planners' experience, rule-based road network generation methods, or traditional transportation network design models. While experience-based planning methods can incorporate actual planning constraints, they depend on manual judgment, resulting in low efficiency in generating solutions and difficulty in systematically comparing a large number of candidate solutions. Rule-based road network generation methods are typically based on preset patterns such as grids, radial structures, or tree structures, making them ill-suited for complex land use boundaries, plot adjacency relationships, and irregular urban spatial forms. Traditional transportation network design methods usually focus on expanding existing roads, optimizing capacity, or reducing congestion, emphasizing continuous or discrete investment optimization on a given network, and rarely addressing planning scenarios that involve "generating road networks from candidate boundaries."

[0004] In recent years, deep reinforcement learning has provided a new technical approach for complex sequential decision-making problems. The road network generation process is inherently sequential; each road construction step alters the accessibility of subsequent candidate roads, network topology, remaining budget, and final traffic service capacity. However, existing reinforcement learning-based road planning or city generation methods still have several shortcomings: First, some methods only focus on road geometry generation or topological connectivity, lacking a feedback loop between road network generation, traffic demand service capacity, and traffic operation costs; second, some methods do not treat construction budgets as hard constraints, leading to generated solutions that may not meet actual planning resource limitations; third, some methods only use simple connectivity indicators to evaluate road networks, failing to reflect the differentiated impact of different OD (Original Demand) quantities on planning schemes; and fourth, some methods lack reasonable resilience normalization references, easily resulting in generated networks that show significant improvement over roadless baselines but lack sufficient differentiation between different budget schemes. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing urban road network planning methods rely on human experience, struggle to handle complex candidate boundary structures, lack hard budget constraints, have insufficient feedback on traffic demand service capacity, and have inadequate resilience evaluation discrimination. This invention provides an intelligent generation method and system for urban road networks based on deep reinforcement learning. By constructing a candidate road boundary graph, the urban road network generation problem is transformed into a sequential decision-making problem on a set of candidate road edges. Hard budget constraints and action feasibility masks ensure that the road generation process meets the construction budget. A graph neural network extracts information such as candidate road topology, node spatial relationships, road construction status, and remaining budget. A deep reinforcement learning strategy network learns road construction decisions. In the final planning stage, training rewards are constructed by combining demand service rate, network connectivity, traffic system cost, unserved demand penalties, and relative resilience improvement indicators, thereby achieving intelligent generation of urban road networks under budget constraints.

[0006] To solve the above-mentioned technical problems, the first technical solution adopted by the present invention is:

[0007] A method for intelligently generating urban road networks based on deep reinforcement learning includes the following steps:

[0008] Step 1: Establish a candidate road boundary map for the urban area to be planned and represent the urban road network generation object as a graph structure composed of candidate nodes and candidate road edges;

[0009] Step 2: Establish an urban road network generation model based on a deep reinforcement learning policy network and determine the corresponding deep reinforcement learning planning environment;

[0010] Step 3: Add budget hard constraints and action feasibility masks to the deep reinforcement learning planning environment to form a budget-feasible deep reinforcement learning planning environment;

[0011] Step 4: Model the process of generating the road network using the urban road network generation model as a sequential decision-making process on the candidate road edge set, and set the road construction budget, road construction status, candidate road action set, and termination conditions for the urban area to be planned.

[0012] Step 5: Encode the topology, node spatial relationships, road construction status, candidate edge attributes, and remaining budget information of the candidate road boundary graph using a graph neural network to obtain a graph representation of the road network generation status; Based on the graph representation, use the urban road network generation model to generate road construction decisions under the budget feasible deep reinforcement learning planning environment to obtain an urban road network generation scheme under budget constraints.

[0013] Step 6: After each round of road network generation, construct or update origin-destination (OD) travel demand based on the generated road network, identify reachable and unreachable OD demand, calculate demand service rate and network topology connectivity, and calculate the traffic system operating cost corresponding to reachable demand based on the traffic assignment model.

[0014] Step 7: Construct a generalized system cost for generating the road network based on the traffic system operating cost and the demand penalty for unreachable ODs, and calculate the relative resilience improvement index through the roadless benchmark and the fully constructed reference benchmark. At the same time, construct a deep reinforcement learning training reward based on the degree of violation of the demand service rate constraint and the network connectivity constraint to train the urban road network generation model, and obtain the trained urban road network generation model.

[0015] Step 8: Based on the trained urban road network generation model, output an urban road network generation scheme that meets the demand service capacity constraint and network connectivity constraint under the given construction budget.

[0016] Furthermore, the specific content of step 1 is as follows: Obtain the set of urban land use units, plot boundary lines, spatial adjacency relationships between plots, node coordinates, candidate road endpoints, and basic traffic data for the planning area; construct a candidate road boundary map by using the public boundaries, navigable boundaries, or planned permitted construction boundaries between land use units as candidate road edges, and the intersections of plot boundaries, road endpoints, or planned control points as candidate nodes, as shown in the following formula.

[0017]

[0018] In the formula, Represents the set of candidate nodes. Represents the set of candidate road edges; for each candidate road edge Set road construction state variables , among which when When it indicates that the candidate roadside is being constructed as a road, The time indicates that the candidate roadside has not been constructed; based on the candidate road boundary map, road construction status and candidate road attributes, a city road network generation state is formed for deep reinforcement learning decision-making.

[0019] Furthermore, the budget hard constraint and action feasibility mask are specifically as follows:

[0020] Assume the total construction budget is , No. The cumulative construction cost before the step was Candidate road actions The corresponding incremental construction cost is Then the candidate action satisfies the following formula

[0021]

[0022] A candidate action is deemed budget-feasible. If a candidate action meets any of the following conditions, it is masked using an action feasibility mask: the candidate road action causes the cumulative construction cost to exceed the total construction budget; the candidate roadside has already been constructed; the candidate roadside does not meet the planning generation rules; the candidate road action does not meet the topological feasibility requirements of the current road network state; the agent's policy network samples or selects only from the set of unmasked candidate actions, thereby ensuring that the road network generation process always meets the construction budget hard constraint; the construction cost directly limits the action feasibility through the budget hard constraint and is not used as a linear penalty term in the training reward function.

[0023] Furthermore, the process of generating the road network using the urban road network generation model is specifically modeled as a sequential decision-making process on the candidate road edge set:

[0024] The urban road network generation process is represented as a Markov decision process, whose states include a candidate road boundary map, a set of constructed roads, a set of unconstructed candidate roads, candidate node characteristics, candidate edge characteristics, cumulative construction cost, and remaining construction budget; its action is represented as selecting one or more candidate road edges from the set of candidate road edges for construction, where the road construction action is represented by the following formula.

[0025]

[0026] In the formula, It represents road construction activities; It is the selected candidate roadside; It is the set of candidate roadside locations;

[0027] Alternatively, it can be represented as a combination of actions involving the candidate roadside and its road class, as shown in the following formula.

[0028]

[0029] In the formula, It refers to the road classification.

[0030] Furthermore, the specific content of step 5 is as follows:

[0031] A graph neural network is used to encode the candidate road boundary graph. The input of the graph neural network includes candidate node features, candidate road edge features, road construction status, node spatial location, candidate road length, road connection relationship, remaining budget ratio, and action mask. The neighborhood information of candidate nodes and candidate edges is aggregated through the message passing mechanism of the graph neural network to obtain node embedding, edge embedding, and graph embedding. The node embedding, edge embedding, and graph embedding are input into a reinforcement learning policy network to obtain the selection probability or action score of each candidate road action. The graph embedding is input into a value network to estimate the state value of the current road network generation state. A deep reinforcement learning algorithm based on policy gradient is used to train the policy network and the value network, enabling the agent to learn a decision-making strategy for selecting candidate road edges and generating urban road networks under budget constraints.

[0032] Furthermore, the calculation of the demand service rate, network topology connectivity, and transportation system operating cost is as follows:

[0033] Generated road network Construct OD demand set Each OD pair Have corresponding travel needs Based on the topological connectivity of the generated road network, identify the set of reachable ODs. and unreachable OD sets The demand service rate is shown in the following formula.

[0034]

[0035] The demand service rate is used to represent the proportion of travel demand that the currently generated road network can serve.

[0036] Network topology connectivity It is obtained by weighting the scores of connected components, road density, and dead ends in the generated road network, and the specific calculation formula is as follows:

[0037]

[0038] in, Score the connected components. To score road density, Score the dead end node; , and These are the weights of the three types of topological indicators.

[0039] The connectivity component score is used to measure whether the generated road network is segmented into multiple disconnected parts. Let... Let represent the number of connected components in the generated road network, then:

[0040]

[0041] When the entire road network has only one connected component ,therefore When a road network is segmented into multiple connected components, The value decreases as the number of connected components increases.

[0042] The road density score is used to measure whether the generated road network has sufficient road service density within a community. Let... This represents the total length of the roads that generated the road network. Indicates the area of ​​the community. Let represent the road density normalization parameter, then:

[0043]

[0044] in, Convert road length to kilometers Converting the community area to square kilometers, the unit for road density is... . Used to normalize road density; when the generated road density reaches or exceeds the reference density... Take 1.

[0045] Dead-end node scores are used to penalize excessive dead ends or nodes with a degree of 1 in a road network. Let... Let represent the number of dead ends in the generated road network, then:

[0046]

[0047] When there are no dead ends in the network ,therefore As the number of dead nodes increases, A gradual decrease indicates a decline in the connectivity of local network structures.

[0048] For reachable OD demand, the link traffic and travel time in the road network are calculated based on the traffic assignment model, and the total system travel time or traffic system operating cost corresponding to the reachable demand is obtained; for unreachable OD demand, an unserved demand penalty is introduced to reflect the system loss caused by the road network's inability to serve some travel demand.

[0049] Furthermore, the specific content of the relative toughness improvement index obtained in step 7 is as follows:

[0050] Suppose we generate a road network The generalized system cost is shown in the following formula.

[0051]

[0052] In the formula, This represents the total system travel time of reachable OD demand on the generated road network. This indicates that the total OD demand cannot be reached. This represents the penalty coefficient for unserved demand;

[0053] Assume the cost of the no-road reference system is as shown in the following formula.

[0054]

[0055] In the absence of roads, all origin-destination (OD) demands are unattainable; a fully constructed reference road network is then constructed. The fully constructed reference road network refers to the reference network after all candidate road edges are constructed to the highest road grade or the default highest construction state. The generalized system cost is shown below.

[0056]

[0057] The fully constructed reference road network is not constrained by the construction budget and serves only as a theoretical reference for the relative resilience improvement index. Based on the roadless benchmark and the fully constructed reference benchmark, the relative resilience improvement index is defined as follows:

[0058]

[0059] In the formula, It is a very small positive number, and its value is determined according to the requirements;

[0060] Furthermore, the demand service rate constraint and network connectivity constraint in step 7 are shown in the following formula:

[0061]

[0062]

[0063] In the formula, It is the minimum demand service rate threshold. It is the minimum network connectivity threshold;

[0064] The deep reinforcement learning training reward is shown in the following formula.

[0065]

[0066]

[0067]

[0068] In the formula, and These are the penalty weights for violating the demand service rate constraint and the penalty weights for violating the network connectivity constraint; the deep reinforcement learning training reward is used to provide training signals for the continuous training of the urban road network generation model.

[0069] Furthermore, the model training process for the urban road network generation model is as follows:

[0070] Initialize the candidate road boundary map, road construction status, construction budget, policy network, value network, and graph neural network parameters. In each training round, the agent selects candidate road construction actions based on the current road network generation status and action feasibility mask. After executing the candidate road construction actions, update the road network topology, road construction status, cumulative construction cost, and remaining construction budget. When the termination condition is met, calculate the demand service rate, network topology connectivity, generalized system cost, relative resilience improvement index, and training reward of the generated road network. Based on the status, actions, rewards, and value estimates collected in the training rounds, update the policy network, value network, and graph neural network parameters using a policy optimization algorithm. Repeat the above training process until the policy converges or the preset number of training rounds is reached to obtain the trained urban road network generation model.

[0071] To solve the above-mentioned technical problems, the second technical solution proposed by this invention is: an intelligent generation system for urban road networks based on deep reinforcement learning, which carries the above-mentioned first technical solution, comprising:

[0072] The candidate road boundary map construction module is used to acquire urban land use units, plot boundaries, spatial adjacency relationships, and basic traffic data to construct candidate road boundary maps.

[0073] The deep reinforcement learning environment construction module is used to establish a sequential decision-making environment for urban road network generation, setting the road construction status, candidate road action set, construction budget, and termination conditions.

[0074] The budget constraint and action mask module is used to generate an action feasibility mask based on the remaining construction budget and road generation rules, and to mask candidate road actions that exceed the budget or do not meet the planning rules.

[0075] The graph neural network encoding module is used to encode the topology, spatial relationships of nodes, edge attributes of candidate roads, road construction status, and remaining budget information of the candidate road boundary graph.

[0076] The road generation decision module is used to output candidate road construction actions based on the reinforcement learning policy network and update the road network generation state.

[0077] The traffic demand and service capacity assessment module is used to construct or update OD demand, identify reachable and unreachable OD demand, and calculate demand service rate and network topology connectivity.

[0078] The traffic system cost calculation module is used to calculate the operating cost of the traffic system based on the reachable demand based on the traffic assignment model, and to calculate the unserved demand penalty based on the unreachable OD demand.

[0079] The resilience reward calculation module is used to calculate training rewards based on the no-path baseline, the full-construction reference baseline, the relative resilience improvement index, the demand service rate violation, and the network connectivity violation.

[0080] The planning scheme output module is used to output urban road network planning schemes that meet budget constraints, demand service capacity constraints, and network connectivity constraints based on the trained road generation strategy.

[0081] The present invention has the following beneficial effects:

[0082] 1. This invention models the urban road network generation process as a sequential decision problem on a set of candidate road edges, which can transform the complex urban road planning process into a graph decision problem that can be solved by deep reinforcement learning, thereby reducing the reliance on human experience and fixed road network templates.

[0083] 2. This invention constructs candidate road boundary maps based on land use units, plot boundaries, and spatial adjacency relationships, enabling the generated road network to be coordinated with the urban spatial structure. It is applicable to scenarios such as new area road planning, area road completion, and low-level road network generation.

[0084] 3. This invention introduces hard budget constraints and action feasibility masks, directly embedding the construction budget into the road generation process. This ensures that the agent can only choose road construction actions that are budget-feasible, thereby guaranteeing that the generated solution meets the construction resource constraints and avoiding the problem of infeasible solutions caused by simply controlling costs through rewards and penalties.

[0085] 4. This invention constrains the final road network scheme by combining demand service rate and network topology connectivity, so that the generated result not only has a certain spatial connectivity skeleton, but can also serve a sufficient proportion of OD travel demand.

[0086] 5. This invention constructs a generalized system cost based on the operating cost of the transportation system and the penalty for unreachable demand, and constructs a relative resilience improvement index through a roadless benchmark and a fully constructed reference benchmark. This can avoid the problem that the reward approaches the upper limit too quickly when normalized only by a roadless benchmark, and improve the ability to distinguish between different budget schemes and different road generation schemes.

[0087] 6. This invention utilizes graph neural networks to perform representation learning on candidate road boundary maps and combines deep reinforcement learning strategy networks to generate road construction decisions, which can effectively handle the topological features, spatial adjacency relationships and long-term sequential decision influences in road network generation. Attached Figure Description

[0088] Figure 1 This is a schematic diagram of the overall process of an intelligent urban road network generation method based on deep reinforcement learning in one embodiment.

[0089] Figure 2 This is a schematic diagram of a road construction decision-making framework based on graph neural networks and deep reinforcement learning in one embodiment.

[0090] Figure 3 This is a diagram of a model architecture based on graph neural networks and deep reinforcement learning in one embodiment. Detailed Implementation

[0091] The present invention will now be described in further detail with reference to the accompanying drawings and specific preferred embodiments.

[0092] In the description of this invention, it should be understood that the terms "left side," "right side," "upper part," "lower part," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. "First," "second," etc., do not indicate the importance of the components, and therefore should not be construed as a limitation of this invention. The specific dimensions used in this embodiment are only for illustrating the technical solution and do not limit the scope of protection of this invention.

[0093] To overcome the problems of high reliance on manual labor, difficulty in handling complex land use boundaries, lack of hard budget constraints, insufficient feedback on demand service capacity, and inadequate discrimination in resilience evaluation in existing urban road network planning methods, this invention proposes an intelligent generation method for urban road networks based on deep reinforcement learning. This method represents the potential construction space of urban roads through candidate road boundary maps, ensures that the road construction process meets resource constraints through hard budget constraints and action feasibility masks, learns the topological and spatial structure features of candidate roads through graph neural networks, gradually generates the road network through a deep reinforcement learning policy network, and evaluates the final solution through demand service rate, network connectivity, transportation system cost, and relative resilience improvement indicators.

[0094] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0095] In one exemplary embodiment, such as Figure 1 As shown, an intelligent generation method for urban road networks based on deep reinforcement learning includes the following steps:

[0096] Step 1: Establish a candidate road boundary map for the urban area to be planned and represent the urban road network generation object as a graph structure composed of candidate nodes and candidate road edges;

[0097] The specific content is as follows: Obtain the set of urban land use units, plot boundary lines, spatial adjacency relationships between plots, node coordinates, candidate road endpoints, and basic traffic data for the planning area; use the public boundaries, navigable boundaries, or planned construction-permitted boundaries between land use units as candidate road edges, and the intersections of plot boundaries, road endpoints, or planning control points as candidate nodes to construct a candidate road boundary map, as shown in the following formula.

[0098]

[0099] In the formula, Represents the set of candidate nodes. Represents the set of candidate road edges; for each candidate road edge Set road construction state variables , among which when When it indicates that the candidate roadside is being constructed as a road, The time indicates that the candidate roadside has not been constructed; based on the candidate road boundary map, road construction status and candidate road attributes, a city road network generation state is formed for deep reinforcement learning decision-making.

[0100] Step 2: Establish an urban road network generation model based on a deep reinforcement learning policy network and determine the corresponding deep reinforcement learning planning environment;

[0101] Step 3: Add budget hard constraints and action feasibility masks to the deep reinforcement learning planning environment to form a budget-feasible deep reinforcement learning planning environment;

[0102] The specific budget hard constraints and action feasibility masks are as follows:

[0103] Assume the total construction budget is , No. The cumulative construction cost before the step was Candidate road actions The corresponding incremental construction cost is Then the candidate action satisfies the following formula

[0104]

[0105] A candidate action is deemed budget-feasible. If a candidate action meets any of the following conditions, it is masked using an action feasibility mask: the candidate road action causes the cumulative construction cost to exceed the total construction budget; the candidate roadside has already been constructed; the candidate roadside does not meet the planning generation rules; the candidate road action does not meet the topological feasibility requirements of the current road network state; the agent's policy network samples or selects only from the set of unmasked candidate actions, thereby ensuring that the road network generation process always meets the construction budget hard constraint; the construction cost directly limits the action feasibility through the budget hard constraint and is not used as a linear penalty term in the training reward function.

[0106] Step 4: Model the process of generating the road network in the urban road network generation model as a sequential decision-making process on the candidate road edge set, and set the road construction budget, road construction status, candidate road action set and termination conditions for the urban area to be planned.

[0107] The process of generating a road network using the urban road network generation model is specifically modeled as a sequential decision-making process on a set of candidate road edges:

[0108] The urban road network generation process is represented as a Markov decision process, whose states include a candidate road boundary map, a set of constructed roads, a set of unconstructed candidate roads, candidate node characteristics, candidate edge characteristics, cumulative construction cost, and remaining construction budget; its action is represented as selecting one or more candidate road edges from the set of candidate road edges for construction, where the road construction action is represented by the following formula.

[0109]

[0110] In the formula, It represents road construction activities; It is the selected candidate roadside; It is the set of candidate roadside locations;

[0111] Alternatively, it can be represented as a combination of actions involving the candidate roadside and its road class, as shown in the following formula.

[0112]

[0113] In the formula, The road level is used; in the current basic implementation, the road construction decision adopts a binary edge selection method, that is, the agent only judges whether to build the candidate road edge; in the extended implementation, the road construction decision further includes the selection of road level or capacity level; after each road construction action is executed, the construction status of the corresponding candidate road edge, the road network topology, the cumulative construction cost and the remaining construction budget are updated until the maximum number of planning steps is reached, the budget is exhausted or there are no more actionable actions.

[0114] Step 5: Encode the topology, node spatial relationships, road construction status, candidate edge attributes, and remaining budget information of the candidate road boundary graph using a graph neural network to obtain a graph representation of the road network generation status; Based on the graph representation, use the urban road network generation model to generate road construction decisions in a budget-feasible deep reinforcement learning planning environment to obtain an urban road network generation scheme under budget constraints.

[0115] The specific details of step 5 are as follows:

[0116] A graph neural network is used to encode the candidate road boundary graph. The input of the graph neural network includes candidate node features, candidate road edge features, road construction status, node spatial location, candidate road length, road connection relationship, remaining budget ratio, and action mask. The neighborhood information of candidate nodes and candidate edges is aggregated through the message passing mechanism of the graph neural network to obtain node embedding, edge embedding, and global graph embedding. The node embedding, edge embedding, and global graph embedding are input into a reinforcement learning policy network to obtain the selection probability or action score of each candidate road action. The graph embedding is input into a value network to estimate the state value of the current road network generation state. A deep reinforcement learning algorithm based on policy gradient is used to train the policy network and the value network, enabling the agent to learn a decision-making strategy for selecting candidate road edges and generating urban road networks under budget constraints.

[0117] Step 6: After each round of road network generation, construct or update origin-destination (OD) travel demand based on the generated road network, identify reachable and unreachable OD demand, calculate demand service rate and network topology connectivity, and calculate the traffic system operating cost corresponding to reachable demand based on the traffic assignment model.

[0118] Step 7: Construct the generalized system cost of the generated road network based on the traffic system operation cost and the demand penalty for unreachable ODs, and calculate the relative resilience improvement index through the roadless benchmark and the fully constructed reference benchmark. At the same time, construct the deep reinforcement learning training reward by combining the degree of violation of the demand service rate constraint and the network connectivity constraint to train the urban road network generation model, and obtain the trained urban road network generation model.

[0119] The calculation of the demand service rate, network topology connectivity, and transportation system operating costs is as follows:

[0120] Generated road network Construct OD demand set Each OD pair Have corresponding travel needs Based on the topological connectivity of the generated road network, identify the set of reachable ODs. and unreachable OD sets The demand service rate is shown in the following formula.

[0121]

[0122] The demand service rate is used to represent the proportion of travel demand that the currently generated road network can serve; network topology connectivity. It is obtained by weighting the scores of connected components, road density, and dead ends in the generated road network, and the specific calculation formula is as follows:

[0123]

[0124] in, Score the connected components. To score road density, Score the dead end node; , and These are the weights of the three types of topological indicators.

[0125] The connectivity component score is used to measure whether the generated road network is segmented into multiple disconnected parts. Let... Let represent the number of connected components in the generated road network, then:

[0126]

[0127] When the entire road network has only one connected component ,therefore When a road network is segmented into multiple connected components, The value decreases as the number of connected components increases.

[0128] The road density score is used to measure whether the generated road network has sufficient road service density within a community. Let... This represents the total length of the roads that generated the road network. Indicates the area of ​​the community. Let represent the road density normalization parameter, then:

[0129]

[0130] in, Convert road length to kilometers Converting the community area to square kilometers, the unit for road density is... . Used to normalize road density; when the generated road density reaches or exceeds the reference density... Take 1.

[0131] Dead-end node scores are used to penalize excessive dead ends or nodes with a degree of 1 in a road network. Let... Let represent the number of dead ends in the generated road network, then:

[0132]

[0133] When there are no dead ends in the network ,therefore As the number of dead nodes increases, A gradual decrease indicates a decline in the connectivity of local network structures.

[0134] For reachable OD demand, the link traffic and travel time in the road network are calculated based on the traffic assignment model, and the total system travel time or traffic system operating cost corresponding to the reachable demand is obtained; for unreachable OD demand, an unserved demand penalty is introduced to reflect the system loss caused by the road network's inability to serve some travel demand.

[0135] The specific details of the relative toughness improvement index are as follows:

[0136] Suppose we generate a road network The generalized system cost is shown in the following formula.

[0137]

[0138] In the formula, This represents the total system travel time of reachable OD demand on the generated road network. This indicates that the total OD demand cannot be reached. This represents the penalty coefficient for unserved demand;

[0139] Assume the cost of the no-road reference system is as shown in the following formula.

[0140]

[0141] In the absence of roads, all origin-destination (OD) demands are unattainable; a fully constructed reference road network is then constructed. The fully constructed reference road network refers to the reference network after all candidate road edges are constructed to the highest road grade or the default highest construction state. The generalized system cost is shown below.

[0142]

[0143] The fully constructed reference road network is not constrained by the construction budget and serves only as a theoretical reference for the relative resilience improvement index. Based on the roadless benchmark and the fully constructed reference benchmark, the relative resilience improvement index is defined as follows:

[0144]

[0145] In the formula, It is a very small positive number, and its value is determined according to the requirements;

[0146] The demand service rate constraint and network connectivity constraint are shown in the following equation.

[0147]

[0148]

[0149] In the formula, It is the minimum demand service rate threshold. It is the minimum network connectivity threshold;

[0150] The deep reinforcement learning training reward is shown in the following formula.

[0151]

[0152]

[0153]

[0154] In the formula, and These are the penalty weights for violating the demand service rate constraint and the penalty weights for violating the network connectivity constraint; the deep reinforcement learning training reward is used to provide training signals for the continuous training of the urban road network generation model.

[0155] The training process for the urban road network generation model is as follows:

[0156] Initialize the candidate road boundary map, road construction status, construction budget, policy network, value network, and graph neural network parameters. In each training round, the agent selects candidate road construction actions based on the current road network generation status and action feasibility mask. After executing the candidate road construction actions, update the road network topology, road construction status, cumulative construction cost, and remaining construction budget. When the termination condition is met, calculate the demand service rate, network topology connectivity, generalized system cost, relative resilience improvement index, and training reward of the generated road network. Based on the status, actions, rewards, and value estimates collected in the training rounds, update the policy network, value network, and graph neural network parameters using a policy optimization algorithm. Repeat the above training process until the policy converges or the preset number of training rounds is reached to obtain the trained urban road network generation model.

[0157] Step 8: Based on the trained urban road network generation model, and given a construction budget, the trained policy network progressively selects candidate road construction actions to generate an urban road network scheme. The output includes the set of constructed roads, road network topology, cumulative construction cost, demand service rate, network connectivity, generalized system cost, and relative resilience improvement value. If the generated scheme satisfies the demand service rate constraint and the network connectivity constraint, then the scheme is determined to be a feasible road network planning scheme.

[0158] In one embodiment, the deep reinforcement learning algorithm may employ a proximal policy optimization algorithm. During training, the agent repeatedly performs road generation actions in multiple training rounds, collecting state, action, reward, and value estimation information, and updating the parameters of the graph neural network, policy network, and value network according to the policy optimization objective. Through multiple rounds of training, the agent gradually learns to generate road network schemes with high demand service capacity, network connectivity, and traffic resilience under different budget conditions and candidate road graph structures.

[0159] In one embodiment, the road construction cost can be determined based on the candidate road length, road grade, construction type, or unit construction cost. In the basic implementation, a uniform construction cost can be set for each candidate road or a construction cost can be set according to length; in the extended implementation, different road grades correspond to different construction costs, capacities, and free-flow times. Regardless of the cost form used, the construction cost controls the feasibility of actions through budget hard constraints, rather than being used as a linear penalty term in the training reward function.

[0160] In one embodiment, the traffic assignment model may be a traffic assignment model based on the BPR function. For reachable OD demand on the generated road network, link travel time is calculated based on road capacity, free-flow time, and traffic volume, and then the total system travel time corresponding to the reachable demand is calculated. Unreachable OD demand does not participate in traffic assignment, but is included in the generalized system cost through unserved demand penalties.

[0161] In one embodiment, the fully-built reference network is used only to construct a relative resilience improvement index and is not constrained by the construction budget. This setting provides a theoretically best reference state, allowing the training reward to reflect the proportion of improvement achieved relative to the "no-path to fully-built" range under the current budget, thereby avoiding the problem of the generator network's reward value approaching its upper limit prematurely after serving most of the demand.

[0162] Based on the same inventive concept, this application also provides an intelligent generation system for urban road networks based on deep reinforcement learning to implement the above-described method. The solution provided by this system is similar to the solution described in the above-described method; therefore, the specific limitations in one or more system embodiments provided below can be found in the limitations of the method embodiments described above, and will not be repeated here.

[0163] In one exemplary embodiment, an intelligent generation system for urban road networks based on deep reinforcement learning is provided, comprising:

[0164] The candidate road boundary map construction module is used to acquire urban land use units, plot boundaries, spatial adjacency relationships, and basic traffic data to construct candidate road boundary maps.

[0165] The deep reinforcement learning environment construction module is used to establish a sequential decision-making environment for urban road network generation, setting the road construction status, candidate road action set, construction budget, and termination conditions.

[0166] The budget constraint and action mask module is used to generate an action feasibility mask based on the remaining construction budget and road generation rules, and to mask candidate road actions that exceed the budget or do not meet the planning rules.

[0167] The graph neural network encoding module is used to encode the topology, spatial relationships of nodes, edge attributes of candidate roads, road construction status, and remaining budget information of the candidate road boundary graph.

[0168] The road generation decision module is used to output candidate road construction actions based on the reinforcement learning policy network and update the road network generation state.

[0169] The traffic demand and service capacity assessment module is used to construct or update OD demand, identify reachable and unreachable OD demand, and calculate demand service rate and network topology connectivity.

[0170] The traffic system cost calculation module is used to calculate the operating cost of the traffic system based on the reachable demand based on the traffic assignment model, and to calculate the unserved demand penalty based on the unreachable OD demand.

[0171] The resilience reward calculation module is used to calculate training rewards based on the no-path baseline, the full-construction reference baseline, the relative resilience improvement index, the demand service rate violation, and the network connectivity violation.

[0172] The planning scheme output module is used to output urban road network planning schemes that meet budget constraints, demand service capacity constraints, and network connectivity constraints based on the trained road generation strategy.

[0173] The modules in the aforementioned intelligent generation system for urban road networks based on deep reinforcement learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0174] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. An intelligent generation method for urban road networks based on deep reinforcement learning, characterized in that: Includes the following steps: Step 1: Establish a candidate road boundary map for the urban area to be planned and represent the urban road network generation object as a graph structure composed of candidate nodes and candidate road edges; Step 2: Establish an urban road network generation model based on a deep reinforcement learning policy network and determine the corresponding deep reinforcement learning planning environment; Step 3: Add budget hard constraints and action feasibility masks to the deep reinforcement learning planning environment to form a budget-feasible deep reinforcement learning planning environment; Step 4: Model the process of generating the road network using the urban road network generation model as a sequential decision-making process on the candidate road edge set, and set the road construction budget, road construction status, candidate road action set, and termination conditions for the urban area to be planned. Step 5: Encode the topology, node spatial relationships, road construction status, candidate edge attributes, and remaining budget information of the candidate road boundary graph using a graph neural network to obtain a graph representation of the road network generation status; Based on the graph representation, use the urban road network generation model to generate road construction decisions under the budget feasible deep reinforcement learning planning environment to obtain an urban road network generation scheme under budget constraints. Step 6: After each round of road network generation, construct or update origin-destination (OD) travel demand based on the generated road network, identify reachable and unreachable OD demand, calculate demand service rate and network topology connectivity, and calculate the traffic system operating cost corresponding to reachable demand based on the traffic assignment model. Step 7: Construct a generalized system cost for generating the road network based on the traffic system operating cost and the demand penalty for unreachable ODs, and calculate the relative resilience improvement index through the roadless benchmark and the fully constructed reference benchmark. At the same time, construct a deep reinforcement learning training reward based on the degree of violation of the demand service rate constraint and the network connectivity constraint to train the urban road network generation model, and obtain the trained urban road network generation model. Step 8: Based on the trained urban road network generation model, output an urban road network generation scheme that meets the demand service capacity constraint and network connectivity constraint under the given construction budget.

2. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 1, characterized in that: Step 1 involves the following steps: Obtaining the urban land use unit set, plot boundary lines, spatial adjacency relationships between plots, node coordinates, candidate road endpoints, and basic traffic data for the planning area; using the public boundaries, navigable boundaries, or planned permitted construction boundaries between land use units as candidate road edges, and the intersections of plot boundaries, road endpoints, or planned control points as candidate nodes, constructing a candidate road boundary map as shown in the following formula. ； In the formula, Represents the set of candidate nodes. Represents the set of candidate road edges; for each candidate road edge Set road construction status variables , among which when When it indicates that the candidate roadside is being constructed as a road, The time indicates that the candidate roadside has not been constructed; based on the candidate road boundary map, road construction status and candidate road attributes, a city road network generation state is formed for deep reinforcement learning decision-making.

3. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 1, characterized in that: The specific budget hard constraints and action feasibility masks are as follows: Assume the total construction budget is , No. The cumulative construction cost before the step was Candidate road actions The corresponding incremental construction cost is Then the candidate action satisfies the following formula ； A candidate action is deemed budget-feasible; if a candidate action meets any of the following conditions, it is masked by an action feasibility mask: the candidate road action causes the cumulative construction cost to exceed the total construction budget; Construction has already begun along the candidate road; The candidate roadside does not meet the planning generation rules; Candidate road actions do not meet the topological feasibility requirements of the current road network state; the agent's policy network samples or selects only from the set of unmasked candidate actions, thereby ensuring that the road network generation process always meets the hard constraint of construction budget; construction cost directly limits action feasibility through the hard constraint of budget and is not used as a linear penalty term in the training reward function.

4. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 1, characterized in that: The process of generating a road network using the urban road network generation model is specifically modeled as a sequential decision-making process on a set of candidate road edges: The urban road network generation process is represented as a Markov decision process, whose states include a candidate road boundary map, a set of constructed roads, a set of unconstructed candidate roads, candidate node characteristics, candidate edge characteristics, cumulative construction cost, and remaining construction budget; its action is represented as selecting one or more candidate road edges from the candidate road edge set for construction, where the road construction action is represented by the following formula. ； In the formula, It represents road construction activities; It is the selected candidate roadside; It is the set of candidate roadside locations; Alternatively, it can be represented as a combination of actions involving the candidate roadside and its road class, as shown in the following formula. ； In the formula, It refers to the road classification.

5. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 1, characterized in that: The specific details of step 5 are as follows: A graph neural network is used to encode the candidate road boundary graph. The input of the graph neural network includes candidate node features, candidate road edge features, road construction status, node spatial location, candidate road length, road connection relationship, remaining budget ratio, and action mask. The neighborhood information of candidate nodes and candidate edges is aggregated through the message passing mechanism of the graph neural network to obtain node embedding, edge embedding, and graph embedding. The node embedding, edge embedding, and graph embedding are input into a reinforcement learning policy network to obtain the selection probability or action score of each candidate road action. The graph embedding is input into a value network to estimate the state value of the current road network generation state. A deep reinforcement learning algorithm based on policy gradient is used to train the policy network and the value network, enabling the agent to learn a decision-making strategy for selecting candidate road edges and generating an urban road network under budget constraints.

6. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 1, characterized in that: The calculation of the demand service rate, network topology connectivity, and transportation system operating costs is as follows: Generated road network Construct OD demand set Each OD pair Have corresponding travel needs Based on the topological connectivity of the generated road network, identify the set of reachable ODs. and unreachable OD sets The demand service rate is shown in the following formula. ； The demand service rate is used to represent the proportion of travel demand that the currently generated road network can serve. The network topology connectivity is shown in the following equation. ； ； ； ； in, Score the connected components. To score road density, Score the dead end node; , and These are the weights of the three types of topological indicators; This indicates the number of connected components in the generated road network; This represents the total length of the roads that generated the road network. Indicates the area of ​​the community. This represents the normalized parameter for road density; This indicates the number of dead ends in the generated road network; For reachable OD demand, the link traffic and travel time in the road network are calculated based on the traffic assignment model, and the total system travel time or traffic system operating cost corresponding to the reachable demand is obtained; for unreachable OD demand, an unserved demand penalty is introduced to reflect the system loss caused by the road network's inability to serve some travel demand.

7. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 6, characterized in that: The specific details of the relative toughness improvement index obtained in step 7 are as follows: Suppose we generate a road network The generalized system cost is shown in the following formula. ； In the formula, This represents the total system travel time of reachable OD demand on the generated road network. This indicates that the total OD demand cannot be reached. This represents the penalty coefficient for unserved demand; Assume the cost of the no-road reference system is as shown in the following formula. ； In the absence of roads, all origin-destination (OD) demands are unattainable; a fully constructed reference road network is then constructed. The fully constructed reference road network refers to the reference network after all candidate road edges are constructed to the highest road grade or the default highest construction state. The generalized system cost is shown below. ； The fully constructed reference road network is not constrained by the construction budget and serves only as a theoretical reference for the relative resilience improvement index. Based on the roadless benchmark and the fully constructed reference benchmark, the relative resilience improvement index is defined as follows: ； In the formula, It is a very small positive number, and its value is determined according to the requirements.

8. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 7, characterized in that: The demand service rate constraint and network connectivity constraint in step 7 are shown in the following formula. ； ； In the formula, It is the minimum demand service rate threshold. It is the minimum network connectivity threshold; The deep reinforcement learning training reward is shown in the following formula. ； ； ； In the formula, and These are the penalty weights for violating the demand service rate constraint and the penalty weights for violating the network connectivity constraint; the deep reinforcement learning training reward is used to provide training signals for the continuous training of the urban road network generation model.

9. The intelligent generation method for urban road networks based on deep reinforcement learning according to claim 8, characterized in that: The training process for the urban road network generation model is as follows: Initialize the candidate road boundary map, road construction status, construction budget, policy network, value network, and graph neural network parameters. In each training round, the agent selects candidate road construction actions based on the current road network generation status and action feasibility mask. After executing the candidate road construction actions, update the road network topology, road construction status, cumulative construction cost, and remaining construction budget. When the termination condition is met, calculate the demand service rate, network topology connectivity, generalized system cost, relative resilience improvement index, and training reward of the generated road network. Based on the status, actions, rewards, and value estimates collected in the training rounds, update the policy network, value network, and graph neural network parameters using a policy optimization algorithm. Repeat the above training process until the policy converges or the preset number of training rounds is reached to obtain the trained urban road network generation model.

10. An intelligent generation system carrying the intelligent generation method according to any one of claims 1-9, characterized in that, include: The candidate road boundary map construction module is used to acquire urban land use units, plot boundaries, spatial adjacency relationships, and basic traffic data to construct candidate road boundary maps. The deep reinforcement learning environment construction module is used to establish a sequential decision-making environment for urban road network generation, setting the road construction status, candidate road action set, construction budget, and termination conditions. The budget constraint and action mask module is used to generate an action feasibility mask based on the remaining construction budget and road generation rules, and to mask candidate road actions that exceed the budget or do not meet the planning rules. The graph neural network encoding module is used to encode the topology, spatial relationships of nodes, edge attributes of candidate roads, road construction status, and remaining budget information of the candidate road boundary graph. The road generation decision module is used to output candidate road construction actions based on the reinforcement learning policy network and update the road network generation state. The traffic demand and service capacity assessment module is used to construct or update OD demand, identify reachable and unreachable OD demand, and calculate demand service rate and network topology connectivity. The traffic system cost calculation module is used to calculate the operating cost of the traffic system based on the reachable demand based on the traffic assignment model, and to calculate the unserved demand penalty based on the unreachable OD demand. The resilience reward calculation module is used to calculate training rewards based on the no-path baseline, the full-construction reference baseline, the relative resilience improvement index, the demand service rate violation, and the network connectivity violation. The planning scheme output module is used to output urban road network planning schemes that meet budget constraints, demand service capacity constraints, and network connectivity constraints based on the trained road generation strategy.

Images