Road rain and sewage pipe network path planning method based on reinforcement learning

Abstract

The invention relates to the technical field of road rain and sewage pipe network path planning, in particular to a road rain and sewage pipe network path planning method based on reinforcement learning. The method comprises the following steps: firstly, based on acquired multi-source heterogeneous data, carrying out preprocessing to obtain a unified space system; based on the unified space system, obtaining state attributes in the unified space system; based on attributes of the states, obtaining a state transition legality function, and judging whether the two states are reachable or not; on the reachable basis, calculating a transfer weight based on a state transfer legality function; then, based on the transfer weight, determining a strategy candidate set; constructing an instant reward function based on the strategy candidate set; on the basis of an instant reward function, a state value is obtained through a dynamic self-regularization Bellman updating formula, then a next state is determined, and finally a path sequence is formed. The technical problems that path hopping is severe, a decision cannot be explained, and control over path continuity and environmental adaptability is lacked are solved.

Description

Technical Field

[0001] This invention relates to the field of road stormwater and sewage pipe network route planning technology, and in particular to a road stormwater and sewage pipe network route planning method based on reinforcement learning. Background Technology

[0002] With the accelerating pace of urbanization and the continuous advancement of infrastructure construction, the scientific planning and rational layout of stormwater and sewage separation pipe networks have become crucial for ensuring urban operational safety and improving environmental quality. Currently, in actual engineering projects, pipe network route planning mostly relies on manual experience and two-dimensional drawing review. The planning process lacks systematic integration of multi-source spatial information such as terrain slope, underground geology, existing pipelines, and planned land use. This results in drainage routes that often have problems such as incompatibility with the terrain, crossing restricted construction areas, inaccessibility during construction, or insufficient drainage efficiency, seriously affecting the implementation and operational efficiency of the plan.

[0003] Traditional path planning methods, such as Dijkstra's algorithm and A* algorithm, can achieve shortest path search on ideal maps, but they cannot model multi-dimensional constraints in space, such as building interference, geological penetration, and drainage directionality, nor can they incorporate topographic flow direction for natural drainage simulation. Most traditional path planning methods employ neural network black-box models, resulting in difficult-to-interpret parameters and reproducible paths, failing to meet the design requirements of "controllable paths, interpretable processes, and adjustable constraints" in engineering projects. Furthermore, existing methods lack the ability to handle issues such as path stability, building density disturbances, and drainage offsets, easily leading to path jumps, convergence difficulties, or planning failures in complex urban environments. Therefore, there is an urgent need to propose a road stormwater and sewage pipe network path planning method based on reinforcement learning to address these problems. Summary of the Invention

[0004] This invention provides a road stormwater and sewage pipe network path planning method based on reinforcement learning to solve the technical problems of drastic path jumps, uninterpretable decisions, and lack of control over path continuity and environmental adaptability.

[0005] The present invention provides a road stormwater and sewage pipe network path planning method based on reinforcement learning, comprising the following steps: S1. Based on the acquired multi-source heterogeneous data, preprocessing is performed to obtain a unified spatial system; based on the unified spatial system, the attributes of the states in the unified spatial system are obtained; based on the attributes of the states, a state transition validity function is obtained to determine whether two states are reachable; based on the reachability, the transition weight is calculated based on the state transition validity function. S2. Based on the transition weights, determine the policy candidate set; based on the policy candidate set, construct the immediate reward function; based on the immediate reward function, obtain the state value through the dynamic self-regularized Bellman update formula, then determine the next state, and finally form a path sequence.

[0006] Preferably, S1 specifically includes: Based on the attributes of the state, a state transition validity function is constructed using a Boolean indicator function; when the value of the state transition validity function is 1, the two states are reachable.

[0007] Preferably, S1 specifically includes: Based on reachability, a state transition function is constructed using the geographic coordinates and state transition validity function in the state attributes, and the transition weights are calculated.

[0008] Preferably, S2 specifically includes: Based on the transition weights, states with transition weights greater than a set dynamic threshold are retained to form a policy candidate set.

[0009] Preferably, S2 specifically includes: Based on the policy candidate set and state attributes, calculate the combined terms of drainage capacity and construction cost, building interference, and path offset.

[0010] Preferably, S2 specifically includes: An instant reward function is constructed based on the combination of drainage capacity and construction cost, building interference, and path offset.

[0011] Preferably, S2 specifically includes: Based on the immediate reward function and policy candidate set, a local path mean is introduced to construct a dynamic self-regularized Bellman update formula to update the state value.

[0012] Preferably, S2 specifically includes: From the adjacent states of the initial drainage starting point state, select the state that maximizes the global right-hand side of the dynamic self-regularized Bellman update formula, and include it in the path sequence.

[0013] The beneficial effects of the technical solution of the present invention are: 1. A state transition mechanism is introduced. By constructing a state transition validity function, the engineering constraints that path expansion must meet, such as elevation difference conditions, non-restricted areas, and geological constructability, are clearly defined. This effectively avoids unreasonable paths that cross buildings, viaducts, railways, or hard rock layers during path planning. The state transition mechanism not only improves the validity of generated paths but also reduces the cost of manual corrections in the subsequent engineering design stage, strictly adheres to gravity drainage and construction rules, and improves the physical rationality of the paths.

[0014] 2. A state transition function is proposed, which models the path extension intention through an exponential decay function. It can adaptively adjust the path exploration space according to the local terrain features of the path, improve learning efficiency, reduce path oscillation, and ensure the directional stability and physical continuity of the path. By designing the state transition function and the state transition legality function, the ability to evaluate path reachability and compress the policy space is enhanced.

[0015] 3. A dynamic self-regularized Bellman update formula with a path stability penalty term is proposed, which effectively suppresses path jumps and improves policy convergence and robustness. For the first time, a local path mean is introduced into the dynamic self-regularized Bellman update formula as a stability constraint, making the policy tend to extend in a continuous and stable path direction. This structural improvement effectively avoids getting trapped in local optima and improves the search ability for the global shortest or optimal path. Attached Figure Description

[0016] Figure 1 This is a flowchart of a road stormwater and sewage pipe network path planning method based on reinforcement learning, as described in this invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a road stormwater and sewage pipe network path planning method based on reinforcement learning provided by this invention.

[0020] See attached document Figure 1 The diagram illustrates a flowchart of a road stormwater and sewage pipe network path planning method based on reinforcement learning, provided by an embodiment of the present invention. The method includes the following steps: S1. Based on the acquired multi-source heterogeneous data, preprocessing is performed to obtain a unified spatial system; based on the unified spatial system, the attributes of the states in the unified spatial system are obtained; based on the attributes of the states, a state transition legality function is obtained to determine whether two states are reachable; based on the reachability, the transition weight is calculated based on the state transition legality function.

[0021] Within the target planning area, multi-source heterogeneous data is acquired. Specifically, digital elevation models of urban areas are obtained through remote sensing mapping or UAV lidar equipment to extract terrain slope and gravity drainage direction; road boundary maps and land use maps are obtained from the natural resources department to define legal construction areas and land use types; urban master plan layers are retrieved from the planning department to identify future non-traversable or reserved control areas; underground geological stratigraphic maps are obtained through field surveys, BIM systems, or geological survey reports to extract information such as soil depth, geological structure, groundwater level, lithology, soil type, and permeability parameters; existing underground pipeline distribution maps are retrieved from water, electricity, and gas pipeline units, including attributes such as pipe diameter, material, burial depth, ownership, and construction year, to avoid existing facilities; and building footprint maps are extracted using urban building databases or remote sensing images, including information such as number of floors, height, and structural type, and building interference factors are constructed using a combination of spatial buffer overlay analysis and building cover density factor methods. The spatial buffer overlay analysis method and the building cover density factor method are well-known processing methods in the field of Geographic Information System (GIS) spatial analysis, and will not be elaborated here. All acquired multi-source heterogeneous data undergoes unified preprocessing using existing GIS data processing tools, such as ESRI and ArcGIS, including coordinate reprojection, boundary clipping, resolution normalization, and layer rasterization or vectorization conversion. The data is stored in standard formats such as GeoTIFF, ESRI Shapefile, and GeoDatabase, and a three-dimensional spatial data model based on raster-vector hybrid representation is constructed. Combined with existing TIN terrain models and underground cross-sectional layered data, a unified spatial system with XYZ spatial structure and engineering attribute fields is achieved. This provides a high-precision input environment for the subsequent reinforcement learning path planning system, incorporating multi-dimensional constraints such as terrain elevation differences, construction feasibility, planning limitations, and environmental interference. Here, X and Y represent two-dimensional geographic coordinates, and Z represents elevation or depth. The preprocessing techniques used are all well-known to those skilled in the art and will not be elaborated here.

[0022] In the unified spatial system, each spatial cell is defined as a state cell and serves as a candidate drainage node. Each state cell is denoted as a state. ,in, Indicates the index of a status or spatial cell. Status The attributes include: the first Geographic coordinates of each status cell Taken from digital elevation model; Elevation value of each status cell Taken from digital elevation model; Construction cost of each status cell Based on multi-source heterogeneous data, GIS spatial overlay analysis and the weighted average cost of capital method were used to obtain the following; Construction penetration index of each status cell This method, based on geological stratigraphic maps and engineering exploration borehole data such as lithology, standard penetration test N-values, and groundwater levels, uses a quantitative scoring of geological parameters combined with spatial interpolation, such as Kriging interpolation or inverse distance weighting, to determine the geological parameters. Building interference factors in each status cell ;No. Current land use label for each status cell Terms such as "roads," "green spaces," and "restricted construction zones" are taken from land use maps and urban master plan layers; Rainwater and sewage zone labels for each status cell The data is taken from the city master plan layer. All state cells are grouped into a path state space, reflecting not only spatial coordinate relationships but also the actual construction feasibility of candidate drainage nodes and the basis for evaluating drainage efficiency. All state attributes have been normalized to eliminate dimensionless issues.

[0023] After the path state space is constructed, the state transition mechanism is defined. The technical purpose of the state transition mechanism is to abstract the path turning behavior into transition actions between states, defining a set of state pairs. Indicates from candidate drainage nodes to adjacent drainage nodes The path extension. To ensure the physical rationality of the path, the action selection must follow the basic principle of drainage, that is, follow the gravity flow from high to low, and avoid crossing geologically inaccessible areas or urban planning restricted areas. Therefore, a state transition legality function is defined. This is used to determine whether any two drainage nodes are reachable. Its value is either 0 or 1. The specific expression is: in, This is a Boolean indicator function that takes the value 1 when the internal condition is met, and 0 otherwise. Indicates the first One drainage node, i.e., the target state; Represents the target state Elevation value; This is the minimum elevation difference threshold, used to represent the required head difference that gravity drainage must meet according to actual needs; Represents the target state The land use label, i.e., the land type; This indicates the type of obstacle, such as buildings, railways, and bridges, which are set as inaccessible areas; Represents the target state The construction penetration index is used to reflect the difficulty of drilling and laying. The minimum penetration threshold for constructability is determined based on underground geological stratigraphy and using drilling layer sampling and standard penetration tests. It is only considered feasible when the target condition meets the requirements of elevation difference, non-restricted area, and geological constructability. Only then is the transfer action allowed.

[0024] Furthermore, define the state transition function. This refers to the transfer weights, which characterize the physical cost of different paths. If the target state... If it is legally reachable, then the state is... Towards the target state The formula for calculating the transfer weight is: in, The target state The geographic coordinates, that is, the geographic coordinates of the corresponding status cell; The path expansion penalty coefficient controls the strength of the suppression of path selection intention by distance. It is determined using existing Bayesian optimization methods, with a reference value range of [value missing]. ; It is a weight decay term; the greater the distance, the faster the decay, which is consistent with the rule that construction paths tend to favor short-distance connections.

[0025] S2. Based on the transition weights, determine the policy candidate set; based on the policy candidate set, construct the immediate reward function; based on the immediate reward function, obtain the state value through the dynamic self-regularized Bellman update formula, then determine the next state, and finally form a path sequence.

[0026] During path generation and reinforcement learning policy execution, the state transition function... Instead of participating in policy sampling as an explicit probabilistic model, it is embedded in the construction logic of the action set as an implicit path selection and penalty structure during value function updates and action selection. Specifically, in the action selection phase of each step, the reinforcement learning agent does not directly consider all adjacent states. Instead of performing an unconditional traversal, it first determines the traversal based on the transition weights. All adjacent states are filtered and sorted, and only those with transition weights greater than the dynamic threshold are retained. State enters policy candidate set Policy candidate set The definition is as follows: in, Representing state The set of policy candidates; Representing state The set of adjacent states; The path exploration threshold, or dynamic threshold, is determined using an empirical distribution truncation method, with a reference range of values. .

[0027] Furthermore, the path unfolds in the form of a state transition chain, and the reinforcement learning agent selects an action at each path decision. The process involves transitioning from the current state to the next state to obtain a corresponding reward; this involves constructing an immediate reward function to evaluate the value of each state transition. The immediate reward function is designed as a nonlinear composite function to comprehensively consider drainage capacity, construction costs, path stability, and environmental adaptability. The immediate reward function is as follows: in, It is the immediate reward function, i.e., the total reward, used to describe the reward from the state. Transfer to target state Timely return assessment; It refers to drainage capacity, indicating the target state. The maximum drainage flow rate of the connected pipe section under design conditions is obtained from the unified spatial system; The target state Construction costs; It is a small constant used to avoid a denominator of 0, such as 0.01; This is the drainage capacity weighting coefficient, used to adjust the contribution of the combined drainage capacity and construction cost to the total reward. It is determined using existing Bayesian optimization methods, with a reference value range of [value missing]. ; This is the building penalty weight coefficient, used to control the impact of building interference items on the total reward. It is determined using existing Bayesian optimization methods, and the reference value range is [value range missing]. ; The target state Building interference factors; This is the path offset penalty weight, used to control the severity of the penalty for path discontinuities and jumps. It is determined using existing Bayesian optimization methods, with a reference value range of [value missing]. ; It is the offset distance, which is the target state. The Euclidean distance between the geographic coordinates and the standard path status location is determined based on historical pipeline data obtained from existing databases, such as location data, using shortest path algorithms such as A* and Dijkstra's algorithm. It is the offset normalization coefficient, which is the upper limit of the baseline offset and is determined by averaging all offset distances obtained from the existing database; This is the offset penalty index, used to control the degree of path jump penalty, i.e., nonlinear compression. It is determined using existing Bayesian optimization methods, with a reference value range of [value missing]. ; This represents a combination of drainage capacity and construction cost. This measure is used to quantify drainage efficiency under construction costs. The higher the drainage capacity, the lower the construction cost, and the greater the reward. Indicates building interference items; It is a penalty item used to control the penalty for path selection in densely built-up areas; It is a path offset term used to control path deviation, such as discontinuity or oscillation, which can cause stability problems.

[0028] To overcome the problem of Bellman equation updates in traditional reinforcement learning paths easily getting trapped in local optima or path oscillations, a dynamic self-regularized Bellman update formula is proposed, which introduces the local path mean as a stability constraint during state value updates: in, It is a state The value function, or state value, is used to represent the state. Starting point: the expected return that can be obtained under the current strategy; This is a discount factor used to control the decay of future returns. It is determined using existing Bayesian optimization methods, with a reference value range of [value missing]. ; It is a state The value function represents the value from the target state. Starting point: the potential gains under the current strategy; This is a stability penalty factor used to control the intensity of the penalty for path jumps. It is determined using existing Bayesian optimization methods, and its reference value range is [range to be filled in]. ; It is a state The number of states in the policy candidate set; Is with state Any adjacent state; Indicates the current path status Among all reachable neighboring states, select the path extension direction that maximizes long-term comprehensive benefits; Indicates from state The expected cumulative future returns of the route after departure; It is a path stability penalty, i.e., the local path mean, which reflects that the greater the degree of path hopping, the heavier the penalty.

[0029] Furthermore, during the path planning and execution phase, the initial drainage starting point state is preset by professional staff. Starting from state Among the adjacent states, select the state that maximizes the global right-hand side of the dynamic self-regularized Bellman update formula. ,Right now: State As the current state The next state, and the state Include it in the path sequence, and then repeat the operation, in order to state. The same maximum value selection process continues for the new current state until the path endpoint meets the drainage access boundary conditions, such as merging into a main pipe or the inlet of a planned sewage treatment facility. Each state transition ensures that the path nodes meet the elevation difference conditions, are not restricted areas, and are geologically feasible, thereby gradually forming a complete stormwater and sewage drainage path that meets the target optimization conditions.

[0030] In summary, a road stormwater and sewage pipe network path planning method based on reinforcement learning has been developed.

[0031] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0032] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0033] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A road stormwater and sewage pipe network path planning method based on reinforcement learning, characterized in that, Includes the following steps: S1. Based on the acquired multi-source heterogeneous data, preprocessing is performed to obtain a unified spatial system; Based on the unified spatial system, the attributes of the state in the unified spatial system are obtained; Based on the attributes of the state, a state transition validity function is obtained to determine whether two states are reachable. Based on reachability, the transition weights are calculated using the state transition legality function; S2. Determine the policy candidate set based on the transition weights; Construct an instant reward function based on the policy candidate set; Based on the immediate reward function, the state value is obtained through the dynamically self-regularized Bellman update formula, the next state is determined, and finally a path sequence is formed.

2. The road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 1, characterized in that, S1 specifically includes: Based on the attributes of the state, a state transition validity function is constructed using a Boolean indicator function; when the value of the state transition validity function is 1, the two states are reachable.

3. The road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 2, characterized in that, S1 specifically includes: Based on reachability, a state transition function is constructed using the geographic coordinates and state transition validity function in the state attributes, and the transition weights are calculated.

4. The road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: Based on the transition weights, states with transition weights greater than a set dynamic threshold are retained to form a policy candidate set.

5. The road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 4, characterized in that, S2 specifically includes: Based on the policy candidate set and state attributes, calculate the combined terms of drainage capacity and construction cost, building interference, and path offset.

6. The road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 5, characterized in that, S2 specifically includes: An instant reward function is constructed based on the combination of drainage capacity and construction cost, building interference, and path offset.

7. A road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 6, characterized in that, S2 specifically includes: Based on the immediate reward function and policy candidate set, a local path mean is introduced to construct a dynamic self-regularized Bellman update formula to update the state value.

8. A road stormwater and sewage pipe network path planning method based on reinforcement learning according to claim 7, characterized in that, S2 specifically includes: From the adjacent states of the initial drainage starting point state, select the state that maximizes the global right-hand side of the dynamic self-regularized Bellman update formula, and include it in the path sequence.

Images