Green way site selection optimization method and device based on graph neural network and reinforcement learning

Abstract

The invention provides a greenway site selection optimization method and device based on a graph neural network and reinforcement learning, and relates to the technical field of urban construction planning, and the method comprises the steps: obtaining to-be-selected line elements and surface elements in a research region; calculating a coverage relationship between a plurality of individual demand points generated in each surface element and the element with the line to be selected; sampling is carried out in the process of cyclically executing Markov decision, an intelligent agent is trained, and a greenway site selection scheme is obtained; in each cycle, the intelligent agent uses the graph neural network to carry out aggregation coding on the current state, and the current state is used as input of the strategy network; in the environment, when the currently selected line element reaches the maximum line element length, target individual demand points with the actual demand number are randomly sampled from each surface element, and the ratio of the number of the target individual demand points covered by the currently selected line element to the total number of the target individual demand points is used as an award. According to the method, the intelligent agent considering uncertainty can be trained, so that a more balanced and comprehensive greenway site selection scheme is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of urban construction planning, and particularly relates to a greenway site selection optimization method and device based on a graph neural network and reinforcement learning. BACKGROUND

[0002] A city greenway refers to a green linear space open in a city, which can provide facilities for citizens to provide leisure, sports, education and other functions. The city greenway is an important green infrastructure in the city, and has important significance for the city image, urban residents and economic benefits. The primary work of the city greenway in carrying out planning and construction is to carry out site selection planning of the city greenway.

[0003] At present, a traditional mixed integer linear programming (MILP) model is usually used for site selection planning of the city greenway. The traditional MILP model mainly focuses on linear elements, especially the best layout of simple linear elements without branches and loops, so that the services provided can cover as many demands as possible. However, in actual application, the organization unit of demand is generally an aggregated surface element. Since the coverage metric provided in the traditional MILP model is binary (covered and not covered), in the case of partially covering demand units, it may cause incorrect calculation of the actual demand covered, thereby causing non-optimality of the solution. SUMMARY

[0004] The present application provides a greenway site selection optimization method and device based on a graph neural network and reinforcement learning, to solve the defect that the MILP model may cause incorrect calculation of the actual demand covered in the case of partially covering demand units, thereby causing non-optimality of the solution, and to achieve the purpose of training an intelligent agent considering uncertainty, thereby obtaining a more optimal greenway site selection scheme.

[0005] The present application provides a greenway site selection optimization method based on a graph neural network and reinforcement learning, comprising: obtaining a plurality of candidate linear elements in a research area and a plurality of surface elements for organizing demand; generating a plurality of individual demand points in each of the surface elements, and calculating the coverage relationship between the individual demand points and the candidate linear elements; building an environment based on the candidate linear elements and their attributes, the surface elements and their attributes, intersection elements between the candidate linear elements and their attributes, the individual demand points and the coverage relationship; Data sampling is performed in the cyclic execution of the Markov decision process to obtain a training data set; at each cycle, the agent aggregates and encodes the current state fed back by the environment using a graph neural network as input of a policy network; when the current selected line element reaches the maximum line element length, the environment randomly samples a target number of individual demand points from the number of individual demand points in each surface element, and takes the ratio of the number of target individual demand points covered by the current selected line element determined based on the coverage relationship to the total number of target individual demand points as the reward fed back to the agent; The agent is trained based on the training data set, and a greenway site selection scheme is obtained by using the trained agent, wherein the greenway site selection scheme includes a plurality of selected line elements and a total amount of covered demand.

[0006] According to the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the application, data sampling is performed in the cyclic execution of the Markov decision process to obtain a training data set, which includes: A preset number of agents and environments are copied; Each agent and environment is assigned a thread, and the agent and the environment cyclically execute a Markov decision process, and data sampling is performed in the cyclic execution of the Markov decision process until the total sampling number of each thread reaches a preset sampling number, thereby obtaining the training data set.

[0007] According to the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the application, the agent and the environment cyclically execute a Markov decision process, and data sampling is performed in the cyclic execution of the Markov decision process, which includes: The following steps are cyclically executed: The agent requests a current state from the environment; the current state includes attributes of the to-be-selected line element, the surface element, and the intersection element between the to-be-selected line element; The agent converts the current state fed back by the environment into a state tensor, and inputs the state tensor into the graph neural network for aggregated encoding to obtain an edge-aggregated encoded tensor; The agent inputs the edge-aggregated encoded tensor into the policy network to obtain probabilities of different actions, and masks illegal actions to obtain a probability distribution; the action is to select a to-be-selected line element; The agent samples a current action based on the probability distribution, and submits the current action to the environment; The environment updates the state based on the current action, and feeds back a next state and a current reward to the agent; The current state, the current action, the probability distribution, the next state, the current reward and a completion state are packaged and stored as training data collected in this cycle.

[0008] According to the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the application, the environment feeds back the current reward to the agent, and the method comprises the following steps of: In the case that the current selected line element is a loop or all actions corresponding to the next state are illegal actions, the environment feeds back a negative value reward to the agent as the current reward. When the current selected line element reaches the maximum line element length, the environment randomly samples the actual number of target individual demand points from the plurality of individual demand points in each surface element, and feeds back the ratio of the number of target individual demand points covered by the current selected line element determined based on the coverage relationship to the total number of target individual demand points to the agent as the current reward.

[0009] According to the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the application, the method comprises the following steps of: The following steps are cyclically executed: Data sampling is performed from the training data set to obtain a plurality of target training data; Each target training data is converted into a plurality of target tensors; For each target tensor, the target tensor is input into the graph neural network, and the graph neural network is used to construct a target topological graph and generate a target encoding tensor of the target topological graph; The target encoding tensor of the target topological graph is input into a value network to obtain a state estimation value; The network parameters of the agent are adjusted based on the state estimation value.

[0010] According to the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the application, the graph neural network comprises a graph generation layer and a plurality of convolution layers; The graph neural network is specifically used for: The graph generation layer is used to extract the selected line element and its attributes, the surface element and its attributes, and the intersection element and its attributes between the selected line element and the surface element from an input tensor, and a topological graph is constructed with the selected line element as an edge, the surface element as a face, and the intersection element as a vertex. The attributes of edges, surfaces and vertices in the topological graph and the attributes of environmental elements are respectively preliminarily coded to obtain attribute coding of the edges, surfaces and vertices and attribute coding of the environmental elements; The following steps are performed using the current convolution layer: For each edge, the attribute coding of the edges, surfaces and vertices adjacent to the edge is respectively aggregated, and the attribute coding of the aggregated edges, surfaces and vertices adjacent to the edge is spliced to obtain the attribute coding of the aggregated edge; For each vertex, the attribute coding of the aggregated edges adjacent to the vertex is aggregated to the attribute coding of the vertex; The attribute coding of the edges, surfaces and vertices finally obtained by the current convolution layer is taken as the input of the next convolution layer; The attribute coding of the edges and vertices output by the last convolution layer and the attribute coding of the environmental elements are used to generate the coding tensor of the topological graph.

[0011] According to the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the application, the distribution mode of the plurality of individual demand points generated in each surface element includes at least one of uniform distribution, edge distribution, center distribution and multi-center distribution.

[0012] The application further provides a greenway site selection optimization device based on a graph neural network and reinforcement learning, comprising: An acquisition module is configured to acquire a plurality of candidate line elements in a study area and a plurality of surface elements for organizing demand; A generation module is configured to generate a plurality of individual demand points in each surface element; A calculation module is configured to calculate a coverage relationship between the individual demand points and the candidate line elements; A building module is configured to build an environment based on the candidate line elements and their attributes, the surface elements and their attributes, intersection elements between the candidate line elements and their attributes, the individual demand points and the coverage relationship; A sampling module is configured to sample data in a loop to perform a Markov decision process to obtain a training data set; in each loop, an agent aggregates and encodes a current state fed back by the environment using a graph neural network as input of a policy network; when a currently selected line element reaches a maximum line element length, the environment randomly samples a target number of individual demand points from the plurality of individual demand points in each surface element, and a ratio of a number of the target individual demand points covered by the currently selected line element based on the coverage relationship to a total number of the target individual demand points is taken as a reward fed back to the agent; A training module is configured to train the agent based on the training data set; An optimization module is configured to obtain a greenway site selection scheme by using the trained agent, the greenway site selection scheme including a plurality of selected line elements and a total demand covered.

[0013] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the greenway site selection optimization method based on the graph neural network and the reinforcement learning.

[0014] The application further provides a non-transitory computer-readable storage medium, which stores a computer program executable by a processor to implement the greenway site selection optimization method based on the graph neural network and the reinforcement learning.

[0015] The application further provides a computer program product, which includes a computer program executable by a processor to implement the greenway site selection optimization method based on the graph neural network and the reinforcement learning.

[0016] The application provides a greenway site selection optimization method and device based on a graph neural network and reinforcement learning. Firstly, a plurality of candidate line elements in a research area and a plurality of surface elements for organizing demand are obtained. Then, a plurality of individual demand points are generated in each surface element, and the covering relationship between the individual demand points and the candidate line elements is calculated. In this way, a large amount of calculation tasks can be completed before training, and the fluency and efficiency of training are improved. Next, the environment is built based on the candidate line elements and their attributes, the surface elements and their attributes, the intersection elements between the candidate line elements and their attributes, the individual demand points and the covering relationship. Data sampling is performed in the Markov decision process to obtain a training data set. In each cycle, the agent aggregates and encodes the current state fed back by the environment using the graph neural network as the input of the policy network, which can simplify and aggregate information, thereby speeding up the process of the agent identifying the environment and learning the strategy. When the current selected line element reaches the maximum line element length, the environment randomly samples a target individual demand point from a plurality of individual demand points in each surface element, and the ratio of the number of target individual demand points covered by the current selected line element to the total number of target individual demand points is determined based on the covering relationship, and is fed back to the agent as a reward. Since the target individual demand points sampled each time are different, the environment needs to dynamically calculate the demand coverage. Finally, the agent is trained based on the training data set, and a greenway site selection scheme is obtained using the trained agent. The greenway site selection scheme includes a plurality of selected line elements and the total amount of demand covered. Since the agent will face millions of demand distribution situations and demand coverage calculation results in the process of interacting with the environment, it can learn a more balanced and comprehensive greenway site selection scheme. Therefore, the application can train an agent considering uncertainty, thereby obtaining a more balanced and comprehensive greenway site selection scheme. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0018] Figure 1 is a binary coverage standard diagram provided by the prior art.

[0019] Figure 2 is a flowchart of the greenway site selection optimization method based on the graph neural network and the reinforcement learning provided by the embodiment of the application.

[0020] Figure 3 is a schematic diagram of the selected line elements which are discontinuous in space, have bifurcated roads and loops.

[0021] Figure 4 is a schematic diagram of calculating a target value provided by an embodiment of the present application.

[0022] Figure 5 is a schematic diagram of a greenway scheme obtained by each model when the distance threshold is 800 meters provided by an embodiment of the present application.

[0023] Figure 6 is a schematic diagram of a greenway scheme obtained by each model when the distance threshold is 1000 meters provided by an embodiment of the present application.

[0024] Figure 7 is a structural schematic diagram of a greenway site selection optimization device based on a graph neural network and reinforcement learning provided by an embodiment of the present application.

[0025] Figure 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0027] A city greenway refers to an open green linear space in a city, which can provide facilities for citizens to relax, exercise, and educate. The greenway is an important green infrastructure in the city, which is of great significance to the city image, urban residents and economic benefits. Specifically, 1) the greenway can improve the ecological environment of the city, protect biodiversity, increase green space, reduce air and noise pollution, reduce the urban heat island effect, and improve the health level of urban residents; 2) the greenway can enrich the landscape level of the city, display the history and culture of the city, enhance the characteristics and charm of the city, and improve the image and quality of the city, attract tourists and investors, and promote the economic and social development of the city; 3) the greenway can meet the diversified needs of urban residents, provide safe and comfortable walking and cycling conditions, increase the choice of traffic, alleviate traffic congestion, reduce traffic accidents, save energy and resources, reduce carbon emissions, and achieve low-carbon travel; 4) the greenway can enhance the sense of community belonging of urban residents, promote communication and interaction among neighbors, improve the participation awareness and responsibility of citizens, cultivate the awareness and action of environmental protection, and build harmonious social relationships; 5) the greenway can also drive the development of related industries such as tourism, catering, and culture, create more employment opportunities and tax revenues, and according to statistics, every 1 yuan invested in greenway construction can bring 1.4 yuan of economic benefits.

[0028] The first work in the planning and construction of urban greenways is to carry out site planning of urban greenways. At present, the traditional MILP model is usually used for site planning of urban greenways. The traditional MILP model mainly focuses on linear elements, especially the optimal layout of simple linear elements without branches and loops, so that the services provided can cover as many demands as possible. However, in practical applications, the organization unit of demand is generally an aggregated area element. Since the coverage metric provided in the traditional MILP model is binary (covered and not covered), in the case of partially covered demand units, it may lead to incorrect calculation of the actual demand covered, thereby causing the non-optimality of the solution.

[0029] As shown in Figure 1 , regardless of the way of coverage metric (arbitrary coverage, center coverage, 50% area coverage or complete coverage), there is a significant difference between the units considered to be covered and the actual covered demand. This feature makes it difficult for the MILP model to handle the uncertainty caused by aggregated demand units.

[0030] Based on this, the embodiment of the present application provides a greenway site selection optimization method and device based on graph neural network and reinforcement learning, which can train an intelligent agent considering uncertainty, thereby obtaining a more balanced and comprehensive greenway site selection scheme.

[0031] First, the greenway site selection optimization method based on graph neural network and reinforcement learning of the present application will be described below. Figures 2-6

[0032] Please refer to Figure 2 , Figure 2 is the flowchart of the greenway site selection optimization method based on graph neural network and reinforcement learning provided by the embodiment of the present application. As shown in Figure 2 , the method comprises steps 201-205.

[0033] Step 201, obtaining a plurality of candidate linear elements and a plurality of area elements for organizing demand in a study area.

[0034] Specifically, the basic geographic vector element data in the study area, including the candidate linear element data and the area element data for organizing demand, is read through the data path set by the user. The intersection elements between the linear elements are calculated according to the vertex coordinates of the linear elements. According to the field set by the user, the original attributes of the linear elements, area elements and intersection elements are read, and the original attributes are normalized for further operation. It should be noted that the attributes mentioned below are normalized attributes.

[0035] Step 202, generating a plurality of individual demand points in each area element, and calculating the coverage relationship between the individual demand points and the candidate linear elements. ​

[0036] Specifically, the demand unit of the urban greenway is an aggregated surface element, and a large number of individual demand points are generated in each surface element according to the demand data in each surface element, and the coverage relationship between the individual demand points and the candidate line elements is calculated, a large amount of calculation tasks can be completed before training, providing data basis for subsequent individual demand point sampling and coverage calculation, and improving the fluency and efficiency of training.

[0037] Optionally, the distribution mode of the plurality of individual demand points generated in each surface element includes at least one of uniform distribution, edge distribution, center distribution and multi-center distribution.

[0038] Step 203, building an environment based on the candidate line elements and their attributes, the surface elements and their attributes, the intersection elements between the candidate line elements and their attributes, the individual demand points and the coverage relationship.

[0039] Specifically, the environment is an important part of reinforcement learning, and the agent learns the strategy by interacting with the environment to solve practical problems. The greenway maximum coverage site selection optimization problem can be regarded as an environment, mainly including: candidate line elements and their attributes, surface elements and their attributes, intersection elements between candidate line elements and their attributes, individual demand points and coverage relationship. In the process of interacting with the agent, the attributes of each part of the environment will change. This dynamic environment provides the basis for learning demand distribution uncertainty for the agent of reinforcement learning, and avoids the partial coverage problem in the general static environment.

[0040] In the greenway maximum coverage site selection optimization problem, it is necessary to select line elements from candidate line elements under the constraint condition to achieve the goal of maximizing the total demand covered. Among them, the constraint conditions include: 1) the total length of the selected line elements is less than or equal to the maximum line element length; 2) the selected line elements must be continuous in space, and branch roads and loops are prohibited. Exemplarily, the three situations limited by the constraint condition 2) are Figure 3 The selected line elements shown are discontinuous in space, have branch roads and loops. The total demand covered is generally calculated by measuring the Euclidean distance between the selected line elements and each surface element with demand attribute value. If the distance is less than the set distance threshold, the demand attribute value of the surface element is added to the total demand. For example Figure 4As shown, the number in each face element represents the demand attribute value. According to the set distance threshold, the coverage range of the selected line element can be calculated, and the distance from any point in the coverage range to the selected line element is less than the set distance threshold. Through the coverage relationship between the coverage range and each face element, the covered face element can be determined. The demand attribute value of the covered face element is summed up to obtain the target value of the line element selection scheme, which is 36 in this example. The final purpose of the maximum coverage site selection optimization problem is to maximize the calculated target value.

[0041] In step 204, data sampling is performed in a loop to execute the Markov decision process to obtain a training data set. In each loop, the agent aggregates and encodes the current state fed back by the environment using a graph neural network as the input of the policy network. The environment randomly samples the target individual demand points from the actual demand quantity in each face element when the current selected line element reaches the maximum line element length. The ratio of the number of target individual demand points covered by the current selected line element determined based on the coverage relationship to the total number of target individual demand points is taken as the reward fed back to the agent.

[0042] Specifically, the Markov decision process (MDP) consists of four basic elements, namely state, action, reward and transition. Among them, 1) state: the layout of elements in the study area and related attribute parameters; 2) action: the agent will make action selection in the candidate line element in the maximum coverage site selection optimization problem in space, and some illegal actions will be shielded; 3) reward: the reward of each intermediate time step is set to 0, and at the final time step, the environment calculates the proportion of covered demand as the settlement value of the reward; 4) transition: the element attributes in the study area will change according to the action selection of the agent. In the Markov decision process, the transition probability and reward function of the current time step only depend on the state of the time step and the decision of the agent, and are independent of the state and decision of the past time step. This feature makes it suitable for decision-making in dynamic environments, and the optimal state value function is calculated to maximize the long-term cumulative reward, thereby determining the optimal strategy and solving the decision-making problem.

[0043] In an embodiment, the data sampling in the loop to execute the Markov decision process in step 204 to obtain the training data set comprises: Step 2041, duplicate a preset number of agents and environments; Step 2042, assign a thread to each agent and environment, and use the agent and environment to execute the Markov decision process in a loop, and perform data sampling in the loop to execute the Markov decision process until the total sampling quantity of each thread reaches a preset sampling quantity to obtain the training data set.

[0044] In the embodiment, a plurality of same agents and environments are copied, a Markov decision process is independently looped for each agent and environment by using an assigned thread, and data sampling is performed in the looped Markov decision process, and the plurality of threads simultaneously perform data sampling, so that the data sampling efficiency can be improved.

[0045] In an embodiment, the graph neural network comprises a graph generation layer and a plurality of convolution layers, and the graph neural network is specifically used to perform the following steps 301-303.

[0046] In step 301, the graph generation layer is used to extract candidate line elements and their attributes, surface elements and their attributes, and intersection elements and their attributes between the candidate line elements from the input tensor, and a topological graph is constructed by taking the candidate line elements as edges, the surface elements as surfaces, and the intersection elements as vertices, so that the normalization of the input data can be realized.

[0047] Specifically, as shown in Table 1, each graph element in the topological graph has its unique attributes.

[0048] Table 1: Attribute table of each graph element in the topological graph

[0049] In the transition caused by the action of the agent, the attribute of the type of dynamic changes, and the reward can be obtained by counting the “demand quantity” attribute of the covered surface.

[0050] Although the state tensor can be converted into a standard topological graph, it still needs a large amount of time and excellent physical device basis to train directly by taking the entire topological graph as data input. Therefore, embedding the attributes of adjacent vertices, edges and surfaces in the topological graph with each other can realize the aggregation and simplification of information, thereby accelerating the process of the agent recognizing the environment and learning the strategy. The encoding process of the topological graph is described in detail below.

[0051] In step 302, the attributes of the edges, surfaces and vertices in the topological graph and the attributes of the environmental elements are respectively preliminarily encoded to obtain the attribute encoding of the edges, surfaces and vertices and the attribute encoding of the environmental elements.

[0052] Specifically, the preliminary encoding can be performed by expressions (1)-(4): (1) (2) (3) (4) wherein, , , respectively represent the attribute of the edge, the face and the vertex in the current topology graph, respectively represent the attribute of the edge, the face and the vertex in the current topology graph, , , , respectively represent the attribute of the edge, the face and the vertex in the current topology graph, 、 、 、 respectively represent the attribute of the edge, the face and the vertex in the current topology graph,

[0053] Step 303, using the current convolutional layer to perform the following steps 3031-3034.

[0054] Step 3031, for each edge, respectively aggregate the attribute codes of the edge, the face and the vertex adjacent to the edge, and splice the attribute codes of the edge, the face and the vertex adjacent to the edge after aggregation to obtain the attribute code of the edge after aggregation.

[0055] Specifically, the attribute codes of the edge, the face and the vertex adjacent to the edge can be respectively aggregated by expressions (5)-(7): (5) (6) (7) wherein, , , represents a linear transformation layer, and || represents a splicing operation, represents the set of faces adjacent to the edge represents the number of faces in .

[0056] Then, the attribute codes of the edge, the face and the vertex adjacent to the edge after aggregation can be spliced by expression (8): (8) wherein, represents the attribute code of the edge after aggregation, represents a linear transformation layer.

[0057] Step 3032, for each vertex, aggregate the attribute codes of the edges adjacent to the vertex into the attribute code of the vertex.

[0058] Specifically, considering the importance of the vertex in the continuity of the online element, the aggregated edges can be re-aggregated into the vertex by expression (9): ​ (9) wherein, represents a linear transformation layer, represents a vertex a set of adjacent edges, represents the number of edges in

[0059] Step 3033, encoding the attributes of the edges, faces and vertices finally obtained by the current convolutional layer as the input of the next convolutional layer.

[0060] Step 3034, generating the encoding tensor of the topological graph based on the attribute encoding of the edges and vertices output by the last convolutional layer and the attribute encoding of the environmental elements.

[0061] Specifically, after passing through convolutional layers, the attribute encoding of the edges and vertices and can be obtained. The encoding of the entire topological graph can be represented by expressions (10)-(12): (10) (11) (12) wherein, represents the encoding of the topological graph.

[0062] In an embodiment, the Markov decision process in step 2042 is executed by using the agent and the environment in a loop, and data sampling is performed in the loop of executing the Markov decision process, including: executing the following steps 401-406 in a loop.

[0063] Step 401, the agent requests the current state from the environment; the current state includes the attributes of the candidate line elements, the face elements and the intersection elements between the candidate line elements.

[0064] Step 402, the agent converts the current state fed back by the environment into a state tensor, and inputs the state tensor into a graph neural network for aggregated encoding to obtain an encoded tensor after edge aggregation.

[0065] Step 403, the agent inputs the encoded tensor after edge aggregation into a policy network to obtain the probabilities of different actions, and screens out illegal actions to obtain a probability distribution; the action is to select a candidate line element.

[0066] Specifically, the policy network is the core of the agent to make decisions. At each time step, it takes the edge-aggregated encoding tensor output by the graph neural network as input, and outputs the probability distribution of the selected action. Specifically, the edge-aggregated encoding tensor is first passed through a multi-layer perceptron (MLP) to obtain a score, and is converted into a probability distribution through a softmax function: (13) (14) wherein, represents the edge-aggregated encoding tensor, represents the score, represents the probability distribution.

[0067] The action selection made by the agent will be sampled from this probability distribution. In addition, for some obvious illegal actions, such as repeated line element selection, or discontinuous line element selection, will be filtered through an action mask to reduce the number of action selections, thereby further improving training efficiency.

[0068] Step 404, the agent samples the current action based on the probability distribution, and submits the current action to the environment.

[0069] Step 405, the environment updates the state based on the current action, and feeds back the next state and the current reward to the agent.

[0070] In an embodiment, the environment in step 405 feeds back the current reward to the agent, including: The environment feeds back a negative reward to the agent as the current reward in the case that the current selected line element is a loop or all actions corresponding to the next state are illegal actions (such as repeated line element selection, or discontinuous line element selection); When the current selected line element reaches the maximum line element length, the environment randomly samples the actual number of target individual demand points from a plurality of individual demand points in each face element, and feeds back the ratio of the number of target individual demand points covered by the current selected line element based on the covering relationship to the total number of target individual demand points to the agent as the current reward.

[0071] Step 406, the current state, the current action, the probability distribution, the next state, the current reward and the completion state are packaged and stored as training data collected in this cycle; the completion state is used to represent that there is no any to-be-selected line element that meets the restriction condition.

[0072] In step 205, the agent is trained based on the training data set, and a greenway site selection scheme is obtained by using the trained agent, the greenway site selection scheme including a plurality of selected line elements and a total demand covered.

[0073] In an embodiment, the training of the agent based on the training data set in step 205 includes: cyclically performing the following steps: sampling data from the training data set to obtain a plurality of target training data; converting each target training data into a plurality of target tensors; for each target tensor, inputting a state input in the target tensor into a graph neural network, constructing a target topology graph by using the graph neural network, and generating a target encoding tensor of the target topology graph; inputting the target encoding tensor of the target topology graph into a value network to obtain a state estimated value; and adjusting network parameters of the agent based on the state estimated value.

[0074] Specifically, the value network is used to predict and evaluate the decisions made by the agent, thereby helping the agent to further improve its decision-making ability. Unlike the policy network, the value network accepts the encoding tensor of the entire topology graph as input and directly obtains the predicted value of the current policy: (15) The policy network and the value network together constitute the reinforcement learning agent. In training, the weight parameters of the multilayer perceptron will be continuously optimized until the generated action approaches the optimal and the predicted policy value approaches the actual value.

[0075] Optionally, proximal policy optimization is the main algorithm used in the training of the agent. The loss function used in the training mainly consists of three parts: policy loss, value loss, and entropy loss. The specific calculation methods of the policy loss, the value loss, and the entropy loss are not described here.

[0076] Obtaining the greenway site selection scheme by using the trained agent specifically includes: S1, finding a line element that can satisfy the constraint condition in all unselected candidate line elements , performing S2; S2, if is not empty, performing S3, otherwise performing S4; S3, selecting a line element from by the agent, and adding it to the selected line elements, and performing S1; S4, calculating the total demand covered, and outputting the selected line elements and the total demand covered.

[0077] The greenway site selection optimization method based on graph neural networks and reinforcement learning provided in this invention first acquires multiple candidate line elements and multiple surface elements for organizing needs within the study area. Then, it generates several individual demand points within each surface element and calculates the coverage relationship between these individual demand points and the candidate line elements. This allows for the completion of a large number of computational tasks before training, improving the smoothness and efficiency of training. Next, it constructs an environment based on the candidate line elements and their attributes, surface elements and their attributes, intersection elements and their attributes, individual demand points, and coverage relationships. Data sampling is performed during the iterative execution of Markov decision-making to obtain a training dataset. In each iteration, the agent uses a graph neural network to aggregate and encode the current state of the environmental feedback, using it as input to the policy network. This simplifies and aggregates information, thereby accelerating the intelligent... The process involves the agent recognizing the environment and learning strategies. When the selected line elements reach their maximum length, the environment randomly samples the target individual demand points from several individual demand points within each surface element. The ratio of the number of target individual demand points covered by the currently selected line elements (determined by coverage relationships) to the total number of target individual demand points is used as a reward for the agent. Since the target individual demand points are different each time they are sampled, the environment needs to dynamically calculate the demand coverage. Finally, the agent is trained based on the training dataset, and the trained agent is used to obtain a greenway site selection scheme. The greenway site selection scheme includes multiple selected line elements and the total covered demand. Because the agent will face millions of demand distribution scenarios and demand coverage calculation results during its interaction with the environment, it learns a more balanced and comprehensive greenway site selection scheme. Therefore, this embodiment of the invention can train an agent that considers uncertainty, thereby obtaining a more balanced and comprehensive greenway site selection scheme.

[0078] The following specific experiments verify the greenway site selection optimization method based on graph neural networks and reinforcement learning (hereinafter referred to as "this method") provided in the embodiments of the present invention.

[0079] When solving the maximum coverage location optimization problem, the optimal solution is usually obtained by establishing a MILP model. In practical application, taking some blocks in a certain area as an example, under two different coverage distance threshold settings, the greenway location schemes obtained by the MILP model using four different coverage measurement methods and the greenway location scheme obtained by this method were compared with those obtained by the MILP model. The test results are shown in Table 2.

[0080] Table 2 Test Results

[0081] Through the above Table 2 and Figure 5 and 6It can be seen that in the test results of the coverage distance of 800 meters and 100 meters, the ratio of the coverage demand to the total demand of the present method is greater than the ratio of the coverage demand to the total demand obtained by using the MILP model with four different coverage metrics. Therefore, no matter how the distance threshold is set, the present method can obtain a better solution with a larger improvement, which shows that the present method can take into account the uncertainty factor and give a better greenway site selection scheme than the traditional MILP model.

[0082] The greenway site selection optimization device based on the graph neural network and the reinforcement learning provided by the present application is described below. The greenway site selection optimization device based on the graph neural network and the reinforcement learning described below can be correspondingly referred to the greenway site selection optimization method based on the graph neural network and the reinforcement learning described above.

[0083] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of the greenway site selection optimization device based on the graph neural network and the reinforcement learning provided by the present application. As Figure 7 shown, the device can include: The acquisition module 10 is configured to acquire a plurality of candidate line elements and a plurality of surface elements for organizing demand in a study area. The generation module 20 is configured to generate a plurality of individual demand points in each surface element. The calculation module 30 is configured to calculate the coverage relationship between the individual demand points and the candidate line elements. The building module 40 is configured to build an environment based on the candidate line elements and their attributes, the surface elements and their attributes, the intersection elements between the candidate line elements and their attributes, the individual demand points, and the coverage relationship. The sampling module 50 is configured to sample data in a loop to execute a Markov decision process to obtain a training data set. In each loop, the agent aggregates and encodes the current state fed back by the environment using the graph neural network as the input of the policy network. The environment randomly samples a target number of individual demand points from a plurality of individual demand points in each surface element when the current selected line element reaches the maximum line element length. The ratio of the number of target individual demand points covered by the current selected line element to the total number of target individual demand points determined based on the coverage relationship is taken as the reward fed back to the agent. The training module 60 is configured to train the agent based on the training data set. The optimization module 70 is configured to obtain a greenway site selection scheme using the trained agent. The greenway site selection scheme includes a plurality of selected line elements and a total amount of demand covered.

[0084] In an embodiment, the sampling module 50 includes: The duplication unit is configured to duplicate a preset number of agents and environments. a sampling unit configured to assign a thread to each agent and environment, execute a Markov decision process in a loop with the agent and the environment, and perform data sampling in the loop of executing the Markov decision process until a total sampling number of the thread reaches a preset sampling number, to obtain a training data set.

[0085] In an embodiment, the sampling unit is specifically configured to: execute the following steps in a loop: the agent requests a current state from the environment; the current state includes attributes of a candidate line element, a face element, and an intersection element between the candidate line element and the face element; the agent converts the current state fed back by the environment into a state tensor, and inputs the state tensor into a graph neural network for aggregated encoding to obtain an edge-aggregated encoded tensor; the agent inputs the edge-aggregated encoded tensor into a policy network to obtain probabilities of different actions, and masks illegal actions to obtain a probability distribution; the action is to select a candidate line element; the agent samples a current action based on the probability distribution, and submits the current action to the environment; the environment updates a state based on the current action, and feeds back a next state and a current reward to the agent; the current state, the current action, the probability distribution, the next state, the current reward, and a completion state are packaged and stored as training data collected in the current loop; the completion state is used to represent that there is no candidate line element that meets the restriction condition.

[0086] In an embodiment, the sampling unit is specifically configured to: the environment feeds back a negative reward to the agent as the current reward in a case where the current selected line element is a loop or all actions corresponding to the next state are illegal actions; the environment randomly samples a number of target individual demand points from a plurality of individual demand points in each face element when the current selected line element reaches a maximum line element length, feeds back, to the agent, a ratio of a number of target individual demand points covered by the current selected line element determined based on a coverage relationship to a total number of target individual demand points as the current reward.

[0087] In an embodiment, the training module 60 is specifically configured to: execute the following steps in a loop: perform data sampling from the training data set to obtain a plurality of target training data; convert each target training data into a plurality of target tensors; for each target tensor, input the target tensor into a graph neural network, use the graph neural network to construct a target topological graph, and generate a target encoded tensor of the target topological graph; The target coding tensor of the target topological graph is input into the value network to obtain a state estimation value; The network parameters of the agent are adjusted based on the state estimation value.

[0088] In an embodiment, the graph neural network comprises a graph generation layer and a plurality of convolution layers; The graph neural network is specifically used for: The graph generation layer is used to extract candidate line elements and their attributes, surface elements and their attributes, and intersection elements and their attributes between the candidate line elements from the input tensor, to construct a topological graph taking the candidate line elements as edges, the surface elements as surfaces, and the intersection elements as vertices; The attributes of the edges, surfaces and vertices in the topological graph and the attributes of the environmental elements are respectively preliminarily coded to obtain the attribute coding of the edges, surfaces and vertices and the attribute coding of the environmental elements; The following steps are performed by using the current convolution layer: For each edge, the attribute coding of the edges, surfaces and vertices adjacent to the edge are respectively aggregated, and the attribute coding of the edges, surfaces and vertices adjacent to the edge after aggregation is spliced to obtain the attribute coding of the edge after aggregation; For each vertex, the attribute coding of the edges adjacent to the vertex after aggregation is aggregated into the attribute coding of the vertex; The attribute coding of the edges, surfaces and vertices finally obtained by the current convolution layer is taken as the input of the next convolution layer; The coding tensor of the topological graph is generated based on the attribute coding of the edges and vertices and the attribute coding of the environmental elements output by the last convolution layer.

[0089] In an embodiment, the distribution mode of the several individual demand points generated in each surface element includes at least one of uniform distribution, edge distribution, center distribution and multi-center distribution.

[0090] Figure 8 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 8As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can invoke the logic instructions in the memory 830 to execute the greenway site selection optimization method based on the graph neural network and reinforcement learning, which includes: acquiring a plurality of candidate line elements in a study area and a plurality of surface elements for organizing demand; generating a plurality of individual demand points in each surface element, and calculating the coverage relationship between the individual demand points and the candidate line elements; building an environment based on the candidate line elements and their attributes, the surface elements and their attributes, the intersection elements between the candidate line elements and their attributes, the individual demand points, and the coverage relationship; data sampling is performed in the cyclic execution of the Markov decision process to obtain a training data set; at each cycle, the agent aggregates and encodes the current state feedback from the environment using the graph neural network as the input of the policy network; when the current selected line element reaches the maximum line element length, the environment randomly samples a target individual demand point from a plurality of individual demand points in each surface element, and the ratio of the number of target individual demand points covered by the current selected line element based on the coverage relationship to the total number of target individual demand points is taken as the reward feedback to the agent; training the agent based on the training data set, and obtaining a greenway site selection scheme using the trained agent, wherein the greenway site selection scheme includes a plurality of selected line elements and a total amount of covered demand.

[0091] In addition, the logic instructions in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the technical solutions that make essential contributions to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0092] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the greenway site selection optimization method based on a graph neural network and reinforcement learning provided by the above-mentioned methods, which comprises: acquiring a plurality of candidate line elements and a plurality of surface elements for organizing demand in a study area; generating a plurality of individual demand points in each surface element and calculating the coverage relationship between the individual demand points and the candidate line elements; building an environment based on the candidate line elements and their attributes, the surface elements and their attributes, the intersection elements between the candidate line elements and their attributes, the individual demand points and the coverage relationship; performing data sampling in the Markov decision process in each cycle to obtain a training data set; in each cycle, the agent aggregates and encodes the current state fed back by the environment using the graph neural network as the input of the policy network; when the current selected line element reaches the maximum line element length, the environment randomly samples a target number of individual demand points from the plurality of individual demand points in each surface element, and takes the ratio of the number of target individual demand points covered by the current selected line element based on the coverage relationship to the total number of target individual demand points as the reward fed back to the agent; training the agent based on the training data set, and obtaining a greenway site selection scheme using the trained agent, wherein the greenway site selection scheme comprises a plurality of selected line elements and the total demand covered.

[0093] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the greenway site selection optimization method based on a graph neural network and reinforcement learning provided by the above-mentioned methods, which comprises: acquiring a plurality of candidate line elements and a plurality of surface elements for organizing demand in a study area; generating a plurality of individual demand points in each surface element and calculating the coverage relationship between the individual demand points and the candidate line elements; building an environment based on the candidate line elements and their attributes, the surface elements and their attributes, the intersection elements between the candidate line elements and their attributes, the individual demand points and the coverage relationship; performing data sampling in the Markov decision process in each cycle to obtain a training data set; in each cycle, the agent aggregates and encodes the current state fed back by the environment using the graph neural network as the input of the policy network; when the current selected line element reaches the maximum line element length, the environment randomly samples a target number of individual demand points from the plurality of individual demand points in each surface element, and takes the ratio of the number of target individual demand points covered by the current selected line element based on the coverage relationship to the total number of target individual demand points as the reward fed back to the agent; training the agent based on the training data set, and obtaining a greenway site selection scheme using the trained agent, wherein the greenway site selection scheme comprises a plurality of selected line elements and the total demand covered.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing greenway location based on graph neural networks and reinforcement learning, characterized in that, include: Obtain multiple candidate line features and multiple polygon features for organizing requirements within the study area; Within each of the surface features, generate several individual demand points and calculate the coverage relationship between the individual demand points and the candidate line features; An environment is built based on the candidate line elements and their attributes, the area elements and their attributes, the intersection elements and their attributes between the candidate line elements, the individual demand points, and the coverage relationships; During the cyclic execution of Markov decision-making, data sampling is performed to obtain a training dataset. In each iteration, the agent uses a graph neural network to aggregate and encode the current state fed back by the environment, which is then used as the input to the policy network. When the currently selected line element reaches the maximum line element length, the environment randomly samples the actual number of target individual demand points from the several individual demand points within each of the surface elements. The ratio of the number of target individual demand points covered by the currently selected line element, determined based on the coverage relationship, to the total number of target individual demand points is used as the reward fed back to the agent. The agent is trained based on the training dataset, and a greenway site selection scheme is obtained using the trained agent. The greenway site selection scheme includes multiple selected route elements and the total demand covered.

2. The greenway site selection optimization method based on graph neural networks and reinforcement learning according to claim 1, characterized in that, The data sampling process during the cyclic execution of Markov decision-making to obtain the training dataset includes: Copy a predetermined number of the intelligent agents and the environment; Each agent and environment is assigned a thread, and the agent and environment are used to cyclically execute a Markov decision process. Data is sampled during the cyclic execution of the Markov decision process until the total number of samples from all threads reaches a preset number, thereby obtaining the training dataset.

3. The greenway site selection optimization method based on graph neural networks and reinforcement learning according to claim 2, characterized in that, The step of cyclically executing a Markov decision process using the agent and the environment, and sampling data during the cyclical execution of the Markov decision process, includes: Repeat the following steps: The agent requests the current state from the environment; the current state includes the attributes of the candidate line features, the polygon features, and the intersection features between the candidate line features; The agent converts the current state fed back by the environment into a state tensor, and inputs the state tensor into the graph neural network for aggregation encoding to obtain an edge-aggregated encoded tensor. The agent inputs the encoded tensor after edge aggregation into the policy network to obtain the probabilities of different actions, and filters out illegal actions to obtain a probability distribution; the action is to select candidate line elements. The agent samples the current action based on the probability distribution and submits the current action to the environment; The environment updates its state based on the current action and feeds back the next state and the current reward to the agent. The current state, the current action, the probability distribution, the next state, the current reward, and the completion state are packaged and stored as training data for this loop. The completion state is used to indicate that no candidate line element meets the restriction conditions at present.

4. The greenway site selection optimization method based on graph neural networks and reinforcement learning according to claim 3, characterized in that, The environment feeds back the current reward to the agent, including: If the environment is such that the currently selected line element is a loop or all actions corresponding to the next state are illegal actions, the negative reward will be fed back to the agent as the current reward. When the selected line element reaches the maximum line element length, the environment randomly samples the actual number of target individual demand points from the plurality of individual demand points within each of the surface elements. The ratio of the number of target individual demand points covered by the currently selected line element to the total number of target individual demand points, determined based on the coverage relationship, is used as the current reward and fed back to the agent.

5. The greenway site selection optimization method based on graph neural networks and reinforcement learning according to claim 1, characterized in that, Training the agent based on the training dataset includes: Repeat the following steps: Data is sampled from the training dataset to obtain multiple target training data sets; The training data for each of the aforementioned targets are converted into multiple target tensors; For each target tensor, the target tensor is input into the graph neural network, and the graph neural network is used to construct a target topology graph and generate a target encoding tensor of the target topology graph; The target encoding tensor of the target topology graph is input into the value network to obtain the state estimation value; The network parameters of the agent are adjusted based on the estimated state value.

6. The greenway site selection optimization method based on graph neural networks and reinforcement learning according to claim 3 or 5, characterized in that, The graph neural network includes a graph generation layer and multiple convolutional layers; The graph neural network is specifically used for: The graph generation layer is used to extract candidate line features and their attributes, surface features and their attributes, and intersection features and their attributes between candidate line features from the input tensor. A topological graph is constructed with the candidate line features as edges, the surface features as surfaces, and the intersection features as vertices. The attributes of edges, faces, and vertices in the topology graph, as well as the attributes of environmental elements, are initially encoded to obtain the attribute codes of edges, faces, and vertices, as well as the attribute codes of environmental elements. Perform the following steps using the currently described convolutional layer: For each edge, the attribute codes of the edges, faces and vertices adjacent to the edge are aggregated separately, and the aggregated attribute codes of the edges, faces and vertices adjacent to the edge are concatenated to obtain the attribute code of the edge after aggregation; For each vertex, the attribute codes of the edges adjacent to the vertex are aggregated and then aggregated onto the attribute code of the vertex. The attribute encodings of the edges, faces, and vertices obtained from the current convolutional layer are used as the input to the next convolutional layer; The encoding tensor of the topological graph is generated based on the attribute encodings of the edges and vertices output from the last convolutional layer and the attribute encodings of the environmental elements.

7. The greenway site selection optimization method based on graph neural networks and reinforcement learning according to claim 1, characterized in that, The distribution pattern of the plurality of individual demand points generated within each of the said surface features includes at least one of the following distribution patterns: uniform distribution, edge distribution, center distribution, and multicenter distribution.

8. A greenway site selection optimization device based on graph neural networks and reinforcement learning, characterized in that, include: The acquisition module is used to acquire multiple candidate line features and multiple polygon features for organizing requirements within the study area; The generation module is used to generate several individual requirement points within each of the surface features; The calculation module is used to calculate the coverage relationship between the individual demand points and the candidate line elements; A construction module is used to build an environment based on the candidate line features and their attributes, the polygon features and their attributes, the intersection features between the candidate line features and their attributes, the individual requirement points, and the coverage relationships; The sampling module is used to sample data during the cyclic execution of Markov decisions to obtain a training dataset. In each iteration, the agent uses a graph neural network to aggregate and encode the current state fed back by the environment, which is then used as the input to the policy network. When the currently selected line element reaches the maximum line element length, the environment randomly samples the actual number of target individual demand points from the plurality of individual demand points within each of the surface elements. The ratio of the number of target individual demand points covered by the currently selected line element, determined based on the coverage relationship, to the total number of target individual demand points is used as the reward fed back to the agent. A training module is used to train the agent based on the training dataset; An optimization module is used to obtain a greenway site selection scheme using a trained agent. The greenway site selection scheme includes multiple selected route elements and the total demand covered.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the greenway location optimization method based on graph neural networks and reinforcement learning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the greenway location optimization method based on graph neural networks and reinforcement learning as described in any one of claims 1 to 7.

Images