Urban facility deployment method and device based on reinforcement learning, and electronic equipment

Abstract

The invention relates to the technical field of urban planning, and provides an urban facility deployment method and device based on reinforcement learning, and electronic equipment, and the method comprises the steps: obtaining an urban region exchange graph corresponding to an initial urban facility deployment scheme, and filtering low-quality edges with the total deployment cost higher than a set cost threshold value in the urban region exchange graph, obtaining a target city region exchange graph; and based on a pre-trained city facility deployment model, generating a target city facility deployment scheme according to the target city region exchange graph, and performing city facility deployment according to the target city facility deployment scheme. According to the method, low-quality edges in an urban area exchange graph are filtered, algorithm attention is guided to high-quality exchange operation, the precision of a target urban facility deployment scheme is improved, the exchange operation is driven through rapid reasoning of a graph neural network and a strategy network to achieve urban facility deployment, heavy calculation is avoided, and the efficiency of urban facility deployment is improved. And the rapid generation of the facility deployment scheme of the target city is realized.

Description

Technical Field

[0001] The present invention relates to the field of urban planning technology, and in particular to a method, device and electronic equipment for deploying urban facilities based on reinforcement learning. Background Art

[0002] The spatial distribution of urban facilities profoundly affects the mobility patterns of billions of residents in cities to access basic public services (such as schools and hospitals). However, in real cities, the layout of facilities often does not match the residents' demand for urban services, resulting in high travel costs and a large gap between the actual layout and the theoretical optimal layout. The undesirable layout of facilities greatly undermines the efficiency of urban operations, leads to excessive carbon emissions and severe traffic congestion, and reduces the accessibility and fairness of citizens seeking basic services. Therefore, it is crucial to solve the above problems by coordinating the layout of urban facilities and optimizing accessibility, which can enable the construction of more sustainable and inclusive cities.

[0003] In fact, the Facility Location Problem (FLP) involves selecting P urban areas from N candidate areas to place facilities. Its own complexity is NP. It is a typical Composite Optimizer (CO) problem and is extremely challenging.

[0004] In actual urban scenarios, N and P are usually very large, and it is difficult to design a reliable algorithm that can provide a solution that meets the accuracy requirements within a reasonable time frame. The solution space of this problem is huge. For example, when (N, P) = (1000, 300), the solution space exceeds 10 263 At the same time, as a real-world optimization problem, there is valuable prior knowledge that needs to be explicitly modeled, including the strategies adopted in existing heuristic algorithms, which should provide a reference for algorithm design.

[0005] Furthermore, unlike the standard FLP setting, there are multiple different types of urban facilities, each with different residential requirements and a different number of facilities, which introduces additional constraints to the problem, such as the number of facilities that can be accommodated in each urban area, making the task more complex.

[0006] FLP is traditionally solved using heuristic and meta-heuristic methods, such as tabu search and variable neighborhood search (VNS). In addition, it can be formulated as a mixed integer optimization problem, which can be solved using commercial solvers such as Gurobi and MOSEK. Although these methods provide near-optimal solutions, their inherent large amount of local search computation makes them only applicable to small-scale problems, and when they are applied to large-scale scenarios in the real world, their solution speed is often slow, and they may not even produce feasible solutions within a few days, making them difficult to apply in practice.

[0007] In recent years, some machine learning models have been proposed to solve FLP. Although they have greatly improved computational efficiency, these machine learning models ignore the rich prior knowledge inherent in FLP, resulting in a significant decrease in solution accuracy compared to traditional methods and unsatisfactory accessibility of the final deployment solution, thus limiting their applicability.

[0008] Therefore, how to solve the problem of slow solution speed and low solution accuracy of existing urban facility deployment methods is an important issue that needs to be urgently addressed in the field of urban planning. Summary of the invention

[0009] The present invention provides a method, device and electronic equipment for urban facility deployment based on reinforcement learning, which are used to overcome the defects of slow solution speed and low solution accuracy of existing urban facility deployment methods, realize the rapid generation of urban facility deployment plans, and ensure the accuracy of urban facility deployment plans.

[0010] On the one hand, the present invention provides a city facility deployment method based on reinforcement learning, comprising: obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0011] Furthermore, the low-quality edges include historically visited edges and negative-benefit edges; accordingly, the step of filtering out the low-quality edges whose total deployment cost in the urban area exchange graph is higher than a set threshold specifically includes: obtaining the historical graph-nodes visited by the historical exchange operation, and determining the graph-edges containing the historical graph-nodes as historically visited edges; obtaining the actual distance between the removed node and the inserted node corresponding to the candidate exchange operation, and calculating the actual benefit and actual loss corresponding to the candidate exchange operation; when the actual distance is greater than the set distance threshold and the actual benefit is lower than the actual loss, determining the graph-edge corresponding to the candidate exchange operation as a negative-benefit edge; filtering the historically visited edges and negative-benefit edges in the urban area exchange graph to obtain the target urban area exchange graph.

[0012] Furthermore, the city facility deployment model based on the pre-trained method generates a target city facility deployment plan according to the target city area exchange graph, including: based on the pre-trained graph neural network, obtaining exchange edge embedding according to the target city area exchange graph; based on the pre-trained strategy network, generating a target city facility deployment plan according to the exchange edge embedding.

[0013] Furthermore, the pre-trained graph neural network obtains exchange edge embeddings according to the target city area exchange graph, including: obtaining node features corresponding to each graph-node in the target city area exchange graph, and encoding the node features into initial node embeddings; updating the initial node embeddings to obtain target node embeddings; fusing the target node embeddings corresponding to the candidate exchange operations with the virtual node embeddings corresponding to the virtual nodes to obtain the exchange edge embeddings; wherein the virtual nodes are connected to all graph-nodes in the target city area exchange graph.

[0014] Furthermore, the policy network includes a multilayer perceptron; accordingly, the pre-trained policy network generates a target city facility deployment plan based on the exchange edge embedding, including: scoring each graph-edge in the target city area exchange graph through a multilayer perceptron based on the exchange edge embedding to obtain an operation score corresponding to the candidate exchange operation; determining the candidate exchange operation corresponding to the highest operation score as the target exchange operation to obtain the target city facility deployment plan.

[0015] Furthermore, training and optimizing the urban facility deployment model specifically includes: obtaining urban geographic spatial data of a real city, and randomly generating the number of candidate areas and the number of facilities to be deployed corresponding to the urban geographic spatial data; according to the number of candidate areas and the number of facilities to be deployed, using a proximal strategy optimization algorithm to iteratively optimize the urban facility deployment model; when the number of iterative optimization reaches a set number of iterations, stopping the training to obtain a trained urban facility deployment model.

[0016] Furthermore, the step of obtaining the urban area exchange graph corresponding to the initial urban facility deployment plan includes: according to given candidate areas and facilities to be deployed, using a greedy algorithm to solve and obtain the initial urban facility deployment plan.

[0017] In a second aspect, the present invention also provides an urban facility deployment device based on reinforcement learning, comprising: a target urban area exchange graph acquisition module, used to obtain the urban area exchange graph corresponding to the initial urban facility deployment plan, and filter the low-quality edges in the urban area exchange graph whose total deployment cost is higher than the set cost threshold to obtain the target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; a target urban facility deployment plan generation module, used to generate a target urban facility deployment plan according to the target urban area exchange graph based on a pre-trained urban facility deployment model, and perform urban facility deployment according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0018] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the urban facility deployment method based on reinforcement learning as described in any one of the above is implemented.

[0019] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described reinforcement learning-based urban facility deployment methods.

[0020] The urban facility deployment method based on reinforcement learning provided by the present invention obtains a target urban area exchange graph by obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold, wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; then based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost. This method filters out low-quality edges in the urban area exchange graph and directs the algorithm's attention to high-quality exchange operations, thereby improving the accuracy of the target city facility deployment plan. At the same time, urban facility deployment is achieved through fast reasoning-driven exchange operations through graph neural networks and policy networks, rather than local exhaustive search, avoiding heavy calculations and achieving rapid generation of target city facility deployment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the urban facility deployment method based on reinforcement learning provided in an embodiment of the present invention.

[0023] Figure 2 Schematic diagram of two urban facility deployment problems provided by embodiments of the present invention.

[0024] Figure 3 It is a schematic diagram of SWAP operation provided by an embodiment of the present invention.

[0025] Figure 4 It is a schematic diagram of training optimization of the urban facility deployment model provided by an embodiment of the present invention.

[0026] Figure 5 It is a schematic diagram of the overall framework of the urban facility deployment method based on reinforcement learning provided in an embodiment of the present invention.

[0027] Figure 6It is a schematic diagram of the overall process of the urban facility deployment method based on reinforcement learning provided in an embodiment of the present invention.

[0028] Figure 7 It is a flow chart of city facility deployment in a city multi-facility deployment scenario provided by an embodiment of the present invention.

[0029] Figure 8 It is a comparison diagram of urban facility deployment results in a city multi-facility deployment scenario provided by an embodiment of the present invention.

[0030] Fig. 9 It is a diagram of the urban facility deployment effect in a single-facility urban deployment scenario provided by an embodiment of the present invention.

[0031] Fig.10 It is a structural diagram of an urban facility deployment device based on reinforcement learning provided in an embodiment of the present invention.

[0032] Fig.11 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be noted that the existing urban facility deployment methods either have a slow solution speed or a low solution accuracy, and it is still impossible to provide an urban facility deployment method that can guarantee both solution speed and solution accuracy.

[0035] In view of this, the present invention proposes a method for urban facility deployment based on reinforcement learning. Specifically, Figure 1 A schematic flow chart of a method for deploying urban facilities based on reinforcement learning provided in an embodiment of the present invention is shown.

[0036] like Figure 1 As shown, the method includes steps S110-S120, and these two steps and related steps will be described in detail below in conjunction with the accompanying drawings.

[0037] S110, obtaining an urban area exchange graph corresponding to the initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes where facilities to be deployed exist and graph-nodes where facilities to be deployed do not exist, and the graph-edges correspond to candidate exchange operations.

[0038] This embodiment aims to solve the problem of urban facility site selection / deployment, especially the problem of large-scale urban facility site selection. The large-scale urban facility site selection problem can be formally described as the following optimization problem (1).

[0039] (1).

[0040] In formula (1), Indicates the facility type number, , Indicates the candidate region number, represents the total number of candidate regions, Represents the candidate region Whether residents go to the candidate area Visit facilities, Represents the candidate region To the candidate area The distance between Represents the candidate region Residents' satisfaction with facilities access needs, Represents the candidate region Whether to deploy facilities , For facility type The total number of , is the facility type number, Indicates the number of types of deployed facilities.

[0041] Correspondingly, Figure 2 A schematic diagram showing two urban facility deployment problems provided by embodiments of the present invention.

[0042] exist Figure 2 middle, Figure 2 The leftmost figure shows the deployment of different urban facilities, where the red and blue dots represent the current deployment of urban facilities 1 and 2 respectively, and the black dots represent candidate areas (residents) that have access needs to urban facilities 1 and 2. Figure 2The second figure from the left shows the access requirements of residents in different candidate areas to urban facilities 1. Figure 2 The third figure from the left shows the access requirements of residents in different candidate areas to urban facilities 2.

[0043] It can be understood that in order to ultimately generate a high-precision target city facility deployment plan, this step first constructs an urban area exchange diagram representing population, residential demand and deployed facilities based on the initial city facility deployment plan.

[0044] Specifically, the initial urban facility deployment plan includes candidate areas and the conditions of currently deployed facilities in the candidate areas (such as the number and type of currently deployed facilities). The candidate areas are determined as graph nodes, and edges are added between graph nodes with facilities to be deployed and graph nodes without facilities to be deployed, i.e., graph edges, so as to construct an urban area exchange graph.

[0045] In this step, the initial urban facility deployment plan can be preliminarily solved by a greedy algorithm, which is not specifically limited here.

[0046] The urban area exchange graph is a SWAP graph. The selection of graph edges in the SWAP graph is equivalent to / corresponds to the selection of SWAP operations (exchange operations, also known as candidate exchange operations). The urban facility site selection / deployment problem to be solved by the present invention can be solved by improving and optimizing the initial urban facility deployment plan through continuous SWAP operations to obtain the final target plan, that is, the target urban facility deployment plan.

[0047] The SWAP graph can be expressed as follows:

[0048] (2).

[0049] In formula (2), represents the SWAP graph, Represents a graph - a collection of nodes, represents a graph - a set of edges, represents a graph-edge in a graph-edge set, Represents an indicator function (1 if the condition in {} is true, otherwise 0), Indicates time Next candidate area Whether to deploy facilities .

[0050] It should be noted here that large cities usually contain thousands of candidate areas and require hundreds of facilities. Generating a city facility deployment plan from scratch requires repeated calculations and takes a lot of time. In order to make full use of the solutions that have been obtained, this embodiment refines the SWAP operation based on the idea of ​​local search and generates the optimal solution (target city facility deployment plan) by fine-tuning the current solution (initial city facility deployment plan). For details, please refer to Figure 3 , Figure 3 A schematic diagram of SWAP operation provided by an embodiment of the present invention is shown.

[0051] On the one hand, by utilizing the idea of ​​local search, the SWAP operation can fine-tune the current solution and move closer to a better solution through continuous iterative adjustments, rather than generating a solution from scratch. This local search strategy helps to quickly find a better solution in the solution space. On the other hand, the SWAP operation can unify single facility deployment and urban multi-facility deployment, and has better versatility.

[0052] For the SWAP operation, specifically, each SWAP operation selects two deployment positions, namely the insertion deployment position and the removal deployment position, and moves the facilities to be deployed at the removal deployment position to the insertion deployment position, which can be specifically expressed as follows (3)-(6): (3).

[0053] (4).

[0054] (5).

[0055] (6).

[0056] In equations (3)-(6), Indicates time facility Deployment plan, Represents the candidate region Are there any facilities deployed? (If deployed, The value is 1; otherwise, The value is 0), Indicates time actions (including removal and insertion), Indicates the insertion node (that is, the insertion deployment location mentioned above), Indicates the removal of a node (that is, the removal of a deployment location mentioned above).

[0057] The SWAP operation provides an efficient and general method for searching the huge solution space of FLP. Through continuous SWAP operations, the initial solution can be iteratively adjusted towards a solution with better performance until the optimal solution is reached.

[0058] The purpose of the SWAP operation is to minimize the total access cost and satisfy the incompatibility constraints. The iterative solution process of FLP using it can be summarized as a Markov decision process containing elements such as state, action, transition and reward.

[0059] Among them, the state summarizes the information of the topological structure and facility layout of the urban area; the action represents selecting an edge in the SWAP graph and exchanging the corresponding facility location; the transition represents using the action Change the facility location from Change to ; Reward is the reduction in access cost defined by the following formula (7).

[0060] (7).

[0061] In formula (7), represents the access cost, Represents the candidate region Whether residents go to the candidate area Visit facilities , Represents the candidate region To the candidate area The distance between Represents the candidate region Residents' satisfaction with facilities access needs.

[0062] However, not all graph-edges on the SWAP graph can reduce the access cost, especially when the urban facility deployment scheme is close to optimal or trapped in a local optimum, only a few SWAP operations can achieve the goal.

[0063] Taking this into consideration, in step S110, after obtaining the initial city area exchange graph (i.e., SWAP graph), some low-quality SWAP operations (low-quality edges) will be filtered. Specifically, this embodiment uses topological dynamic edge connection technology to preliminarily evaluate the exploration value of each SWAP operation to filter out some graph-edges that contribute less to the optimal solution, thereby obtaining the target city area exchange graph.

[0064] Among them, a small contribution to the optimal solution means that the total deployment cost corresponding to the graph-edge is higher than the set cost threshold. The set cost threshold can be set according to actual conditions and is not specifically limited here.

[0065] The total deployment cost refers to the total access cost of residents in the candidate area with access needs to the facilities to be deployed after executing the candidate exchange operation / candidate SWAP operation, that is, .

[0066] It should be noted that the process of filtering low-quality edges whose total deployment cost in the urban area exchange graph / SWAP graph is higher than the set cost threshold will be elaborated in detail in the embodiments below and is not specifically limited here.

[0067] S120, based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in a candidate exchange operation, and the target urban facility deployment plan corresponds to a minimum total deployment cost.

[0068] It can be understood that, on the basis of filtering out low-quality edges in the urban area exchange graph whose total deployment cost is higher than the set cost threshold in step S110 to obtain the target urban area exchange graph, further, a target urban facility deployment plan is generated based on the pre-trained urban facility deployment model, and urban facilities are deployed according to the target urban facility deployment plan.

[0069] Specifically, the pre-trained urban facility deployment model includes a graph neural network and a policy network, wherein the graph neural network is used to extract and encode features of graph edges in the target urban area exchange graph, and the policy network is used to score each valid graph edge in the target urban area exchange graph according to the output of the graph neural network, so as to select the most suitable candidate exchange operation as the target exchange operation from multiple candidate exchange operations corresponding to multiple graph edges in the target urban area exchange graph, thereby completing the deployment of the facilities to be deployed by executing the target exchange operation.

[0070] The process of executing the target exchange operation can be seen in the above formulas (3)-(6). The target exchange operation is selected from the candidate exchange operations, and the candidate exchange operations correspond to the graph-edges in the target city area exchange graph. Therefore, the two graph-nodes of the graph-edge corresponding to the target exchange operation are the insertion node and the removal node respectively.

[0071] It should be noted that, whether it is the initial city-region exchange graph or the target city-region exchange graph, their graph-nodes represent candidate regions, and their graph-edges correspond to candidate exchange operations (candidate SWAP operations).

[0072] It should also be noted that the urban facility deployment method based on reinforcement learning provided in this embodiment is executed for each type of facility to be deployed each time, thereby completing the deployment of multiple facilities to be deployed.

[0073] In this embodiment, a target city area exchange graph is obtained by obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment costs are higher than a set cost threshold; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; then, based on a pre-trained urban facility deployment model, a target city facility deployment plan is generated according to the target city area exchange graph, and urban facilities are deployed according to the target city facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost. This method filters out low-quality edges in the urban area exchange graph and directs the algorithm's attention to high-quality exchange operations, thereby improving the accuracy of the target city facility deployment plan. At the same time, urban facility deployment is achieved through fast reasoning-driven exchange operations through graph neural networks and policy networks, rather than local exhaustive search, avoiding heavy calculations and achieving rapid generation of target city facility deployment plans.

[0074] On the basis of the above-mentioned embodiment, the process of filtering low-quality edges in the urban area exchange graph will be described in detail below.

[0075] Low-quality edges include historically visited edges and negative-benefit edges; accordingly, the step of filtering out low-quality edges whose total deployment costs in the urban area exchange graph are higher than a set threshold specifically includes: obtaining the historical graph-nodes visited by the historical exchange operation, and determining the graph-edges containing the historical graph-nodes as historically visited edges; obtaining the actual distance between the removed node and the inserted node corresponding to the candidate exchange operation, and calculating the actual benefit and actual loss corresponding to the candidate exchange operation; when the actual distance is greater than the set distance threshold and the actual benefit is lower than the actual loss, determining the graph-edge corresponding to the candidate exchange operation as a negative-benefit edge; filtering the historically visited edges and negative-benefit edges in the urban area graph to obtain the target urban area exchange graph.

[0076] It is understandable that the graph-edges in the initial urban area exchange graph / SWAP graph correspond to candidate exchange operations / candidate SWAP operations, but not all graph-edges on the SWAP graph can make the urban facility deployment plan closer to the optimal solution, especially when the urban facility deployment plan is close to the optimal solution or falls into the local optimal solution, only a few graph-edges corresponding to the candidate SWAP operations can achieve the goal. Therefore, accurately filtering out low-quality edges is crucial to improving the quality of urban facility deployment plans, but current machine learning methods ignore this point, resulting in low accuracy.

[0077] In this embodiment, a topological dynamic edge connection technology is used to increase the probability of sampling high-quality SWAP operations and improve the search speed. Specifically, in this embodiment, the topological dynamic edge connection technology is used to filter low-quality edges that contribute less to the optimal solution, including but not limited to historical access edges (hereinafter referred to as Tabu edges) and negative benefit edges (Negative BenefitEdges).

[0078] Tabu edges come from Tabu Search, which is a widely used heuristic algorithm. The graph nodes visited by the historical swap operation / historical SWAP operation will be recorded as historical graph nodes / Tabu nodes, and the graph edges containing Tabu nodes will be regarded as Tabu edges. Tabu edges can be expressed by the following equations (8)-(9).

[0079] (8).

[0080] (9).

[0081] In equations (8)-(9), Represents a set of Tabu nodes. , Indicates the inserted nodes and removed nodes visited by the historical SWAP operation. represents a Tabu edge. The parameters here are Controlling the number of tabu nodes and filtering tabu edges can help the algorithm escape from the local optimal state and avoid repeated cycles.

[0082] Negative benefit edges represent candidate SWAP operations with significant negative benefits. After that, the change of total access cost / total deployment cost can be divided into three parts: Gains when placing facilities; only from removing nodes Losses when removing a facility; additional items caused by placing and removing it simultaneously.

[0083] It is worth noting that if the candidate SWAP operation corresponds to the insertion node and remove nodes If the distance between the inserted node and the removed node is far enough, the additional term is always equal to zero. Therefore, if the actual distance between the inserted node and the removed node is greater than the set distance threshold, and the actual benefit brought by the corresponding candidate SWAP operation is lower than the actual loss, then executing the candidate SWAP operation will increase the total access cost. This implementation determines the graph-edge corresponding to this candidate SWAP operation as a negative benefit edge.

[0084] Specifically, the negative return edge can be described by the following equation (10).

[0085] (10).

[0086] In formula (10), represents a negative return edge, Represents the actual distance between the inserted node and the removed node, To set the distance threshold, Indicates the actual benefit of inserting a node, Represents the actual loss caused by removing the node.

[0087] Filtering the negative benefit edges in the graph-edges will direct the algorithm's attention to the candidate SWAP operations that improve the urban facility deployment plan, thereby significantly narrowing the space of urban facility deployment plans to high-quality areas. Based on the above, the topological dynamic edge connection technology highlights the high-quality SWAP operations of urban facility deployment and ensures the accuracy of the obtained urban facility deployment plan.

[0088] In this embodiment, by obtaining the historical graph-nodes accessed by the historical exchange operation, the graph-edges containing the historical graph-nodes are determined as historical access edges, and the actual distance between the removed node and the inserted node corresponding to the candidate exchange operation is obtained, and the actual benefit and actual loss corresponding to the candidate exchange operation are calculated. Then, when the actual distance is greater than the set distance threshold and the actual benefit is lower than the actual loss, the graph-edge corresponding to the candidate exchange operation is determined as a negative benefit edge, thereby filtering the historical access edges and negative benefit edges in the urban area exchange graph to obtain the target urban area exchange graph. This method effectively improves the accuracy of the target city facility deployment plan by filtering the low-quality edges in the urban area exchange graph and directing the algorithm's attention to high-quality exchange operations.

[0089] On the basis of the above-mentioned embodiments, the process of generating a target city facility deployment plan using a graph neural network and a policy network will be described in detail below.

[0090] Based on a pre-trained urban facility deployment model, a target city facility deployment plan is generated according to a target city area exchange graph, including: based on a pre-trained graph neural network, obtaining exchange edge embedding according to the target city area exchange graph; based on a pre-trained strategy network, generating a target city facility deployment plan according to the exchange edge embedding.

[0091] It can be understood that this embodiment uses a graph neural network and a policy network. The graph neural network is used to obtain exchange edge embeddings based on the target city exchange graph, and the policy network is used to generate a target city facility deployment plan based on the exchange edge embeddings.

[0092] In a specific embodiment, a process of obtaining exchange edge embeddings using a graph neural network is described in detail.

[0093] Based on a pre-trained graph neural network, an exchange edge embedding is obtained according to a target city area exchange graph, including: obtaining node features corresponding to each graph-node in the target city area exchange graph, and encoding the node features into initial node embeddings; updating the initial node embeddings to obtain target node embeddings; fusing the target node embeddings corresponding to the candidate exchange operations with the virtual node embeddings corresponding to the virtual nodes to obtain exchange edge embeddings; wherein the virtual nodes are connected to all graph-nodes in the target city area exchange graph.

[0094] Specifically, the traditional heuristic method performs a local exhaustive search for the SWAP operation, which cannot be extended to the actual FLP. This embodiment designs a model based on a graph neural network and a policy network to perform the SWAP operation, thereby avoiding time-consuming local search.

[0095] Effective graph-edge selection should consider both the global distribution of urban areas and facilities, as well as the local areas where residents access facilities. Therefore, the graph neural network in this embodiment considers both the topological relationship and facility distribution of candidate areas to learn the embedding of high-quality edges on the target SWAP graph / target urban area exchange graph.

[0096] Specifically, first, for each graph-node in the target city area exchange graph, rich node features are obtained, including but not limited to the coordinates of the candidate area represented by the node, access requirements, distance to facilities, etc. Then, these node features are encoded into graph-node embedding, that is, initial node embedding, as shown in the following formula (11).

[0097] (11).

[0098] In formula (11), represents the initial node embedding, represents the learnable weights, Represents the original node features.

[0099] Then, the rich global information and neighbor information are combined to calculate the representation of the graph-node. Through the linear transformation layer and nonlinear activation function, the neighbor information is summarized and the initial node embedding is updated, as shown in the following formula (12).

[0100] (12).

[0101] In formula (12), Represents the graph neural network The target node embedding of the layer output, Represents the graph neural network The node embeddings of the layer output, represents the learnable weights, Represents the graph neural network Neighbor node embeddings output by the layer.

[0102] In addition, this embodiment also adds a virtual node connecting all graph nodes to collect global information of the target city area exchange graph. For details, please refer to the virtual node embedding described in the following formula (13).

[0103] (13).

[0104] In formula (13), Represents the graph neural network The virtual node embedding of the layer output, Represents the graph neural network The virtual node embedding of the layer output, , Represents a graph-node, represents the learnable weights, Represents the graph neural network Neighbor node embeddings output by the layer.

[0105] Next, the target node embedding corresponding to the candidate swap operation and the virtual node embedding corresponding to the virtual node are fused to obtain the representation of the edge, i.e., the swap edge embedding. For details, see the following formula (14).

[0106] (14).

[0107] In formula (14), represents the exchange edge embedding, , They represent the target node embeddings of the two graph-nodes corresponding to a graph-edge of a candidate swap operation, Represents an embedded concatenation.

[0108] Through propagation and aggregation operations, the obtained swap edge embedding can not only perceive the global distribution information of candidate areas and facilities, but also capture the local relations between urban areas, supporting the subsequent target SWAP operation selection.

[0109] In another specific embodiment, a process of generating a target city facility deployment plan using a policy network is described in detail.

[0110] The policy network includes a multi-layer perceptron; accordingly, based on the pre-trained policy network, a target city facility deployment plan is generated according to the exchange edge embedding, including: scoring each graph-edge in the target city area exchange graph through the multi-layer perceptron according to the exchange edge embedding, and obtaining the operation score corresponding to the candidate exchange operation; determining the candidate exchange operation corresponding to the highest operation score as the target exchange operation, and obtaining the target city facility deployment plan.

[0111] Specifically, this embodiment adopts a policy network to select the graph-edge of each SWAP operation. The policy network embeds the swap edge output by the graph neural network as input and selects the target SWAP operation that can effectively reduce the access cost.

[0112] First, the policy network scores each valid graph-edge based on the exchange edge embedding through a multi-layer perceptron (MLP), as shown in the following equations (15)-(16).

[0113] (15).

[0114] (16).

[0115] In formulas (15)-(16), represents a valid graph-edge, that is, a graph-edge in the target city-region exchange graph, represents the historical access edge, represents a negative return edge, Represents the scores of different valid graphs-edges output by the policy network, that is, the operation scores corresponding to the candidate exchange operations. It ensures that historical access edges and negative benefit edges will not be selected.

[0116] With the help of rich prior knowledge in the swap edge embedding, the policy network selects the target swap operation / target SWAP operation according to the calculated operation score. Specifically, the candidate swap operation corresponding to the highest operation score is used as the target swap operation.

[0117] Then, the city facility deployment is completed by performing a target exchange operation.

[0118] In this embodiment, based on the pre-trained graph neural network, the exchange edge embedding is obtained according to the target city area exchange graph, and based on the pre-trained policy network, the target city facility deployment plan is generated according to the exchange edge embedding, and then the city facilities are deployed according to the target city facility deployment plan. This method realizes the deployment of city facilities by fast reasoning-driven exchange operations of the graph neural network and the policy network, rather than local exhaustive search, avoiding heavy calculations and realizing the rapid generation of the target city facility deployment plan.

[0119] On the basis of the above-mentioned embodiments, the training optimization of the urban facility deployment model will be further described in detail below.

[0120] Training and optimizing the urban facility deployment model specifically includes: obtaining urban geographic spatial data of a real city, and randomly generating the number of candidate areas and the number of facilities to be deployed corresponding to the urban geographic spatial data; according to the number of candidate areas and the number of facilities to be deployed, using a proximal strategy optimization algorithm to iteratively optimize the urban facility deployment model; when the number of iterative optimization reaches the set number of iterations, stopping the training to obtain a trained urban facility deployment model.

[0121] The number of iterations can be set according to actual needs and is not specifically limited here.

[0122] In a specific embodiment, the urban geospatial data of real New York City is used to train the urban facility deployment model to solve the FLP.

[0123] Figure 4 A schematic diagram of training optimization of the urban facility deployment model provided by an embodiment of the present invention is shown.

[0124] like Figure 4 As shown in the figure, this embodiment randomly generates a new FLP instance for each round of urban facility deployment model training. Specifically, at the beginning of each round of training, we first randomly generate the number of candidate regions N and the number of facilities to be deployed p. Then, a region (CBG) and its nearest N-1 CBGs are extracted as candidate regions. Finally, considering the residents' needs for the deployed facilities, the algorithm performs a SWAP operation to improve the initial urban facility deployment plan obtained by the greedy algorithm for a specific FLP instance.

[0125] In this embodiment, Proximal Policy Optimization is used to train and optimize the urban facility deployment model.

[0126] It is worth noting that during the model training process, the algorithm provided in this embodiment will attempt to solve thousands of different synthetic FLP instances, thereby generating a universal solver (i.e., the urban facility deployment model) that can handle any input FLP.

[0127] On the basis of the above embodiment, further, before obtaining the urban area exchange graph corresponding to the initial urban facility deployment plan, it includes: according to the given candidate areas and facilities to be deployed, using a greedy algorithm to solve and obtain the initial urban facility deployment plan.

[0128] Specifically, when using the greedy algorithm to solve the problem, the input can be the distance between candidate areas and the number of facilities to be deployed, or the number of candidate areas, the distance between candidate areas, and the access requirements of residents in candidate areas for the facilities to be deployed. The output is an initial urban facility deployment plan, which includes the current deployment status of the facilities to be deployed in each candidate area.

[0129] A greedy algorithm is a strategy that takes the locally optimal choice at each step, hoping that such a choice will lead to a globally optimal solution. This algorithm does not always lead to a globally optimal solution, but it usually finds a reasonably good solution quickly.

[0130] Since the greedy algorithm is already a relatively mature technology in the prior art, it will not be described in detail here.

[0131] In this embodiment, a greedy algorithm is used to solve the initial urban facility deployment plan based on the given candidate areas and facilities to be deployed, and then based on the initial urban facility deployment plan, the SWAP operation is used to improve and optimize it to obtain the target urban facility deployment plan, rather than generating the final plan from scratch. This can help to quickly find a better solution in the solution space.

[0132] In some embodiments, Figure 5 A schematic diagram of the overall framework of the urban facility deployment method based on reinforcement learning provided in an embodiment of the present invention is shown.

[0133] like Figure 5 As shown, this embodiment adopts a "divide and conquer" framework and uses a unified reinforcement learning algorithm based on SWAP to handle the FLP problem.

[0134] The algorithm consists of two consecutive stages. In stage 1, the initial urban facility deployment plan obtained by the greedy algorithm is improved through the SWAP operation, and the final deployment location is independently searched for each type of facility to be deployed.

[0135] Since the independent solution may conflict with the incompatible constraints of the urban area, in stage 2, the conflict is eliminated through SWAP operation to further improve the independent solution until all incompatible constraint conflicts are resolved, and the final solution of FLP is obtained.

[0136] The incompatible constraints here mainly refer to the restrictions on the number and type of facilities to be deployed in the candidate area. For example, it is required that the number of a certain type of facility in the candidate area can only be 1.

[0137] Among them, for the selection of SWAP operation, it is first necessary to construct the SWAP graph corresponding to the initial urban facility deployment plan, and filter the Tabu edges and negative benefit edges in the SWAP graph to obtain the target SWAP graph.

[0138] Furthermore, based on the target SWAP graph, the graph neural network is used to encode the swap edge embeddings, and the MLP in the policy network is used to score each swap edge embedding, and then the appropriate SWAP operation, that is, the target swap operation, is selected.

[0139] Finally, the target exchange operation is performed to complete the deployment of the facilities to be deployed in the candidate area.

[0140] at the same time, Figure 6 The overall process diagram of the urban facility deployment method based on reinforcement learning provided by an embodiment of the present invention is shown.

[0141] like Figure 6 As shown in the figure, first, an urban area exchange graph (SWAP graph) corresponding to the initial urban facility deployment plan is constructed, and then the topological dynamic edge connection technology is used to filter the Tabu edges and negative benefit edges in the SWAP graph to obtain a high-quality solution space SWAP graph, that is, the target urban area exchange graph or target SWAP graph.

[0142] Furthermore, using graph neural networks (i.e. Figure 6 ), obtain the swap edge embedding corresponding to each graph-edge in the target SWAP graph.

[0143] Next, on the one hand, the exchange edge embedding is input into the policy network, and the policy network scores each graph-edge in the target SWAP graph, and then selects the target SWAP operation based on the scoring result. On the other hand, the exchange edge embedding can be input into the value network, and the value network will calculate the total deployment cost / total access cost corresponding to each graph-edge in the target SWAP graph. To calculate, and use this to determine the effect of the corresponding SWAP operation.

[0144] In other embodiments, the urban facility deployment method based on reinforcement learning provided by the present invention is used in a multi-facility deployment scenario in an urban area, and corresponding technical effects are given.

[0145] Specifically, for the multi-facility deployment scenario in the city, similarly, the target city facility deployment solution is calculated for each type of facility through the divide-and-conquer framework, and then the same city facility deployment model is used to resolve conflicts between different types of facilities, all of which are completed through a unified SWAP operation. For details, see Figure 7 , Figure 7 A flow chart of city facility deployment in a city multi-facility deployment scenario provided by an embodiment of the present invention is shown.

[0146] at the same time, Figure 8 A comparison diagram of urban facility deployment results in an urban multi-facility deployment scenario provided by an embodiment of the present invention is also shown.

[0147] exist Figure 8 The comparison object includes the urban facility deployment method based on reinforcement learning provided by the present invention (i.e. Figure 8 DRL-GNN in ), as well as existing urban facility deployment methods such as GA, TS, LS, VNS, CardNN, Gurobi (120s), and Gurobi (optimal).

[0148] Applicable cities include New York, Boston, Los Angeles and Chicago.

[0149] The comparison indicators are AC (Actual Cost) and RT (Response-time).

[0150] according to Figure 8 It can be seen that the urban facility deployment method based on reinforcement learning provided by the present invention can generate an urban facility deployment plan that is close to the solver accuracy at an extremely fast solving speed.

[0151] In some other embodiments, the urban facility deployment method based on reinforcement learning provided by the present invention is used in a single urban facility deployment scenario, and corresponding technical effects are given.

[0152] Specifically, for the single-facility deployment scenario in a city, the city facility deployment model can be used directly for fast reasoning, and the target city facility deployment plan can be obtained after continuous SWAP operations. Different from the greedy algorithm used to construct the initial deployment plan during model training, here we use the current city deployment plan as the initial city facility deployment plan.

[0153] Taking the redeployment of a single hospital facility in Brooklyn, New York as an example, the present invention can effectively improve the accessibility of facility services by about 39%. Fig. 9 , Fig. 9 A diagram showing the deployment effect of urban facilities in a single-facility urban deployment scenario provided by an embodiment of the present invention is shown.

[0154] In addition, it is worth mentioning that the urban facility deployment method based on reinforcement learning provided in the embodiment of the present invention has the following three advantages.

[0155] First, the deployment of a single facility and the deployment of multiple facilities in a city can be completed through a unified SWAP operation, killing two birds with one stone. The urban facility deployment model has strong versatility.

[0156] Secondly, the urban facility deployment model that combines reinforcement learning (policy network) with graph neural network has fast forward reasoning speed and high problem-solving efficiency. The model can quickly solve large-scale urban facility deployment problems in real scenarios.

[0157] Third, the introduction of prior knowledge of the FLP problem has significantly higher solution accuracy than the existing machine learning-based methods, and the performance is close to the solution obtained by the solver.

[0158] Corresponding to the urban facility deployment method based on reinforcement learning described in the above embodiments, the present invention also provides an urban facility deployment device based on reinforcement learning, specifically, Fig.10 A schematic structural diagram of a reinforcement learning-based urban facility deployment device provided in an embodiment of the present invention is shown.

[0159] like Fig.10 As shown, the device includes: a target city area exchange graph acquisition module 1010, which is used to obtain the city area exchange graph corresponding to the initial city facility deployment plan, and filter the low-quality edges in the city area exchange graph whose total deployment cost is higher than the set cost threshold to obtain the target city area exchange graph; wherein the city area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; a target city facility deployment plan generation module 1020, which is used to generate a target city facility deployment plan according to the target city area exchange graph based on a pre-trained city facility deployment model, and perform city facility deployment according to the target city facility deployment plan; wherein the city facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost.

[0160] In this embodiment, the city area exchange graph corresponding to the initial city facility deployment plan is obtained through the target city area exchange graph acquisition module 1010, and the low-quality edges in the city area exchange graph whose total deployment cost is higher than the set cost threshold are filtered to obtain the target city area exchange graph; wherein the city area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize the candidate areas, the graph-edges are used to connect the graph-nodes with the facilities to be deployed and the graph-nodes without the facilities to be deployed, and the graph-edges correspond to the candidate exchange operations; the target city facility deployment plan is generated according to the target city area exchange graph based on the pre-trained city facility deployment model through the target city facility deployment plan generation module 1020, and the city facilities are deployed according to the target city facility deployment plan; wherein the city facility deployment model is constructed based on the graph neural network and the policy network, the target city facility deployment plan includes the target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost. The device filters out low-quality edges in the urban area exchange graph and directs the algorithm's attention to high-quality exchange operations, thereby improving the accuracy of the target city facility deployment plan. At the same time, urban facility deployment is achieved through fast reasoning-driven exchange operations through graph neural networks and policy networks, rather than local exhaustive search, avoiding heavy calculations and achieving rapid generation of target city facility deployment plans.

[0161] It should be noted that the urban facility deployment device based on reinforcement learning provided in this embodiment can be referenced to each other with the urban facility deployment method described in the above embodiment, and will not be described in detail here.

[0162] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11As shown, the electronic device may include: a processor (processor) 1110 , a communication interface (Communications Interface) 1120 , a memory (memory) 1130 and a communication bus 1140 , wherein the processor 1110 , the communication interface 1120 , and the memory 1130 communicate with each other via the communication bus 1140 . The processor 1110 can call the logic instructions in the memory 1130 to execute the urban facility deployment method based on reinforcement learning, which includes: obtaining an urban area exchange graph corresponding to the initial urban facility deployment plan, and filtering the low-quality edges in the urban area exchange graph whose total deployment cost is higher than the set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0163] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0164] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the urban facility deployment method based on reinforcement learning provided by the above-mentioned methods, the method comprising: obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph comprises graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes where facilities to be deployed exist and graph-nodes where facilities to be deployed do not exist, and the graph-edges correspond to candidate exchange operations; based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0165] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for urban facility deployment based on reinforcement learning, characterized in that: include: Obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges whose total deployment costs are higher than a set cost threshold in the urban area exchange graph to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes where facilities to be deployed exist and graph-nodes where facilities to be deployed do not exist, and the graph-edges correspond to candidate exchange operations; Based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in a candidate exchange operation, and the target urban facility deployment plan corresponds to a minimum total deployment cost.

2. The urban facility deployment method based on reinforcement learning according to claim 1 is characterized in that: The low-quality edges include historical access edges and negative benefit edges; Accordingly, the step of filtering low-quality edges whose total deployment cost in the urban area exchange graph is higher than a set threshold specifically includes: Obtaining a historical graph-node accessed by the historical exchange operation, and determining a graph-edge containing the historical graph-node as a historical access edge; Obtaining the actual distance between the removed node and the inserted node corresponding to the candidate exchange operation, and calculating the actual gain and actual loss corresponding to the candidate exchange operation; When the actual distance is greater than the set distance threshold and the actual benefit is lower than the actual loss, determining the graph edge corresponding to the candidate exchange operation as a negative benefit edge; The historical visit edges and negative benefit edges in the city area exchange graph are filtered to obtain the target city area exchange graph.

3. The urban facility deployment method based on reinforcement learning according to claim 1 is characterized in that: The method of generating a target city facility deployment plan based on the pre-trained city facility deployment model and the target city area exchange map comprises: Based on the pre-trained graph neural network, obtaining the exchange edge embedding according to the target city area exchange graph; Based on the pre-trained policy network, a target city facility deployment plan is generated according to the exchange edge embedding.

4. The urban facility deployment method based on reinforcement learning according to claim 3 is characterized in that: The pre-trained graph neural network based on the target city area exchange graph obtains the exchange edge embedding, including: Obtain node features corresponding to each graph-node in the target city area exchange graph, and encode the node features into initial node embeddings; Updating the initial node embedding to obtain a target node embedding; Merging the target node embedding corresponding to the candidate exchange operation with the virtual node embedding corresponding to the virtual node to obtain the exchange edge embedding; Wherein, the virtual node is connected with all graph-nodes in the target city area exchange graph.

5. The urban facility deployment method based on reinforcement learning according to claim 3 is characterized in that: The policy network includes a multi-layer perceptron; Accordingly, the pre-trained strategy network generates a target city facility deployment plan based on the exchange edge embedding, including: According to the exchange edge embedding, scoring each graph-edge in the target city area exchange graph by a multi-layer perceptron to obtain an operation score corresponding to the candidate exchange operation; The candidate exchange operation corresponding to the highest operation score is determined as the target exchange operation, and the target city facility deployment plan is obtained.

6. The urban facility deployment method based on reinforcement learning according to any one of claims 1 to 5, characterized in that: Training and optimizing the urban facility deployment model, specifically including: Obtaining urban geographic spatial data of a real city, and randomly generating the number of candidate areas and the number of facilities to be deployed corresponding to the urban geographic spatial data; According to the number of candidate areas and the number of facilities to be deployed, the urban facility deployment model is iteratively optimized using a proximal strategy optimization algorithm; When the number of iterative optimization reaches the set number of iterations, the training is stopped to obtain the trained urban facility deployment model.

7. The urban facility deployment method based on reinforcement learning according to any one of claims 1 to 5, characterized in that: The step of obtaining the urban area exchange graph corresponding to the initial urban facility deployment plan includes: According to the given candidate areas and facilities to be deployed, a greedy algorithm is used to solve and obtain the initial urban facility deployment plan.

8. An urban facility deployment device based on reinforcement learning, characterized in that: include: A target city area exchange graph acquisition module is used to acquire the city area exchange graph corresponding to the initial city facility deployment plan, and filter the low-quality edges in the city area exchange graph whose total deployment cost is higher than the set cost threshold to obtain the target city area exchange graph; wherein the city area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent the candidate areas, the graph-edges are used to connect the graph-nodes with the facilities to be deployed and the graph-nodes without the facilities to be deployed, and the graph-edges correspond to the candidate exchange operations; A target city facility deployment plan generation module is used to generate a target city facility deployment plan based on a pre-trained city facility deployment model and the target city area exchange graph, and to deploy city facilities according to the target city facility deployment plan; wherein the city facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in a candidate exchange operation, and the target city facility deployment plan corresponds to a minimum total deployment cost.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the urban facility deployment method based on reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the urban facility deployment method based on reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images