Urban facility deployment method and device based on reinforcement learning and electronic equipment

Abstract

The application relates to the technical field of urban planning, and provides a city facility deployment method and device based on reinforcement learning and electronic equipment, wherein the method comprises the following steps: acquiring a city area exchange graph corresponding to an initial city facility deployment scheme, and filtering low-quality edges with total deployment costs higher than a set cost threshold in the city area exchange graph to obtain a target city area exchange graph; based on a pre-trained city facility deployment model, a target city facility deployment scheme is generated according to the target city area exchange graph, and city facility deployment is performed according to the target city facility deployment scheme. According to the method, the low-quality edges in the city area exchange graph are filtered, the attention of the algorithm is guided to high-quality exchange operations, the precision of the target city facility deployment scheme is improved, the exchange operations are driven by the rapid reasoning of the graph neural network and the policy network to realize the city facility deployment, the heavy calculation is avoided, and the rapid generation of the target city facility deployment scheme is realized.

Description

Technical Field

[0001] The present invention relates to the field of urban planning technology, and in particular to a method, device and electronic equipment for deploying urban facilities based on reinforcement learning. Background Art

[0002] The spatial distribution of urban facilities profoundly influences the mobility patterns of billions of residents seeking basic public services within cities, such as schools and hospitals. However, in real cities, the layout of facilities often does not match residents' demand for urban services, resulting in high travel costs and a significant gap between actual layouts and theoretical optimal layouts. This suboptimal facility layout significantly undermines the efficiency of urban operations, leading to excessive carbon emissions and severe traffic congestion, and reducing the accessibility and equity of citizens seeking basic services. Therefore, optimizing accessibility through the coordinated layout of urban facilities is crucial to addressing these issues and enabling the construction of more sustainable and inclusive cities.

[0003] In fact, the Facility Location Problem (FLP) involves selecting P urban areas from N candidate areas to place facilities. Its complexity itself is NP. It is a typical Composite Optimizer (CO) problem and is extremely challenging.

[0004] In real urban scenarios, N and P are usually very large, making it difficult to design a reliable algorithm that can provide a solution that meets the accuracy requirements within a reasonable time frame. The solution space for this problem is huge. For example, when (N, P) = (1000, 300), the solution space exceeds 10 263 At the same time, as a real-world optimization problem, there is valuable prior knowledge that needs to be explicitly modeled, including the strategies adopted in existing heuristic algorithms, which should provide a reference for algorithm design.

[0005] Furthermore, unlike the standard FLP setting, there are many different types of urban facilities, each with different residential requirements and a different number of facilities. This introduces additional constraints to the problem, such as the number of facilities that each urban area can accommodate, making the task more complex.

[0006] FLP has traditionally been solved using heuristic and metaheuristic methods, such as tabu search and variable neighborhood search (VNS). It can also be formulated as a mixed-integer optimization problem, making it amenable to commercial solvers such as Gurobi and MOSEK. While these methods provide near-optimal solutions, their inherently computationally intensive local search makes them suitable only for small-scale problems. When applied to large-scale, real-world scenarios, their solution speed is often slow, sometimes failing to produce a feasible solution within days, making them difficult to implement in practice.

[0007] In recent years, some machine learning models have been proposed to solve FLP. Although they have greatly improved computational efficiency, these machine learning models ignore the rich prior knowledge inherent in FLP, resulting in a significant decrease in solution accuracy compared to traditional methods and unsatisfactory accessibility of the final deployment solution, thus limiting their applicability.

[0008] Therefore, how to solve the problem of slow solution speed and low solution accuracy of existing urban facility deployment methods is an important issue that needs to be urgently addressed in the field of urban planning. Summary of the Invention

[0009] The present invention provides a reinforcement learning-based urban facility deployment method, device and electronic equipment, which are used to overcome the defects of the existing urban facility deployment methods, such as slow solution speed and low solution accuracy, to achieve the rapid generation of urban facility deployment plans and ensure the accuracy of urban facility deployment plans.

[0010] On the one hand, the present invention provides a city facility deployment method based on reinforcement learning, comprising: obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation contained in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0011] Furthermore, the low-quality edges include historical access edges and negative benefit edges; accordingly, the step of filtering out low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set threshold specifically includes: obtaining the historical graph-nodes accessed by historical exchange operations, and determining the graph-edges containing the historical graph-nodes as historical access edges; obtaining the actual distance between the removed node and the inserted node corresponding to the candidate exchange operation, and calculating the actual benefit and actual loss corresponding to the candidate exchange operation; when the actual distance is greater than the set distance threshold and the actual benefit is lower than the actual loss, determining the graph-edge corresponding to the candidate exchange operation as a negative benefit edge; filtering the historical access edges and negative benefit edges in the urban area exchange graph to obtain the target urban area exchange graph.

[0012] Furthermore, the pre-trained urban facility deployment model generates a target city facility deployment plan according to the target city area exchange graph, including: based on a pre-trained graph neural network, obtaining exchange edge embedding according to the target city area exchange graph; based on a pre-trained strategy network, generating a target city facility deployment plan according to the exchange edge embedding.

[0013] Furthermore, the pre-trained graph neural network obtains exchange edge embeddings according to the target city area exchange graph, including: obtaining node features corresponding to each graph-node in the target city area exchange graph, and encoding the node features into initial node embeddings; updating the initial node embeddings to obtain target node embeddings; fusing the target node embeddings corresponding to the candidate exchange operations with the virtual node embeddings corresponding to the virtual nodes to obtain the exchange edge embeddings; wherein, the virtual nodes are connected to all graph-nodes in the target city area exchange graph.

[0014] Furthermore, the policy network includes a multilayer perceptron; accordingly, the pre-trained policy network generates a target city facility deployment plan based on the exchange edge embedding, including: scoring each graph-edge in the target city area exchange graph through the multilayer perceptron based on the exchange edge embedding to obtain operation scores corresponding to candidate exchange operations; determining the candidate exchange operation corresponding to the highest operation score as the target exchange operation to obtain the target city facility deployment plan.

[0015] Furthermore, the urban facility deployment model is trained and optimized, specifically including: obtaining urban geographic spatial data of a real city, and randomly generating the number of candidate areas and the number of facilities to be deployed corresponding to the urban geographic spatial data; according to the number of candidate areas and the number of facilities to be deployed, the urban facility deployment model is iteratively optimized using a proximal strategy optimization algorithm; when the number of iterative optimization reaches a set number of iterations, the training is stopped to obtain a trained urban facility deployment model.

[0016] Furthermore, the obtaining of the urban area exchange graph corresponding to the initial urban facility deployment plan includes: using a greedy algorithm to solve the initial urban facility deployment plan based on given candidate areas and facilities to be deployed.

[0017] In a second aspect, the present invention also provides an urban facility deployment device based on reinforcement learning, comprising: a target urban area exchange graph acquisition module, used to obtain an urban area exchange graph corresponding to an initial urban facility deployment plan, and filter out low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein, the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; a target urban facility deployment plan generation module, used to generate a target urban facility deployment plan according to the target urban area exchange graph based on a pre-trained urban facility deployment model, and perform urban facility deployment according to the target urban facility deployment plan; wherein, the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0018] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the urban facility deployment method based on reinforcement learning as described above is implemented.

[0019] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described urban facility deployment methods based on reinforcement learning.

[0020] The reinforcement learning-based urban facility deployment method provided by the present invention obtains a target urban area exchange graph by obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan and filtering out low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to characterize candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; then, based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein, the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation contained in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost. This method filters out low-quality edges in the urban area exchange graph and directs the algorithm's attention to high-quality exchange operations, thereby improving the accuracy of the target city facility deployment plan. At the same time, urban facility deployment is achieved through fast reasoning-driven exchange operations using graph neural networks and policy networks, rather than local exhaustive search, avoiding heavy calculations and achieving rapid generation of the target city facility deployment plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flowchart of the urban facility deployment method based on reinforcement learning provided by an embodiment of the present invention.

[0023] Figure 2 Schematic diagram of two urban facility deployment problems provided by embodiments of the present invention.

[0024] Figure 3 This is a schematic diagram of the SWAP operation provided by an embodiment of the present invention.

[0025] Figure 4 2 is a schematic diagram of training optimization of the urban facility deployment model provided by an embodiment of the present invention.

[0026] Figure 5 It is a schematic diagram of the overall framework of the urban facility deployment method based on reinforcement learning provided by an embodiment of the present invention.

[0027] Figure 6It is a schematic diagram of the overall process of the urban facility deployment method based on reinforcement learning provided by an embodiment of the present invention.

[0028] Figure 7 This is a flow chart of urban facility deployment in a multi-facility urban deployment scenario provided by an embodiment of the present invention.

[0029] Figure 8 This is a comparison diagram of urban facility deployment results in a multi-facility urban deployment scenario provided by an embodiment of the present invention.

[0030] Figure 9 This is a diagram of the urban facility deployment effect in a single-facility urban deployment scenario provided by an embodiment of the present invention.

[0031] Figure 10 It is a structural diagram of the urban facility deployment device based on reinforcement learning provided by an embodiment of the present invention.

[0032] Figure 11 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0034] It should be noted that the existing urban facility deployment methods either have a slow solution speed or a low solution accuracy, and it is still impossible to provide an urban facility deployment method that can simultaneously guarantee the solution speed and solution accuracy.

[0035] In view of this, the present invention proposes a method for urban facility deployment based on reinforcement learning. Specifically, Figure 1 A flow chart of a method for deploying urban facilities based on reinforcement learning provided in an embodiment of the present invention is shown.

[0036] like Figure 1 As shown, the method includes steps S110-S120. The following will describe these two steps and related steps in detail with reference to the accompanying drawings.

[0037] S110, obtaining an urban area exchange graph corresponding to the initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein, the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent candidate areas, the graph-edges are used to connect graph-nodes where facilities to be deployed exist and graph-nodes where facilities to be deployed do not exist, and the graph-edges correspond to candidate exchange operations.

[0038] This embodiment aims to solve the problem of urban facility site selection / deployment, especially the problem of large-scale urban facility site selection. The large-scale urban facility site selection problem can be formally described as the following optimization problem (1).

[0039] (1).

[0040] In formula (1), Indicates the facility type number, 、 Indicates the candidate region number, represents the total number of candidate regions, Represents the candidate region Whether residents go to the candidate area Visit facilities, Represents the candidate region To the candidate area The distance between Represents the candidate region Residents' satisfaction with facilities access needs, Represents the candidate region Whether to deploy facilities , For facility type The total number of 、 is the facility type number, Indicates the number of types of deployed facilities.

[0041] Correspondingly, Figure 2 Schematic diagrams showing two urban facility deployment problems provided by embodiments of the present invention.

[0042] exist Figure 2 middle, Figure 2 The leftmost figure shows the deployment of different urban facilities, where the red and blue dots represent the current deployment of urban facilities 1 and urban facilities 2 respectively, and the black dots represent candidate areas (residents) that have access needs to urban facilities 1 and urban facilities 2. Figure 2The second figure from the left shows the access requirements of residents in different candidate areas to urban facilities 1. Figure 2 The third figure from the left shows the access requirements of residents in different candidate areas to urban facilities 2.

[0043] It can be understood that in order to ultimately generate a high-precision target city facility deployment plan, this step first constructs an urban area exchange map representing population, residential demand and deployed facilities based on the initial city facility deployment plan.

[0044] Specifically, the initial urban facility deployment plan includes candidate regions and the current status of deployed facilities in the candidate regions (e.g., the number and type of currently deployed facilities). The candidate regions are identified as graph nodes, and edges (graph edges) are added between graph nodes with facilities to be deployed and graph nodes without facilities to be deployed, thereby constructing an urban region exchange graph.

[0045] In this step, the initial urban facility deployment plan can be preliminarily solved by a greedy algorithm, which is not specifically limited here.

[0046] The urban area exchange graph is a SWAP graph. The selection of graph edges in the SWAP graph is equivalent to / corresponds to the selection of SWAP operations (exchange operations, i.e., candidate exchange operations). The urban facility site selection / deployment problem to be solved by the present invention can be solved by improving and optimizing the initial urban facility deployment plan through continuous SWAP operations to obtain the final target plan, i.e., the target urban facility deployment plan.

[0047] The SWAP graph can be expressed as follows:

[0048] (2).

[0049] In formula (2), represents the SWAP graph, Represents a graph - a collection of nodes, represents a graph - a set of edges, represents a graph-edge in a graph-edge set, Represents an indicator function (1 if the condition in {} is true, otherwise 0), Indicates time Next candidate area Whether to deploy facilities .

[0050] It should be noted that large cities usually contain thousands of candidate areas and require hundreds of facilities. Generating a city facility deployment plan from scratch requires repeated calculations and consumes a lot of time. In order to make full use of the solutions already obtained, this embodiment refines the SWAP operation based on the idea of ​​local search and generates the optimal solution (target city facility deployment plan) by fine-tuning the current solution (initial city facility deployment plan). For details, please refer to Figure 3 , Figure 3 A schematic diagram of SWAP operation provided by an embodiment of the present invention is shown.

[0051] On the one hand, by leveraging the concept of local search, the SWAP operation fine-tunes the current solution, moving iteratively toward a more optimal solution, rather than generating a solution from scratch. This local search strategy helps quickly find optimal solutions within the solution space. On the other hand, the SWAP operation unifies single-facility deployment and multi-facility deployment across a city, offering greater versatility.

[0052] For the SWAP operation, specifically, each SWAP operation selects two deployment locations, namely the insertion deployment location and the removal deployment location, and moves the facilities to be deployed at the removal deployment location to the insertion deployment location. Specifically, it can be expressed as follows (3)-(6):

[0053] (3).

[0054] (4).

[0055] (5).

[0056] (6).

[0057] In formulas (3)-(6), Indicates time facility Deployment plan, Represents the candidate region Are there facilities deployed? (If deployed, then The value is 1; otherwise, The value is 0), Indicates time Actions (including removal and insertion), Indicates the insertion node (also known as the insertion deployment location mentioned above), Indicates removing a node (also referred to as removing a deployment location).

[0058] The SWAP operation provides an efficient and general method for searching the huge solution space of FLP. Through continuous SWAP operations, the initial solution can be iteratively adjusted towards a better solution until the optimal solution is reached.

[0059] The purpose of the SWAP operation is to minimize the total access cost and satisfy the incompatibility constraints. The iterative solution process of FLP using it can be summarized as a Markov decision process containing elements such as state, action, transition and reward.

[0060] Among them, the state summarizes the information of the urban area topology and facility layout; the action represents selecting an edge in the SWAP graph and exchanging the corresponding facility location; the transition represents using the action Change the facility location from Change to ; Reward is the reduction in access cost defined by the following formula (7).

[0061] (7).

[0062] In formula (7), represents the access cost, Represents the candidate region Whether residents go to the candidate area Visit facilities , Represents the candidate region To the candidate area The distance between Represents the candidate region Residents' satisfaction with facilities access needs.

[0063] However, not all graph-edges on the SWAP graph can reduce the access cost, especially when the urban facility deployment plan is close to optimal or trapped in a local optimum, only a few SWAP operations can achieve the goal.

[0064] Taking this into consideration, in step S110, after obtaining the initial urban area exchange graph (i.e., SWAP graph), some low-quality SWAP operations (low-quality edges) will also be filtered. Specifically, this embodiment uses topological dynamic edge connection technology to preliminarily evaluate the exploration value of each SWAP operation to filter out some graph-edges that contribute less to the optimal solution, thereby obtaining the target urban area exchange graph.

[0065] Among them, a small contribution to the optimal solution means that the total deployment cost corresponding to the graph-edge is higher than the set cost threshold. The set cost threshold can be set according to actual conditions and is not specifically limited here.

[0066] The total deployment cost refers to the total access cost of residents in the candidate area with access needs to the facilities to be deployed after executing the candidate exchange operation / candidate SWAP operation, that is, .

[0067] It should be noted that the process of filtering low-quality edges whose total deployment cost in the urban area exchange graph / SWAP graph is higher than the set cost threshold will be elaborated in detail in the embodiments below and is not specifically limited here.

[0068] S120, based on the pre-trained urban facility deployment model, generate a target urban facility deployment plan according to the target urban area exchange graph, and deploy urban facilities according to the target urban facility deployment plan; wherein, the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0069] It can be understood that in step S110, low-quality edges with total deployment costs higher than the set cost threshold in the urban area exchange graph are filtered out to obtain the target urban area exchange graph. Further, a target urban facility deployment plan is generated based on the pre-trained urban facility deployment model, and urban facilities are deployed according to the target urban facility deployment plan.

[0070] Specifically, the pre-trained urban facility deployment model includes a graph neural network and a policy network, wherein the graph neural network is used to extract and encode features of graph edges in the target urban area exchange graph, and the policy network is used to score each valid graph edge in the target urban area exchange graph according to the output of the graph neural network, so as to select the most suitable candidate exchange operation as the target exchange operation from multiple candidate exchange operations corresponding to multiple graph edges in the target urban area exchange graph, thereby completing the deployment of the facilities to be deployed by executing the target exchange operation.

[0071] The process of executing the target exchange operation can be seen in formulas (3)-(6) above. The target exchange operation is selected from the candidate exchange operations, and the candidate exchange operations correspond to the graph-edges in the target city area exchange graph. Therefore, the two graph-nodes of the graph-edge corresponding to the target exchange operation are the insertion node and the removal node, respectively.

[0072] It should be noted that, whether it is the initial city-region exchange graph or the target city-region exchange graph, their graph-nodes represent candidate regions, and their graph-edges correspond to candidate exchange operations (candidate SWAP operations).

[0073] It should also be noted that the urban facility deployment method based on reinforcement learning provided in this embodiment is executed for each type of facility to be deployed each time, thereby completing the deployment of multiple facilities to be deployed.

[0074] In this embodiment, a target city area exchange graph is obtained by obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan and filtering out low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold; wherein, the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent candidate areas, the graph-edges are used to connect graph-nodes where facilities to be deployed exist and graph-nodes where facilities to be deployed do not exist, and the graph-edges correspond to candidate exchange operations; then, based on a pre-trained urban facility deployment model, a target city facility deployment plan is generated according to the target city area exchange graph, and urban facilities are deployed according to the target city facility deployment plan; wherein, the urban facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost. This method filters out low-quality edges in the urban area exchange graph and directs the algorithm's attention to high-quality exchange operations, thereby improving the accuracy of the target city facility deployment plan. At the same time, urban facility deployment is achieved through fast reasoning-driven exchange operations using graph neural networks and policy networks, rather than local exhaustive search, avoiding heavy calculations and achieving rapid generation of the target city facility deployment plan.

[0075] Based on the above embodiment, the process of filtering low-quality edges in the city area exchange graph will be described in detail below.

[0076] Low-quality edges include historically visited edges and negative-benefit edges; accordingly, the step of filtering out low-quality edges whose total deployment cost in the urban area exchange graph is higher than a set threshold specifically includes: obtaining the historical graph-nodes visited by the historical exchange operation, and determining the graph-edges containing the historical graph-nodes as historically visited edges; obtaining the actual distance between the removed node and the inserted node corresponding to the candidate exchange operation, and calculating the actual benefit and actual loss corresponding to the candidate exchange operation; when the actual distance is greater than the set distance threshold and the actual benefit is lower than the actual loss, determining the graph-edge corresponding to the candidate exchange operation as a negative-benefit edge; filtering the historically visited edges and negative-benefit edges in the urban area graph to obtain the target urban area exchange graph.

[0077] It's understandable that while the graph edges in the initial urban area swap graph / SWAP graph correspond to candidate swap operations / candidate SWAP operations, not all graph edges on the SWAP graph can bring the urban facility deployment solution closer to the optimal solution. In particular, when the urban facility deployment solution is near optimal or stuck in a local optimum, only a small number of graph edges corresponding to candidate SWAP operations can achieve the desired solution. Therefore, accurately filtering out low-quality edges is crucial to improving the quality of urban facility deployment solutions. However, current machine learning methods ignore this point, resulting in low accuracy.

[0078] In this embodiment, topology dynamic edge connection technology is used to increase the probability of sampling high-quality SWAP operations and improve search speed. Specifically, this technology is used to filter out low-quality edges that have little contribution to the optimal solution, including but not limited to historically accessed edges (hereinafter referred to as Tabu edges) and negative benefit edges.

[0079] Tabu edges are derived from Tabu Search, a widely used heuristic algorithm. Graph nodes visited by historical swap operations / swap operations are recorded as historical graph nodes / tabu nodes, and graph edges containing a tabu node are considered tabu edges. Tabu edges can be expressed as follows: (8)-(9).

[0080] (8).

[0081] (9).

[0082] In formulas (8)-(9), Represents a Tabu node set, 、 Indicates the inserted nodes and removed nodes visited by the historical SWAP operation. Represents a Tabu edge. The parameters here are Controlling the number of tabu nodes and filtering tabu edges can help the algorithm escape from the local optimal state and avoid repeated cycles.

[0083] Negative benefit edges represent candidate SWAP operations with significant negative benefits. After that, the change of total access cost / total deployment cost can be divided into three parts: Gains when placing facilities; only from removing nodes Losses when removing a facility; additional items resulting from placing and removing it simultaneously.

[0084] It is worth noting that if the candidate SWAP operation corresponds to the insertion node and remove nodes If the distance between the inserted and removed nodes is sufficiently large, the additional term is always zero. Therefore, if the actual distance between the inserted and removed nodes is greater than the set distance threshold, and the actual benefit of the corresponding candidate SWAP operation is lower than the actual loss, then executing the candidate SWAP operation will increase the total access cost. In this implementation, the graph edge corresponding to this candidate SWAP operation is determined to be a negative benefit edge.

[0085] Specifically, the negative return edge can be described by the following formula (10).

[0086] (10).

[0087] In formula (10), represents a negative return edge, Indicates the actual distance between the inserted node and the removed node, To set the distance threshold, Indicates the actual benefit of inserting a node, Indicates the actual loss caused by removing the node.

[0088] Filtering out negatively impacted edges in the graph directs the algorithm's attention to candidate SWAP operations that improve urban facility deployment plans, significantly narrowing the space of urban facility deployment solutions to high-quality areas. Based on this, topological dynamic edge connection technology highlights high-quality SWAP operations in urban facility deployment and ensures the accuracy of the resulting urban facility deployment solutions.

[0089] In this embodiment, historical graph nodes accessed by historical swap operations are obtained, graph edges containing these historical graph nodes are identified as historically visited edges, and the actual distance between the removed and inserted nodes corresponding to candidate swap operations is obtained. The actual gain and actual loss corresponding to the candidate swap operation are calculated. If the actual distance is greater than a set distance threshold and the actual gain is lower than the actual loss, the graph edge corresponding to the candidate swap operation is determined as a negative gain edge. This method filters the historically visited and negative gain edges in the city-area swap graph to obtain the target city-area swap graph. By filtering low-quality edges in the city-area swap graph, this method directs the algorithm's attention to high-quality swap operations, effectively improving the accuracy of the target city facility deployment plan.

[0090] Based on the above embodiments, the following will further describe in detail the process of generating a target city facility deployment plan using a graph neural network and a policy network.

[0091] Based on a pre-trained urban facility deployment model, a target city facility deployment plan is generated according to the target city area exchange graph, including: based on a pre-trained graph neural network, obtaining exchange edge embedding according to the target city area exchange graph; based on a pre-trained policy network, generating a target city facility deployment plan according to the exchange edge embedding.

[0092] It can be understood that this embodiment uses a graph neural network and a policy network. The graph neural network is used to obtain exchange edge embeddings based on the target city exchange graph, and the policy network is used to generate a target city facility deployment plan based on the exchange edge embeddings.

[0093] In a specific embodiment, a process of obtaining exchange edge embeddings using a graph neural network is described in detail.

[0094] Based on a pre-trained graph neural network, an exchange edge embedding is obtained according to the target city-region exchange graph, including: obtaining the node features corresponding to each graph-node in the target city-region exchange graph, and encoding the node features into an initial node embedding; updating the initial node embedding to obtain the target node embedding; fusing the target node embedding corresponding to the candidate exchange operation with the virtual node embedding corresponding to the virtual node to obtain the exchange edge embedding; wherein the virtual node is connected to all graph-nodes in the target city-region exchange graph.

[0095] Specifically, traditional heuristic methods perform local exhaustive searches on SWAP operations and cannot be extended to actual FLPs. This embodiment designs a model based on a graph neural network and a policy network to perform SWAP operations, thereby avoiding time-consuming local searches.

[0096] Effective graph-edge selection must consider both the global distribution of urban areas and facilities, as well as the local areas where residents access facilities. Therefore, the graph neural network in this embodiment considers both the topological relationships of candidate areas and the distribution of facilities to learn high-quality edge embeddings on the target SWAP graph / target urban area exchange graph.

[0097] Specifically, first, for each graph-node in the target city area exchange graph, we obtain rich node features. These node features include but are not limited to the coordinates of the candidate area represented by the node, access requirements, and distance to facilities. These node features are then encoded into a graph-node embedding, i.e., the initial node embedding, as shown in Equation (11).

[0098] (11).

[0099] In formula (11), represents the initial node embedding, represents the learnable weights, Represents the original node features.

[0100] Then, the graph-node representation is calculated by combining rich global information and neighbor information. Through linear transformation layers and nonlinear activation functions, the neighbor information is summarized and the initial node embedding is updated, as shown in the following equation (12).

[0101] (12).

[0102] In formula (12), Represents the graph neural network The target node embedding of the layer output, Represents the graph neural network The node embeddings of the layer output, represents the learnable weights, Represents the graph neural network Neighbor node embeddings output by the layer.

[0103] In addition, this embodiment also adds a virtual node connecting all graph nodes to collect global information of the target city area exchange graph. For details, please refer to the virtual node embedding described in the following formula (13).

[0104] (13).

[0105] In formula (13), Represents the graph neural network The virtual node embeddings of the layer output, Represents the graph neural network The virtual node embeddings of the layer output, 、 Represents a graph-node, represents the learnable weights, Represents the graph neural network Neighbor node embeddings output by the layer.

[0106] Next, the target node embedding corresponding to the candidate swap operation is fused with the virtual node embedding corresponding to the virtual node to obtain the edge representation, i.e., the swap edge embedding. For details, see the following equation (14).

[0107] (14).

[0108] In formula (14), represents the exchange edge embedding, 、 They represent the target node embeddings of the two graph-nodes corresponding to a certain candidate swap operation, Represents an embedded concatenation.

[0109] Through propagation and aggregation operations, the obtained swap edge embedding can not only perceive the global distribution information of candidate areas and facilities, but also capture the local relations between urban areas, supporting the subsequent target SWAP operation selection.

[0110] In another specific embodiment, a process of generating a target city facility deployment plan using a policy network is described in detail.

[0111] The policy network includes a multilayer perceptron; accordingly, based on the pre-trained policy network, a target city facility deployment plan is generated according to the exchange edge embedding, including: scoring each graph-edge in the target city area exchange graph through the multilayer perceptron according to the exchange edge embedding, and obtaining operation scores corresponding to candidate exchange operations; determining the candidate exchange operation corresponding to the highest operation score as the target exchange operation, and obtaining the target city facility deployment plan.

[0112] Specifically, this embodiment adopts a policy network to select the graph-edge for each step of the SWAP operation. The policy network takes the swap edge embedding output by the graph neural network as input and selects the target SWAP operation that can effectively reduce the access cost.

[0113] First, the policy network scores each valid graph-edge based on the exchange edge embedding through a multi-layer perceptron (MLP), as shown in the following equations (15)-(16).

[0114] (15).

[0115] (16).

[0116] In formulas (15)-(16), represents a valid graph-edge, that is, a graph-edge in the target city-region exchange graph, represents the historical visit edge, represents a negative return edge, Represents the scores of different valid graphs-edges output by the policy network, that is, the operation scores corresponding to the candidate exchange operations. This ensures that historically visited edges and negatively benefited edges will not be selected.

[0117] With the help of rich prior knowledge in the swap edge embedding, the policy network selects the target swap operation / target SWAP operation based on the calculated operation score. Specifically, the candidate swap operation corresponding to the highest operation score is used as the target swap operation.

[0118] Then, the city facility deployment is completed by performing a target exchange operation.

[0119] In this embodiment, a pre-trained graph neural network is used to obtain exchange edge embeddings based on the target city region exchange graph. Based on these exchange edge embeddings, a pre-trained policy network is used to generate a target city facility deployment plan. Urban facilities are then deployed according to the target city facility deployment plan. This method achieves urban facility deployment by driving exchange operations through rapid inference between the graph neural network and the policy network, rather than through local exhaustive search. This avoids heavy computation and enables the rapid generation of target city facility deployment plans.

[0120] Based on the above embodiments, the training optimization of the urban facility deployment model will be described in detail below.

[0121] Training and optimizing the urban facility deployment model specifically includes: obtaining urban geographic spatial data of a real city, and randomly generating the number of candidate areas and the number of facilities to be deployed corresponding to the urban geographic spatial data; according to the number of candidate areas and the number of facilities to be deployed, using a proximal strategy optimization algorithm to iteratively optimize the urban facility deployment model; when the number of iterative optimization reaches the set number of iterations, stopping the training and obtaining a trained urban facility deployment model.

[0122] The number of iterations can be set according to actual needs and is not specifically limited here.

[0123] In a specific embodiment, real New York City urban geospatial data is used to train an urban facility deployment model to solve the FLP.

[0124] Figure 4 A schematic diagram of training optimization of the urban facility deployment model provided by an embodiment of the present invention is shown.

[0125] like Figure 4 As shown, this embodiment randomly generates a new FLP instance for each round of urban facility deployment model training. Specifically, at the beginning of each training round, we first randomly generate the number of candidate regions N and the number of facilities to be deployed p. Then, we extract a region (CBG) and its nearest N-1 CBGs as candidate regions. Finally, considering residents' needs for deployed facilities, the algorithm performs a SWAP operation to improve the initial urban facility deployment plan obtained using the greedy algorithm for a specific FLP instance.

[0126] In this embodiment, Proximal Policy Optimization is used to train and optimize the urban facility deployment model.

[0127] It is worth noting that during the model training process, the algorithm provided in this embodiment will attempt to solve thousands of different synthetic FLP instances, thereby generating a universal solver (i.e., the urban facility deployment model) that can handle any input FLP.

[0128] On the basis of the above embodiment, further, before obtaining the urban area exchange graph corresponding to the initial urban facility deployment plan, it includes: according to the given candidate areas and facilities to be deployed, using a greedy algorithm to solve and obtain the initial urban facility deployment plan.

[0129] Specifically, when using the greedy algorithm for solving the problem, the input can be the distance between candidate areas and the number of facilities to be deployed, or the number of candidate areas, the distance between candidate areas, and the access needs of residents in the candidate areas for the deployed facilities. The output is the initial urban facility deployment plan, which includes the current deployment status of the facilities to be deployed in each candidate area.

[0130] A greedy algorithm is a strategy that takes the locally optimal choice at each step, hoping that this choice will lead to a globally optimal solution. This algorithm does not always lead to a globally optimal solution, but it can usually find a reasonably good solution quickly.

[0131] Since the greedy algorithm is already a relatively mature technology in the existing technology, it will not be described in detail here.

[0132] In this embodiment, a greedy algorithm is used to solve the initial urban facility deployment plan based on the given candidate areas and facilities to be deployed. Then, based on the initial urban facility deployment plan, the SWAP operation is used to improve and optimize it to obtain the target urban facility deployment plan, rather than generating the final plan from scratch. This can help to quickly find a better solution in the solution space.

[0133] In some embodiments, Figure 5 A schematic diagram of the overall framework of the urban facility deployment method based on reinforcement learning provided by an embodiment of the present invention is shown.

[0134] like Figure 5 As shown, this embodiment adopts a "divide and conquer" framework and uses a unified reinforcement learning algorithm based on SWAP to handle the FLP problem.

[0135] The algorithm consists of two consecutive stages. In stage 1, the initial urban facility deployment plan obtained by the greedy algorithm is improved through the SWAP operation, and the final deployment location is independently searched for each type of facility to be deployed.

[0136] Since the independent solution may conflict with the incompatible constraints of the urban area, in stage 2, the conflicts are eliminated through SWAP operations, thereby further improving the independent solution until all incompatible constraint conflicts are resolved, and the final solution of the FLP is obtained.

[0137] The incompatibility constraints here mainly limit the number and type of facilities to be deployed in the candidate area. For example, the number of a certain type of facility in the candidate area can only be 1.

[0138] Among them, for the selection of SWAP operation, it is first necessary to construct the SWAP graph corresponding to the initial urban facility deployment plan, and filter the Tabu edges and negative benefit edges in the SWAP graph to obtain the target SWAP graph.

[0139] Furthermore, based on the target SWAP graph, the graph neural network is used to encode the swap edge embeddings, and the MLP in the policy network is used to score each swap edge embedding, and then the appropriate SWAP operation, that is, the target swap operation, is selected.

[0140] Finally, the target exchange operation is performed to complete the deployment of the facilities to be deployed in the candidate area.

[0141] at the same time, Figure 6 The figure shows the overall process of the urban facility deployment method based on reinforcement learning provided by an embodiment of the present invention.

[0142] like Figure 6 As shown in the figure, first, an urban area exchange graph (SWAP graph) corresponding to the initial urban facility deployment plan is constructed, and then the topological dynamic edge connection technology is used to filter the Tabu edges and negative benefit edges in the SWAP graph to obtain a high-quality solution space SWAP graph, which is also the target urban area exchange graph or target SWAP graph.

[0143] Furthermore, using graph neural networks (i.e. Figure 6 ), obtain the swap edge embedding corresponding to each graph-edge in the target SWAP graph.

[0144] Next, on the one hand, the exchange edge is embedded into the policy network, which scores each graph-edge in the target SWAP graph and selects the target SWAP operation based on the scoring result. On the other hand, the exchange edge can be embedded into the value network, which calculates the total deployment cost / total access cost corresponding to each graph-edge in the target SWAP graph. To calculate, and use this to determine the effect of the corresponding SWAP operation.

[0145] In other embodiments, the urban facility deployment method based on reinforcement learning provided by the present invention is used in urban multi-facility deployment scenarios, and corresponding technical effects are given.

[0146] Specifically, for the multi-facility deployment scenario in a city, similarly, the divide-and-conquer framework is used to calculate the target city facility deployment plan for each facility type. The same city facility deployment model is then used to resolve conflicts between different types of facilities. This is accomplished through a unified SWAP operation. For details, see Figure 7 , Figure 7 A flow chart of urban facility deployment in a multi-facility urban deployment scenario provided by an embodiment of the present invention is shown.

[0147] at the same time, Figure 8 Also shown is a comparison diagram of urban facility deployment results in an urban multi-facility deployment scenario provided by an embodiment of the present invention.

[0148] exist Figure 8 The comparison objects include the urban facility deployment method based on reinforcement learning provided by the present invention (i.e. Figure 8 DRL-GNN in ), as well as existing urban facility deployment methods such as GA, TS, LS, VNS, CardNN, Gurobi (120s), and Gurobi (optimal).

[0149] Applicable cities include New York, Boston, Los Angeles and Chicago.

[0150] The comparison indicators are AC (Actual Cost) and RT (Response-time).

[0151] according to Figure 8 It can be seen that the urban facility deployment method based on reinforcement learning provided by the present invention can generate an urban facility deployment plan with an extremely fast solution speed close to the solver accuracy.

[0152] In some other embodiments, the urban facility deployment method based on reinforcement learning provided by the present invention is used in a single urban facility deployment scenario, and corresponding technical effects are given.

[0153] Specifically, for single-facility deployment scenarios in cities, we can directly leverage the city facility deployment model for rapid inference. After continuous SWAP operations, we can obtain the target city facility deployment plan. Unlike the greedy algorithm used to construct the initial deployment plan during model training, we use the current city deployment plan as the initial city facility deployment plan.

[0154] Taking the redeployment of a single hospital facility in Brooklyn, New York as an example, the present invention can effectively improve the accessibility of facility services by approximately 39%. For details, please refer to Figure 9 , Figure 9 A diagram showing the deployment effect of urban facilities in a single-facility urban deployment scenario provided by an embodiment of the present invention is shown.

[0155] In addition, it is worth mentioning that the urban facility deployment method based on reinforcement learning provided by the embodiment of the present invention has the following three advantages.

[0156] First, the deployment of a single facility and multiple facilities in a city can be completed through a unified SWAP operation, killing two birds with one stone. The urban facility deployment model is highly versatile.

[0157] Secondly, the urban facility deployment model that combines reinforcement learning (policy network) with graph neural network has fast forward reasoning speed and high problem-solving efficiency. The model can quickly solve large-scale urban facility deployment problems in real scenarios.

[0158] Third, the introduction of prior knowledge of the FLP problem can significantly improve the solution accuracy compared to existing machine learning-based methods, and the performance is close to the solution obtained by the solver.

[0159] Corresponding to the urban facility deployment method based on reinforcement learning described in the above embodiments, the present invention also provides an urban facility deployment device based on reinforcement learning, specifically, Figure 10 A schematic structural diagram of an urban facility deployment device based on reinforcement learning provided by an embodiment of the present invention is shown.

[0160] like Figure 10 As shown, the device includes: a target city area exchange graph acquisition module 1010, which is used to obtain the city area exchange graph corresponding to the initial city facility deployment plan, and filter the low-quality edges in the city area exchange graph whose total deployment cost is higher than the set cost threshold to obtain the target city area exchange graph; wherein, the city area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; a target city facility deployment plan generation module 1020, which is used to generate a target city facility deployment plan according to the target city area exchange graph based on a pre-trained city facility deployment model, and deploy city facilities according to the target city facility deployment plan; wherein, the city facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost.

[0161] In this embodiment, the city area exchange graph corresponding to the initial city facility deployment plan is obtained by the target city area exchange graph acquisition module 1010, and the low-quality edges in the city area exchange graph whose total deployment cost is higher than the set cost threshold are filtered to obtain the target city area exchange graph; wherein, the city area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; the target city facility deployment plan is generated according to the target city area exchange graph based on a pre-trained city facility deployment model by the target city facility deployment plan generation module 1020, and city facilities are deployed according to the target city facility deployment plan; wherein, the city facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost. The device filters out low-quality edges in the urban area exchange graph and directs the algorithm's attention to high-quality exchange operations, thereby improving the accuracy of the target city facility deployment plan. At the same time, it realizes urban facility deployment through fast reasoning-driven exchange operations of graph neural networks and policy networks, rather than local exhaustive search, avoiding heavy calculations and achieving rapid generation of target city facility deployment plans.

[0162] It should be noted that the urban facility deployment device based on reinforcement learning provided in this embodiment can be referenced to each other with the urban facility deployment method described in the above embodiment, and will not be described in detail here.

[0163] Figure 11 An example of a physical structure diagram of an electronic device is shown below. Figure 11As shown, the electronic device may include: a processor (processor) 1110 , a communication interface (Communications Interface) 1120 , a memory (memory) 1130 and a communication bus 1140 , wherein the processor 1110 , the communication interface 1120 , and the memory 1130 communicate with each other via the communication bus 1140 . The processor 1110 can call the logic instructions in the memory 1130 to execute the urban facility deployment method based on reinforcement learning, which includes: obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph-nodes and graph-edges, the graph-nodes are used to represent candidate areas, the graph-edges are used to connect graph-nodes with facilities to be deployed and graph-nodes without facilities to be deployed, and the graph-edges correspond to candidate exchange operations; based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0164] Furthermore, the logic instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0165] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the urban facility deployment method based on reinforcement learning provided by the above-mentioned methods, the method comprising: obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph nodes and graph edges, the graph nodes are used to represent candidate areas, the graph edges are used to connect graph nodes with facilities to be deployed and graph nodes without facilities to be deployed, and the graph edges correspond to candidate exchange operations; based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation contained in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0167] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for urban facility deployment based on reinforcement learning, characterized in that: include: Obtaining an urban area exchange graph corresponding to an initial urban facility deployment plan, and filtering low-quality edges in the urban area exchange graph whose total deployment cost exceeds a set cost threshold to obtain a target urban area exchange graph; wherein the urban area exchange graph includes graph nodes and graph edges, wherein the graph nodes are used to represent candidate areas, the graph edges are used to connect graph nodes where facilities to be deployed exist and graph nodes where facilities to be deployed do not exist, and the graph edges correspond to candidate exchange operations; Based on a pre-trained urban facility deployment model, a target urban facility deployment plan is generated according to the target urban area exchange graph, and urban facilities are deployed according to the target urban facility deployment plan; wherein, the urban facility deployment model is constructed based on a graph neural network and a policy network, the target urban facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target urban facility deployment plan corresponds to the minimum total deployment cost.

2. The urban facility deployment method based on reinforcement learning according to claim 1, characterized in that: The low-quality edges include historical visit edges and negative benefit edges; Accordingly, the step of filtering low-quality edges in the urban area exchange graph whose total deployment cost is higher than a set threshold specifically includes: Obtaining a historical graph-node accessed by the historical exchange operation, and determining a graph-edge containing the historical graph-node as a historical access edge; Obtaining the actual distance between the removed node and the inserted node corresponding to the candidate swap operation, and calculating the actual gain and actual loss corresponding to the candidate swap operation; When the actual distance is greater than a set distance threshold and the actual gain is lower than the actual loss, determining the graph edge corresponding to the candidate exchange operation as a negative gain edge; The historical visit edges and negative benefit edges in the city area exchange graph are filtered to obtain the target city area exchange graph.

3. The urban facility deployment method based on reinforcement learning according to claim 1, characterized in that: The method of generating a target city facility deployment plan based on the pre-trained city facility deployment model and the target city area exchange graph includes: Based on the target city area exchange graph, the exchange edge embedding is obtained based on the pre-trained graph neural network; Based on the pre-trained policy network, a target city facility deployment plan is generated according to the exchange edge embedding.

4. The urban facility deployment method based on reinforcement learning according to claim 3 is characterized in that: The pre-trained graph neural network is used to obtain exchange edge embeddings according to the target city area exchange graph, including: Obtain node features corresponding to each graph-node in the target city-region exchange graph, and encode the node features into initial node embeddings; Updating the initial node embedding to obtain a target node embedding; fusing the target node embedding corresponding to the candidate swap operation and the virtual node embedding corresponding to the virtual node to obtain the swap edge embedding; The virtual node is connected to all graph-nodes in the target city area exchange graph.

5. The urban facility deployment method based on reinforcement learning according to claim 3, characterized in that: The policy network includes a multilayer perceptron; Accordingly, the pre-trained policy network generates a target city facility deployment plan based on the exchange edge embedding, including: Scoring each graph-edge in the target city area exchange graph using a multi-layer perceptron based on the exchange edge embedding to obtain an operation score corresponding to the candidate exchange operation; The candidate exchange operation corresponding to the highest operation score is determined as the target exchange operation, and the target city facility deployment plan is obtained.

6. The urban facility deployment method based on reinforcement learning according to any one of claims 1 to 5, characterized in that: Training and optimizing the urban facility deployment model, specifically including: Obtaining urban geographic spatial data of a real city, and randomly generating the number of candidate areas and the number of facilities to be deployed corresponding to the urban geographic spatial data; Iteratively optimizing the urban facility deployment model using a proximal strategy optimization algorithm according to the number of candidate areas and the number of facilities to be deployed; When the number of iterative optimization reaches the set number of iterations, the training is stopped and a trained urban facility deployment model is obtained.

7. The urban facility deployment method based on reinforcement learning according to any one of claims 1 to 5, characterized in that: The step of obtaining the urban area exchange graph corresponding to the initial urban facility deployment plan includes: According to the given candidate areas and facilities to be deployed, a greedy algorithm is used to solve and obtain the initial urban facility deployment plan.

8. An urban facility deployment device based on reinforcement learning, characterized in that: include: a target city area exchange graph acquisition module, configured to acquire a city area exchange graph corresponding to the initial city facility deployment plan, and filter low-quality edges in the city area exchange graph whose total deployment cost exceeds a set cost threshold to obtain a target city area exchange graph; wherein the city area exchange graph includes graph nodes and graph edges, wherein the graph nodes are used to represent candidate areas, the graph edges are used to connect graph nodes where facilities to be deployed exist with graph nodes where facilities to be deployed do not exist, and the graph edges correspond to candidate exchange operations; A target city facility deployment plan generation module is used to generate a target city facility deployment plan based on a pre-trained city facility deployment model and the target city area exchange graph, and to deploy city facilities according to the target city facility deployment plan; wherein, the city facility deployment model is constructed based on a graph neural network and a policy network, the target city facility deployment plan includes a target exchange operation included in the candidate exchange operation, and the target city facility deployment plan corresponds to the minimum total deployment cost.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the urban facility deployment method based on reinforcement learning according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the urban facility deployment method based on reinforcement learning according to any one of claims 1 to 7 is implemented.

Images