Low-altitude traffic infrastructure network layout optimization method based on multi-agent simulation

Abstract

The application relates to the technical field of data processing, in particular to a low-altitude traffic infrastructure network layout optimization method based on multi-agent simulation, which comprises the following steps: generating a layout through a generative model; simulating the dynamic behaviors of agents such as persons, vehicles and aircrafts in the layout by using the generated layout to perform reinforcement learning algorithm simulation on the multi-agent; and optimizing the generative model by using the agent after reinforcement learning, so that the generative model generates a layout which makes each agent output the maximum value. The "combination explosion" problem of low-altitude infrastructure layout is effectively solved, and efficient utilization of air-ground resources is ensured.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a method for optimizing the layout of low-altitude transportation infrastructure networks based on multi-agent simulation. Background Technology

[0002] The low-altitude economy is an emerging industrial system. An efficient and safe low-altitude transportation network is the cornerstone for unleashing the potential of the low-altitude economy. Its rational layout is directly related to the operational efficiency, safety and sustainability of future urban three-dimensional transportation.

[0003] However, the core challenge in the current network layout planning of low-altitude transportation infrastructure lies in the "ultra-complexity" of the problem. First, the planning process suffers from a severe "fragmentation" problem. Traditional planning methods largely rely on static urban data and expert experience, isolating low-altitude infrastructure planning from existing ground transportation systems, urban spatial structures, and airspace management rules, creating "information silos." This "static prediction and expert experience"-dominated model makes it difficult to achieve seamless integration between air and ground transportation, resulting in low efficiency in air-ground coordination. A more fundamental challenge is that low-altitude infrastructure layout itself is a typical "high-dimensional, multi-constraint, multi-objective" complex optimization problem, highly susceptible to "combinatorial explosion." The decision variables are extremely dimensional, and the constraints are numerous and intertwined, causing the number of feasible planning schemes to grow exponentially.

[0004] Faced with this complexity, traditional traffic planning models and discrete optimization algorithms (such as linear programming and genetic algorithms) have proven inadequate. These methods are prone to getting stuck in local optima and failing to find better solutions. Moreover, they essentially "search" within a finite set of pre-defined solutions, lacking the ability to evaluate the macroscopic characteristics of the planning solutions in actual operation. Consequently, they are unable to address the "combinatorial explosion" problem and achieve integrated and collaborative planning of air and land resources. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for optimizing the layout of low-altitude transportation infrastructure networks based on multi-agent simulation.

[0006] The low-altitude transportation infrastructure network layout optimization method based on multi-agent simulation of the present invention adopts the following technical solution:

[0007] One embodiment of the present invention provides a method for optimizing the layout of low-altitude transportation infrastructure networks based on multi-agent simulation, the method comprising the following steps:

[0008] Create several intelligent agents;

[0009] A generative model is used to generate layout F1, which is used as the environment of each agent. The DDPG algorithm is used to perform reinforcement learning on each agent. The DDPG algorithm includes an A network and a C network. The C network is used to output the value of each agent in each state and action. The A network updates its parameters based on the value output by the C network. The maximum value output by the C network of each agent after reinforcement learning is used as the evaluation index Q1 of each agent.

[0010] The layout F2 is generated again using the generative model. F2 is used as the environment of each agent and reinforcement learning is performed again to obtain the evaluation index Q2 of each agent. When reinforcement learning is performed again, the A network of each agent uses the Q1 of all agents to update the parameters so that the difference of Q2 between different agents is minimized.

[0011] Based on the differences between Q2 and Q1 for each agent, determine whether F2 is a label; when F2 is a label, use the differences between Q2 of different agents to fit F1 output by the generator model into F2.

[0012] Generative models are used to obtain the network layout of low-altitude transportation infrastructure.

[0013] Preferably, when performing reinforcement learning again, the A network of each agent uses the Q1 of all agents to update its parameters, so as to minimize the difference in Q2 between different agents. The specific steps include the following:

[0014] During the reinforcement learning process, each agent takes turns undergoing reinforcement training.

[0015] For any agent, the average sampled value of the C network output after each reinforcement training iteration is expressed as: For the i-th agent other than the current agent, the average sampled value of the C network output after the most recent reinforcement training of the i-th agent is denoted as . ;

[0016] The loss function of the agent's A network Depend on and The weighted summation is used to obtain N, which represents the number of agents other than the one in question, using the loss function. The parameters of network A are updated, where the weights in the weighted summation are obtained from the evaluation metric Q1 of all agents.

[0017] Preferably, the specific steps for obtaining the weights in the weighted summation are as follows:

[0018] For any given agent, the agent's corresponding The weight w in the weighted summation is negatively correlated with the evaluation index Q1 of the agent, and positively correlated with the evaluation index Q1 of all agents other than the agent.

[0019] Preferably, the specific steps for determining whether F2 is a label based on the difference between Q2 and Q1 for each agent are as follows:

[0020] Obtain the average of the evaluation metrics Q1 of all agents, and denote it as the first average. Obtain the average of the evaluation metrics Q2 of all agents, and denote it as the second average. When the second average is greater than or equal to the first average, determine that F2 is a label.

[0021] Preferably, when F2 is a label, the specific steps for fitting the F1 output by the generator model to F2 by utilizing the differences between Q2 of different agents are as follows:

[0022] For the layout F1 output by the generative model, the cross-entropy loss function is constructed using the F1 output by the generative model and the label F2. The cross-entropy loss function is multiplied by (1+h) and used as the loss function of the generative model. The parameters of the generative model are updated using the loss function of the generative model, where h represents the gradient update coefficient.

[0023] Preferably, the specific steps for obtaining the gradient update coefficients are as follows:

[0024] The variance of the evaluation index Q1 for all agents is denoted as q1, and the variance of the evaluation index Q2 for all agents is denoted as q2; the gradient update coefficient is negatively correlated with q2 and positively correlated with q1.

[0025] Preferably, when the second mean is less than the first mean, F1 is determined to be a label;

[0026] For the layout F2 output by the generative model, a cross-entropy loss function is constructed using F2 and the label F1, and used as the loss function of the generative model; the parameters of the generative model are updated using the loss function of the generative model.

[0027] Preferably, the intelligent agent includes a person, a vehicle, a drone, or a vertical takeoff and landing manned aircraft; the state of each intelligent agent includes the position of the intelligent agent in the layout, and the action of each intelligent agent includes the magnitude and direction of the speed of the intelligent agent in the layout.

[0028] Preferably, the specific steps for obtaining the maximum value of the C network output of each agent after reinforcement learning are as follows:

[0029] For each agent in the reinforcement learning process, the values ​​most recently output by the C network of each agent constitute a value sequence. The value sequence is filtered for maximum value, and the standard deviation of the filtered value sequence is recorded as the training error. When the training errors of all agents are less than or equal to a preset threshold, the reinforcement learning process ends, and the value output by the C network of each agent is maximized. The mean of the filtered value sequence is taken as the maximum value.

[0030] Preferably, the specific steps for obtaining the average sample value are as follows:

[0031] During reinforcement learning, each agent samples several states s1 from the experience replay pool of the DDPG algorithm during each reinforcement training session. Each state s1 is input into the A network of each agent, and the A network outputs action a1. The state s1 and action a1 are input into the C network of each agent, and the value output by the C network is recorded as the first value. The average of all first values ​​obtained from sampling several states is recorded as the average sampled value of each agent.

[0032] The beneficial effects of the technical solution of the present invention are:

[0033] Traditional optimization methods search within a finite set of preset solutions, making them prone to getting trapped in local optima. In contrast, the generative model at the core of this invention learns the complex constraints inherent in urban data (such as road, building, and pedestrian distribution), thereby directly "creating" entirely new, physically feasible, and rule-compliant layout schemes. This generative capability frees it from the limitations of the scale of preset solutions, effectively exploring high-dimensional solution spaces inaccessible to traditional algorithms. It methodologically solves the "combinatorial explosion" problem caused by numerous constraints, offering the possibility of finding globally optimal solutions.

[0034] This invention achieves accurate deduction and evaluation of the real-world performance of layout schemes through multi-agent collaborative simulation. The generated layout (F1, F2) serves as the environment for multiple agents (representing people, vehicles, drones, aircraft, etc.), and the DDPG reinforcement learning algorithm simulates the dynamic behavior of these agents within the layout. Each agent quantifies the degree to which the layout satisfies its own needs (e.g., commuting efficiency, safety) through the maximum value output (evaluation metrics Q1, Q2) of its C-network output. This method transforms layout evaluation from a static, rule-based judgment into a dynamic, simulation-based comprehensive performance assessment, resulting in more objective and accurate evaluation results.

[0035] The core advancement of this invention lies in establishing a reinforcement learning feedback mechanism to optimize the generative model. Specifically, in the second round of simulation, each agent's A network updates its parameters using the evaluation metric Q1 from the previous round across all agents, aiming to minimize the difference in evaluation metric Q2 obtained by different agents in this round. This mechanism forces each agent to consider not only its own interests but also the performance of other agents when making decisions, thereby spontaneously promoting collaboration among different traffic participants such as people, vehicles, and aircraft during simulation. Finally, the system judges the superiority or inferiority of the layout based on the average of Q2 and Q1 (whether F2 is a label), and uses this judgment to train the generative model. This closed loop ensures that the generated layout is not only efficient but also highly collaborative, enabling integrated and efficient utilization of air and ground resources.

[0036] In summary, this invention achieves collaborative planning and optimization of air and ground resources through a closed loop of "generating models to create schemes, multi-agent simulation and deduction, and reinforcement learning feedback optimization," ensuring the integrated and efficient utilization of air and ground resources. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 The flowchart illustrates the steps of a low-altitude transportation infrastructure network layout optimization method based on multi-agent simulation, as provided in an embodiment of the present invention.

[0039] Figure 2 This is a flowchart illustrating the steps of updating the reinforcement learning model and the generative model in this invention. Detailed Implementation

[0040] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the low-altitude transportation infrastructure network layout optimization method based on multi-agent simulation proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0042] The following description, in conjunction with the accompanying drawings, details the specific scheme of the low-altitude transportation infrastructure network layout optimization method based on multi-agent simulation provided by this invention.

[0043] Example 1:

[0044] Please see Figure 1 The diagram illustrates a flowchart of a method for optimizing the layout of low-altitude transportation infrastructure networks based on multi-agent simulation, according to an embodiment of the present invention. The method includes the following steps:

[0045] Step S1: Create a generative model.

[0046] When planning the layout of low-altitude transportation infrastructure networks, the conventional approach is to design the network layout based on the existing urban layout and using relevant expert experience or discrete optimization algorithms. However, the layout of low-altitude infrastructure is a complex optimization problem with "high dimensions, multiple constraints, and multiple objectives" and many constraints. When dealing with such ultra-complex problems, human experience methods and discrete optimization algorithms (such as linear programming and genetic algorithms) are prone to getting trapped in local optima.

[0047] This embodiment uses a generative model to directly generate the low-altitude transportation infrastructure network layout (hereinafter referred to as the layout) based on the city's existing transportation network. This method can initially avoid the above-mentioned problems.

[0048] The layout in this embodiment refers to the layout of take-off and landing sites in low-altitude traffic, specifically including take-off and landing sites for UAVs and aircraft.

[0049] Step S2: Create several intelligent agents.

[0050] Each intelligent agent represents a type of physical entity in the low-altitude traffic network, used to simulate its state and actions within the low-altitude traffic network environment. In this embodiment, people, vehicles, drones, and vertical takeoff and landing manned aircraft (referred to as aircraft) in the low-altitude traffic network are each created as an intelligent agent. The constrained operation processes of people, vehicles, drones, and aircraft in the low-altitude traffic network are mapped to the simulation process of the intelligent agents.

[0051] This embodiment will subsequently combine the generated model with the simulation of the intelligent agent to solve the problems mentioned above.

[0052] Step S3: Using the different layouts generated by the generative model, perform reinforcement learning on all agents under different layouts, and use the reinforced agents to optimize the generative model so that the generative model generates the layout that maximizes the output value of each agent.

[0053] The purpose of reinforcement learning for all agents is to enable them to simulate the normal traffic activities of entities such as people, vehicles, drones, and aircraft under the constraints of the layout generated by the generative model, which is equivalent to realizing the deduction of the traffic operation of the layout.

[0054] The purpose of using reinforcement learning agents to optimize the generative model is to evaluate whether the layout meets the normal traffic activity needs of entities such as people, vehicles, drones, and aircraft through the simulation process of all agents, and to update and optimize the generative model until the layout generated by the generative model can meet the normal traffic activity needs of entities such as people, vehicles, drones, and aircraft to the greatest extent (that is, to maximize the output value of each agent).

[0055] Meeting the needs of normal traffic activities means that low-altitude transportation can operate efficiently, such as meeting the needs of minimal delays, safety, and convenient and fast travel.

[0056] Step S4: Optimize and transform the layout of the low-altitude transportation infrastructure network using the layout generated by the generative model.

[0057] Steps S1-S4 above employ generative artificial intelligence and construct an intelligent feedback loop of "generation-deduction-optimization," achieving a leap from the traditional "optimization" mode (searching among finite solutions) to the "generative decision-making" mode (intelligently creating an infinite number of new solutions). Generative AI excels at learning and creating in high-dimensional, multi-constraint spaces, effectively solving the "combinatorial explosion" problem that traditional optimization algorithms cannot handle. Simultaneously, it coordinates and optimizes air and land resources, ensuring the integrated and efficient utilization of these resources.

[0058] Furthermore, this embodiment provides a further description of step S3; please refer to [link / reference]. Figure 2 Step S3 includes:

[0059] Step S301: Generate layout F1 using the generative model, and use F1 as the environment of each agent. Use the DDPG algorithm to perform reinforcement learning on each agent. The maximum value output by the C network of each agent after reinforcement learning is used as the evaluation index Q1 of each agent.

[0060] The city's road distribution map, building distribution map, and pedestrian flow distribution map are used as urban reference data to describe the current traffic situation or traffic environment of the city. In this embodiment, the urban reference data is considered a constant and remains unchanged throughout the embodiment.

[0061] Generate a random vector, denoted as R1. Input the random vector R1 and the city reference data into the generative model. The generative model outputs a layout, which is a two-channel binary graph. In one channel, each connected component represents the take-off and landing field of the UAV, and in the other channel, each connected component represents the take-off and landing field of the aircraft.

[0062] It should be noted that in this embodiment, different random vectors are subsequently generated to enable the generation model to generate different layouts.

[0063] The generated layout is denoted as F1, and F1 serves as the environment for each agent. The environment also includes city reference data. The DDPG algorithm is used to perform reinforcement learning on each agent.

[0064] The principle of the DDPG algorithm is as follows:

[0065] (1) The DDPG algorithm contains an Actor network (A network for short) and a Critic network (C network for short).

[0066] (2) Input the state s of each agent (including the position of each agent in the layout) into network A, and network A outputs the action a of each agent (including the movement speed of each agent); then each agent executes action a in state s to obtain the next state. (Including the position of each agent after performing action a), based on the environment of each agent, the state. The agent is assigned a reward r. Next, each agent's state s and action a are input into the C network. The C network outputs the value of each agent after performing action a in state s; the quadruple is then processed. It serves as an experience storage for the DDPG algorithm's experience replay pool.

[0067] (3) Sample several (e.g., 64) experiences from the experience replay pool, update the C network based on the sampled experiences using the temporal difference function (TD-error), and update the A network based on the average value output by the sampled experiences after the C network is updated.

[0068] This completes one intensive training session.

[0069] (4) After the A and C networks are updated, Assign a value to s, and repeat (2)~(3) to complete multiple reinforcement training processes, forming a cyclical process of "collecting first, learning later, and collecting while learning". Finally, the reinforcement learning process of each agent is completed.

[0070] Additionally, it should be noted that, as described in (3), the A network is updated based on the average value output by the sampled experience after the C network is updated. The specific method for this process is as follows: For each sampled experience, there is a corresponding quadruple. Here, the first data in the quadruple is denoted as state s1. s1 is input into the A network, and the A network outputs action a1. State s1 and action a1 are input into the C network, and the value output by the C network is denoted as the first value. The average of the first values ​​obtained from all sampled experiences is denoted as the average sampled value for each agent. ,Will As the loss function of network A for each agent, the parameters of network A are updated using the Adam optimizer; the purpose is to make the actions output by network A obtain the greatest possible value.

[0071] The above (1) to (4) are summaries of existing DDPG algorithms, which are all well-known technologies. This embodiment will not elaborate on them.

[0072] After the above reinforcement learning process is completed, the value of the C network output of each agent is maximized. In this embodiment, the maximized value is used as the evaluation index Q1 of each agent.

[0073] At this point, the evaluation index Q1 for all intelligent agents has been obtained. The evaluation index Q1 is used to describe whether the layout F1 meets the normal traffic activity needs of entities such as people, vehicles, drones, and aircraft. The larger the Q1 is, the better the layout F1 can meet the normal traffic activity needs of each intelligent agent, or in other words, the more reasonable the design of the layout F1 is for each intelligent agent.

[0074] Step S302: Generate layout F2 again using the generative model, use F2 as the environment of each agent, perform reinforcement learning again, and obtain the evaluation index Q2 of each agent; wherein, when performing reinforcement learning again, the A network of each agent uses the Q1 of all agents to update the parameters.

[0075] A new random vector, denoted as R2, is generated. This random vector R2 and the city reference data are input into the generative model. The layout output by the generative model is denoted as F2. F2 is used as the environment for each agent, which also includes the city reference data. In this environment, the DDPG algorithm is used to re-encode reinforcement learning for each agent. After the reinforcement learning process, the value of the C network output for each agent is maximized. In this embodiment, the maximized value is used as the evaluation metric Q2 for each agent.

[0076] This process is the same as step S301 above, and will not be described in detail in this embodiment.

[0077] In this step, when re-performing reinforcement learning for each agent, it is considered that in order to simultaneously plan and optimize air and ground resources in a coordinated manner and ensure the integrated and efficient utilization of air and ground resources, the reinforcement learning process of all agents needs to be considered at the same time. It is necessary to minimize the differences between the evaluation indicators of different agents so that the evaluation indicators of different agents can reach the largest possible value at the same time.

[0078] Based on this, the process of reinforcement learning for each agent in this step differs from step S301 in the following ways:

[0079] (1) After each agent undergoes a reinforcement training process, the other agents also undergo a reinforcement training process. This achieves the goal of all agents taking turns undergoing reinforcement training.

[0080] (2) For any agent, the average sampled value of the agent during each reinforcement training process is expressed as: For the i-th agent other than the current agent, the average sampled value obtained by the i-th agent during the most recent reinforcement training process is denoted as . .

[0081] The loss function of the agent's A network Where w represents the attention weight, w is negatively correlated with the evaluation metric Q1 corresponding to the agent, and N represents the number of agents other than the agent in question.

[0082] The purpose of the loss function design described above is twofold: firstly, when each agent updates network A, it should not only focus on its own reinforcement training process but also pay attention to the reinforcement training process of other agents to a certain extent; secondly, when the evaluation metric Q1 obtained by the same agent in the previous reinforcement learning is relatively small, it should pay more attention to its own reinforcement training process in the next reinforcement training process to avoid the evaluation metric Q2 obtained after reinforcement training being too small; when the evaluation metric Q1 obtained by the same agent in the previous reinforcement learning is relatively large, it should pay more attention to the reinforcement training process of other agents in the next reinforcement training process, so that the agent can further cooperate with other agents.

[0083] As an example, w is negatively correlated with the evaluation metric Q1 corresponding to the agent, as shown in the following formula:

[0084] The mean of the evaluation index Q1 of all agents other than this agent is denoted as k1, and w = k1 / (Q1 + k1).

[0085] In summary, the aforementioned differences can suppress excessive differences in the evaluation metric Q2 among different agents, tightly integrate the reinforcement learning processes of all agents, minimize the differences in the evaluation metric Q2 obtained by all agents after reinforcement learning, enable further collaboration among agents, and ensure that any layout generated by the generative model is optimized based on the reasonable combination of all agents, providing a technical foundation for generating better layouts in the future.

[0086] Step S303: Based on the differences between Q2 and Q1 for each agent, determine whether F2 is a label. When F2 is a label, use the differences between Q2 of different agents to train a generative model, and fit the F1 output by the generative model to F2.

[0087] Obtain the average of the evaluation index Q1 of all agents, and denote it as the first average. Obtain the average of the evaluation index Q2 of all agents, and denote it as the second average. When the second average is greater than or equal to the first average, it means that the generated layout F2 is better able to meet the needs of normal traffic activities than layout F1.

[0088] At this point, layout F2 is used as the label for layout F1, and then the parameters of the generated model are updated.

[0089] As an optional example, the parameters of the generated model are updated, including:

[0090] For the layout F1 output by inputting R1 and city reference data into the generative model, F2 is used as the label to train the generative model. That is, the cross-entropy loss function is constructed using the F1 output by the generative model and the label F2. The parameters of the generative model are updated using this loss function and the Adam optimizer, and then the F1 output by the generative model is fitted to F2 for output.

[0091] The pair (R2, F1) is used as a training sample and its label.

[0092] As a preferred example, updating the parameters of the generative model includes:

[0093] For all agents, the evaluation metric Q2 is denoted as q2. The cross-entropy loss function of the above optional example is multiplied by (1+h) to obtain the loss function of this preferred example. The parameters of the generator model are updated using the Adam optimizer with this loss function, where h represents the gradient update coefficient and is negatively correlated with q2.

[0094] The triple (R2, F1, h) is used as a training sample and its label.

[0095] The purpose of designing the loss function in this preferred example is that when the difference between evaluation metrics Q2 is small (i.e., when q2 is small), the gradient update coefficient h is larger. In this case, the parameters of the generator model are updated with a larger gradient, allowing the generator model to escape local optima and update parameters in the direction that can strengthen the cooperative relationship between the agents. When the difference between evaluation metrics Q2 is large (i.e., when q2 is large), the gradient update coefficient h is smaller. This indicates that although all agents under layout F2 have a large second mean, the cooperation between all agents is insufficient. In this case, the generator model does not need to update drastically to avoid missing or moving away from the strong cooperative relationship between the agents.

[0096] As an example, the steps for obtaining the gradient update coefficients h include:

[0097] For all agents, the evaluation metric Q1 is denoted as q1, and the variance of Q1 is set as h = exp(-q2 / (q1+0.1)). Here, exp() represents an exponential function with respect to the natural constant. The purpose of including q1 in the denominator is twofold: firstly, to remove the order of magnitude of q2, and secondly, to describe the magnitude of q2 relative to q1. Adding 0.1 to the denominator avoids a denominator of 0; in other embodiments, 0.1 can be replaced with other values ​​within the interval (0.05, 0.2). Alternatively, in some examples, the formula q2 / (q1+0.1) is replaced with (q2+0.1) / (q1+0.1).

[0098] Specifically, when h is less than 0.3, let h = 0.1. In this example, the purpose is to update the parameters of the generating model with a relatively small gradient when q2 is too large (when there is insufficient coordination among all agents).

[0099] Furthermore, when the second mean is less than the first mean, it indicates that the generated layout F2 is better able to meet the needs of normal traffic activities compared to layout F1.

[0100] At this point, layout F1 is used as the label for layout F2, and then the parameters of the generated model are updated. As an example, this process includes:

[0101] For the layout F2 output by the generative model after inputting R2 and city reference data, F1 is used as the label to train the generative model. That is, the cross-entropy loss function is constructed using the F2 output by the generative model and the label F1, and the parameters of the generative model are updated using the Adam optimizer using this loss function.

[0102] The pair (R2, F1) is used as a training sample and its label.

[0103] This completes one parameter update for the generative model and yields a training sample.

[0104] Assign R2 and F2 to R1 and F2 respectively; assign the evaluation index Q2 of all agents to the evaluation index Q1 of the corresponding agents (the purpose is to provide a data basis for subsequent repeated execution).

[0105] Step S304: Continuously train the generative model.

[0106] Restart from step S302. Repeat steps S302 to S303 to continuously generate different layouts (i.e., F2) and perform reinforcement learning, and then gradually update the generated model.

[0107] In this embodiment, the training of the generative model ends after steps S302 to S303 are repeated 1000 times.

[0108] In other embodiments, to significantly reduce reinforcement learning time, before step S302 is executed, the initial parameters of each agent A network and C network are set to the parameters of each agent A network and C network after the last execution of step S302; specifically, if the last execution was step S301 instead of step S302, then the parameters are set to the parameters of each agent A network and C network after executing step S301. The parameters of each agent A network and C network in step S301 are randomly initialized.

[0109] In some embodiments, whenever the generative model performs a parameter update but reinforcement learning is not yet complete, the generative model is trained again using all the training samples and their labels (training is performed using the loss function in step S303). Specifically, when there are fewer than 10 pairs of training samples and their labels, it is necessary to wait for reinforcement learning to be completed before updating the parameters of the generative model (i.e., executing step S303).

[0110] Step S4: Optimize and transform the low-altitude transportation infrastructure network layout using the layout generated by the generative model. This process specifically includes...

[0111] Several random vectors (e.g., 20) are randomly generated. These random vectors and city reference data are then input into the trained generative model to obtain several layouts.

[0112] In one example, one layout is manually selected from all layouts for optimization and transformation of the low-altitude transportation infrastructure network layout, or to provide a reference for layout optimization and transformation (i.e., construction or transformation of take-off and landing sites).

[0113] The advantage of this example is that, in practical applications, there is no single best method for optimizing and transforming the layout of low-altitude transportation infrastructure networks. This example can provide multiple layouts for reference at once, and each layout can achieve collaborative planning and optimization of air and land, ensuring the integrated and efficient utilization of air and land resources, and providing greater flexibility in layout optimization and transformation.

[0114] In another example, for each layout, reinforcement learning is performed again according to the method in step S301 to obtain the average value of the evaluation index of all agents, which is recorded as the optimal index for each layout. The layout with the largest optimal index is used as the basis for layout optimization and modification (i.e., building or modifying the take-off and landing field).

[0115] The advantage of this example is that it allows for the highest possible integration of air and ground resources.

[0116] This concludes the example.

[0117] Example 2:

[0118] In step S301 of Embodiment 1, the method for obtaining the city's road distribution map, building distribution map, and pedestrian flow distribution map is as follows:

[0119] In this embodiment, a city map image is captured from map software, and the road distribution is marked on it to form a road distribution map. The road distribution map includes all roads used by motor vehicles. The road distribution map is a binary image, in which road areas are marked as 1 and non-road areas are marked as 0. In other embodiments, the road distribution map can also be marked based on a high-precision GIS map.

[0120] In the city map image, the areas where buildings exist are marked to construct a building distribution map. In this map, the grayscale value of the pixels in the building areas is set to 1, and the grayscale value of other areas is set to 0.

[0121] In a city map image, for each pixel's neighborhood (e.g., an 8-neighborhood), the pedestrian density within the corresponding geographical area is calculated and recorded as the pedestrian density for each pixel. The maximum pedestrian density across all pixels is obtained, and the pedestrian density for each pixel is divided by this maximum value to obtain the normalized pedestrian density for that pixel. This normalized pedestrian density is then used as the new grayscale value for the pixel. The image composed of these new grayscale values ​​for all pixels in the city map image is called the pedestrian flow distribution map.

[0122] As an example, the method for obtaining the population density in this geographical area is as follows: For 5G and 4G base stations in this geographical area, obtain the number of users connected to all base stations at each time point. In the past month, the number of users is obtained once every hour. The average number of users at all times is calculated, and the average is divided by the area of ​​the geographical area. The result is used as the population density in this geographical area.

[0123] In this embodiment, the city map image, road distribution map, building distribution map, and pedestrian flow distribution map are all 2048×1024 image sizes. The road distribution map, building distribution map, and pedestrian flow distribution map are stacked into a three-channel image and used as city reference data.

[0124] The city reference data is used to describe the current traffic operation environment of the city. In other embodiments, other methods can also be used to obtain city map images, road distribution maps, and building distribution maps. This embodiment does not impose specific limitations.

[0125] Example 3:

[0126] This embodiment illustrates the methods for obtaining the state, actions, and rewards of each agent:

[0127] For the intelligent agent corresponding to the vehicle (taking a taxi as an example in this embodiment):

[0128] The agent's state includes its location and its environment. The environment includes a layout (a two-channel binary image) and city reference data, where the layout image size and the city reference data image size are the same. In this embodiment, considering that the city reference data is a constant, it is not included as the agent's state to reduce data volume. The location refers to the pixel position within the layout.

[0129] The agent's actions include its velocity (including magnitude and direction) and acceleration (including magnitude and direction).

[0130] How to obtain rewards:

[0131] (1) In the layout, an initial position is randomly assigned to the agent on the city road, and the velocity and acceleration are randomly initialized (that is, a state is initialized, where the direction of the velocity and acceleration is along the road direction), and a take-off and landing field (limited to the take-off and landing field of the aircraft) is randomly selected as the endpoint (specifically, the road endpoint closest to the outline of the connected domain of the take-off and landing field). In this embodiment, the process of the agent moving from the initial position to the endpoint is recorded as a random simulation process.

[0132] (2) Randomly select several road segments (e.g., one-tenth of the total number of road segments and rounded down), where a road segment refers to the road between any two adjacent intersections (or ramps). Mark the selected road segments as congested road segments, and generate a random congestion time (e.g., a time randomly generated within 1 minute to 2 hours). The road segments will not be remarked as congested road segments within 20 minutes after the congestion time ends.

[0133] In other embodiments, traffic lights are simulated at each intersection, and the traffic lights at each intersection are synchronized with the traffic lights in the real city.

[0134] (3) Assign a pick-up time to the agent. The assignment method is as follows: obtain the shortest road between the initial position and the destination, divide the total length of the shortest road by the average speed limit of the shortest road to obtain the estimated time T, generate a random number p greater than or equal to 1.5 and less than or equal to 2.5, and add p×T to the time when the agent is in the initial position as the pick-up time.

[0135] (4) Upon reaching the destination, the random simulation process ends. After the random simulation process ends, steps (1) to (3) are repeated to perform the next random simulation process, and so on, allowing the agent to continuously simulate. During each random simulation, the agent's actions and states are changed according to the method described in step (2) of S301. It should be noted that when the agent's speed is greater than the maximum speed limit of the road segment, its speed is set to the maximum speed limit.

[0136] Each time the state of the agent is obtained, that is, in step (2) of S301, the next state is obtained. Then: Initialize r to 0; when the agent's position is outside the road area, decrement r by 100, and restart from the initial position; when the agent's position is in a congested section, decrement r by 1. Let r + d / s, where d represents the length of the trajectory between the two most recent states, and s represents the average length of all road segments. Let r + f, where f equals the average grayscale value of the pedestrian flow distribution map within the 5×5 neighborhood of the current pixel. If the agent's arrival time at the destination is greater than the pick-up time, then reduce all rewards r obtained during this random simulation and stored in the experience replay pool by 0.1. If the agent's arrival time at the destination is greater than or equal to the pick-up time, then increase all rewards r obtained during this random simulation and stored in the experience replay pool by 0.1.

[0137] For the intelligent agent corresponding to a person (taking a pedestrian as an example in this embodiment), the same principle applies as for a vehicle, except that: (1) the action of the intelligent agent does not include acceleration, and when the speed is greater than 1.5 meters per second, the speed is set to be equal to 1.5 meters per second. (2) No more congested road sections are set. (3) The destination is limited to the take-off and landing field of the drone. It should be noted that in this embodiment, the drone is only used to transport small goods (such as express delivery). (4) When calculating the pick-up time, the average speed limit is set to 1.5 meters per second. (5) When the straight-line distance between the location of the intelligent agent and the destination location is small (for example, less than 10 pixels), when the location of the intelligent agent is outside the road area, r is no longer reduced by 100.

[0138] For the intelligent agent corresponding to the drone or aircraft, the same principle applies as for the vehicle, except that: (1) When the speed of the intelligent agent is greater than the maximum speed limit (e.g., the maximum speed limit of the drone is 19 m / s and the maximum speed limit of the aircraft is 40 m / s), its speed is set to the maximum speed limit. In addition, this embodiment ignores the vertical take-off process. The position of the intelligent agent is its projection position on the ground. This embodiment only focuses on its motion projection on the ground. (2) Instead of setting congested road sections, obstacles are randomly generated at a preset distance in the direction of the intelligent agent's movement (e.g., at 20 pixels). For example, within three hours after the intelligent agent starts moving for one minute, three time periods are randomly sampled, and obstacles are generated at the three time periods. (3) The initial position and the endpoint of the drone are the take-off and landing fields of the drone, and the initial position and the endpoint of the aircraft are the take-off and landing fields of the aircraft. (4) When the position of the intelligent agent is outside the road area, r is no longer reduced by 100. When the Euclidean distance between the intelligent agent and the obstacle is less than 10 pixels, r-1 is set to simulate the obstacle avoidance and detour of the aircraft. (5) Replace r+f with rf to ensure that the agent avoids dense crowds and thus avoids safety hazards.

[0139] In other embodiments, the values ​​set in the above examples are replaced with other values, or other methods are designed to simulate the operation of different intelligent agents. Since the above are all conventional techniques, they will not be described in detail in this embodiment.

[0140] It should be noted that the inputs to networks A and C are multimodal (including image data, such as layout, and vector data, such as vectors composed of the agent's position or actions). Based on this, networks A and C are multimodal neural networks. In this embodiment, networks A and C utilize the LLaVA model, which is a model that integrates image and text modalities. In this embodiment, it is only necessary to replace the embedding vectors corresponding to the text in the LLaVA model with vectors composed of the agent's position or vectors composed of position and action. The LLaVA model is a well-known technology and will not be described in detail in this embodiment.

[0141] It should also be noted that in this embodiment, the learning rates of network A and network C are set to 0.03 and 0.01 respectively during the reinforcement learning process, the soft update coefficient is set to 0.01, the capacity of the experience replay pool is set to 1000, and the number of sampled experience items is set to 64. In particular, if there are less than 64 experience items in the experience replay pool, then the update of network A and network C is not performed, that is, the process of step S301 (3) is skipped. At this time, it is equivalent to the agent performing random movement.

[0142] For each agent in the reinforcement learning process, the 30 most recent outputs of the C network for each agent constitute a value sequence. This value sequence is then filtered for maximum value using a filter kernel with a kernel length of 5. The standard deviation of the filtered value sequence is denoted as the training error. When the training errors of all agents are less than or equal to th1, the reinforcement learning process ends. At this point, the mean of the filtered value sequence is taken as the value at which the value is maximized, and is denoted as the agent's evaluation metric. This embodiment uses th1=0.2 as an example, with a preferred value range of (0.1, 0.3). When the training error of all agents exceeds th1, reinforcement training continues for each agent.

[0143] Example 4:

[0144] As an example, the generative model is a convolutional neural network with a DeepLabv3 network structure. Other embodiments may use the UNet network structure; this embodiment is not specifically limited. The network's input is city reference data and a random vector, and its output is a two-channel binary image describing the layout, which has the same size as the image corresponding to the city reference data.

[0145] It should be noted that in this example, the dimension D of the random vector is 16. Before inputting the random vector into the generation model, it needs to be converted into a random image. Specifically, a convolutional layer in the generation model is manually specified. For the feature map input to this convolutional layer, the feature map is divided into D sub-blocks. In this embodiment, the feature map is equally divided into D rectangular sub-blocks. This embodiment is not limited to the method of using D sub-blocks; as long as there are D sub-blocks, it is acceptable. A single-channel image of the same size as the feature map is constructed. This image also corresponds to D sub-blocks, with each sub-block corresponding one-to-one with each dimension of the random vector. For example, the first sub-block corresponds to the first dimension of the random vector, the second sub-block corresponds to the second dimension, and so on. The grayscale value of all pixels within each sub-block of this image is equal to the value of the corresponding dimension in the random vector. This image is denoted as a random image. This random image and the feature map are stacked together (i.e., as a new channel in the feature map) and then input into the convolutional layer. In this embodiment, the specified convolutional layer is the first convolutional layer, and at this point, the feature map is the city reference data.

[0146] As an example, the method for generating a random vector is as follows: construct a normal distribution with a mean of 0 and a variance of 1, use it as a probability density function, and sequentially sample D values ​​from it to form a random vector. The method of numerical sampling based on the probability density function is well-known and will not be elaborated upon in this embodiment.

[0147] Before running step S301 of Example 1, the generative model is pre-trained. As an example, the process includes:

[0148] The areas permitted for the planning of drone take-off and landing sites and the areas permitted for the planning of aircraft take-off and landing sites are manually marked in the urban reference data, and are denoted as the first area and the second area, respectively. The first area is distributed on rooftops, and the second area is distributed in the developable area outside the building area. In this embodiment, the number of first areas is set to 30 to 40, and the number of second areas is set to 5 to 10.

[0149] An integer m is randomly generated between 30 and 40. Within the first region, m pixels are randomly selected. Each pixel's 8-neighborhood is used as a takeoff and landing field region, resulting in m takeoff and landing field regions. A binary image of the same size as the city reference data image is constructed. In this image, the grayscale value of the pixels in the m takeoff and landing field regions is set to 1, and the grayscale value of other pixels is set to 0. This image represents the takeoff and landing field distribution map of a randomly generated UAV.

[0150] An integer n is randomly generated between 5 and 10. Within the first region, n pixels are randomly selected. Each pixel's 5×5 neighborhood is used as a takeoff and landing field region, resulting in n takeoff and landing field regions. A binary image of the same size as the city reference data image is constructed. In this image, the grayscale value of the pixels in the n takeoff and landing field regions is set to 1, and the grayscale value of other pixels is set to 0. This image represents a randomly generated takeoff and landing field distribution map of an aircraft.

[0151] The above method generates a large number of takeoff and landing field distribution maps for UAVs and aircraft. Any two UAV and aircraft takeoff and landing field distribution maps are stacked to form a two-channel image, which serves as a label. If two labels are identical, only one is retained. This embodiment uses the above method to generate a large number of labels (e.g., 1000 labels). In this embodiment, a random vector is generated for each label, which serves as the sample corresponding to the label. Specifically, if two samples are identical, they are deleted and a new sample is generated.

[0152] All samples and labels constitute a dataset, which is used to train a generative model. The loss function used for training is the cross-entropy loss function, the optimizer is Adam, the batch size is set to 6, and the training ends after the generative model is trained 20,000 times (that is, after the generative model parameters are updated 20,000 times).

[0153] This completes the pre-training process of the generative model.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing the layout of low-altitude transportation infrastructure networks based on multi-agent simulation, characterized in that, The method includes the following steps: Create several intelligent agents; the intelligent agents include people, vehicles, drones, and vertical take-off and landing manned aircraft; the state of each intelligent agent includes the position of the intelligent agent in the layout, and the action of each intelligent agent includes the speed magnitude and speed direction of the intelligent agent in the layout; A generative model is used to generate layout F1, which is used as the environment of each agent. The DDPG algorithm is used to perform reinforcement learning on each agent. The DDPG algorithm includes an A network and a C network. The C network is used to output the value of each agent in each state and action. The A network updates its parameters based on the value output by the C network. The maximum value output by the C network of each agent after reinforcement learning is used as the evaluation index Q1 of each agent. The layout F2 is generated again using the generative model. F2 is used as the environment of each agent and reinforcement learning is performed again to obtain the evaluation index Q2 of each agent. When reinforcement learning is performed again, the A network of each agent uses the Q1 of all agents to update the parameters so that the difference of Q2 between different agents is minimized. Based on the differences between Q2 and Q1 for each agent, determine whether F2 is a label; when F2 is a label, use the differences between Q2 of different agents to fit F1 output by the generator model into F2. Generative models are used to obtain the network layout of low-altitude transportation infrastructure.

2. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 1, characterized in that, When performing reinforcement learning again, each agent's A network updates its parameters using the Q1 values ​​of all agents to minimize the Q2 differences between different agents. The specific steps include the following: During the reinforcement learning process, each agent takes turns undergoing reinforcement training. For any agent, the average sampled value of the C network output after each reinforcement training iteration is expressed as: ; For the i-th agent other than the current agent, the average sampled value of the C network output after the most recent reinforcement training of the i-th agent is denoted as . ; The loss function of the agent's A network Depend on and The weighted summation is used to obtain N, which represents the number of agents other than the one in question, using the loss function. The parameters of network A are updated, where the weights in the weighted summation are obtained from the evaluation metric Q1 of all agents.

3. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 2, characterized in that, The specific steps for obtaining the weights in the weighted summation are as follows: For any given agent, the agent's corresponding The weight w in the weighted summation is negatively correlated with the evaluation index Q1 of the agent, and positively correlated with the evaluation index Q1 of all agents other than the agent.

4. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 1, characterized in that, The specific steps for determining whether F2 is a label based on the difference between Q2 and Q1 for each agent are as follows: Obtain the average of the evaluation metrics Q1 of all agents, and denote it as the first average. Obtain the average of the evaluation metrics Q2 of all agents, and denote it as the second average. When the second average is greater than or equal to the first average, determine that F2 is a label.

5. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 4, characterized in that, When F2 is the label, the specific steps for fitting the F1 output of the generator model to F2 by utilizing the differences between Q2 of different agents are as follows: For the layout F1 output by the generative model, a cross-entropy loss function is constructed using F1 and label F2. The cross-entropy loss function is multiplied by (1+h) and used as the loss function of the generative model. The parameters of the generative model are updated using the loss function of the generative model, where h represents the gradient update coefficient.

6. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 5, characterized in that, The specific steps for obtaining the gradient update coefficients are as follows: The variance of the evaluation index Q1 for all agents is denoted as q1, and the variance of the evaluation index Q2 for all agents is denoted as q2; the gradient update coefficient is negatively correlated with q2 and positively correlated with q1.

7. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 4, characterized in that, When the second mean is less than the first mean, F1 is determined to be the label; For the layout F2 output by the generative model, a cross-entropy loss function is constructed using F2 and the label F1, and used as the loss function of the generative model; the parameters of the generative model are updated using the loss function of the generative model.

8. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 1, characterized in that, The specific steps for obtaining the maximum value of the C network output of each agent after reinforcement learning are as follows: For each agent in the reinforcement learning process, the values ​​most recently output by the C network of each agent constitute a value sequence. The value sequence is filtered for maximum value, and the standard deviation of the filtered value sequence is recorded as the training error. When the training errors of all agents are less than or equal to a preset threshold, the reinforcement learning process ends, and the value output by the C network of each agent is maximized. The mean of the filtered value sequence is taken as the maximum value.

9. The method for optimizing the layout of low-altitude transportation infrastructure network based on multi-agent simulation according to claim 2, characterized in that, The specific steps for obtaining the average sample value are as follows: During reinforcement learning, each agent samples several states s1 from the experience replay pool of the DDPG algorithm during each reinforcement training session. Each state s1 is input into the A network of each agent, and the A network outputs action a1. The state s1 and action a1 are input into the C network of each agent, and the value output by the C network is recorded as the first value. The average of all first values ​​obtained from sampling several states is recorded as the average sampled value of each agent.

Images