Large-scale road network multi-vehicle path load balancing method based on reinforcement learning graph neural network

Abstract

The application provides a large-scale road network multi-vehicle path load balancing method based on a reinforcement learning graph neural network, and comprises the following steps: acquiring road network data in a real scene, modeling the road network, and representing the road network as a graph structure; the graph structure is split into multiple subgraph structures, and each subgraph structure corresponds to a sub-road network; a reinforcement learning adaptive graph neural network is constructed and trained; for each sub-road network of the road network, the trained reinforcement learning adaptive graph neural network is called to generate corresponding road selection probabilities in parallel; the generated sub-road network road selection probabilities are merged according to the topological structure of the original road network to restore each road selection probability in the complete original road network; and road load balancing is realized based on each road selection probability in the complete original road network. In a large-scale road network scene, the application realizes efficient path replanning, reduces road network load in real time, solves the problem of road congestion, and improves traffic efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence algorithms, specifically to the technical fields of graph neural networks, reinforcement learning and load balancing. BACKGROUND

[0002] In the real world, with the continuous advancement of urbanization, road traffic congestion problems are becoming increasingly serious. In response to this problem, various road network optimization methods and traffic management strategies have been proposed.

[0003] However, in large-scale road network scenarios, traditional methods often face challenges such as high computational complexity, poor real-time performance and poor adaptability. SUMMARY

[0004] The present application proposes a large-scale road network multi-vehicle load balancing method based on reinforcement learning graph neural networks, aiming to solve the problem of road congestion in real-world scenarios.

[0005] The present application is based on the idea of data flow dispersion transmission in load balancing, and analogizes network nodes to traffic intersections and transmission lines to lanes, and applies it to the task of traffic dispersion in real-world traffic scenarios. By training a reinforcement learning adaptive graph neural network, it adapts to road network graph structures of different sizes. In the application layer, it divides large road networks into small sub-road networks, simultaneously calls the trained reinforcement learning adaptive graph neural network to generate corresponding road selection probabilities in parallel, and merges the generated sub-road selection probabilities according to the topology of the original graph to restore the road selection probabilities in the complete large road network. Vehicles adjust their driving paths in real time according to the complete large road network road selection probabilities, thereby achieving efficient path re-planning and ultimately achieving load balancing, significantly shortening the planning time.

[0006] Technical solution:

[0007] The large-scale road network multi-vehicle path load balancing method based on reinforcement learning graph neural networks includes the following steps:

[0008] Step 1: Obtain road network data in real-world scenarios, model the road network and represent it as a graph structure;

[0009] Step 2: Split the graph structure into multiple sub-graph structures, each corresponding to a sub-road network;

[0010] Step 3: Build and train a reinforcement learning adaptive graph neural network;

[0011] Step 4: For each sub-road network of the road network, call the trained reinforcement learning adaptive graph neural network to generate corresponding road selection probabilities in parallel, and merge the generated sub-road selection probabilities according to the topology of the original road network to restore the road selection probabilities in the complete original road network;

[0012] Step 5: Based on the road selection probability in the complete original road network, road load balancing is realized.

[0013] The beneficial effects of the present application are:

[0014] In the large-scale road network scenario, the present application can realize efficient path re-planning, reduce road network load in real time, solve road congestion problems, and improve traffic efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 Schematic diagram for selecting subgraphs by geographical distribution and radiation for the embodiment.

[0016] Figure 2 Schematic diagram of the large graph segmentation and recombination restoration process for the embodiment.

[0017] Figure 3 Schematic diagram of the training principle of the embedded graph neural network for reinforcement learning.

[0018] Figure 4 Schematic diagram of the method flow of the present application. DETAILED DESCRIPTION

[0019] A graph neural network is a machine learning model specifically designed for processing graph-structured data, which can effectively perform representation learning on nodes and edges in a graph. In road network optimization and traffic management, the road network is regarded as a graph, where nodes represent intersections and edges represent roads connecting various intersections. By training a graph neural network, road selection probabilities for vehicles to choose can be generated, thereby realizing more intelligent and dynamic path planning, which can optimize the driving route of vehicles according to real-time traffic conditions.

[0020] Reinforcement learning is a machine learning method that aims to learn the optimal decision-making strategy through the interaction between the agent and the environment. In the present application, the reinforcement learning method is applied to train an embedded model that can guide the path selection of multiple vehicles according to real-time road congestion data and road network weights. Through continuous interaction with the environment, the embedded model can learn the optimal path selection strategy that adapts to different road congestion conditions and make real-time adjustments in the real-time environment.

[0021] Load balancing is a key technology for balancing traffic transmission tasks. In the present application, load balancing is applied to reduce the load of large road networks to solve the problem of road congestion in real-world scenarios. By dispersing the traffic of vehicles to different roads, the rational use of resources and the optimization of network performance can be achieved. The application of network load balancing can effectively improve the road capacity and traffic efficiency.

[0022] The application realizes a large road network multi-vehicle path load balancing method based on a reinforcement learning graph neural network by comprehensively using three aspects of technologies of a graph neural network, reinforcement learning and network load balancing, so as to reduce road network load, solve road congestion problems and improve traffic efficiency.

[0023] The technical solutions provided by the application will be further described below with reference to specific embodiments and drawings thereof. The advantages and features of the application will be more apparent in combination with the following description.

[0024] Embodiments

[0025] The large road network multi-vehicle path load balancing method based on the reinforcement learning graph neural network comprises the following steps:

[0026] Step 1: Obtain road network data in a real scene, model the road network and represent it as a graph structure.

[0027] 1.1 Collect road network data in a real scene, including the latitude and longitude of an intersection, the starting and ending intersection of a road and the corresponding latitude and longitude position.

[0028] 1.2 Model the road network as a graph structure with intersections as nodes and roads as edges, represented as a graph G = (V, E) , wherein: V is a node set, including intersection coordinate information; E is an edge set, wherein the elements correspond to each road and contain the starting node and the ending node of the road.

[0029] Step 2: Split the graph structure into multiple subgraph structures, each corresponding to a sub-road network.

[0030] Select the center node of each subgraph according to the dispersion degree of the geographical position, judge whether the current node is within the coverage range of the center node by calculating the distance between each node and the center node, and determine the effective range of the subgraph.

[0031] Since the action space corresponds to the road selection probability in reinforcement learning, and the road selection probability is related to the number of roads, the action space is closely related to the number of edges in the graph structure. Therefore, the larger the graph structure is, the greater the corresponding calculation speed grows linearly. Therefore, by dividing the graph structure into small subgraphs, calculating the road selection probability of each subgraph in parallel and recombining, the calculation efficiency is improved.

[0032] The graph structure splitting method specifically is: according to geographical distribution, select the center node of a static subgraph, and determine the range of the subgraph according to the radiation of the center node to the surrounding. For example, Figure 1As shown. Let the central node be I, and its relationship with its adjacent node J include their distance and radiation range. The distance between the central node I and its adjacent node J varies depending on the size of the original image. The radiation range needs to be adjusted accordingly. Corresponding changes are also needed, including the number of reference nodes. and actual map size The distance between the center node I and the adjacent node J and radiation range for:

[0033]

[0034]

[0035] Where γ is the map extent adjustment parameter, λ is the radiation range adjustment parameter, and "[]" is the rounding function. That is:

[0036] The distance between the center node I and the adjacent node J The map extent adjustment parameter γ is multiplied by the square root of the number of nodes and the map size;

[0037] Radiation range The number of hops is the integer result of multiplying the map range adjustment parameter λ by half the spacing (the number of hops is the minimum number of nodes that a node needs to pass through to reach the current node).

[0038] like Figure 2 As shown, the map is divided into four sub-maps according to geographical regions, namely: Figure 1 ,son Figure 2 ,son Figure 3 ,son Figure 4 At this point, since the edges connecting the subgraphs do not yet have corresponding structures, the subgraphs are... Figure 1 ,son Figure 2 ,son Figure 3 ,son Figure 4 Treating them as the fifth subgraph nodes and their connecting edges as the edges of the fifth subgraph, the connection relationship can be abstracted into a new subgraph, namely subgraph 5.

[0039] Step 3: Construct and train a reinforcement learning adaptive graphical neural network

[0040] Each subgraph has a different size and structure, so it is necessary to design a reinforcement learning adaptive graph neural network to satisfy the input of subgraphs of different sizes and structures and the corresponding road congestion levels.

[0041] 3.1 Constructing an adaptive graph neural network for reinforcement learning.

[0042] like Figure 3As shown, the reinforcement learning adaptive graph neural network includes two graph convolutional layers, namely the first graph convolutional layer GraphConv1 and the second graph convolutional layer GraphConv2, and one fully connected layer FC. The activation functions of the first graph convolutional layer GraphConv1 and the second graph convolutional layer GraphConv2 are ReLU, and the activation functions of the second graph convolutional layer GraphConv2 and the fully connected layer FC are ReLU.

[0043] The reinforcement learning adaptive graph neural network takes road congestion data of a subgraph as input and outputs the probability of edge selection (lane) in the subgraph. The higher the road congestion level, the lower the corresponding road selection probability, thereby reducing the load on that road.

[0044] In this embodiment, the first graph convolutional layer, GraphConv1, has an input size equal to the embedding layer vector. The first graph has an input size of 128 and an output size of 128; the second graph is a convolutional layer GraphConv2 with an input size of 128 and an output size of action vectors. Size; Fully connected layer (FC), input size is the action vector Size, output size is the action vector Size. Reinforcement learning adaptive graph neural network, through input vector Generate a set of action vectors , The probability of choosing a road.

[0045] 3.2 Training the reinforcement learning adaptive graph neural network.

[0046] We use the reinforcement learning PPO (Proximal Policy Optimization) algorithm to train an adaptive graph neural network and its evaluation network. Figure 3 As shown, the embedding vector contains hidden subgraph structure information and lane congestion information. The input is fed into a reinforcement learning adaptive graph neural network, which passes through the first graph convolutional layer GraphConv1, the second graph convolutional layer GraphConv2, and a fully connected layer to produce the output, which is the road selection probability corresponding to the subgraph.

[0047] Specifically, the training of the reinforcement learning adaptive graph neural network is implemented based on a simulator (system). The road network corresponding to the subgraph structure is mapped into the simulator (system), and vehicles are generated in the simulator (system) to simulate the vehicle driving process. During vehicle driving, the simulator (system) reads the current congestion level of each lane in real time. The lane congestion level is calculated as: number of vehicles × vehicle length / road length, where 0 represents an empty lane and 1 represents a lane that is full of vehicles. The lane congestion level changes in real time as the traffic flow simulation in the simulator (system) changes.

[0048] 3.2.1 Start the simulator, create the vehicle, initialize the current lane information and vehicle information, get the corresponding current lane congestion degree information, and embed it as the current time vector .

[0049] 3.2.2 At each time step t, get the current graph structure and the corresponding road congestion degree, generate the vector corresponding to the t time . Input the reinforcement learning adaptive graph neural network, output the current time , that is, the road selection probability. The process includes:

[0050] 3.2.2.1 Get the subgraph corresponding to the sub-road network and its corresponding road congestion degree information, and embed it as a vector , and as the features of all lanes, represented as state (State) in the reinforcement learning algorithm , s represents the state and t represents the current time step.

[0051] 3.2.2.2 Generate action (Action) through action network (reinforcement learning adaptive graph neural network) , that is, the road selection probability of each lane in the subgraph.

[0052] 3.2.2.3 In the simulator (system), when the vehicle gets the current road selection probability, update the path of each vehicle itself in each subgraph, get the road corresponding to the current vehicle driving section, and select the next driving road according to , so as to modify the trajectory in real time.

[0053] 3.2.2.4 Update the reinforcement learning adaptive graph neural network parameters according to the reward iteration.

[0054] 3.2.3 Repeat step 3.2.2 for N time steps (example: 500 time steps), complete a complete set of training.

[0055] 3.2.4 Repeat 3.2.2 and 3.2.3 for M sets (example: 200 sets), save the reinforcement learning adaptive graph neural network parameters. Complete the training.

[0056] The following is the detailed definition of state state , action action and reward function reward in 3.2.2.

[0057] The State refers to the embedding vector used as input for reinforcement learning, and its value varies depending on the graph structure. As input to the state, reinforcement learning algorithms need to obtain road congestion information from graphs of different sizes; therefore, this information is unified into a vector of the same size. In order to meet the needs of reinforcement learning training.

[0058] The Action, as the output of reinforcement learning, is related to the number of edges. same size output , which corresponds to the probability of choosing a road.

[0059] The reward function is designed to consist of three parts: the first reward, the second reward, and the third reward.

[0060] The first part of the reward is determined by the lane congestion level. The lane (edge) congestion level is defined as the ratio of the total length of all vehicles to the total length of the lane, denoted as [formula missing]. . The value reflects the congestion level of lane i, where i ∈ E, and E is the set of edges. For example, if a car is 2m long, there are 50 cars in total, and the lane length is 200m, then... A higher congestion value indicates a higher degree of congestion. Therefore, the first part of the reward is recorded as follows:

[0061]

[0062] The second part, the reward, is determined by the BPR value. In traffic engineering, the BPR value is an impedance function, derived from... Determined. Where T is the actual travel time in the current lane, T0 is the free-flow travel time in the lane, D is the number of vehicles in the current lane, and C is the vehicle capacity of the current lane. a and b are parameters, typically a=0.15 and b=4. A larger T value indicates greater road resistance, resulting in a longer travel time. This application measures the second reward function value by calculating the T-T0 value, letting... Let T-T0 be the value of road i, and the second part, reward, be...

[0063]

[0064] The third part, reward, is related to whether the path selected by the vehicle is an edge on the shortest path. It checks whether the next road selected by the vehicle in the current situation is an edge on the shortest path; if so, its reward value is set to 1; otherwise, it is 0. H is the set of vehicles. Indicates vehicle v The reward value for whether the currently selected path is the shortest path edge.

[0065]

[0066] Combining the above three reward parts, the final reward value function is:

[0067]

[0068] Step 4: For each sub-road network of the road network, call the trained reinforcement learning adaptive graph neural network to generate the corresponding road selection probability in parallel, and merge the generated sub-road network road selection probability according to the topology of the original large road network to restore the road selection probability of each road in the original large road network.

[0069] After splitting, since the subgraph corresponds to local road network information, it needs to be reassembled according to the results of local calculation after splitting into subgraphs (i.e. subgraph road selection probability) to form the road selection probability corresponding to the original large graph, meeting the vehicle's route-finding task on the complete graph while balancing the degree of road congestion.

[0070] At different times, the generated original complete road network road selection probability will change. Since the degree of road congestion will change according to the number of vehicles in the current road at different times, the input of the reinforcement learning adaptive graph neural network will also change, and the generated edge selection probability will also change accordingly. The reinforcement learning adaptive graph neural network will identify the degree of edge congestion and generate the edge selection probability, for example, if the edge is heavily congested, the edge selection probability will decrease, and vice versa.

[0071] As shown in Figure 2 After the example is processed by the reinforcement learning adaptive graph neural network, the edge selection probability of each subgraph is [1a, 1b, 1c, 1d, 1e, 1f, 1g], [2a, 2b, 2c, 2d, 2e, 2f, 2g], [3a, 3b, 3c, 3d, 3e, 3f, 3g], [4a, 4b, 4c, 4d, 4e], [5a, 5b, 5c, 5d, 5e, 5f, 5g, 5h]. Paste and reassemble to get the edge selection probability of each edge of the original large graph. Vehicles will select roads according to the road selection probability of the complete road network as their actual driving path.

[0072] Step 5: Realize road load balancing. After each vehicle obtains the road selection probability corresponding to the complete road network, it makes a detour or original road driving selection for the next driving road according to the spatial shortest path, meeting its own travel demand while reducing the overall road traffic load and improving road traffic efficiency.

[0073] When the vehicle arrives at the intersection, it obtains the road selection probability of all roads connected to the current intersection, and then selects the next road to be driven into. According to two influence factors, the next road to be driven into is selected:

[0074] Influence factor 1: whether the next road belongs to the shortest path of the corresponding vehicle in the space of the original graph.

[0075] Influence factor 2: whether the next road is the road with the maximum edge selection probability connected to the current intersection.

[0076] In the case of few vehicles, the road with the shortest spatial distance is selected as the road to be driven into according to the influence shadow 1, thereby reducing the driving resource consumption of the vehicle.

[0077] In the case of many vehicles and congested roads, the road with the maximum edge selection probability connected to the current intersection is selected as the road to be driven into according to the influence shadow 2, that is, the road with a lower congestion degree is selected as the road to be driven into.

[0078] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the above disclosed technical content shall be regarded as an equivalent effective embodiment, and shall fall within the scope of protection of the technical scheme of the present application.

Claims

1. A large-scale road network multi-vehicle path load balancing method based on reinforcement learning graph neural network, characterized in that, Comprising the following steps: Step 1: Obtain road network data in a real scene, model the road network, and represent it as a graph structure; Step 2: Split the graph structure into multiple subgraph structures, each corresponding to a sub-road network; Step 3: Build and train a reinforcement learning adaptive graph neural network; Step 4: For each sub-road network of the road network, call the trained reinforcement learning adaptive graph neural network to generate the corresponding road selection probability in parallel, and merge the generated sub-road network road selection probability according to the topological structure of the original road network to restore the road selection probability of each road in the original road network; Step 5: Based on the road selection probability of each road in the original road network, realize road load balancing; The training of the reinforcement learning adaptive graph neural network is realized based on the simulator; the road network corresponding to the subgraph structure is mapped into the simulator, and vehicles are generated in the simulator to simulate the driving process; during the driving process, the simulator reads the congestion degree of each lane in real time, and the lane congestion degree changes in real time with the simulation of vehicle flow; The training process includes: 3.2.1 Start the simulator, create the vehicle, initialize the current lane information and vehicle information, get the corresponding current lane congestion degree information, and embed it as a current time vector ; 3.2.2 At each time step t, obtain the current graph structure and the corresponding road congestion degree, and generate the vector corresponding to the time t , input the reinforcement learning adaptive graph neural network, and output the road selection probability at the current time , that is, the road selection probability, and the process includes: 3.2.2.1 Obtain the subgraph corresponding to the sub-road network and its corresponding road congestion degree information, and embed it as a vector , will be represented as a state (State) in the reinforcement learning algorithm as a feature of all lanes , s represents the state, and t represents the current time step; 3.2.2.2 Action generation by reinforcement learning adaptive graph neural network i.e., the road selection probability of each lane in the subgraph; 3.2.2.3 In the simulator, when the vehicle gets the current road selection probability, in each subgraph, for each vehicle's own path, update the current vehicle driving road, and according to select the next driving road, so as to modify the own trajectory in real time; 3.2.2.4 According to reward iterating, updating the reinforcement learning adaptive graph neural network parameters; 3.2.3 Repeat step 3.2.2 N times to complete a complete set of training; 3.2.4 Repeat step 3.2.2 and step 3.2.3 M times to save the reinforcement learning adaptive graph neural network parameters; The function design of the reward consists of three parts, namely the first part reward, the second part reward, and the third part reward: The first part of the reward is determined by the congestion degree of the lane, and the congestion degree of the lane is defined as the ratio of the total length of all vehicles to the total length of the lane, denoted as ; The value reflects the congestion degree of the lane i, i∈E, E is an edge set; the larger the congestion value is, the higher the congestion degree is; and the first part of the reward is recorded as The second part of reward is determined by BPR value, which is impedance function, determined by ; wherein, T is the actual travel time of current lane, T0 is the travel time of free flow speed of lane, D is the number of vehicles on current lane, C is the vehicle capacity of current lane, and a and b are parameters; the greater the value of T is, the greater the impedance of road is, and thus the longer the travel time is; the value of second reward function is measured by calculating T-T0 value, and let be the T-T0 value of road i, and the second part of reward is The third part reward, the value of which is related to whether the path selected by the vehicle is the shortest path edge; detecting whether the next road selected by the vehicle under the current situation is an edge on the shortest path, if yes, setting the reward value to 1, otherwise 0; H is a set of vehicles, representing the vehicle v reward value of whether the currently selected path is the shortest path edge: Combining the above three reward parts, the final reward value function is:

2. The method of claim 1, wherein, The step 1 includes: 1.1 Collect road network data in a real scene, including the latitude and longitude of the intersection, the starting and ending intersection of the road, and the corresponding latitude and longitude position; 1.2 Take the intersection as the node, the road as the edge, model the road network as the graph structure, and express it as graph Wherein: is the node set, including the intersection coordinate information; is the edge set, The element in the above corresponds to each road, and contains the starting node and the terminal node of the road.

3. The method of claim 1, wherein, The step 2 selects the center node of each subgraph according to the dispersion degree of the geographical position, determines the effective range of the subgraph by calculating the distance between each node and the center node to determine whether the current node is within the coverage range of the center node; Let the center node be I, and the relationship of the adjacent node J to it include the distance and radiation range between them; reference node quantity and actual map size , the distance between the center node I and the adjacent node J and radiation range are: Wherein, γ is the map range adjustment parameter, λ is the radiation range adjustment parameter, and "[]" is the rounding function.

4. The method of claim 1, wherein, The step 3, The reinforcement learning adaptive graph neural network includes two graph convolution layers, namely the first graph convolution layer GraphConv1 and the second graph convolution layer GraphConv2, and a fully connected layer FC; the activation function of the first graph convolution layer GraphConv1 and the second graph convolution layer GraphConv2 is ReLU, and the activation function of the second graph convolution layer GraphConv2 and the fully connected layer FC is ReLU; The input of the reinforcement learning adaptive graph neural network is the road congestion data of the subgraph, and the output is the road selection probability of the subgraph.

5. The method of claim 1, wherein, The step 5 is specifically implemented as: When the vehicle arrives at the intersection, the road selection probability of all roads connected to the current intersection is obtained, and the next road to be entered is selected; the next road to be entered is selected according to two influence factors: Influence factor 1: whether the next road belongs to the shortest path of the corresponding vehicle in the original graph; Influence factor 2: whether the next road is the road with the maximum edge selection probability connected to the current intersection; In the case of less vehicles, the road with the shortest spatial distance is selected as the road to be entered according to the influence shadow 1; In the case of more vehicles and more congested roads, the road with the lowest congestion degree is selected as the road to be entered according to the influence shadow 2.

Images